Title: Fast Encoder-Based 3D from Casual Videos via Point Track Processing
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dynamic Videos

Dynamic 3D Reconstruction

Equivariance

Symmetries
Abstract: This paper addresses the long-standing challenge of reconstructing 3D structures from videos with dynamic content. Current approaches to this problem were not designed to operate on casual videos recorded by standard cameras or require a long optimization time. 
  Aiming to significantly improve the efficiency of previous approaches, we present TracksTo4D, a learning-based approach that enables inferring 3D structure and camera positions from dynamic content originating from casual videos using a single efficient feed-forward pass. To achieve this, we propose operating directly over 2D point tracks as input and designing an architecture tailored for processing 2D point tracks. Our proposed architecture is designed with two key principles in mind: (1) it takes into account the inherent symmetries present in the input point tracks data, and (2) it assumes that the movement patterns can be effectively represented using a low-rank approximation. TracksTo4D is trained in an unsupervised way on a dataset of casual videos utilizing only the 2D point tracks extracted from the videos, without any 3D supervision. Our experiments show that TracksTo4D can reconstruct a temporal point cloud and camera positions of the underlying video with accuracy comparable to state-of-the-art methods, while drastically reducing runtime by up to 95\%.  We further show that TracksTo4D  generalizes well to unseen videos of unseen semantic categories at inference time.

Fast Encoder-Based 3D from Casual Videos via Point Track Processing

Abstract

This paper addresses the long-standing challenge of reconstructing 3D structures from videos with dynamic content. Current approaches to this problem were not designed to operate on casual videos recorded by standard cameras or require a long optimization time. Aiming to significantly improve the efficiency of previous approaches, we present TRACKSTO4D, a learning-based approach that enables inferring 3D structure and camera positions from dynamic content originating from casual videos using a single efficient feed-forward pass. To achieve this, we propose operating directly over 2D point tracks as input and designing an architecture tailored for processing 2D point tracks. Our proposed architecture is designed with two key principles in mind: (1) it takes into account the inherent symmetries present in the input point tracks data, and (2) it assumes that the movement patterns can be effectively represented using a low-rank approximation.TRACKSTO4D is trained in an unsupervised way on a dataset of casual videos utilizing only the 2D point tracks extracted from the videos, without any 3D supervision. Our experiments show that TRACKSTO4D can reconstruct a temporal point cloud and camera positions of the underlying video with accuracy comparable to state-of-the-art methods, while drastically reducing runtime by up to 95%. We further show that TRACKSTO4D generalizes well to unseen videos of unseen semantic categories at inference time. Project page: https://tracks-to-4d.github.io .

1 Introduction

Predicting 3D geometry in dynamic scenes is a challenging problem. In this problem setup, we are given access to multiple images of a scene taken sequentially, e.g., from a monocular video camera, where both the content in the scene and the camera are moving. Our task is to reconstruct the dynamic 3D positions of the points seen in the images and the camera poses. This fundamental problem has gained significant interest from the research community over the years [7, 37, 26, 64], mainly due to its important applications in many fields such as robot navigation, autonomous driving and 3D reconstruction of general environments [19]. Importantly, in contrast to static scenes where the epipolar geometry constraints hold between the corresponding points of different views [17], determining the depth of a moving point from monocular views is an ill-posed problem [3]. This causes standard Structure from Motion techniques [42, 55, 33] to be inadequate in this setup [25].

Previous work and limitations. Many existing approaches for the above problem make simplifying assumptions that limit their applicability to real-world scenarios. Methods based on orthographic camera models and low-rank assumptions use matrix factorization techniques [7, 26], but the orthographic camera assumption might not be realistic and may cause reconstruction errors. Techniques that incorporate depth priors often require lengthy optimization processes in order to make the depth estimates across frames consistent [25, 64]. Other physics-based approaches make assumptions

2

Per-frame 3D Point Clouds

𝑋

!

, … , 𝑋

"

∈ ℝ

# × %

Positional

Encoding

𝑀 ∈ ℝ

! × # × $

𝑁×𝑃×256

Frames

Attention

Points

attention

𝑁×𝑃×256

Frames

Attention

Points

attention

Equivaraint network architecture

Invariant Layers

Input 2D

Point-track

Tensor

Equivariant

Invariant

𝑃

Movement

Level Values

𝛾

!

…,𝛾

"

∈ ℝ

𝐾

bases

𝐵

!

…𝐵

#

∈ ℝ

"×%

Linear coefficients

∈ ℝ

&×(#(!)

𝑁

Camera Poses

𝑅

!

, 𝑡

!

, … , 𝑅

&

, 𝑡

&

Reprojection

Error

about rigid bones [60, 58] or isometric deformable surfaces [37] and typically involve complex, slow optimization per video. In addition, they may require foreground-background segmentation of the moving content, which is not always easily obtained. Alternatively, some methods are specifically tailored to certain object classes like humans [52], restricting their domain to those limited cases. Consequently, these prior methods are either not directly applicable to casual videos, or require long optimization time per video.

Our approach. We propose TRACKSTO4D, 1 a novel approach for fast reconstruction of sparse dynamic 3D point clouds and camera poses from casual videos. Our main idea is to train a neural network on multiple videos to learn the mapping from the input image sequence to a sequence of the scene's 3D point clouds and camera poses. After training, the trained network can be efficiently applied to new image sequences using a single feed-forward pass, avoiding costly optimization.

To enhance the method's ability to generalize across different types of videos and scenes, we made a crucial design choice: our approach processes point track tensors as input, rather than operating directly on the image sequence. Specifically, each entry ( n, p ) in these tensors represents the 2D position of a tracked point p in a specific video frame n [7]. Our main insight is that point track tensors may exhibit more common motion patterns across casual video domains compared to image pixels. In other words, we argue that processing the raw point track data rather than scene-specific pixels or features may enable learning class and scene-agnostic internal feature representations for improved generalization. Importantly, recent advances in point tracking [11, 20] enable efficiently inferring these point tracks from casual videos using pre-trained models. These two properties make point track matrices an attractive input for our learning method.

Following this design choice, we design our architecture according to two principles: (1) process point track tensors, which have a unique structure, and (2) encode meaningful prior knowledge about the reconstruction problem, as the problem is ill-posed in general. In the following, we address these desired properties.

First, we design a network architecture that can effectively and efficiently handle point track inputs. To do that, we propose a novel layer design that takes into account the symmetries of the problem: the mapping we aim to learn, from point track matrices to 3D point clouds and camera poses, preserves two natural symmetries: (i) the points being tracked can be arbitrarily permuted without affecting the problem; (ii) the frames containing these points exhibit temporal structure, adhering to an approximate time-translation symmetry. Following the Geometric Deep Learning paradigm [8], we build upon recent theoretical advances in equivariant learning [30] and integrate these two symmetries into our network architecture using dedicated attention and positional encoding mechanisms.

Second, a key challenge in predicting 3D dynamic motion and camera poses from 2D point tracks is that this problem is inherently ill-posed without additional constraints [3]. To address this, we integrate a low-rank movement assumption into our architecture, following the seminal work of [7] which constrained output point clouds to be linear combinations of basis elements. Specifically, given an input point track tensor, our architecture equivariantly predicts a small set of input-specific basis elements. The output point clouds at each time frame are then defined as a linear combination of these basis elements, with the coefficients also predicted by the network. Notably, the first basis is assumed to fully represent the 3D static points in the video, while the remaining basis elements capture the 3D dynamic deviations. This structure effectively restricts the predicted point clouds to have a more specific form, making the problem more constrained.

Our network is trained on a dataset of extracted point track matrices [20] from raw videos without any 3D supervision by simply minimizing the reprojection errors, aiming to predict output point clouds that, after undergoing a perspective projection, will return the original 2D point tracks. In our experiments, TRACKSTO4D is trained on the Common Pets dataset [44]. We evaluate our method on test data with GT camera poses and GT depth information for point tracks, and demonstrate that it produces comparable results to state-of-the-art methods, while having a significantly shorter inference time by up to 95% . In addition, we show the method's ability to generalize to out-of-domain videos.

Contributions. In summary, our contributions are (1) A novel modeling of the dynamic reconstruction problem via learning on point tracks without 3D supervision; (2) A novel deep learning architecture incorporating two key principles: accounting for the symmetry of the data and encoding low-rank structure in the predicted point clouds (3) Experiments demonstrating extremely fast inference time compared to baselines, accurate results, and strong generalization across other categories.

2 Method

Problem formulation. Given a video of N frames, let M ∈ R N × P × 3 be a pre-extracted 2 D point tracks tensor (Fig. 1, left side). This tensor represents the two-dimensional information about a set of P world points that are tracked throughout the video. Each element in the tensor, M i,j, : , stores three values: ( x, y, o ) where x, y ∈ R are respectively the observed horizontal and vertical locations of point j in frame i , and o ∈ { 0 , 1 } indicates whether point j is observed in frame i or not. Our goal is to train a deep neural network to map the input point tracks tensor M into a set of per-frame camera poses { R i ( M ) , t i ( M ) } N i =1 and per-frame 3D points { X i ( M ) } N i =1 , where R i ( M ) ∈ SO (3) , t i ( M ) ∈ R 3 , X i ( M ) ∈ R P × 3 (Fig. 1, right side).

Overview of our approach. Our method receives M ∈ R N × P × 3 as input. This tensor is being processed by a neural architecture composed of multi-head attention layers where the attention is applied in an alternating fashion on the P and the N dimensions in each layer. These layers are defined in Sec. 2.1. After a composition of several such layers, the network uses the resulting features in R N × P × d to predict N camera poses in SO 3 × R 3 and N point clouds in R N × P × 3 . These N point clouds are parameterized as a linear combination of K input specific point cloud bases B 1 ( M ) , . . . B K ( M ) ∈ R P × 3 . This is discussed in detail in Sec. 2.2. Our network is trained in an unsupervised way on a dataset of videos by minimizing the reprojection error and other regularization losses (Sec. 2.3) that are used to update the model parameters. Our pipeline is illustrated in Fig. 1

2.1 Equivariant layers for point track tensors

Following the geometric deep learning paradigm, our goal is to design a neural architecture that respects the underlying symmetries and structure of the data.

Symmetry analysis. Our input is a tensor M ∈ R N × P × 3 representing a sequence of N frames each with P point coordinates. This structure gives rise to two key symmetries: First, the order of

the P points within each frame does not matter - in other words, permuting this axis results in an equivalent problem [30, 39, 63] 2 .

Formally, this axis has a permutation symmetry S P where S P is the symmetric group on P elements. Second, along the temporal N axis, we have an approximate translation symmetry arising from the ordered video sequence. This means that shifting the time frames is required to result in the same shift in our output. We model this with a cyclic group C N of order N . Both symmetries are illustrated in Fig. 2. We note that while the cyclic group assumption may not be entirely accurate, we still find it useful as it helps us to derive appropriate parametric layers for our data, similar to how the convolutional layer is derived for data with translational symmetries such as images. Taken together, the full symmetry group of the input space is the direct product G = C N × S P combining these time and point permutation symmetries, acting on R N × P × 3 by (( t, σ ) · M ) n,p,j = M t -1 ( n ) ,σ -1 ( p ) ,j for ( t, σ ) ∈ G 3 . Next, we will design an architecture equivariant to G , to ensure that the model takes into account the symmetries above.

Linear equivariant layers. Point track tensors can be viewed as a collection of N individual point tracks, each of which exhibits translational symmetry. The scenario where an object comprises a set of elements with their own symmetry group, such as a set of images or graphs, was previously explored in [30]. In that work, the authors characterized the general linear equivariant layer structure in such cases, termed the Deep Sets for Symmetric Elements (DSS) layer. Building on the DSS approach, our basic linear equivariant layer for the point track tensors M would take the form:

where L i are linear translation equivariant function (i.e. convolutions), M : ,j ∈ R N × d are the columns of M representing all the inputs for a specific tracked point, F ( M ) : ,j ∈ R N × d ′ is the output column and d, d ′ are the input and output feature channels respectively. To construct a neural network, these layers can be interleaved with pointwise nonlinearities, similar to basic convolutional neural networks.

Implementation via transformers and positional encoding. While the linear layer design is reasonable, it may not be the optimal choice. To enhance the model, we design a new layer whose structure follows Equation (1), but incorporates nonlinear layers in the form of transformers [51]. We note that the idea of using linear layers as inspiration for non-linear layers aligns with common practices in geometric deep learning, as described in several previous works [30, 4, 14].

Specifically, our layer F is formulated similarly to Equation (1), but instead of convolutions ( L i ) and summations, it utilizes two self-attention mechanisms and suitable temporal positional encoding across the N dimension. Formally, our basic layer F : R N × P × d → R N × P × d ′ is computed via four steps, which are described below:

Here, M i,j ∈ R d are the features associated with the j -th point in the i -th frame. The attention mechanism defined in the first equation above (2) is augmented with standard temporal positional

encoding in the first layer and replaces the translation equivariant function L i applied to the columns of M (Eq.(1)). The attention in the second equation (3) implements the set aggregation (summation) (also in Eq.(1)) applied to the rows of M . As commonly done, we use transformers with 16 attention heads [51].

2.2 Constraining 3D motion and camera poses via low-rank assumption

Given our 2D tracks, we aim to characterize the motion of the points by decomposing them into the global camera motion and the 3D motion of objects in the scene. The 2D motion of static scene points provides useful constraints for estimating the camera motion. However, as previously mentioned, predicting camera and dynamic 3D motion solely from 2D motion is an ill-posed problem without additional constraints [3]. We tackle this challenge by adding two mechanisms to our architecture: (1) low-rank movement assumption; and (2) specific modeling of the static scene for camera estimation.

Low-rank movement assumption. First, motivated by classical orthographic Non-Rigid Structure from Motion [7], we constrain the output points to be formulated by a linear combination of inputspecific basis elements. Specifically, given the input 2D point tracks, M ∈ R N × P × 3 , our network predicts K point clouds: B 1 ( M ) , . . . , B K ( M ) ∈ R P × 3 and N ( K -1) linear coefficients, { c 1 k ( M ) } K k =2 , . . . { c Nk ( M ) } K k =2 , such that

where X i ( M ) ∈ R P × 3 is the 3D point cloud at frame i . The point clouds and coefficients are computed by taking the output of the last equivariant layer as defined in the previous section and applying invariant aggregations on the respective dimension resulting in equivariant and invariant outputs. See more details in the appendix. We note that we deliberately chose the coefficient of B 1 ( M ) to be the constant 1 , the reason is explained in the next paragraph.

Specific modeling of the static scene for camera estimation. Frequently, casual video data of dynamic scenes contains many static regions, which can be used to determine camera poses [66]. We leverage this observation by treating the first basis element B 1 ( M ) ∈ R P × 3 as a static approximation for all scene points and encourage B 1 ( M ) as well as the output camera poses to explain the 2D observations according to this approximation using a "static" reprojection loss ( L Static, defined in the next section). We note, however, that a static point cloud is not likely to produce low reprojection errors for the non-static components, thus the reprojection error necessitates robustness to substantial errors from the non-static elements. To address this, our network predicts (equivariantly) P motion level values γ 1 ( M ) , . . . , γ P ( M ) ∈ R + , one for each point in our dynamic point cloud, which we use to weight the reprojection errors from B 1 ( M ) . The main idea is to give less weight to non-static points so that the static projection loss can disregard them. Specifically, inspired by [64], each γ i ( M ) defines a Cauchy distribution that models the reprojection errors for its associated world point, such that a world point with higher γ is expected to produce a wider error distribution. Empirically, as noted by [64], the Cauchy distribution tends to be more robust for modeling reprojection error uncertainties compared to Gaussian noise modeling [22]. Then, L Static, minimizes the negative log-likelihood under this assumption. See details in Sec. 2.3.

Comparison to supervised learning setups. A central aspect of our method is addressing the ill-posed nature of this problem through an architecture that heavily regularizes the output dynamic point clouds, enabling training without supervision. Alternatively, one could use video data with ground truth depth for supervised training, potentially allowing the output to have a more general form. Unfortunately, such supervision is both challenging to acquire and may lead to poor generalization on novel motion patterns.

2.3 Training and losses

Model outputs. Given the input 2D point tracks M ∈ R N × P × 3 , our network produces outputs as a function of M : linear bases and coefficients,

B 1 ( M ) , . . . , B K ( M ) ∈ R P × 3 , { c 1 k ( M ) } K k =2 , . . . , { c Nk ( M ) } K k =2 ∈ R which define a dynamic point cloud X 1 ( M ) , · · · , X N ( M ) ∈ R P × 3 , γ 1 ( M ) , . . . , γ P ( M ) ∈ R + movement level values, and ( R 1 ( M ) , t 1 ( M )) , . . . , ( R N ( M ) , t N ( M )) ∈ SO (3) × R 3 camera poses.

We use these network outputs to define a self-supervised loss function with respect to the current network weights and M which is defined by:

Reprojection loss. The reprojection loss encourages the consistency between the output 3D point clouds and camera poses, to the input 2D observations:

where R ( X ij , R i , t i , M xy ij ) is the reprojection error when projecting the point X ij with the camera pose ( R i , t i ) with respect to the measured point M xy ij :

Static loss. As discussed in Sec. 2.2, to better constrain the camera poses, the first predicted basis element B 1 ( M ) ∈ R P × 3 defines a static (fixed in time) point cloud approximation. Our network also predicts a movement coefficient γ j ( M ) for each world point that defines a zero-mean Cauchy distribution. Given γ j and the reprojection error r ij = R ( B 1 j , R i , t i , M xy ij ) 4 of the j th point of B 1 that is projected by the i th camera, the negative log-likelihood of r ij distributed according to the γ j -zero-mean Cauchy distribution is proportional to:

Note, that this loss reduces the contribution of the reprojection errors for points with high γ , but also encourages γ to be small, i.e. encouraging the static point cloud to represent the dynamic scene when possible. Our static loss is the mean negative log-likelihood over all observed points in all frames:

Regularization losses. As in [32] we regularize the observed points to be in front of the camera by:

We further find it beneficial to regularize the deviation from the static approximation B 1 to be sparse for static points, i.e. points with low γ values:

where γ is detached from the gradient calculation for this loss.

3 Experiments

In this section, we conduct experiments to verify our proposed network's performance on real-world casual videos. We began by training the network on specific domains and then evaluated its accuracy and running time on unseen videos from both, training and unseen domains.

Training procedure. We trained our network on the cat and dog partitions from the COP3D dataset [44], which contains a diverse set of casual real-world videos of pets. Our networks were trained from scratch three times to test our generalization capability between semantic categories: once on the cat partition, once on the dog partition, and once on both partitions combined. Training technical details are provided in the appendix. We use K = 12 bases in all our experiments (Sec. 2.2).

Evaluation data. To assess our framework's performance on pet videos, we curated a new dataset 5 consisting of 21 casual videos of dogs and cats, each video containing 50 frames. These videos were captured using an RGBD (RGB-Depth) sensor. The depth maps were used as ground truth for evaluating the reconstructed structure. We extracted the cameras by running COLMAP on the images while masking out the pet areas with dilatated masks provided by [71]. The cameras were scaled to millimeter units using the provided GT depth. Note that our network did not see this test data during training and it was not used to tune our hyperparameters.

Additionally, to evaluate our method on out-of-domain evaluation data, we used the Nvidia Dynamic Scenes Dataset [62]. Specifically, while our network was trained on pet videos, this dataset contains other dynamic object types, e.g. human, balloon, truck, and umbrella, with a different camera motion type and a variety of motion profiles. The dataset contains 8 dynamic scenes which are captured by 12 synchronized cameras, enabling accurate depth estimation which is treated as GT for evaluating monocular depth estimation. The ground truth camera poses were calculated by [42] with the synchronized multiview camera rig and the ground truth dynamic masks. Similarly to [27] we

simulated 8 monocular dynamic video sequences using the camera rig, each with 24 frames, and used them for evaluation.

Evaluation results. Qualitative visualizations are presented in Fig. 3. 6 We also show a visualization of the movement level values, γ in Fig. 4. For comparisons, we chose state-of-the-art methods that as our method, can be applied to raw casual videos that were captured by standard pinhole camera models and do not need any static or dynamic segmentation. We evaluated both, the camera poses and the structure accuracies. Comparison results for the pet-test-set and out-of-domain dataset are presented in Tables 1 and 2 respectively. The camera poses are evaluated compared to the GT, using the Absolute Translation Error(ATE), the Relative Translation Error(RTE), and the Relative Rotation Error(RRE) metrics after coordinates system alignment. We compare three training configurations of our method of training only on cats, only on dogs, and on both. As can be seen in the tables, the results are similar in all 3 cases. Our output camera poses as inferred by the network are already accurate and outperform some of the prior methods. We further show the results of our method after a single and short round of Bundle Adjustment, which makes our method better than all baselines on the pet sequences, and comparable on the out-of-domain cases.

Importantly, Tables 1 and 2 also compare the method's runtime. It can be seen that our method, even with bundle adjustment, is the fastest camera prediction method. Note that our method's runtimes include the point tracking time that is performed by [20] as a pre-process. In the appendix (Tab. 8), we present tracking time versus inference time, showing that most of the time is spent on tracking, while our inference is very fast. Tables 1 and 2 also show structure evaluation with the depth evaluation metrics [12] on the sampled point tracks. They demonstrate that our inferred structure is almost comparable to the state-of-the-art [64] while taking significantly shorter running times (a few seconds for our method versus more than two hours for [64] on pet videos). Short (0.6-5 minutes), per-sequence fine-tuning makes our method's accuracy comparable to [64]. In the appendix, we present per-scene output accuracy, demonstrating our ability to generalize across different speed profiles. In terms of running time, our method is a bit slower than MiDaS [5] which only provides depth maps without cameras, but achieves much better results 7 . We note that in contrast to the other methods that predict the dynamic depth, ours does not use any depth-from-single-image prior.

Ablation study To evaluate the contribution of our different method parts we run an ablation study which is presented in Tab. 3. In this study, the training was always done on the cat partition from COP3D and evaluated on our test data which contains dogs and cats. First, we performed an ablation study on our transformer architecture by taking the architecture suggested by [32] ("Set of Sets") or the DSS architecture that uses only linear layers [30] ("DSS"). As the table shows our architecture ("Full") achieved significantly better results. To test the losses in our framework, we also evaluated the following: (1) ignoring the γ outputs and using regular reprojection error on B 1 for all points ("No γ "); (2) removing our sparsity loss ("No L Sparse"); and (3) removing the static loss ("No L Static"). In all cases, the error increased whereas in the later one, the results became much worse. We further ablate the choice of K = 12 as the number of linear bases, by trying 2 extreme numbers of K = 30 , K = 2 (we saw no significant differences when we used nearby choices such as K = 11 ). As can be seen in the table, when K = 30 the output is not regularized enough and produces a higher

70!

depth error for the dynamic part. For K = 2 the depth is regularized but the reprojection error ("Rep. (pix.)") gets higher due to over-regularization. Overall, this study justifies our design choices ("Full").

4 Related Work

Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SfM) SfM pipelines seek to recover static 3D structure and camera poses from unordered images.[49, 45, 42, 1, 55]. Learning-free pipelines [42] are effective but require repeated applications of Bundle Adjustment (BA) [50]. [32, 9] presented a method for learning prior from a dataset of multiview image sets, to accelerate SfM pipelines by using equivariant deep networks. Monocular Simultaneous Localization and Mapping (SLAM) methods [33, 34, 13, 61, 6, 54, 67, 69, 46] extract camera poses from video sequences while defining a scene map with keyframes. These methods assume static scenes, fail to produce the cameras in scenes with large portions of dynamic motion, and cannot reproduce dynamic parts of the scene. DROID-SLAM [48] used synthetic data with ground truth 3D supervision for learning to predict camera poses via deep-based BA on keyframes while excluding dynamic objects. ParticleSfM [66] filters out 2D dynamic content for reproducing the cameras in dynamic scenes, using its pre-trained motion prediction network. Both, [48, 66] do not infer the dynamic 3D structure.

Orthographic Non-Rigid SfM (NRSfM) Bregler et al. [7] introduced a factorization method for computing a non-rigid structure and rotation matrices from a point track matrix, by assuming a low

dimensional basis model. While follow-up papers improved accuracies with different regularizations [26, 10, 36, 18] or neural representations [35, 24, 43], the orthographic camera model assumption is in general not valid for casual videos. Furthermore, these methods often assume background subtraction as a preprocessing. Even though a follow-up work proposed factorization solutions for pinhole cameras [16], its sensitivity to noise [19], makes it impractical for casual videos.

Test-time optimization for dynamic scenes Recent methods [29, 25, 64, 65] finetuned the monocular depth estimation from a pre-trained model [41, 40] using optical flow constraints [47], for obtaining consistent dense depth maps for a monocular video. [64] further optimized motion maps for handling scenes with large dynamic motion. [56, 15] use depth from single image estimations, to improve novel view synthesis in dynamic scenes. [28] further optimizes for the unknown camera poses together with the dynamic radiance field optimization. [37, 38] model a single deformable surface from a monocular video by applying isometric constraints. LASR [58], ViSER [59] and BANMo [60] optimize for a dynamic surface by assuming rigid bones and linear blend skinning weights. However, all the above-mentioned methods require per-scene optimization, resulting in slow inference. Recently, [44] presented the Common Pets in 3D (COP3D) dataset that contains casual, in-the-wild videos of pets, and used it to learn priors for novel view synthesis in dynamic scenes.

Point tracking There has been a recent advance in 2D point tracking by learning [20, 11], or optimization [53] techniques. Concurrently, [57] presented a method for jointly performing 2D tracking and 3D lifting, by learning to track with depth supervision while applying an as-rigid-aspossible loss. However, their method cannot predict camera poses or identify static parts directly.

5 Conclusions and limitations

We presented TRACKSTO4D, a novel deep-learning framework that directly maps 2D motion tracks from casual videos into their corresponding dynamic structure and camera motion. Our approach features a deep learning architecture that considers the symmetries in the problem with designed intrinsic constraints to handle the ill-posed nature of this problem. Notably, our network was trained using only raw supervision of 2D point tracks extracted by an off-the-shelf method [20] without any 3D supervision. Yet, it implicitly learned to predict camera poses and 3D structures while identifying the dynamic parts. During inference, our method demonstrates significantly faster processing times compared to previous methods while achieving comparable accuracy. Furthermore, our network exhibits strong generalization capabilities, performing well even on semantic categories that were not present in the training data.

Limitations and future work. While our experiments demonstrated that our network is efficient, accurate, and capable of generalizing to unseen video categories, there are several limitations and future work directions that we would like to address. First, our method is limited in handling videos with rapid motion, as it depends on the accuracy of the tracking method used, specifically CoTracker [20], which has limitations in tracking fast movements. We observed that [20] often fails in automotive scenes due to high motion blur, particularly on the road, limiting our ability to perform large-scale evaluations on automotive datasets given the current tracking method performance. We note that any future improvements with point tracking in terms of accuracy and inference time will directly improve our method as well. Second, our method assumes enough motion parallax to constrain the depth values and fails to generate accurate camera poses without it. An interesting direction for future work would be to leverage depth models and 3D tracking techniques, which could potentially improve accuracy, especially in cases with minimal motion parallax. Yet, our current attempts to incorporate MiDaS-inferred depth as an additional input and apply a depth loss relative to MiDaS have not yielded any performance improvements, indicating that this approach needs further exploration. Third, while we found K = 12 basis elements to be effective for our evaluation set, balancing complexity reduction and motion representation, we acknowledge this fixed number may not capture all possible scene dynamics. Future work could explore automatically inferring the optimal number of bases per scene. Lastly, our network can handle up to about 1000 point tracks in 50 frames in one inference step when running on a single GPU. A possible extension to handle denser point clouds and longer videos could involve querying point tracks iteratively while maintaining a shared state, but this approach remains to be explored.

Acknowledgments HMis the Robert J. Shillman Fellow, and is supported by the Israel Science Foundation through a personal grant (ISF 264/23) and an equipment grant (ISF 532/23).

References

Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S.M., Szeliski, R.: Building rome in a day. Communications of the ACM 54 (10), 105-112 (2011)

Agarwal, S., Mierle, K., Team, T.C.S.: Ceres Solver (10 2023), https://github.com/ ceres-solver/ceres-solver

Akhter, I., Sheikh, Y ., Khan, S., Kanade, T.: Nonrigid structure from motion in trajectory space. Advances in neural information processing systems 21 (2008)

Bevilacqua, B., Frasca, F., Lim, D., Srinivasan, B., Cai, C., Balamurugan, G., Bronstein, M.M., Maron, H.: Equivariant subgraph aggregation networks. arXiv preprint arXiv:2110.02910 (2021)

Birkl, R., Wofk, D., Müller, M.: Midas v3.1 - a model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307.14460 (2023)

Bloesch, M., Czarnowski, J., Clark, R., Leutenegger, S., Davison, A.J.: Codeslam-learning a compact, optimisable representation for dense visual slam. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2560-2568 (2018)

Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3d shape from image streams. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662). vol. 2, pp. 690-696. IEEE (2000)

Bronstein, M.M., Bruna, J., Cohen, T., Veliˇ ckovi´ c, P.: Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 (2021)

Brynte, L., Iglesias, J.P., Olsson, C., Kahl, F.: Learning structure-from-motion with graph attention networks. arXiv preprint arXiv:2308.15984 (2023)

Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. International Journal of Computer Vision 107 , 101-122 (2014)

Doersch, C., Yang, Y., Vecerik, M., Gokay, D., Gupta, A., Aytar, Y., Carreira, J., Zisserman, A.: Tapir: Tracking any point with per-frame initialization and temporal refinement. arXiv preprint arXiv:2306.08637 (2023)

Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems 27 (2014)

Engel, J., Schöps, T., Cremers, D.: Lsd-slam: Large-scale direct monocular slam. In: European conference on computer vision. pp. 834-849. Springer (2014)

Frasca, F., Bevilacqua, B., Bronstein, M., Maron, H.: Understanding and extending subgraph gnns by rethinking their symmetries. Advances in Neural Information Processing Systems 35 , 31376-31390 (2022)

Gao, C., Saraf, A., Kopf, J., Huang, J.B.: Dynamic view synthesis from dynamic monocular video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5712-5721 (2021)

Hartley, R., Vidal, R.: Perspective nonrigid shape and motion recovery. In: Computer VisionECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part I 10. pp. 276-289. Springer (2008)

Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)

Iglesias, J.P., Olsson, C., Valtonen Örnhag, M.: Accurate optimization of weighted nuclear norm for non-rigid structure from motion. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16. pp. 21-37. Springer (2020)

Jensen, S.H.N., Doest, M.E.B., Aanæs, H., Del Bue, A.: A benchmark and evaluation of non-rigid structure from motion. International Journal of Computer Vision 129 (4), 882-899 (2021)

Karaev, N., Rocco, I., Graham, B., Neverova, N., Vedaldi, A., Rupprecht, C.: Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635 (2023)

Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Repurposing diffusionbased image generators for monocular depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems 30 (2017)

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

Kong, C., Lucey, S.: Deep non-rigid structure from motion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1558-1567 (2019)

Kopf, J., Rong, X., Huang, J.B.: Robust consistent video depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1611-1621 (2021)

Kumar, S., Van Gool, L.: Organic priors in non-rigid structure from motion. In: European Conference on Computer Vision. pp. 71-88. Springer (2022)

Li, Z., Niklaus, S., Snavely, N., Wang, O.: Neural scene flow fields for space-time view synthesis of dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6498-6508 (2021)

Liu, Y.L., Gao, C., Meuleman, A., Tseng, H.Y., Saraf, A., Kim, C., Chuang, Y.Y., Kopf, J., Huang, J.B.: Robust dynamic radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13-23 (2023)

Luo, X., Huang, J.B., Szeliski, R., Matzen, K., Kopf, J.: Consistent video depth estimation. ACM Transactions on Graphics (ToG) 39 (4), 71-1 (2020)

Maron, H., Litany, O., Chechik, G., Fetaya, E.: On learning sets of symmetric elements. In: International conference on machine learning. pp. 6734-6744. PMLR (2020)

Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65 (1), 99-106 (2021)

Moran, D., Koslowsky, H., Kasten, Y., Maron, H., Galun, M., Basri, R.: Deep permutation equivariant structure from motion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5976-5986 (2021)

Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics 31 (5), 1147-1163 (2015)

Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: Dtam: Dense tracking and mapping in real-time. In: 2011 international conference on computer vision. pp. 2320-2327. IEEE (2011)

Novotny, D., Ravi, N., Graham, B., Neverova, N., Vedaldi, A.: C3dpo: Canonical 3d pose networks for non-rigid structure from motion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7688-7697 (2019)

Oskarsson, M., Batstone, K., Astrom, K.: Trust no one: Low rank matrix factorization using hierarchical ransac. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5820-5829 (2016)

Parashar, S., Pizarro, D., Bartoli, A.: Isometric non-rigid shape-from-motion with riemannian geometry solved in linear time. IEEE transactions on pattern analysis and machine intelligence 40 (10), 2442-2454 (2017)

Parashar, S., Pizarro, D., Bartoli, A.: Robust isometric non-rigid structure-from-motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (10), 6409-6423 (2021)

Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652-660 (2017)

Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. ICCV (2021)

Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V .: Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (3) (2022)

Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4104-4113 (2016)

Sidhu, V., Tretschk, E., Golyanik, V., Agudo, A., Theobalt, C.: Neural dense non-rigid structure from motion with latent space constraints. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16. pp. 204-222. Springer (2020)

Sinha, S., Shapovalov, R., Reizenstein, J., Rocco, I., Neverova, N., Vedaldi, A., Novotny, D.: Common pets in 3d: Dynamic new-view synthesis of real-life deformable categories. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4881-4891 (2023)

Sturm, P., Triggs, B.: A factorization based algorithm for multi-image projective structure and motion. In: Computer Vision-ECCV'96: 4th European Conference on Computer Vision Cambridge, UK, April 15-18, 1996 Proceedings Volume II 4. pp. 709-720. Springer (1996)

Teed, Z., Deng, J.: Deepv2d: Video to depth with differentiable structure from motion. arXiv preprint arXiv:1812.04605 (2018)

Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pp. 402-419. Springer (2020)

Teed, Z., Deng, J.: Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems 34 , 16558-16569 (2021)

Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. International journal of computer vision 9 , 137-154 (1992)

Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment-a modern synthesis. In: Vision Algorithms: Theory and Practice: International Workshop on Vision Algorithms Corfu, Greece, September 21-22, 1999 Proceedings. pp. 298-372. Springer (2000)

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)

Wan, Z., Li, Z., Tian, M., Liu, J., Yi, S., Li, H.: Encoder-decoder with multi-level attention for 3d human shape and pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13033-13042 (2021)

Wang, Q., Chang, Y.Y., Cai, R., Li, Z., Hariharan, B., Holynski, A., Snavely, N.: Tracking everything everywhere all at once. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19795-19806 (2023)

Wang, W., Hu, Y., Scherer, S.: Tartanvo: A generalizable learning-based vo. In: Conference on Robot Learning. pp. 1761-1772. PMLR (2021)

Wu, C.: Towards linear-time incremental structure from motion. In: 2013 International Conference on 3D Vision-3DV 2013. pp. 127-134. IEEE (2013)

Xian, W., Huang, J.B., Kopf, J., Kim, C.: Space-time neural irradiance fields for free-viewpoint video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9421-9431 (2021)

Xiao, Y., Wang, Q., Zhang, S., Xue, N., Peng, S., Shen, Y., Zhou, X.: Spatialtracker: Tracking any 2d pixels in 3d space. arXiv preprint arXiv:2404.04319 (2024)

Yang, G., Sun, D., Jampani, V., Vlasic, D., Cole, F., Chang, H., Ramanan, D., Freeman, W.T., Liu, C.: Lasr: Learning articulated shape reconstruction from a monocular video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15980-15989 (2021)

Yang, G., Sun, D., Jampani, V., Vlasic, D., Cole, F., Liu, C., Ramanan, D.: Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. Advances in Neural Information Processing Systems 34 , 19326-19338 (2021)

Yang, G., Vo, M., Neverova, N., Ramanan, D., Vedaldi, A., Joo, H.: Banmo: Building animatable 3d neural models from many casual videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2863-2873 (2022)

Yang, N., Stumberg, L.v., Wang, R., Cremers, D.: D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1281-1292 (2020)

Yoon, J.S., Kim, K., Gallo, O., Park, H.S., Kautz, J.: Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5336-5345 (2020)

Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. Advances in neural information processing systems 30 (2017)

Zhang, Z., Cole, F., Li, Z., Rubinstein, M., Snavely, N., Freeman, W.T.: Structure and motion from casual videos. In: European Conference on Computer Vision. pp. 20-37. Springer (2022)

Zhang, Z., Cole, F., Tucker, R., Freeman, W.T., Dekel, T.: Consistent depth of moving objects in video. ACM Transactions on Graphics (TOG) 40 (4), 1-12 (2021)

Zhao, W., Liu, S., Guo, H., Wang, W., Liu, Y.J.: Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild. In: European Conference on Computer Vision. pp. 523-542. Springer (2022)

Zhao, W., Liu, S., Shu, Y., Liu, Y.J.: Towards better generalization: Joint depth-pose learning without posenet. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9151-9161 (2020)

Zheng, Y., Harley, A.W., Shen, B., Wetzstein, G., Guibas, L.J.: Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In: ICCV (2023)

Zhou, H., Ummenhofer, B., Brox, T.: Deeptam: Deep tracking and mapping. In: Proceedings of the European conference on computer vision (ECCV). pp. 822-838 (2018)

Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5745-5753 (2019)

Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Wang, J., Wang, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. Advances in Neural Information Processing Systems 36 (2024)

A Supplementary results

A.1 Video results

We provide supplementary video outputs of several cases from our test set. Each video presents the input video frames with a set of pre-extracted point tracks that are used as input to our network and presented in corresponding colors (left side), and the output cameras and dynamic 3D structure (right side). The output camera trajectory is presented as gray frustums, whereas the current camera is marked in red. The reconstructed 3D scene points are presented in corresponding colors to the input tracks. Note that the outputs presented in the videos were obtained at inference time, with a single feed-forward prediction, without any optimization or fine-tuning, on unseen test cases.

A.2 Extended quantitative evaluation

Per sequence quantitative numbers Per-sequence and mean quantitative comparisons for our 21 pet test videos are presented in Tab. 9 and Tab. 10. Tables with similar structure for the out-of-domain dataset are presented in Tab. 11 and Tab. 12. Tables 11 and 12 also demonstrate our robustness across various levels of dynamic motion. However, some differences in tolerance to motion patterns are evident. For instance, our depth accuracy on 'Skating' is slightly higher than on 'Truck'.

Handling tracking errors We note that the input point tracks extracted by CoTracker [18] are imperfect and contain noise and outliers. Nevertheless, our model shows significant robustness to these errors. Our model handles imperfect input through two features:

Static scene modeling: Ensures that only 2D motion that can be represented by a camera motion and truly static points is modeled statically. This makes our camera estimation robust to errors while pushing the non-modelled errors to the dynamic part.

Dynamic scene modeling: Using limited basis elements for dynamic parts inherently resists outliers and extreme anomalies.

To quantify this robustness, we conducted tests by adding Gaussian and uniform noise to CoTracker points. The results (see Tab. 4) confirm that our method tolerates significant noise levels, further validating its effectiveness in handling imperfect input data.

Robustness to tracking method Additionally, we tested using point tracks extracted by TAPIR [11] as input at inference, instead of the CoTracker point tracks [20]. This evaluates our method's ability to generalize to a different tracking method at inference than the one used during training. The results are shown in Tab. 5. Although we observed some degradation in accuracy, the results are still good especially after finetunning, demonstrating our method's robustness to different point trackers.

Synthetic Data Evaluation We ran an evaluation on multiple test cases from PointOdyssey [68] and compared them with CasualSAM [64], which is the most accurate baseline. The results are presented in Tab. 6. We observed that our method generalizes well to these cases while taking much less time than CasualSAM and maintaining high accuracy.

Comparsion with Marigold We also add new experiments with the Marigold depth estimation method [21]. The numbers are presented in Tab. 7, demonstrating that our method is more accurate in terms of depth accuracy.

B Implementation details

B.1 Architecture technical details

For learning high frequencies we map each input coordinate to sinusoidal functions as in [31] with L = 12 . We use 3 pairs of attention layers, each of frames attention followed by point attention. Each point (after sinusoidal functions embedding) is mapped into R 256 with a linear layer. Each attention layer is a function of the form F : R N × P × 256 → R N × P × 256 (see details above). Each attention uses 16 heads with K,Q,V ∈ R ( N or P ) × 64 followed by a fully connected network with 1

hidden layer of 2048 features. We then average over the rows to get per-point features P 0 ∈ R P × 256 and over the columns to get per-frame features F 0 ∈ R N × 256 . Finally, we map P 0 to per-point outputs P 1 ∈ R P × (3 K +1) ( K basis points and γ ) with a linear layer, and F 0 into per-camera outputs F 1 ∈ R N × (6+3+ K -1) ( 6 for the rotation parameters [70], 3 for the camera center, and K -1 linear coefficients) using a convolutional layer with a kernel size of 31 .

B.2 Training details

In total, we used 733 cat videos and 753 dog videos for training. We trained our networks for 7000 and 3500 epochs for the single-class and multi-class setups respectively. Training our method lasts about one week on a single Tesla V100 GPU with 32GB memory. We used the Adam optimizer [23] with a learning rate of 10 -4 . Our method assumes known camera internal parameters which are provided by the dataset and used to normalize the point tracks as a preprocessing step.

B.3 Licenses and links for existing assets

CoTracker[20] The code is available here: https://github.com/facebookresearch/co-tracker. It is released under Creative Commons Attribution-NonCommercial 4.0 International Public License

Common pets dataset [44] The data is available here: https://github.com/facebookresearch/cop3d. It is released under Creative Commons Attribution-NonCommercial 4.0 International Public License.

Nvidia Dynamic Scenes Dataset [62] The data is available here:

https://gorokee.github.io/jsyoon/dynamic_synth/. It is released under the CC_BY-NC-ND License.

B.4 Other implementation details

Point tracks sampling For building M ∈ R N × P × 3 we use the implementation of [20]. We sample a uniform grid of 15 × 15 2D points, starting from frame number 0 , 20 , 40 , . . . , and then track these points throughout the entire video (backward and forward). In Tab. 8 we show the effect of reducing the grid size at inference time, in terms of camera pose accuracy and running time. During training, at each iteration, we randomly sample 20-50 frames from the training videos and 100 point tracks, i.e. 20 ≤ N ≤ 50 and P = 100 . When sampling cameras and point tracks of size N × P × 3 from a larger tensor of size N ′ × P ′ × 3 we only take a point track if its starting tracking time is in the range [ t -N 2 , t + 3 N 2 ] , where t is the first sampled index. At inference time we take all the available point tracks. In both, training and inference time, we keep only point tracks that are observed in more than 10 frames.

Finetuning details For our fine-tuning (FT) in the main paper, we applied per-sequence fine-tunning of 500,100 iterations starting from our final checkpoint, for pets,out-of-domain data respectively. The fine-tuning is done as a post-processing by minimizing the original loss function on the given test video.

Test Set We used the RGBD camera of the iPhone 11 to record our 21 test videos of dogs and cats. Each frame has a resolution of 640 × 480 pixels. Note that the training set contained various types of resolutions. All pet owners who were photographed gave their permission for the animals to be photographed. For evaluation only, we define a point track as dynamic if its associated GT mask value is 1 for at least 40 frames. The GT masks are obtained by running [71] and searching for labels of dogs and cats. They were only used for evaluation and not used by our method at all. We verified that this data includes enough dynamic motion, by also including several videos that COLMAP failed to reconstruct without the masks. We further verified manually that the camera trajectories look reasonable.

The out-of-domain dataset contains video sequences with 24 frames, each of resolution of 546 × 288 . The GT dynamic masks are provided by the dataset. For this dataset, for evaluation only, we define a point track as dynamic if its associated GT mask value is 1 for at least 15 frames.

Bundle Adjustment After inference, as optional refinement, we take the output static approximation B 1 ∈ R P × 3 and the output camera poses { R 1 , . . . , R N } , { t 1 , . . . , t N } and apply Bundle Adjustment (BA). We use a 3D world point from B 1 if its associated γ is below 0 . 008 for the pets dataset and 0 . 005 for out-of-domain dataset. We optimize reprojection errors of a given observation M i,j , only if it is observed, i.e. M o i,j = 1 and if the initial reprojection error is below 10 pixels. We use the BA implementation provided by [32], which is based on the Ceres package [2].

Running times All inference running times were computed on a machine with NVIDIA RTX A6000 GPU and Intel(R) Core(TM) i7-9800X 3.80GHz CPU. Extracting point tracks with [20] took

8.6 and 2.5 seconds for each video on the pet-videos and out-of-domain videos respectively and included in the running time tables as part of our method inference time.

Training technical details In all training setups, we used: λ Reprojection = 50 . 0 , λ Static = 1 . 0 , λ Negative = 1 . 0 , λ Sparse = 0 . 001 . At the beginning of the training, we pre-train the camera poses to be located behind and facing the origin. This prevents cases in which the cameras are located in the middle of the initial point cloud s.t. many points have negative depths, which may result in bad convergence. More specifically, the pre-train loss is: L Pretrain = 1 N ∑ N i =1 1 100 ∥ ∥ t i -[0 , 0 , -15] T ∥ ∥ 2 + ∥ R i -I ∥ 2 F .

The pretrain runs until convergence ( L Pretrain < 10 -4 ) . During the main training, we detach gradients from B 1 and ( R 1 , t 1 ) . . . , ( R N , t N ) for L Reproject to stabilize the training. Until epoch 50 we sample sequences of N in the range [20 , 22] , and then we increase the range to [20 , 50] .

C Broader Impacts

Our work focuses on mapping video data to dynamic 3D point clouds and camera motion. Our network efficiently takes this video input and generates dynamic point cloud representations that model the evolving 3D structure as a set of points in space, enabling analysis of geometry changes across the video frames. This could improve applications in domains where a fast understanding of the spatiotemporal evolution of 3D structure from video data is valuable, such as motion capture, robotic perception, augmented reality, and dynamic scene reconstruction. As robotic capabilities advance, it could fundamentally reshape how society perceives and interacts with the domains that become automated by robots, as well as the individuals previously employed in those domains being replaced.

Seq13

δ <

1

.

25

↑

Balloon2

Table 12: Camera poses accuracy for out-of-domain data [62]. We show a comparison to previous methods on the predicted camera poses. We compare 3 ways of running our method. Ours (C): Our inference time outputs. Ours (C)+BA: Our inference time outputs, followed by a short Bundle Adjustment for a model that was trained only on cats. Ours (C) FT: The outputs of the model that was trained only on cats after fine-tuning our losses for each specific video

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] ,

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope and are demonstrated in our experiments (Section 3).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] ,

Justification: The paper discusses the limitations of the work in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All our architecture details and experimental setup are provided in Sections 2,3 respectively. Appendix B provides supplementary technical details. This allows reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the code once the paper is published.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training and test details are provided in Section 3, and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The standard deviation numbers for the different experiments are provided in the supplementary tables of Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Inference times are provided in the tables of Section 3. Training times and hardware details are provided in appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We verified that the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader impacts are discussed in Appendix C.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not see such risks in our paper, that models geometry from video sequences.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited the original papers that produce the code packages and datasets. We include the URLs and licenses in Appendix B.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets for now.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. We only run experiments on human datasets that have been collected and released by other papers.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. We only run experiments on human datasets that have been collected and released by other papers.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents TRACKSTO4D, a fast, encoder-based method for reconstructing 3D structures and camera positions from casual videos with dynamic content. It processes 2D point tracks using a single feed-forward pass, leveraging inherent symmetries and low-rank approximations. TRACKSTO4D is trained unsupervised, reducing runtime by up to 95% compared to state-of-the-art methods while maintaining accuracy. It generalizes well to unseen videos of different categories, offering an efficient solution for 3D reconstruction in applications like robot navigation and autonomous driving.\n\nSTRENGTHS:\nMethod:\n\n1. The method considers the symmetry of tracked points and the temporal sequence of the video. By designing an equivariant layer based on transformers and positional encoding, it effectively takes advantage of these properties.\n2. It also uses a low-rank movement assumption to decompose the motion of the 2D tracked points into the global camera motion and the 3D motion of objects in the scene. This converts an originally ill-posed problem into a solvable and intuitive problem.\n\nThe method incorporates hard constraints into the design, resulting in more efficient training and more accurate and constrained results. Overall, I think the method is very solid.\n\n\n\nResults:\n\n1. TRACKSTO4D significantly reduces the runtime for 3D reconstruction from casual videos by up to 95%, making it highly efficient and practical for real-time applications.\n2. The method generalizes well to unseen videos and different semantic categories, demonstrating robustness and versatility.\n3. The use of unsupervised training without the need for 3D supervision minimizes the dependency on annotated datasets, which are often expensive and time-consuming to create.\n\nThe evaluation results are fairly comprehensive.\n\nWEAKNESSES:\n1. The paper primarily focuses on runtime and accuracy compared to state-of-the-art methods, but it lacks a detailed evaluation on other important aspects like robustness to noise or handling of occlusions. There is also limited discussion on how the method scales with increasing video length, number of objects, or more complex scenes. Scalability is crucial for deploying the method in large-scale real-world environments. In particular, I'm very curious how the chosen $K$ would affect the performance since $K$ is like an indirect descriptor of how complicated the point cloud's components are in my opinion.\n2. The method's performance is heavily dependent on the quality of 2D point tracks extracted from videos. Poor quality or incorrect point tracking can adversely affect the 3D reconstruction accuracy. I'm curious how robust the method is against failing 2D point tracks.\n3. While the low-rank assumption helps in reducing complexity, it may not capture the full variability of highly dynamic or complex scenes, potentially limiting the method’s applicability in such scenarios.\n\nQUESTIONS:\n1. How robust is TRACKSTO4D to variations in the quality of 2D point tracks? For instance, how does it perform when the point tracks contain noise, are partially occluded, or have tracking errors? Does it strictly follow the 2D point tracks, or does it try to rectify them to some extent? Understanding this robustness is crucial for assessing its applicability in real-world scenarios where perfect point tracking is often unattainable.\n2. Can the method be extended to handle dense point clouds or full 3D reconstructions rather than sparse point clouds? If so, what modifications would be necessary, and what impact would this have on runtime and accuracy?\n\nLIMITATIONS:\n1. The method assumes that the movement patterns of the dynamic 3D points can be effectively represented using a low-rank approximation. This assumption might not hold true for all types of motion, particularly those involving complex deformations or highly non-linear dynamics.\n2. There is an assumption on the motion parallax, without which the method fails to generate accurate camera poses.\n3. The accuracy and robustness of the method are more or less dependent on the quality of the 2D point tracks extracted from the videos. Noisy or incomplete point tracks could adversely impact the 3D reconstruction and camera pose estimation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method for fast 3D reconstruction of dynamic structures (or 4D reconstruction) from monocular video. The model is a transformer architecture that takes a set of 2D point tracks as input, and lifts them to 3D. It is learned with re-projection losses on 2d tracks without 3D ground-truth. The method is trained on COP3D (containing cat/dog videos) and evaluated on two test datasets: (1) 21 self-collected videos of dogs and cats and (2) NVIDIA dynamic scene dataset, using depth metrics and camera tracking metrics. It is shown to perform on par with an optimization-based (CasualSAM) but much faster.\n\nSTRENGTHS:\n- The method tries to tackle a challenging and significant problem, and presents a viable approach that works decently well and very fast. \n- It does not need 3D ground-truth, or depth supervision, and shows good generalization ability to novel scenarios. \n- The authors also collected a new test set that has more accurate depth ground-truth than COP3D.\n\nWEAKNESSES:\n- The presentation can be improved. Specifically, I had difficulty understanding the points made in Sec 2.1. \n  - The permutation invariance of point sets has been discussed in early literature (e.g., pointnet), and it might be better to connect to that instead of starting from symmetry analysis. \n  - I also don't really understand what is the relation of \"linear equivariance\" and cyclic group, and how that relates to the proposed architecture. Is there any cyclic operation that moves the end of a sequence to the beginning?\n  - It seemed to me the final architecture is a standard transformer with some modifications. It would be good to highlight which part enforces translation equivariance.\n\n- Since the method does not require 3D ground-truth. In theory, it can be trained using any video. However, the model is only trained on cats and dogs in practice. I'd like to encourage the authors to train it on larger scale dataset and observe whether there is any performance gain on the held-out set.\n- It uses some hand-crafted shape and motion prior (low-rank). I'd be interested in seeing an analysis of how changing the flexibility of the model (by changing the number of basis) affects the results.\n- The evaluation can be further enhanced by reporting 3D end-point-errors, since depth only evaluates one aspect of 4D reconstruction. Tapvid3d [A] or synthetic datasets like Kubrics, PointOdyssey could provide such ground-truth data.\n\n[A] Koppula, Skanda, et al. \"TAPVid-3D: A Benchmark for Tracking Any Point in 3D.\" arXiv preprint arXiv:2407.05921 (2024).\n\nQUESTIONS:\n- Is there any mechanism to handle the errors in co-tracker's 2d trajectory?\n- Spatialtracker [B] is another relevant work, although then train on 3D ground-truth and rely on depth inputs. It would be helpful to discuss what are the pros and cons of using unlabelled/3D data to train the model.\n\n[B] Xiao, Yuxi, et al. \"SpatialTracker: Tracking Any 2D Pixels in 3D Space.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper proposes a feed-forward network that takes a set of 2D TAP curves as input, outputs the 3D curves as well as extracts the camera rigid SE(3) poses.  This paper exploits the permutation and time shift equivariance when designing the encoding network. To train the model, the main loss is the re-projection loss (similar to BA). Experiments show in-category improvement and novel category generalization.

STRENGTHS:
- The reviewer really likes the paper's style, exploiting the underlying structure of dynamic scenes, although it seems the symmetries exploited here are simple.
- The problem this paper is solving is very important and meaningful for the community. Although I do see more promising ways to solve the problem (see weakness), I do like this paper's perspective on this problem.
- The model learned on cats and dogs is somehow generalizable to new categories, verifying their effectiveness.

WEAKNESSES:
- I do like the equivariance part, but given the current advances in the community, I do find that maybe 3D-track or exploiting depth models may lead to better performance instead of learning everything from scratch.
- Solving a single scene efficiently: one important loss that supports the model learning is the reprojection error, which seems to conduct an implicit BA when training the model over many sequences, the reviewer is curious whether the same techniques can work on a single video efficiently. This is to justify the necessity of the feedforward network if the same technique can fit a single video efficiently as well.

QUESTIONS:
Please see weakness

LIMITATIONS:
discussed in last paragraph.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents TracksTo4D, a feed-forward approach for estimating 3D structure and camera poses from 2D point tracks. Authors propose a novel architecture that directly processes 2D point tracks, takes into account symmetries present in the data, and assumes movement patterns can be represented by a low-rank approximation. Notably, this approach achieves similar performance to state-of-the-art methods with significantly reduced runtime. Despite training on videos of animals, TracksTo4D generalizes to unseen videos and unseen semantic categories at inference time.\n\nSTRENGTHS:\nProblem Motivation. This paper clearly describes the problem they are trying to solve and explicitly outlines the inputs and outputs of their method. Learning to predict camera poses and 3D structure from 2D point tracks is a challenging and relevant problem of interest in the community. Few methods currently solve both subproblems using feed forward approaches. \n\nAblations. Authors extensively ablate different losses used in training their method and highlight the impact of training on different subsets of COP3D (e.g. cat vs. dog vs. cat + dog)\n\nClear Writing and Informative Visuals. The paper is well written and easy to follow. The provided visuals in the supplement clearly demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\nLimited Evaluation. Although many methods do not tackle jointly predicting 3D structure and camera poses, authors evaluate methods that support a subset of these tasks. However, these evaluations can be supplemented. MiDaS is quite old at this point, and there has been significant improvements in monocular depth estimation in recent years. Instead, one should evaluate multi-view or video depth estimation methods. Other simple baselines can be included to provide more context like Chained RAFT3D [1] and Lifted CoTracker [2].\n\nSmall Test Set Size. Authors evaluate on a subset of COP3D and the NVIDIA Dynamics dataset, both of which only have a limited number of videos. Instead, authors should consider evaluating on synthetic datasets like PointOddessey [3], which can allow authors to explore other dimensions of their method like data scaling (which is unique to this method because it is much faster than competitive methods). \n\n[1] Teed et. al. Raft3D: Scene Flow using Rigid Motion Embeddings. \n[2] Karaev et. al. CoTracker: It Is Better to Track Together.\n\nQUESTIONS:\nEvaluating Robustness to 2D Point Tracking Noise. Although authors use CoTracker in this work, it would be interesting to characterize how this method works when evaluated using a different point tracker at test-time. \n\nEvaluating Robustness to Speed of Dynamic Motion. Although authors highlight that their method does not perform well given rapid motion, it would be useful to quantify this to benchmark performance for future methods. It would be useful to show off performance on automotive datasets that have ground truth depth (via LiDAR) and precise camera poses. \n\nImpact of Data Scaling. Given the speed and generalizability of this approach, it would be interesting to explore how this method scales with more data, particularly since this method doesn't require labeled 3D data. \n\nDefining Out-of-Distribution. Since this paper takes 2D points tracks (instead of raw images) as input, its I would not consider different datasets and semantic classes to be out-of-distribution, since the 2D point tracks don't have any notion of semantics. Instead, it would be useful to benchmark on data with different speed profiles, and provide breakdown analysis by speed buckets.\n\nContextualizing Runtimes. Since this method runs CoTracker before processing, run times should take this into consideration as well. Although the proposed method is quite fast, I believe the 2D point tracker can be a limiting factor on real-world performance. \n\nSupplemental Baseline Comparisons. Although SpaTracker and DuST3R (CVPR 2024) are considered concurrent work, I think this paper can be further strengthen by comparing against such methods. Note that a lack of comparison to these approaches does not influence my ratings.\n\nLIMITATIONS:\nYes, authors highlight that their approach can't handle rapid motion and is limited by the tracking noise imputed by CoTracker.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}