Fast Encoder-Based 3D from Casual Videos via Point Track Processing

Authors: yoni  kasten
wuyue  lu
haggai  maron
nvidia  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:47:57

Abstract: This paper addresses the long-standing challenge of reconstructing 3D structures from videos with dynamic content. Current approaches to this problem were not designed to operate on casual videos recorded by standard cameras or require a long optimization time. 
  Aiming to significantly improve the efficiency of previous approaches, we present TracksTo4D, a learning-based approach that enables inferring 3D structure and camera positions from dynamic content originating from casual videos using a single efficient feed-forward pass. To achieve this, we propose operating directly over 2D point tracks as input and designing an architecture tailored for processing 2D point tracks. Our proposed architecture is designed with two key principles in mind: (1) it takes into account the inherent symmetries present in the input point tracks data, and (2) it assumes that the movement patterns can be effectively represented using a low-rank approximation. TracksTo4D is trained in an unsupervised way on a dataset of casual videos utilizing only the 2D point tracks extracted from the videos, without any 3D supervision. Our experiments show that TracksTo4D can reconstruct a temporal point cloud and camera positions of the underlying video with accuracy comparable to state-of-the-art methods, while drastically reducing runtime by up to 95\%.  We further show that TracksTo4D  generalizes well to unseen videos of unseen semantic categories at inference time.

Main paper: 
 introduction: 
 Predicting 3D geometry in dynamic scenes is a challenging problem. In this problem setup, we are given access to multiple images of a scene taken sequentially, e.g., from a monocular video camera, where both the content in the scene and the camera are moving. Our task is to reconstruct the dynamic 3D positions of the points seen in the images and the camera poses. This fundamental problem has gained significant interest from the research community over the years [7, 37, 26, 64] , mainly due to its important applications in many fields such as robot navigation, autonomous driving and 3D reconstruction of general environments [19] . Importantly, in contrast to static scenes where the epipolar geometry constraints hold between the corresponding points of different views [17] , determining the depth of a moving point from monocular views is an ill-posed problem [3] . This causes standard Structure from Motion techniques [42, 55, 33] to be inadequate in this setup [25] .Previous work and limitations. Many existing approaches for the above problem make simplifying assumptions that limit their applicability to real-world scenarios. Methods based on orthographic camera models and low-rank assumptions use matrix factorization techniques [7, 26] , but the orthographic camera assumption might not be realistic and may cause reconstruction errors. Techniques that incorporate depth priors often require lengthy optimization processes in order to make the depth estimates across frames consistent [25, 64] . Other physics-based approaches make assumptions Figure 1 : We present TRACKSTO4D, a method for mapping a set of 2D point tracks extracted from casual dynamic videos into their corresponding 3D locations and camera motion. At inference time, our network predicts the dynamic structure and camera motion in a single feed-forward pass. Our network takes as input a set of 2D point tracks (left) and uses several multi-head attention layers while alternating between the time dimension and the track dimension (middle). The network predicts cameras, per-frame 3D points, and per-world point movement value (right). The 3D point internal colors illustrate the predicted 3D movement level values, such that points with high/low 3D motion are presented in red/purple colors respectively. These outputs are used to reproject the predicted points into the frames for calculating the reprojection error losses. See details in the text. The reader is encouraged to watch the supplementary video visualizations.about rigid bones [60, 58] or isometric deformable surfaces [37] and typically involve complex, slow optimization per video. In addition, they may require foreground-background segmentation of the moving content, which is not always easily obtained. Alternatively, some methods are specifically tailored to certain object classes like humans [52] , restricting their domain to those limited cases. Consequently, these prior methods are either not directly applicable to casual videos, or require long optimization time per video.Our approach. We propose TRACKSTO4D,1 a novel approach for fast reconstruction of sparse dynamic 3D point clouds and camera poses from casual videos. Our main idea is to train a neural network on multiple videos to learn the mapping from the input image sequence to a sequence of the scene's 3D point clouds and camera poses. After training, the trained network can be efficiently applied to new image sequences using a single feed-forward pass, avoiding costly optimization.To enhance the method's ability to generalize across different types of videos and scenes, we made a crucial design choice: our approach processes point track tensors as input, rather than operating directly on the image sequence. Specifically, each entry (n, p) in these tensors represents the 2D position of a tracked point p in a specific video frame n [7] . Our main insight is that point track tensors may exhibit more common motion patterns across casual video domains compared to image pixels. In other words, we argue that processing the raw point track data rather than scene-specific pixels or features may enable learning class and scene-agnostic internal feature representations for improved generalization. Importantly, recent advances in point tracking [11, 20] enable efficiently inferring these point tracks from casual videos using pre-trained models. These two properties make point track matrices an attractive input for our learning method.Following this design choice, we design our architecture according to two principles: (1) process point track tensors, which have a unique structure, and (2) encode meaningful prior knowledge about the reconstruction problem, as the problem is ill-posed in general. In the following, we address these desired properties.First, we design a network architecture that can effectively and efficiently handle point track inputs. To do that, we propose a novel layer design that takes into account the symmetries of the problem: the mapping we aim to learn, from point track matrices to 3D point clouds and camera poses, preserves two natural symmetries: (i) the points being tracked can be arbitrarily permuted without affecting the problem; (ii) the frames containing these points exhibit temporal structure, adhering to an approximate time-translation symmetry. Following the Geometric Deep Learning paradigm [8] , we build upon recent theoretical advances in equivariant learning [30] and integrate these two symmetries into our network architecture using dedicated attention and positional encoding mechanisms.Second, a key challenge in predicting 3D dynamic motion and camera poses from 2D point tracks is that this problem is inherently ill-posed without additional constraints [3] . To address this, we integrate a low-rank movement assumption into our architecture, following the seminal work of [7] which constrained output point clouds to be linear combinations of basis elements. Specifically, given an input point track tensor, our architecture equivariantly predicts a small set of input-specific basis elements. The output point clouds at each time frame are then defined as a linear combination of these basis elements, with the coefficients also predicted by the network. Notably, the first basis is assumed to fully represent the 3D static points in the video, while the remaining basis elements capture the 3D dynamic deviations. This structure effectively restricts the predicted point clouds to have a more specific form, making the problem more constrained.Our network is trained on a dataset of extracted point track matrices [20] from raw videos without any 3D supervision by simply minimizing the reprojection errors, aiming to predict output point clouds that, after undergoing a perspective projection, will return the original 2D point tracks. In our experiments, TRACKSTO4D is trained on the Common Pets dataset [44] . We evaluate our method on test data with GT camera poses and GT depth information for point tracks, and demonstrate that it produces comparable results to state-of-the-art methods, while having a significantly shorter inference time by up to 95%. In addition, we show the method's ability to generalize to out-of-domain videos.Contributions. In summary, our contributions are (1) A novel modeling of the dynamic reconstruction problem via learning on point tracks without 3D supervision; (2) A novel deep learning architecture incorporating two key principles: accounting for the symmetry of the data and encoding low-rank structure in the predicted point clouds (3) Experiments demonstrating extremely fast inference time compared to baselines, accurate results, and strong generalization across other categories. 
method: 
 Problem formulation. Given a video of N frames, let M ∈ R N ×P ×3 be a pre-extracted 2D point tracks tensor (Fig. 1 , left side). This tensor represents the two-dimensional information about a set of P world points that are tracked throughout the video. Each element in the tensor, M i,j,: , stores three values: (x, y, o) where x, y ∈ R are respectively the observed horizontal and vertical locations of point j in frame i, and o ∈ {0, 1} indicates whether point j is observed in frame i or not. Our goal is to train a deep neural network to map the input point tracks tensor M into a set of per-frame camera poses {R i (M ), t i (M )} N i=1 and per-frame 3D points{X i (M )} N i=1 , where R i (M ) ∈ SO(3), t i (M ) ∈ R 3 , X i (M ) ∈ R P ×3 (Fig. 1, right side).Overview of our approach. Our method receives M ∈ R N ×P ×3 as input. This tensor is being processed by a neural architecture composed of multi-head attention layers where the attention is applied in an alternating fashion on the P and the N dimensions in each layer. These layers are defined in Sec. 2.1. After a composition of several such layers, the network uses the resulting features in R N ×P ×d to predict N camera poses in SO 3 × R 3 and N point clouds in R N ×P ×3 . These N point clouds are parameterized as a linear combination of K input specific point cloud bases B 1 (M ), . . . B K (M ) ∈ R P ×3 . This is discussed in detail in Sec. 2.2. Our network is trained in an unsupervised way on a dataset of videos by minimizing the reprojection error and other regularization losses (Sec. 2.3) that are used to update the model parameters. Our pipeline is illustrated in Fig. 1 2.1 Equivariant layers for point track tensors Following the geometric deep learning paradigm, our goal is to design a neural architecture that respects the underlying symmetries and structure of the data. Symmetry analysis. Our input is a tensor M ∈ R N ×P ×3 representing a sequence of N frames each with P point coordinates. This structure gives rise to two key symmetries: First, the order of the P points within each frame does not matter -in other words, permuting this axis results in an equivalent problem [30, 39, 63] 2 .Formally, this axis has a permutation symmetry S P where S P is the symmetric group on P elements. Second, along the temporal N axis, we have an approximate translation symmetry arising from the ordered video sequence. This means that shifting the time frames is required to result in the same shift in our output. We model this with a cyclic group C N of order N . Both symmetries are illustrated in Fig. 2 . We note that while the cyclic group assumption may not be entirely accurate, we still find it useful as it helps us to derive appropriate parametric layers for our data, similar to how the convolutional layer is derived for data with translational symmetries such as images. Taken together, the full symmetry group of the input space is the direct product G = C N × S P combining these time and point permutation symmetries, acting on R N ×P ×3 by ((t, σ) • M ) n,p,j = M t -1 (n),σ -1 (p),j for (t, σ) ∈ G3 . Next, we will design an architecture equivariant to G, to ensure that the model takes into account the symmetries above. Linear equivariant layers. Point track tensors can be viewed as a collection of N individual point tracks, each of which exhibits translational symmetry. The scenario where an object comprises a set of elements with their own symmetry group, such as a set of images or graphs, was previously explored in [30] . In that work, the authors characterized the general linear equivariant layer structure in such cases, termed the Deep Sets for Symmetric Elements (DSS) layer. Building on the DSS approach, our basic linear equivariant layer for the point track tensors M would take the form:EQUATIONwhere L i are linear translation equivariant function (i.e. convolutions), M :,j ∈ R N ×d are the columns of M representing all the inputs for a specific tracked point, F (M ) :,j ∈ R N ×d ′ is the output column and d, d ′ are the input and output feature channels respectively. To construct a neural network, these layers can be interleaved with pointwise nonlinearities, similar to basic convolutional neural networks.Implementation via transformers and positional encoding. While the linear layer design is reasonable, it may not be the optimal choice. To enhance the model, we design a new layer whose structure follows Equation (1), but incorporates nonlinear layers in the form of transformers [51] . We note that the idea of using linear layers as inspiration for non-linear layers aligns with common practices in geometric deep learning, as described in several previous works [30, 4, 14] . Specifically, our layer F is formulated similarly to Equation (1), but instead of convolutions (L i ) and summations, it utilizes two self-attention mechanisms and suitable temporal positional encoding across the N dimension. Formally, our basic layer F : R N ×P ×d → R N ×P ×d ′ is computed via four steps, which are described below:qij = W Q M ij , kij = W K M ij , vij = W V M ij ⇒ Mij = N i ′ =1 exp(q ij • ki ′ j ) N l=1 exp(q ij • klj ) vi ′ j (2)q ij = W Q Mij , k ij = W K Mij , v ij = W V Mij ⇒ F (M ) ij = P j ′ =1 exp(q ij • k ij ′ ) P l=1 exp(q ij • k il ) v ij ′ (3)Here, M i,j ∈ R d are the features associated with the j-th point in the i-th frame. The attention mechanism defined in the first equation above ( 2) is augmented with standard temporal positional encoding in the first layer and replaces the translation equivariant function L i applied to the columns of M (Eq.( 1)). The attention in the second equation (3) implements the set aggregation (summation) (also in Eq.( 1)) applied to the rows of M . As commonly done, we use transformers with 16 attention heads [51] .Given our 2D tracks, we aim to characterize the motion of the points by decomposing them into the global camera motion and the 3D motion of objects in the scene. The 2D motion of static scene points provides useful constraints for estimating the camera motion. However, as previously mentioned, predicting camera and dynamic 3D motion solely from 2D motion is an ill-posed problem without additional constraints [3] . We tackle this challenge by adding two mechanisms to our architecture: (1) low-rank movement assumption; and (2) specific modeling of the static scene for camera estimation.Low-rank movement assumption. First, motivated by classical orthographic Non-Rigid Structure from Motion [7] , we constrain the output points to be formulated by a linear combination of inputspecific basis elements. Specifically, given the input 2D point tracks, M ∈ R N ×P ×3 , our network predicts K point clouds: B 1 (M ), . . . , B K (M ) ∈ R P ×3 and N (K -1) linear coefficients,EQUATIONwhere X i (M ) ∈ R P ×3 is the 3D point cloud at frame i. The point clouds and coefficients are computed by taking the output of the last equivariant layer as defined in the previous section and applying invariant aggregations on the respective dimension resulting in equivariant and invariant outputs. See more details in the appendix. We note that we deliberately chose the coefficient of B 1 (M ) to be the constant 1, the reason is explained in the next paragraph.Specific modeling of the static scene for camera estimation. Frequently, casual video data of dynamic scenes contains many static regions, which can be used to determine camera poses [66] . We leverage this observation by treating the first basis element B 1 (M ) ∈ R P ×3 as a static approximation for all scene points and encourage B 1 (M ) as well as the output camera poses to explain the 2D observations according to this approximation using a "static" reprojection loss (L Static , defined in the next section). We note, however, that a static point cloud is not likely to produce low reprojection errors for the non-static components, thus the reprojection error necessitates robustness to substantial errors from the non-static elements. To address this, our network predicts (equivariantly) P motion level values γ 1 (M ), . . . , γ P (M ) ∈ R + , one for each point in our dynamic point cloud, which we use to weight the reprojection errors from B 1 (M ). The main idea is to give less weight to non-static points so that the static projection loss can disregard them. Specifically, inspired by [64] , each γ i (M ) defines a Cauchy distribution that models the reprojection errors for its associated world point, such that a world point with higher γ is expected to produce a wider error distribution. Empirically, as noted by [64] , the Cauchy distribution tends to be more robust for modeling reprojection error uncertainties compared to Gaussian noise modeling [22] . Then, L Static , minimizes the negative log-likelihood under this assumption. See details in Sec. 2.3.Comparison to supervised learning setups. A central aspect of our method is addressing the ill-posed nature of this problem through an architecture that heavily regularizes the output dynamic point clouds, enabling training without supervision. Alternatively, one could use video data with ground truth depth for supervised training, potentially allowing the output to have a more general form. Unfortunately, such supervision is both challenging to acquire and may lead to poor generalization on novel motion patterns.Model outputs. Given the input 2D point tracks M ∈ R N ×P ×3 , our network produces outputs as a function of M : linear bases and coefficients,B 1 (M ), . . . , B K (M ) ∈ R P ×3 , {c 1k (M )} K k=2 , . . . , {c N k (M )} K k=2 ∈ R which define a dynamic point cloud X 1 (M ), • • • , X N (M ) ∈ R P ×3 ,γ 1 (M ), . . . , γ P (M ) ∈ R + movement level values, and (R 1 (M ), t 1 (M )), . . . , (R N (M ), t N (M )) ∈ SO(3) × R 3 camera poses.We use these network outputs to define a self-supervised loss function with respect to the current network weights and M which is defined by:EQUATION)Reprojection loss. The reprojection loss encourages the consistency between the output 3D point clouds and camera poses, to the input 2D observations:EQUATIONwhereR(X ij , R i , t i , M xy ij )is the reprojection error when projecting the point X ij with the camera pose (R i , t i ) with respect to the measured point M xy ij :EQUATIONStatic loss. As discussed in Sec. 2.2, to better constrain the camera poses, the first predicted basis element B 1 (M ) ∈ R P ×3 defines a static (fixed in time) point cloud approximation. Our network also predicts a movement coefficient γ j (M ) for each world point that defines a zero-mean Cauchy distribution. Given γ j and the reprojection error 4 of the j th point of B 1 that is projected by the i th camera, the negative log-likelihood of r ij distributed according to the γ j -zero-mean Cauchy distribution is proportional to:r ij = R(B 1j , R i , t i , M xy ij )EQUATIONNote, that this loss reduces the contribution of the reprojection errors for points with high γ, but also encourages γ to be small, i.e. encouraging the static point cloud to represent the dynamic scene when possible. Our static loss is the mean negative log-likelihood over all observed points in all frames:EQUATIONRegularization losses. As in [32] we regularize the observed points to be in front of the camera by:EQUATIONWe further find it beneficial to regularize the deviation from the static approximation B 1 to be sparse for static points, i.e. points with low γ values:EQUATION)where γ is detached from the gradient calculation for this loss. 
experiments: 
 In this section, we conduct experiments to verify our proposed network's performance on real-world casual videos. We began by training the network on specific domains and then evaluated its accuracy and running time on unseen videos from both, training and unseen domains. Training procedure. We trained our network on the cat and dog partitions from the COP3D dataset [44] , which contains a diverse set of casual real-world videos of pets. Our networks were trained from scratch three times to test our generalization capability between semantic categories: once on the cat partition, once on the dog partition, and once on both partitions combined. Training technical details are provided in the appendix. We use K = 12 bases in all our experiments (Sec. 2.2).Evaluation data. To assess our framework's performance on pet videos, we curated a new dataset5 consisting of 21 casual videos of dogs and cats, each video containing 50 frames. These videos were captured using an RGBD (RGB-Depth) sensor. The depth maps were used as ground truth for evaluating the reconstructed structure. We extracted the cameras by running COLMAP on the images while masking out the pet areas with dilatated masks provided by [71] . The cameras were scaled to millimeter units using the provided GT depth. Note that our network did not see this test data during training and it was not used to tune our hyperparameters.Additionally, to evaluate our method on out-of-domain evaluation data, we used the Nvidia Dynamic Scenes Dataset [62] . Specifically, while our network was trained on pet videos, this dataset contains other dynamic object types, e.g. human, balloon, truck, and umbrella, with a different camera motion type and a variety of motion profiles. The dataset contains 8 dynamic scenes which are captured by 12 synchronized cameras, enabling accurate depth estimation which is treated as GT for evaluating monocular depth estimation. The ground truth camera poses were calculated by [42] with the synchronized multiview camera rig and the ground truth dynamic masks. Similarly to [27] we Evaluation results. Qualitative visualizations are presented in Fig. 3 . 6 We also show a visualization of the movement level values, γ in Fig. 4 . For comparisons, we chose state-of-the-art methods that as our method, can be applied to raw casual videos that were captured by standard pinhole camera models and do not need any static or dynamic segmentation. We evaluated both, the camera poses and the structure accuracies. Comparison results for the pet-test-set and out-of-domain dataset are presented in Tables 1 and 2 respectively. The camera poses are evaluated compared to the GT, using the Absolute Translation Error(ATE), the Relative Translation Error(RTE), and the Relative Rotation Error(RRE) metrics after coordinates system alignment. We compare three training configurations of our method of training only on cats, only on dogs, and on both. As can be seen in the tables, the results are similar in all 3 cases. Our output camera poses as inferred by the network are already accurate and outperform some of the prior methods. We further show the results of our method after a single and short round of Bundle Adjustment, which makes our method better than all baselines on the pet sequences, and comparable on the out-of-domain cases.Importantly, Tables 1 and 2 also compare the method's runtime. It can be seen that our method, even with bundle adjustment, is the fastest camera prediction method. Note that our method's runtimes include the point tracking time that is performed by [20] as a pre-process. In the appendix (Tab. 8), we present tracking time versus inference time, showing that most of the time is spent on tracking, while our inference is very fast. Tables 1 and 2 also show structure evaluation with the depth evaluation metrics [12] on the sampled point tracks. They demonstrate that our inferred structure is almost comparable to the state-of-the-art [64] while taking significantly shorter running times (a few seconds for our method versus more than two hours for [64] on pet videos). Short (0.6-5 minutes), per-sequence fine-tuning makes our method's accuracy comparable to [64] . In the appendix, we present per-scene output accuracy, demonstrating our ability to generalize across different speed profiles. In terms of running time, our method is a bit slower than MiDaS [5] which only provides depth maps without cameras, but achieves much better results 7 . We note that in contrast to the other methods that predict the dynamic depth, ours does not use any depth-from-single-image prior.Ablation study To evaluate the contribution of our different method parts we run an ablation study which is presented in Tab. 3 . In this study, the training was always done on the cat partition from COP3D and evaluated on our test data which contains dogs and cats. First, we performed an ablation study on our transformer architecture by taking the architecture suggested by [32] ("Set of Sets") or the DSS architecture that uses only linear layers [30] ("DSS"). As the table shows our architecture ("Full") achieved significantly better results. To test the losses in our framework, we also evaluated the following: (1) ignoring the γ outputs and using regular reprojection error on B 1 for all points ("No γ"); (2) removing our sparsity loss ("No L Sparse "); and (3) removing the static loss ("No L Static "). In all cases, the error increased whereas in the later one, the results became much worse. We further ablate the choice of K = 12 as the number of linear bases, by trying 2 extreme numbers of K = 30, K = 2 (we saw no significant differences when we used nearby choices such as K = 11).As can be seen in the table, when K = 30 the output is not regularized enough and produces a higher In each video sequence, we show the input tracks, where each color visualizes its movement level value, γ. Purple marks static points with low γ whereas red marks dynamic points with high γ. Note, that our network did not get any direct supervision for these values, but only the raw point tracks predictions from [20] .The γ visualizations for cats were produced by the model that was only trained on dogs and vice versa. We note that our model generalizes well to out-of-domain (non-pet) cases as well.depth error for the dynamic part. For K = 2 the depth is regularized but the reprojection error ("Rep. (pix.)") gets higher due to over-regularization. Overall, this study justifies our design choices ("Full"). 
related work: 
 Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SfM) SfM pipelines seek to recover static 3D structure and camera poses from unordered images. [49, 45, 42, 1, 55] . Learning-free pipelines [42] are effective but require repeated applications of Bundle Adjustment (BA) [50] . [32, 9] presented a method for learning prior from a dataset of multiview image sets, to accelerate SfM pipelines by using equivariant deep networks. Monocular Simultaneous Localization and Mapping (SLAM) methods [33, 34, 13, 61, 6, 54, 67, 69, 46] extract camera poses from video sequences while defining a scene map with keyframes. These methods assume static scenes, fail to produce the cameras in scenes with large portions of dynamic motion, and cannot reproduce dynamic parts of the scene. DROID-SLAM [48] used synthetic data with ground truth 3D supervision for learning to predict camera poses via deep-based BA on keyframes while excluding dynamic objects. ParticleSfM [66] filters out 2D dynamic content for reproducing the cameras in dynamic scenes, using its pre-trained motion prediction network. Both, [48, 66] do not infer the dynamic 3D structure.Orthographic Non-Rigid SfM (NRSfM) Bregler et al. [7] introduced a factorization method for computing a non-rigid structure and rotation matrices from a point track matrix, by assuming a low dimensional basis model. While follow-up papers improved accuracies with different regularizations [26, 10, 36, 18] or neural representations [35, 24, 43] , the orthographic camera model assumption is in general not valid for casual videos. Furthermore, these methods often assume background subtraction as a preprocessing. Even though a follow-up work proposed factorization solutions for pinhole cameras [16] , its sensitivity to noise [19] , makes it impractical for casual videos.Test-time optimization for dynamic scenes Recent methods [29, 25, 64, 65] finetuned the monocular depth estimation from a pre-trained model [41, 40] using optical flow constraints [47] , for obtaining consistent dense depth maps for a monocular video. [64] further optimized motion maps for handling scenes with large dynamic motion. [56, 15] use depth from single image estimations, to improve novel view synthesis in dynamic scenes. [28] further optimizes for the unknown camera poses together with the dynamic radiance field optimization. [37, 38] model a single deformable surface from a monocular video by applying isometric constraints. LASR [58] , ViSER [59] and BANMo [60] optimize for a dynamic surface by assuming rigid bones and linear blend skinning weights. However, all the above-mentioned methods require per-scene optimization, resulting in slow inference. Recently, [44] presented the Common Pets in 3D (COP3D) dataset that contains casual, in-the-wild videos of pets, and used it to learn priors for novel view synthesis in dynamic scenes.Point tracking There has been a recent advance in 2D point tracking by learning [20, 11] , or optimization [53] techniques. Concurrently, [57] presented a method for jointly performing 2D tracking and 3D lifting, by learning to track with depth supervision while applying an as-rigid-aspossible loss. However, their method cannot predict camera poses or identify static parts directly. 
conclusions and limitations: 
 We presented TRACKSTO4D, a novel deep-learning framework that directly maps 2D motion tracks from casual videos into their corresponding dynamic structure and camera motion. Our approach features a deep learning architecture that considers the symmetries in the problem with designed intrinsic constraints to handle the ill-posed nature of this problem. Notably, our network was trained using only raw supervision of 2D point tracks extracted by an off-the-shelf method [20] without any 3D supervision. Yet, it implicitly learned to predict camera poses and 3D structures while identifying the dynamic parts. During inference, our method demonstrates significantly faster processing times compared to previous methods while achieving comparable accuracy. Furthermore, our network exhibits strong generalization capabilities, performing well even on semantic categories that were not present in the training data.Limitations and future work. While our experiments demonstrated that our network is efficient, accurate, and capable of generalizing to unseen video categories, there are several limitations and future work directions that we would like to address. First, our method is limited in handling videos with rapid motion, as it depends on the accuracy of the tracking method used, specifically CoTracker [20] , which has limitations in tracking fast movements. We observed that [20] often fails in automotive scenes due to high motion blur, particularly on the road, limiting our ability to perform large-scale evaluations on automotive datasets given the current tracking method performance. We note that any future improvements with point tracking in terms of accuracy and inference time will directly improve our method as well. Second, our method assumes enough motion parallax to constrain the depth values and fails to generate accurate camera poses without it. An interesting direction for future work would be to leverage depth models and 3D tracking techniques, which could potentially improve accuracy, especially in cases with minimal motion parallax. Yet, our current attempts to incorporate MiDaS-inferred depth as an additional input and apply a depth loss relative to MiDaS have not yielded any performance improvements, indicating that this approach needs further exploration. Third, while we found K = 12 basis elements to be effective for our evaluation set, balancing complexity reduction and motion representation, we acknowledge this fixed number may not capture all possible scene dynamics. Future work could explore automatically inferring the optimal number of bases per scene. Lastly, our network can handle up to about 1000 point tracks in 50 frames in one inference step when running on a single GPU. A possible extension to handle denser point clouds and longer videos could involve querying point tracks iteratively while maintaining a shared state, but this approach remains to be explored. hidden layer of 2048 features. We then average over the rows to get per-point features P 0 ∈ R P ×256 and over the columns to get per-frame features F 0 ∈ R N ×256 . Finally, we map P 0 to per-point outputs P 1 ∈ R P ×(3K+1) ( K basis points and γ) with a linear layer, and F 0 into per-camera outputs F 1 ∈ R N ×(6+3+K-1) (6 for the rotation parameters [70] , 3 for the camera center, and K -1 linear coefficients) using a convolutional layer with a kernel size of 31. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The training and test details are provided in Section 3, and Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The standard deviation numbers for the different experiments are provided in the supplementary tables of Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We verified that the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Broader impacts are discussed in Appendix C.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]  
 
 Appendix: In total, we used 733 cat videos and 753 dog videos for training. We trained our networks for 7000 and 3500 epochs for the single-class and multi-class setups respectively. Training our method lasts about one week on a single Tesla V100 GPU with 32GB memory. We used the Adam optimizer [23] with a learning rate of 10 -4 . Our method assumes known camera internal parameters which are provided by the dataset and used to normalize the point tracks as a preprocessing step.CoTracker [20] The code is available here: https://github. Table 8 : Tracking Grid Size Effect Quantitative evaluation of the effect of reducing the number of sampled point tracks at inference time on our model that was trained on cats (C). We measure the camera pose accuracy and the running time. We also mention the point tracks extraction time in parenthesis (e.g. +8.6 seconds) which is performed by [20] as a preprocess. As can be seen, our method can handle a smaller of points but the accuracy slightly drops with fewer sampled points 8.6 and 2.5 seconds for each video on the pet-videos and out-of-domain videos respectively and included in the running time tables as part of our method inference time.Training technical details In all training setups, we used: λ Reprojection = 50.0, λ Static = 1.0, λ Negative = 1.0, λ Sparse = 0.001. At the beginning of the training, we pre-train the camera poses to be located behind and facing the origin. This prevents cases in which the cameras are located in the middle of the initial point cloud s.t. many points have negative depths, which may result in bad convergence. More specifically, the pre-train loss is:L Pretrain = 1 N N i=1 1 100 t i -[0, 0, -15] T 2 + ∥R i -I∥ 2 F .The pretrain runs until convergence (L Pretrain < 10 -4 ). During the main training, we detach gradients from B 1 and (R 1 , t 1 ) . . . , (R N , t N ) for L Reproject to stabilize the training. Until epoch 50 we sample sequences of N in the range [20, 22] , and then we increase the range to [20, 50] .Our work focuses on mapping video data to dynamic 3D point clouds and camera motion. Our network efficiently takes this video input and generates dynamic point cloud representations that model the evolving 3D structure as a set of points in space, enabling analysis of geometry changes across the video frames. This could improve applications in domains where a fast understanding of the spatiotemporal evolution of 3D structure from video data is valuable, such as motion capture, robotic perception, augmented reality, and dynamic scene reconstruction. As robotic capabilities advance, it could fundamentally reshape how society perceives and interacts with the domains that become automated by robots, as well as the individuals previously employed in those domains being replaced.RCVD [25] MiDaS [5] CasualSAM [64] Ours 10 : Camera poses accuracy for pets. We show a comparison to previous methods on the predicted camera poses. We compare 3 ways of running our method. Ours (C): Our inference time outputs (total inference time of 0.16 seconds) for a model that was trained only on cats. Ours (C)+BA: Our inference time outputs, followed by a short Bundle Adjustment (total inference time of 0.4 seconds) for a model that was trained only on cats. Ours (C) FT: The outputs of the model that was trained only on cats after fine-tuning our losses for each specific video (total running time of about 5 minutes). As can be seen, after BA, our results are the most accurate compared to the other methods, and fine-tuning improves our accuracy even more.RCVD [25] MiDaS [5] CasualSAM [64] Ours 12 : Camera poses accuracy for out-of-domain data [62] . We show a comparison to previous methods on the predicted camera poses. We compare 3 ways of running our method. Ours (C): Our inference time outputs. Ours (C)+BA: Our inference time outputs, followed by a short Bundle Adjustment for a model that was trained only on cats. Ours (C) FT:The outputs of the model that was trained only on cats after fine-tuning our losses for each specific video• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.4D since we have three Euclidean coordinates with an additional time coordinateOur work builds upon the pioneering works PointNet[39] and Deep Sets[63] that studied permutations symmetries in general and for point clouds in particular. As we shall see next, in our case, the symmetry group is a bit more complicated and fits nicely in the setup of[30].This is different from the symmetry group studied in[32], where the temporal structure was not exploited.We denote the j th 3D point of B k ∈ R P ×3 by B kj ∈ R 3 . The 3 elements of this point are denoted by B kj1 , B kj2 , B kj3 ∈ R (see(11)).While the COP3D dataset provides cameras that were extracted by COLMAP[42], we note that this evaluation data is insufficient in our case. This is because the dynamic structure was captured as well in part of their reconstruction which indicates that its reconstruction might not be accurate. Furthermore, the coordinates system units of these reconstructions are unknown. Finally, this dataset does not have any depth map information for evaluating the dynamic structure.The reader is encouraged to watch the supplementary videos for better 4D perception.Our comparisons were made using MiDaS 3.1, specifically the dpt_beit_large_512 version(Birkl et al. 2023). This is an improved version of MiDaS that utilizes the DPT architecture, representing a more current state of the method. 


Decision: ## Summary

TracksTo4D is a feed-forward approach for estimating 3D structure and camera poses from 2D point tracks. It uses a novel architecture that directly processes 2D point tracks, considers symmetries, and assumes movement patterns can be represented by a low-rank approximation. The approach achieves similar performance to state-of-the-art methods with significantly reduced runtime. The method generalizes well to unseen videos and unseen semantic categories at inference time. The method is trained on COP3D and evaluated on two test datasets. TRACKSTO4D is a fast, encoder-based method for reconstructing 3D structures and camera positions from casual videos with dynamic content.

## Strengths
- The paper aims to predict camera poses and 3D structure from 2D point tracks. 
- The method considers the symmetry of tracked points and the temporal sequence of the video.
- It uses a low-rank movement assumption to decompose the motion of the 2D tracked points into the global camera motion and the 3D motion of objects in the scene.
- Few methods currently solve these subproblems using feed forward approaches.
- The authors extensively ablate different losses used in training their method.
- The impact of training on different subsets of COP3D is highlighted.
- The paper is well written and easy to follow
- TRACKSTO4D significantly reduces the runtime for 3D reconstruction from casual videos by up to 95%.

## Weaknesses
- The paper primarily focuses on runtime and accuracy compared to state-of-the-art methods.
- The method's performance is heavily dependent on the quality of 2D point tracks extracted from videos.
- The low-rank assumption may not capture the full variability of highly dynamic or complex scenes.
- Authors evaluate methods supporting a subset of 3D structure and camera poses.
- MiDaS is outdated and monocular depth estimation has improved.
- Multi-view or video depth estimation methods should be evaluated.
- Other baselines like Chained RAFT3D and Lifted CoTracker can provide more context.
- Authors evaluate on a subset of COP3D and the NVIDIA Dynamics dataset, which have a limited number of videos.
- Authors should consider evaluating on synthetic datasets like PointOddessey to explore other dimensions of their method.
## Overall
Based on the discussion, the paper should be accepted once all the points raised by the reviewers are addressed in the final version of the paper

Reviews: review: paper_summary: paper_summary: This paper presents TracksTo4D, a feed-forward approach for estimating 3D structure and camera poses from 2D point tracks. Authors propose a novel architecture that directly processes 2D point tracks, takes into account symmetries present in the data, and assumes movement patterns can be represented by a low-rank approximation. Notably, this approach achieves similar performance to state-of-the-art methods with significantly reduced runtime. Despite training on videos of animals, TracksTo4D generalizes to unseen videos and unseen semantic categories at inference time.
strength_weakness: strength_weakness: Limited Evaluation. Although many methods do not tackle jointly predicting 3D structure and camera poses, authors evaluate methods that support a subset of these tasks. However, these evaluations can be supplemented. MiDaS is quite old at this point, and there has been significant improvements in monocular depth estimation in recent years. Instead, one should evaluate multi-view or video depth estimation methods. Other simple baselines can be included to provide more context like Chained RAFT3D [1] and Lifted CoTracker [2].

Small Test Set Size. Authors evaluate on a subset of COP3D and the NVIDIA Dynamics dataset, both of which only have a limited number of videos. Instead, authors should consider evaluating on synthetic datasets like PointOddessey [3], which can allow authors to explore other dimensions of their method like data scaling (which is unique to this method because it is much faster than competitive methods). 

[1] Teed et. al. Raft3D: Scene Flow using Rigid Motion Embeddings. 
[2] Karaev et. al. CoTracker: It Is Better to Track Together.
questions: questions: Evaluating Robustness to 2D Point Tracking Noise. Although authors use CoTracker in this work, it would be interesting to characterize how this method works when evaluated using a different point tracker at test-time. 

Evaluating Robustness to Speed of Dynamic Motion. Although authors highlight that their method does not perform well given rapid motion, it would be useful to quantify this to benchmark performance for future methods. It would be useful to show off performance on automotive datasets that have ground truth depth (via LiDAR) and precise camera poses. 

Impact of Data Scaling. Given the speed and generalizability of this approach, it would be interesting to explore how this method scales with more data, particularly since this method doesn't require labeled 3D data. 

Defining Out-of-Distribution. Since this paper takes 2D points tracks (instead of raw images) as input, its I would not consider different datasets and semantic classes to be out-of-distribution, since the 2D point tracks don't have any notion of semantics. Instead, it would be useful to benchmark on data with different speed profiles, and provide breakdown analysis by speed buckets.

Contextualizing Runtimes. Since this method runs CoTracker before processing, run times should take this into consideration as well. Although the proposed method is quite fast, I believe the 2D point tracker can be a limiting factor on real-world performance. 

Supplemental Baseline Comparisons. Although SpaTracker and DuST3R (CVPR 2024) are considered concurrent work, I think this paper can be further strengthen by comparing against such methods. Note that a lack of comparison to these approaches does not influence my ratings.
limitations: limitations: Yes, authors highlight that their approach can't handle rapid motion and is limited by the tracking noise imputed by CoTracker.
paper_summary: paper_summary: This paper proposes a feed-forward network that takes a set of 2D TAP curves as input, outputs the 3D curves as well as extracts the camera rigid SE(3) poses.  This paper exploits the permutation and time shift equivariance when designing the encoding network. To train the model, the main loss is the re-projection loss (similar to BA). Experiments show in-category improvement and novel category generalization.
strength_weakness: strength_weakness: - I do like the equivariance part, but given the current advances in the community, I do find that maybe 3D-track or exploiting depth models may lead to better performance instead of learning everything from scratch.
- Solving a single scene efficiently: one important loss that supports the model learning is the reprojection error, which seems to conduct an implicit BA when training the model over many sequences, the reviewer is curious whether the same techniques can work on a single video efficiently. This is to justify the necessity of the feedforward network if the same technique can fit a single video efficiently as well.
questions: questions: Please see weakness
limitations: limitations: discussed in last paragraph.
paper_summary: paper_summary: The paper introduces a method for fast 3D reconstruction of dynamic structures (or 4D reconstruction) from monocular video. The model is a transformer architecture that takes a set of 2D point tracks as input, and lifts them to 3D. It is learned with re-projection losses on 2d tracks without 3D ground-truth. The method is trained on COP3D (containing cat/dog videos) and evaluated on two test datasets: (1) 21 self-collected videos of dogs and cats and (2) NVIDIA dynamic scene dataset, using depth metrics and camera tracking metrics. It is shown to perform on par with an optimization-based (CasualSAM) but much faster.
strength_weakness: strength_weakness: - The presentation can be improved. Specifically, I had difficulty understanding the points made in Sec 2.1. 
  - The permutation invariance of point sets has been discussed in early literature (e.g., pointnet), and it might be better to connect to that instead of starting from symmetry analysis. 
  - I also don't really understand what is the relation of "linear equivariance" and cyclic group, and how that relates to the proposed architecture. Is there any cyclic operation that moves the end of a sequence to the beginning?
  - It seemed to me the final architecture is a standard transformer with some modifications. It would be good to highlight which part enforces translation equivariance.

- Since the method does not require 3D ground-truth. In theory, it can be trained using any video. However, the model is only trained on cats and dogs in practice. I'd like to encourage the authors to train it on larger scale dataset and observe whether there is any performance gain on the held-out set.
- It uses some hand-crafted shape and motion prior (low-rank). I'd be interested in seeing an analysis of how changing the flexibility of the model (by changing the number of basis) affects the results.
- The evaluation can be further enhanced by reporting 3D end-point-errors, since depth only evaluates one aspect of 4D reconstruction. Tapvid3d [A] or synthetic datasets like Kubrics, PointOdyssey could provide such ground-truth data.

[A] Koppula, Skanda, et al. "TAPVid-3D: A Benchmark for Tracking Any Point in 3D." arXiv preprint arXiv:2407.05921 (2024).
questions: questions: - Is there any mechanism to handle the errors in co-tracker's 2d trajectory?
- Spatialtracker [B] is another relevant work, although then train on 3D ground-truth and rely on depth inputs. It would be helpful to discuss what are the pros and cons of using unlabelled/3D data to train the model.

[B] Xiao, Yuxi, et al. "SpatialTracker: Tracking Any 2D Pixels in 3D Space." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents TRACKSTO4D, a fast, encoder-based method for reconstructing 3D structures and camera positions from casual videos with dynamic content. It processes 2D point tracks using a single feed-forward pass, leveraging inherent symmetries and low-rank approximations. TRACKSTO4D is trained unsupervised, reducing runtime by up to 95% compared to state-of-the-art methods while maintaining accuracy. It generalizes well to unseen videos of different categories, offering an efficient solution for 3D reconstruction in applications like robot navigation and autonomous driving.
strength_weakness: strength_weakness: 1. The paper primarily focuses on runtime and accuracy compared to state-of-the-art methods, but it lacks a detailed evaluation on other important aspects like robustness to noise or handling of occlusions. There is also limited discussion on how the method scales with increasing video length, number of objects, or more complex scenes. Scalability is crucial for deploying the method in large-scale real-world environments. In particular, I'm very curious how the chosen $K$ would affect the performance since $K$ is like an indirect descriptor of how complicated the point cloud's components are in my opinion.
2. The method's performance is heavily dependent on the quality of 2D point tracks extracted from videos. Poor quality or incorrect point tracking can adversely affect the 3D reconstruction accuracy. I'm curious how robust the method is against failing 2D point tracks.
3. While the low-rank assumption helps in reducing complexity, it may not capture the full variability of highly dynamic or complex scenes, potentially limiting the method’s applicability in such scenarios.
questions: questions: 1. How robust is TRACKSTO4D to variations in the quality of 2D point tracks? For instance, how does it perform when the point tracks contain noise, are partially occluded, or have tracking errors? Does it strictly follow the 2D point tracks, or does it try to rectify them to some extent? Understanding this robustness is crucial for assessing its applicability in real-world scenarios where perfect point tracking is often unattainable.
2. Can the method be extended to handle dense point clouds or full 3D reconstructions rather than sparse point clouds? If so, what modifications would be necessary, and what impact would this have on runtime and accuracy?
limitations: limitations: 1. The method assumes that the movement patterns of the dynamic 3D points can be effectively represented using a low-rank approximation. This assumption might not hold true for all types of motion, particularly those involving complex deformations or highly non-linear dynamics.
2. There is an assumption on the motion parallax, without which the method fails to generate accurate camera poses.
3. The accuracy and robustness of the method are more or less dependent on the quality of the 2D point tracks extracted from the videos. Noisy or incomplete point tracks could adversely impact the 3D reconstruction and camera pose estimation.

Comments: title: Keep my original recommendation
Keep my original recommendation
comment: We thank the reviewer for the response and the positive feedback.

We agree with the reviewer's comment that our method opens the door to exciting future directions, particularly for the fast handling of longer videos with dense point cloud outputs. We will include this in the conclusion section of the revised version.
We thank the reviewer for the response and the positive feedback.

**Q**: The limiting factor is that the low-rank motion prior typically makes the model biased to simple motion. At the same time, as the authors observed, increasing K leads to worse results due to lack of regularization.



**A**: We found K=12 basis elements effective for our evaluation set, balancing complexity reduction and motion representation. However, we acknowledge this fixed number may not capture all possible scene dynamics. Future work could explore automatically inferring the optimal number of bases per scene. We'll add this point to our limitations and future work section.

**Q**:  I still think it is worth discussing using 3D ground-truth vs unlabelled video + hand-crafted motion prior.

**A**: We will add a discussion on the pros and cons of using unlabeled data and 3D ground truth data to train the model in the revised version.
We thank the reviewer for the response and the positive feedback.
We thank the reviewer for the response and the positive feedback.
I read the rebuttal and would like to keep my score. I find this work novel as it uses low-rank motion regularization to learn 3d tracking from unlabeled videos. 

The limiting factor is that the low-rank motion prior typically makes the model biased to simple motion. At the same time, as the authors observed, increasing K leads to worse results due to lack of regularization. 

Furthermore, I still think it is worth discussing using 3D ground-truth vs unlabelled video + hand-crafted motion prior.
Authors have sufficiently addressed my questions. I recommend this paper be accepted.
Thank the authors for providing more insights into their work. I find this paper solid enough to lay a foundation for future long-sequence and dense point cloud tracking system. I would like to keep my original recommendation.
After reading the reviews and rebuttals, I think this paper makes enough contribution to the community, I keep my original recommendation.