RGB-Only Supervised Camera Parameter Optimization in Dynamic Scenes
Authors: Fang Li
Hao Zhang
Narendra Ahuja
Venue: NeurIPS 2025 spotlight ()
Keywords: 4D reconstruction
camera parameter optimization; visual localization
Primary area: applications

Abstract: Although COLMAP has long remained the predominant method for camera parameter optimization in static scenes, it is constrained by its lengthy runtime and reliance on ground truth (GT) motion masks for application to dynamic scenes. Many efforts attempted to improve it by incorporating more priors as supervision such as GT focal length, motion masks, 3D point clouds, camera poses, and metric depth, which, however, are typically unavailable in casually captured RGB videos. In this paper, we propose a novel method for more accurate and efficient camera parameter optimization in dynamic scenes solely supervised by a single RGB video, dubbed $\textbf{\textit{ROS-Cam}}$. Our method consists of three key components: (1) Patch-wise Tracking Filters, to establish robust and maximally sparse hinge-like relations across the RGB video. (2) Outlier-aware Joint Optimization, for efficient camera parameter optimization by adaptive down-weighting of moving outliers, without reliance on motion priors. (3) A Two-stage Optimization Strategy, to enhance stability and optimization speed by a trade-off between the Softplus limits and convex minima in losses. We visually and numerically evaluate our camera estimates. To further validate accuracy, we feed the camera estimates into a 4D reconstruction method and assess the resulting 3D scenes, and rendered 2D RGB and depth maps.
We perform experiments on 4 real-world datasets (NeRF-DS, DAVIS, iPhone, and TUM-dynamics) and 1 synthetic dataset (MPI-Sintel), demonstrating that our method estimates camera parameters more efficiently and accurately with a single RGB video as the only supervision.

RGB-Only Supervised Camera Parameter Optimization in Dynamic Scenes
Fang Li
Hao Zhang
University of Illinois at Urbana-Champaign
University of Illinois at Urbana- Champaign
Champaign, IL 61820 fangli3Gillinos edu
Champaign, 61820 haoz190illing edu
Narendra Ahuja
University of Illinois at Urbana-Channn Chaamaaan
Champaign, IL 61820 n--ahjaa@illin edu
Secendio
OBRUNSTAN
ESHN753
65
SSIM D
LMPMS t4
ESHASS
Dorh ESNPOS 74
Porametor Opiiiiztiio
Tat DERRE Conses
COL
by Curs
Rendering Compatsons
Figure 1: (a) Overview of our RGB-only supervised camera parameter optimization. (b) Front view of the 3D Gaussian field reconstructed by our camera estimates at time t. (c) 2D renderings (RGB and depth) time with quantitative metrics. Our optimization is not only significantly more efficient and accurate, but also avoids overfitting the reconstruction to specific viewpoints. Record3D is mobile app that factory-calibrates the intrinsic and uses LiDAR sensors collect metric depth for camera pose estimates, thus does not have valid runtime.
Abstract
Although COLMAP has long remained the predominant method for camera pa- rameter optimization in static scenes, it is constrained by its lengthy runtime and reliance on ground truth (GT) motion masks for application dynamic scenes. Many efforts attempted to improve it by incorporating more priors as supervision such as GT focal length, motion masks, 3D point clouds, camera poses, and metric depth, which, however, are typically unavailable casually captured RGB videos. In this paper, we propose novel method for parameter optimization in dynamic scenes solely supervised by single RGB video, dubbed ROS-Cam. Our method consists of three key components: (1) Patch-wise Tracking Filters, to establish robust and maximally sparse hinge-like relations across the RGB video. (2) Outier-aware Joint Optimization, for efficient camera parameter optimization by adaptive down fown-weighting of moving outliers, without reliance on motion priors. (3) A Two-stage Optimization Strategy, to enhance sta- bility and optimization speed by trade-off between the Softplus limits and convex minima in losses. We visually and numerically evaluate our camera estimates. To
accurate and efficient camera
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

further validate accuracy, we feed the camera estimates into a 4D reconstruction method and assess the resulting 3D scenes, and rendered 2D RGB and depth maps. We perform experiments on 4 real-world datasets (NeRF-DS, DAVIS, iPhone, and TUM-dynamies) and 1 synthetic dataset (MPl-Sintel),, demonstrating that our method estimates camera parameters more efficiently and accurately with single RGB video as the only supervision.
Introduction
Despite recent progress in visual odometry, efficiently and accurately optimizing camera parameter (focal length + rotation&&taassltion) from casually collected RGB dynamic-scene videos remains a big challenge. Although the most predominant COLMAP (2 method is RGB-only supervised, it
suffers from its lengthy runtime and requisite of GT motion masks to mask out the outlier moving
stuff. In Table [ most recent approaches 66 45 3 42
44 attempted to improve through being supervised by additional GT priors such as focal length, metric depth, 3D point clouds, camera poses, and motion masks, which are typically unavailable in casually collected videos. We cannot help but ask a natural question: Is it possible to accurately and efficiently estimate camera parameters in dynamic scenes in an RGB-only supervised manner- the most minimal form of supervision?
59
46
56
59 3 make obvious improvements, but they compensate for the inaccuracies of individual pseudo-superrision sources, resulting in performance degradation if any of them fails. They also cannot adaptively exclude moving outliers without GT motion supervision. Besides, their high computational latency always leads to lengthy runtimes, Further discussion of related work provided in Section 2
Existing RGB-only supervised methods
20
mostly rely multiple pre-trained dense prediction models 33 13 31
Table 1: Categorization of supervision of current methods. Ours, casualS (58. and Robust- CVD [6] are RGB-only supervised, while our performance is the best as shown in section 4
Supervision
Statle Semn
Dynamic Scene
GT 3D Point Clond & Camera Pose
Dusch
4M
Fr
S3
Morthr
Spannár
VGGT 41
Monst 3d
Curke
StereoAD
Faiiär
GT Focal Length
CP
3DGS
b Nops NeRt 1 LocalNoRI
22
Metric Depil:
DROID SLAM MV
GT Priors
GHoww [5) LEAR-VO
Motion Prioes
RoDyaRe
ParticlSSM
RGB-Onll
VGGSIM
+
FlowMap
Spslan
COL Atrmm
Robas-CVD
IB
CM
SS Ours (ROS-Cam)
Based on these insights, we propose ROS-Cam, an RGB-only supervised, accurate and efficient camera parameter optimization method, with brief performance overview in Figure Specifically, to minimize reliance on pre-trained dense prediction models while still establishing robust and maximally sparse hinge-like relations across the video as accurate pseudo-supervision (bottom right corner in Figure 2 we propose the novel patch-wise tracking filters built solely on a pre-trained point tracking (PT) model. This formulation effectively avoids inaccurate tracking trajectories extracted across frames and computational latency induced by the noisy dense prediction as pseudo-supervision.
However, the extracted superviinn includes portion of trajectories belonging to moving outliers. To eliminate the influence of such outliers, we introduce a learnable uncertainty associ- ated with each calibration point, where each is learnable 3D position in the world coordinates, corresponding to one extracted tracking trajectory. We model such uncertainty parameters with the Cauchy distribution, which can deal with heavy tails better than, e.g., the Gaussian distribution, and propose the novel Average Cumulative Projection error and Cauchy loss for the outlier-aware joint optimization of the calibration points, focal length, rotation, translation, and uncertainty parameters. Unlike ca and LEA.PVO, which assign uncertainty parameters to 2D pixels, our approach associates uncertainties with sparse 3D calibration points, resulting in significantly fewer learnable parameters and reduced runtime, as shown in Table 3.
Such joint optimization is prone to getting trapped in local minima. To address this, we analyze the asymptotic behavior of the Softplus function and the analytical minima of the inner convex term in
I Like all existing methods in table we also assume a pinhole camera. We denote the COLMAP using motion masks COL w/ mask and the one w/o motion masks COL weo mask

---

losses to propose a two-stage optimization strategy to accelerate and stabilize the optimization. We evaluate the performance our method through extensive experiments on 5 popular public datasets NeRF-DS [50], DAVIS [28] iPhone ? MPI-Sintel [2] and TUM-dynamics [36]. demonstrating our superior performance. Our contributions can be summarized as follows.
We propose the first RGB-only supervised, accurate, and efficient camera parameter opti- mization method in dynamic scenes with three key components: (1) patch-wise tracking filters; (2) outlier-aware joint optimization; and (3) a two-stage optimization strategy.
We present exhaustive quantitative and qualitative experiments and extensive ablation studies that demonstrate the superior performance of our proposed method and the contribution of each component.
2 Related Works
Dynamic Scene Reconstruction/Nove View Synthesis (NVS). Existing methods for reconstructing
objects and scenes use a variety of 3D representations, including planar [8] 9 mesh [51] [51,
point cloud [8
57). neural field [23
50
37
29 21], and the recently introduced Gaussian explicit
representations
[4
10
47
53
NeRF 23 enables high-fidelity NVS. Some methods [25] 26
29
50
13
also extend NeRF dynamic scenes, while others 51 55 54
37 build on
them to extract high-quality meshes. However, NeRF-based methods have the limitation of a long training time. Recently, 3DGS [14] effectively addressed this issue by using 3D Gaussian-based representations and presented Differential-Gausiin -aaaaa--aaaaiaaaa-ttiitttiii Rasterizatioon in CUDA. 3DGS optimizes 3D Gaussian ellipsoids dynamic scene representations associated with attributes such as position, orientation, opacity, scale, and color. Several studies 47 53 also have used 3DGS for dynamic scenes, achieving near real-time dynamic scene novel view synthesis. However, both NeRF-based and 3DGS-buss based methods heavily rely on COL w/ mask to estimate camera parameters.
Camera Parameter Optimization, Many efforts have been made overcome the shortcomings of COLMAP, particularly for dynamic scenes. But each suffers from some constraints. In Table [ we present categorization of supervision of current SOTA methods. Supervised by additional Gocal length, CF-3DGS [6), Nope-NeRE [[] and LocalNeRF (22) leverage pre-trained monocular depth
estimation model [3] to estimate camera poses and the static scene jointly. The most representative SLAM-based method
DROID-SLAM [39] leverages both GT focal length and metric depth as
supervision. GFlow and LEAP-VO
45 S
extend it to dynamic scenes with both GT focal length and
motion priors as supervision. Although VGGSIM, FlowMap. InstantSplat, COL w/o mask [42 [35] 32]
eliminate the GT focal length requirement by leveraging pre-trained PT models [3, they cannot handle the moving objects in dynamic scenes. RoDynRF (20. COL w/ mask [32]. and ParticleSMM (59]
simply tackle such a problem by incorporating GT motion supervision like GFlow. Recently, DUSt3R-
based methods [46] 52 18,
41 and their dynamic-scent counterparts [56 [1 4 explored
feed-forward camera parameter prediction by training on large-scale static and dynamic scene datasets, respectively, in a fully supervised manner that using GT 3D point clouds and camera poses as supervision, requiring several days' training on high-end GPUs However, unlike LLMs that benefit from abundant language data, such metric 3D supervision is relatively scarce in the vision area, leading to frequent domain gaps when these models are applied to unseen data. In contrast, Robust-CVD [16). casualSAM [58) and our method conduct camera parameter optimization for dynamic scenes in a more general RGB-only supervised way. However, as shown in Section 4 their performance is significantly worse than ours.
3 Method
Under RGB-only supervision, RGB frames F., i E N - (N is frame count) are given. Our proposed patch-wise tracking filters (Section 3.1 extract H robust and maximally sparse hinge- like tracking trajectories as pseudo-supervision, where each corresponds to one calibration point Peats E 3 h [0, H] in the world coordinates. Under such pseudo-supervision and our newly proposed ACP error and Cauchy loss, the calibration points peali, focal length ʃ T R. quaternion matrix Q E N translation E Vx3 and motion uncertainty parameters r E R are jointly optimized (Section 3.2p. I is the scale parameter of the Cauchy distribution which used to model such uncertainty parameters, associated with each calibration point, to reduce the erroncous
3

---

influence of moving outliers. By analyzing the Softplus limits and convex minima in losses, we propose a simple but effective two-stage optimization strategy (Section 3.3) to enhance the stability and optimization speed.
3.1 Patch-wise Tracking Filters
Built on pre-trained PT model, we observe that its attention mechanism assigns higher attention weights to pixels with more accurate tracking results which are always texture-rich pixels with large gradient norms. Inspired by it, as shown in Figure 2 we propose the patch-wise texture filter to identify the high-texture patches within F2 and the patch-wise gradient filter to select the pixel with the highest gradient norm within each identified patch. While tracking such identified points, the visibility filter keeps removing trajectories that become invisible and the patch-wise distribution filter keeps the one with the largest gradient norm when multiple moving points enter the same patch. As shown in appendix 1 fig. 10 our method only retains the robust and accurate trajectories as pseudo-supervision.
Vasliit Filter?PT) PaTch wiee Enstisutin Filter
Visiility Filter(F)
PETCh-WiE Distmbutiin Filter
Py-A
Patch-wise Gradd
Ratch- satch--aise Grodiontt Filter
Patch wise Texture
Patchwatio Texture Filte
Figure 2: Patch-wise tracking filters. (1) Partitioning Fo into patches of size w x the patch-wise texture filter computes the texture map To and marks the high-texture patches in gray; (2) Within each high-texture patch, the patch- wise gradient filter selects one potential tracking point with the highest gradient norm. (3) The visibility filter removes the entire trajectory of a point if it becomes invisible at any time
kept trajectories; o, D. A + removed trajectories); (4) The patch-wise distribution filter only keeps the one with the largest gradient norm when multiple trajectories fall into the same patch. P and I are the location and index of trajectory, and is the trajectory range.
Patch-wise Texture Filter. Highly distinguishable points, that can be tracked reliably, belong to highly nonuniform (textrred) neighhorhoods. To identify such neighhorhoods, our patch-wise texture filter computes texture map T, 1 B///w//w giving a measure of texture level for each w x w patch where H and W denote the height & width of F. We represent the texture level of a patch by
T [. n] = [m, n] > Tvar o²}
(1)
where Σ; E /// is the intensity variance, o" = max Ei), Tear is the percentage threshold of minimum variance for the patch to be selected, and m,n n E H/w - 1], [0, W //w 1]. The texture levels of patch are represented by 1 for the selected patches and 0 for the others.
Patch-wise Gradient Filter. Within the identified patches, our patch-wise gradient filter computes the intensity gradient norm map G, E Hx of F,, and selects the point with the largest gradient norm within each patch. This yields the pool of potentially distinguishable points, forming potential trajectories, namely,
protential
arg max
( G.G...muu [mw mw + w. nww : nw + ww), p - pixel locations
(2)
P
Visibility Filter. We find that current PT models B 12 30
still tend suffer from reduced tracking
accuracy when a point becomes occluded and later reappears, due to the disruption of temporal feature continuity. Thus, if any P in any F; becomes invisible, our visibility filter deletes it by the dot product P V. V E 1) ~ P. where V = 0 if a point is invisible.
Patch-wise Distribution Filter. This filter enforces more even point distribution within each frame, preventing them from clustering into a small region as the viewpoint changes. also helps reduce susceptibility to loss of resolution which might result in triangulation errors. We keep the highest-gradient tracking point P in each patch Patm,n of F2, as follows:

---

pd arg max G. E Palman,, if > 1(p Palm.)) P
(3)
As shown in Figure locations and indices of P are stored in P. E RBx2 and I, E RR [0, H - 1]. acting as pseudo-supervision in the outlier-aware joint optimization. Each iteration starts F,,t arg mine( I,), and ends until each frame contains exactly B tracked points.
3.2 Outlier-aware Joint Optimization
Outlier-aware Joint Optimization Mechanism. Under the obtained pseudo- suauuvd-uperisionn, Peali f. Q. t and r are jointly optimized. We first project peali-homo E RR#x (the homogeneous coordinates of poati obtained by concatenating 1) onto each frame by
P aprof-homo homo
pack-homoy homo [..]
R. 0
t.
K"
(4)
prres pomo ; / (omo [5,3]
(5)
where i E N -
1] and Peroj E RNXxx2 pproj-homo E Nxx4 denotes the homogeneous
2D location of the projection pproj, The perspective projection matrix K E Rxx is derived from S. and the world-to-cameet transformation matrix consists of rotation R, and translation t,. We assume constant f like SOTA [59 58 20 Notably, we learn the quaternion matrix Q: instead of optimizing the R, and additional constraints. This optimization approach cireumvents the tefficult-tt-enforee orthogonality and #1 determinant constraints required for rotation matrices during optimization.
>4h
Running (hours)
4h -3h
Resth Traceng Fifire
K
R.
200
400
800
8000
1000
Figure 3: Outlier-aware Joint Optimization. represents P, and Ptr on each frame. The static samples proalé and p " can establish concrete triangulation relations with their corresponding Pt, Pr. and cameras, resulting in lower Y and y". contrast, the dynamic sample !'lculéé ex- hibits the opposite behavior.
Number of Frames
Figure 4: Runtime Trends. As the frame count increases, runtime grows almost linearly, whereas COL w/omask scales exponentially. The runtime casualSAM is too large to fit in this figure. The complete runtime is in Table 3 and appendix B.I.I (table table table
However, the extracted pseudo-supervision always contains moving outliers. To mitigate its impact, without any GT motion priors, we identify such outliers by modeling the uncertainty their presence may cause in the observed distributions of the inlier points. We introduce the uncertainty T E R associated with peati E R and incorporate the Cauchy distribution [2; r) ""
T >> 0 to model the uncertainty parameter since this distribution
can better handle the heavy tails than, e.g., the Gaussian distribution. As depicted in Figure 3 during optimization, inliers are expected have low uncertainty, while outliers have high uncertainty. Since the scale parameter I in [x;; Zo. T' is required to be strictly positive, we introduce a new parameter Traw which we obtain I from using the Softplus function T = log(1 c r Traw E RR This effectively ensures I Ry is differentiable and has smooth gradients.
Losses. To down-weight outliers by learned r, we replace the commonly used projection error Eurej [prroj 2 with our proposed Average Cumulative Projection (ACP) error, defined as:
E1 P P
ACC 0
(6)
5

---

where EACP E R and o denotes the clement-wise matrix multiplication. For each Paut, we accumulate the errors between its corresponding projection and tracking locations across the video, then take the average as EAAPP -1] Furthermore, we propose the novel Cauchy loss Leauchy in terms of the negative-log-lkkellood log (T+ (ar =m)" of fa; zo, D) where we replace x - 2o with 1
EACP as eq. 77. Our total loss Crotal in Equation 8 consists of Leauthy and a depth regularization term deptth encourage positive depth. With the estimated camera parameters,
4DGS |47
for scene reconstruction. Reconstruction and loss derivation details are in appendix[AAaal appendix B
H Σ log (T +
(BACPy)
Learchy
(7)
H
Ciotal = Leaathy + Recpthh Rdepth
PP. mnoy homo [;,3))
(8)
3.3 Two-stage Optimization Strategy
To avoid convergence to local minima, we propose this strategy based on analysis of the asymptotic behavior of the Softplus function and the analytical minima of the inner convex term Leanthy- Stage 1 focuses rapid convergence, while Stage 2 aims for stable convergence by initializing rro to the ACP error after Stage 1. The effectiveness of it is concretely demonstrated in Table
Stage 1. In the Softplus function, r log(1 e ) Draw Frow _ +.0. So in Stage 1,
fix Frow = 1 and optimize only peali, f. Q, and t for quick convergence. The loss will converge to a certain value beyond the global minimum, as there is no proper I' to down-weight outliers.
Stage 2. The inner term 4- = x + O 0 0 of Leauchy is convex. Assuming a constant O E R and solving for min, (xx), we have xx = /O. Similarly, in Stage if Traw is randomly initialized to values largely different from stegel (the ACP error from Stage 1), convergence will be highly unstable. Therefore, we initialize praw EACP and optimize peati f, Q, t, and r jointly.
stagel
Experiments
To demonstrate the superiority of our method, we show extensive quantitative and qualitative results in this section. For NeRF-DS (50], DAVIS (28), and iPhone [7] datasets without GT camera parameters,
we feed the camera parameters from different methods 4DGS [47) while keeping all other factors the same, and evaluate each NVS performance (PSNR, SSIM, and LPIPS). Regarding the MPI-Sintel 2 and TUM-dynamics (36] datasets with GT camera parameters, we directly evaluate methods by ATE, RPE trans, and RPE rot metrics. In all tables, the best and second-best results are bold and underline. More about datasets, and evaluation metrics are in appendix C and appendix D
4.1 Implementation Details
The optimization is conducted NVIDIA A100 40GB GPU with Adam (27 optimizer and learning rates lQ = 0.01, 4 = 0.01, 1 1.0, 4peats = 0.01, and freww = 0.01. We also choose to build our patch-wise tracking filters on CoTracker 13 and load its pre-training weights. The hyperparameters of our patch-wise tracking filters are set at Tear = 0.1, B = 100, WNERF-DS, DAVIS, Simell = 12, and WiPhone, TUM 24. Notably, is only related the frame size. Besides, throughout experiments, we have 200 and 50 iterations in Stage, and Stage, respectively.
Table 5: Camera Pose Evaluation on MPI-Sintel (2). (ATE//PPE trans.//RP rot.) We achieve better results than A (58) and exclude COL w/o musk due to its failure.
Method
alley 1
alley
ambush.
ambush_S
ambush.. 6
market 2
market
CaRaaISAM
53
0.028A 055
0.6400.0588..22
0.302-0068/2200
0.010N0.010000
0.239/0
Our
8.1198.4991.30
BUNGNBOEWW..U 192
0.0N00112992.1
0.0090000000000
Method
Shaman
sleeping_ I
sleeping 2
temple 2
mountain
bumboo_
bambou,
AAM 59
0.008 00900.000
0.01 000068011
0.013400022 10..002500.100
0,00500 00400.30 380
0.003/000040) 182
0.033N0.019K00C
0.005/0.003 00850.003/0000
Oues
GUIIS 10011007 ULARE-I..BEA..IIS
1.002.1.01100 N1.144
0.007-0.004410869

---

Table 2:
NVS Evaluation
NeRE-DS 50 and DAVIS [28]. Table 4: Camera Pose Evaluation on TUM- (PSNRT/SSIM//LPIPSS)
is super- dynamics [36]. Other results are from Cut3r
44
vised by additional GT priors. Ours is
and Monst3r [56]. Performance of DROID-SLAM
39
the best among these datasets.
is from casualSAM [58]. Our method achieves the best overall performance among RGB-only supervised
Method
NeRF-DS
DAVIS
methods, and even better than the ones supervised by additional GT priors.
RaDynRF 201
23.0330 74940. 385
COL saake
92.1740092200.11
CDL
29. 3480 875/0. 234
9.196A1 2360)455
casualSAM
5X
38.562/0.535001118
21,2300 6861). 463
22.292N0.70//0.22
Supervision
ATT!
trans.
rot.
Monst3
56
0.098
0.019
0.935
Point Cloud
Dust3r
46
0.083
0.017
3.567
Table 3: Runtime Evaluation on NeRF- DS [50), DAVIS [28), and iPhone [7],
& Camera Pose
Mastee 18
0,038
0.012
0.448
Cut3r
44
0.046
0.015
0.473
covering frame count from 50 to 900.
GT Focal ength
GT Motion Prior
LEAP-VO R
0.046
0.027
is supervised by additional GT priors.
GT Focal Length
Our method is the most efficient.
Metric Depth
DROID SLAM 39
0.043
Method
NeRF-DS
DAVIS
iPhone
GT Motion Priors
ParticleSIM 59
RoDynRF
20
29.6h
27.4h
28.5h
Robust
Cobust-CVD CVD
16
0.153
0.026
3.528
COL
1,5h
RGB-Only
CasualSAM
58
0.071
0,010
1.712
Ours
0.065
0.010
0.987
COL Mil maac
1.8h
0.51h
9.53h
C
58
10.5h
0.28h
4.07h
Ours
0.83h
0.03h
0.33h
4.2 Time Efficiency Evaluation
In T we present the average runtime evaluations. Our average runtime on NeRF-DS, DAVIS, and iPhone 55%, 11%, and 8% of that of the second-fastest methods, while keeping the best performance shown in table 2 We attribute it to three main reasons: (1) Our method only leverages the maximally sparse pseudo-supervision extracted by our proposed patch- wise tracking filters under the RGB-only supervision. (2) Our uncertainty parameters are associated with the 3D calibration points rather than 2D uncertainty maps (58), significantly reducing the number of learnable parameters. For Plate video (424 frames) in NeRF-DS, casualSAM has (424x270x480) uncertainties, whereas our method only has 440 uncertainties, one per peali 3) The two-stage optimization strategy highly accelerates the optimization speed. As seen from Table omitting the two-stage strategy leads to a dramatic performance drop after the same iterations, indicating more iterations, and thus time, are needed to achieve the same performance.
ATE 6312 DERHO
ATE-0000 FIm 2012.
RPC 6.00
Haaidoee R008
OBL
ATC 6.07
marees
45G DM
natee_2
ATE 0.00
May_1
ATE: 0004
boc
Figure 5: Qualitative NVS Results on DAVIS (28). Our performance is the best because of our accurate camera estimates. More in ap- pendix E.2.2 fig. fig. 15] fig. 16 and fig. 17
Figure 6: Qualitative Results of Camera Pose on MPI-Sintel [2]. represents our camera esti- mates; represents the GT. Our estimated camera trajectories almost perfectly align with the GT.
Besides, in Figure we see that our method exhibits a linear growth (at the rate of about 1/800 hours per frame) vs COL COLw/omos. w/mookk whose runtime growth is roughly exponential. This difference will be increasingly significant the video length increases, which can also demonstrate the superior
7

---

Table NVS Evaluation on iPhone 77. (SSNRI/SSIM/LLPIPS)) Record3D is a paid mobile app obtaining camera results by LiDAR sensors, where are provided by M. Ours is the best among RGB- supervised methods and surpasses LiDAR-basee Record3D sometimes.
Method
Apple
Paperr..mlll
Space-ont
Backpack
Block
Creeper
Teddy
Record30
06.3550.7700.88
23.91/0.7300 24
27.12/0 77/0.35
20. 79/0 56/0.10
23.72/0.71/0.38
21.8000663/00
19.7 02.720059/0..1 11
Wh raok
22.45/0.69/0.
22.74/0.722 72/0
24.33/0 74/00.7440.55 38
58/5 39/0
18,49/0 600.
18.13/0.4300
16.5640.000055
casulSAM 5X
19. 00.00990000000
18.850 39/0 54
22.1990 677
1841003600 55
19. 19.100.5911. 52
29.0626
15.6991.42)..8 58
Ours
25, 99960.4000..5
24.09007400.22
28-4220.990.31
21.22/0.400.32
23,28/0 6910.38
21,6700.533 6310
20.780.60N0.41
Method
Handwavy
Haru-sit
Mochi-..five
Spin
Sriracha
Pillow
Wheel
RecordāD
27.80/0.8600 24
29.86/1.5700. 22
34.3//091/02 24
24.85/0.9900.S 38
31.15/0 25
20.86/0.6300.41
20.78/0.60N041
CDt.
15 15.906661100555 55
25. 25.58/0. 80/0. 30
22.47/0.7700 38
19.27/0.5/0044 49
28.41/088600 28
07750.4661.55
20.78/0.6000.4
casuaiSAM SS
20 87/0 6880 46
19 89,880 69/0.41
26. 34/4 84/0.5
19.33004500.5 57
23.20/0 13.20007330.42
1.950..110..
14 69.1-7700.77
Ours
28.02/0.860022
28.01/0
34.56/0.2/00.22
24.81/0.77/0399
32.49/0.990.25
20.630.6110044
20.4200.700.37
time efficiency of our method compared with other RGB-only supervised methods. We exclude the casualSAM here since its runtime is too large to fit here.
4.3 Camera Pose Evaluation
We follow the same evaluation setup of Cut3r 44 and Monst3r [56] on TUM-dynamics 33 and evaluate all videos of the synthetic MPI-Sintel dataset.
Quantitative Evaluation. In Table 5 and Table our method has the best performance among all RGB-only supervised approaches. Our method also achieves comparable even better results than others that require additional GT priors as supervision. We attribute it to our accurate and robust pseudo-supervision, derived from RGB-only input, enabling effective outlier-aware joint optimization. Besides, our uncertainty modeling and loss design effectively down-weight the impact of moving outliers. Since COL w/ mask and COL weo mask always fail on MPI-Sintel [2], as observed by us and 120 [56]. we exclude comparisons with them here. However, RGB-only supervised methods including ours perform not very well in some special cases, which discussed in the limitations.
Qualitative Evaluation. In Figure 0 we show our estimated camera trajectories alongside the GT on MPI-Sintel (2) Our estimated camera trajectories can perfectly overlap with the GT, which provides qualitative support to the higher accuracies seen in the quantitative results in Table
4.4
NVS Evaluation
Since NeRF-De DS
DAVIS 28 and iPhone 77 datasets do not provide GT camera parameters, by inputting camera estimates of different methods into the same 4D reconstruction pipeline- 4DGS (47), and evaluate the NVS performance. Such NVS performance reveals the quality of the camera parameter estimation.
50
we follow
6
20
45
19
Quantitative Evaluation. In Table 2, our method is the best on NeRF-DS 50 (long vidcos w/ little blur, textureless regions, and specular moving objects) and DAVIS 28 (short vidcos w/ low parallax and rapid object movement), demonstrating our more accurate camera estimates. We skip COLW/ mask and RoDynRF DAVIS (8 because they not RGB-only supervised methods and require supervision beyond RGB frames, and have already underperformed compared to ours on NeRF-DS [50]. Besides, our pseudo-supervise supervision extraction built the PT models [13] [2] performs better on low-parallax videos, which remains challenging for the pre-trained depth model 31)
Regarding the iPhone 7 dataset (videos w/ irregular camera movement and object movement), it provides so-called 'GT' camera parameters obtained by Record3D which is a paid mobile app obtaining camera results by LiDAR sensors. However, parameters are occasionally unreliable. As shown in Table and and fig. besides being the best among all RGB-only supervised methods, our method can occasionally beat Record3D.
observe that such so-called 'GT" camera
Qualitative Evaluation. We evaluate the quality of the rendered RGB images and depth maps in Figure Figure & and Figure Beyond superior RGB renderings, our camera estimates yield the highest-quality depth maps, offering more convincing evidence of accurate scene geometry than RGB renderings. It indicates that our estimated camera parameters enable the model to learn the correct dynamic scene representations rather than overfitting to training views. the first row of Figure 7 ours performs the best (surpassing even Record3D), especially in rendered depth; whereas
8

---

RGB
Dept
ROB
Death
RGB
Darait
HOB
Depin
PGNRLEATT
FGN 22.09
PCNN-26.3
PGNA:29
65IMS
ESIM 67
BBIMOo 7d
Soyat
LFIE5/0.47
LPIPS 0.40
P 0:31
RGB
Dept
R3B
Decff
ROB
Depth
RGB
Depfi
PBNR:21
FSNR-16 16.43
PENR B
PSNIERT
ESIMO.
56IM0. 29
SEIMOO
SSIMO G
RIPSo
PIPS
AGB
Dopt
RGB
Death
FGB
Depti
RGB
Deppy
PSNR:34.24
FSNR22634
2S
PSNIR3
BBIMIC 99
SSIM B4
35I:0. 77
SSIMO BE
LPIPS:0.2 24
LPIPS.0.00
LP
GT RIBB
Record30
CRCUDESAM
OT),
Our
Figure 7: Qualitative NVS Results on iPhone (7). Our method outperforms other SOTA RGB-only supervised approaches and even surpasses LiDAR --bsedd Record3D when the movement in scenes with large motion (top row). More are appendix E.2.2 fig.
fig. and fig. 13
Table 7: Ablation Study on NeRF-DS (0)
GT RGB (Plane)
GT RGB
Part 1. We conduct ablation studies different
(Sasin)
Scene optimization strategies and individual com- ponents of our proposed model.
method P1Am
Our method
PAWROM TD
COL 25
COL Sriott
Optimization
Camera Optimization
PSNR
SSIM1
LPIPS,
n5 mad
0.1609
D-DDG 3
206 40 PON03333
CDL Dovent
Ours (full)
32,45
0.9336
0.1312
COL
29.35
0.8748
0.2240
Ours (full)
33.55
0.9388
0.1182
tvo-stage
0.2668
4DGS
17
wlo I
26.44
0.8667
0.2327
EACP
w/o inxtrre filt
25.99
0.8356
0.2536
RGB
gradient filt.
26.04
0.8393
0.2404
w/o distrib. filt.
26.02
0.8382
0.2497
Figure 8:
Qualitative NVS results on NeRF-
DS [50] Our renderings are the most plausible.
More are in appendix E.2.2 figg. 18).
in the second row, our method does not match Record3D, but is still better than other RGB-only supervised works. This is because Record3D is not originally designed for dynamic scenes, so when scene contains larger irregular movements, its performance will be worse (such observations are also supported by the numerical results in 6). In contrast, our method is more robust in various scenarios, consistently maintaining high standards.
Ablation Study. In Table 7 the loss of any filter results in less robust relations across video, leading to poor camera estimates and NVS performance. Further, the removal of any of r, IAAC or the two-stage strategy will harm the results due to outliers. This indicates that w/o such a strategy, increas- ing training iterations is necessary but not sufficient condition for comparable results. We also overcome the limitations of COLMAP
Table Ablation Study on NeRF-DS 50
Part We conducted ablation studies on different PT models.
Scene Optimization
Camera Optimization
LPIPS.
(PT model choice)
huilt on CoTracker
4DGS
Ours
built on CoTracker3
0.1180
by improving the performance of different scene optimization models [53 47 with our camera estimates. As reported in CoTracker3 [12] CoTracker 13 performs worse than CoTracker3. However, in table 8 the performance of our proposed method is nearly independent of building the particular PT model. This further sup-
32

---

kmmp 3
market
D
1,060, 6,226
ATE: D. 125 RPE trans. 0.265
ATE: 2.027, RPE trans: 0.617
ATE: 0.782, RPE trars: 0.147
Figure Failure cases of ours and casualSAM on MPI-Sintel (2
ports our claim that our patch-wise tracking filters effectively exact only the accurate trajectories as secudo-uuperisionn
Conclusion and Limitation
We proposed
RGB-only supervised. accurate, and efficient camera parameter optimization method casually collected dynamic-scene videos. Our method effectively tackles the challenge of precisely and efficiently estimating per-frame camera parameters under the situation of having no additional GT supervision (e.g. GT motion masks, focal length, 3D point clouds, metric depth, and camera poses) other than RGB videos, which is the most common scenario in real-worle and consumer grade reconstructions. Our method may serve as a step towards high-fidelity dynamic scene reconstruction from casually captured videos.
Although our proposed method is currently the most state-of-the-art RGB-only supervised, accurate, and efficient camera parameter optimization method in dynamic scenes, there are still several limitations. We assume a constant focal length throughout the video. While this assumption is reasonable and currently common to SOTA, the task of accurate and efficient camera parameter optimization for dynamic scene videos with zooming effects under RGB-only supervision remains open problem. Another common challenge for RGB-on -nly supervised methods, not addressed in this paper, is maintaining robustness scenes dominated by large moving objects. As shown in fig. [9] the screen space is occupied by the moving human and dragon. It is challenging for method to establish robust and maximally sparse hinge-like relations as accurate pseudo-supervision because most of the extracted trajectories belong outliers. CasualSAM (58) struggles due the rapid changes in depth maps from frame to frame, making 3D space alignment difficult. We plan to maintain consistency in our input setup and address these challenges part of future research.
10

---

Acknowledgements
The support of the Office of Naval Research under grant 1-000011-20--22444 and of USDA National In- stitute of Food and Agriculture under grant 2020-67021-33 67021-12799/102247788 are gratefully acknowledged. This research used the Delta system at NCSA through allocation CIS240124 from the ACCESS program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #21 #2138296.
References
[1] Wenjing Bian, Zirui Wang. Kejie Li. Jia- Ma--ang Bian, and Victor Adrian Prisacariu. Nope-nert nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4160-4169, 2023.
[2] Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for optical flow evaluation. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy October 7-13, 2012, Proceedings, Part VI 12, pages 611-625, Springer, 2012.
[3] Weirong Chen, Le Chen, Rui Wang, and Marc Pollefeys. Leap-vo: for visual odometry. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19844-19853 2024.
Long-term effective any point tracking
[4] Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, and Anpei Chen. Easi3r: Estimating disentangled motion from dust3r without training. arXiv preprint arXiv 2025.
[5] Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding. Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds. arXiv preprint 2403v:2403.20309 2024.
[6] Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, and Xiaolong Wang. Colmap- free 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20796-20805, June 2024.
[7] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Advances in Neural Information Processing Syxtems, 35:33768-33780, 2022.
[8] Derek Hoiem, Alexei A Efros, and Martial Hebert. Automatic photo pop-up. In ACM SIGGRAPH 2005 Papers, pages 577-584. 2005.
[9] Youichi Horry, Ken-Ichi Anjyo, and Kiyoshi Arai. Tour into the picture: using spidery mesh interface to make animation from a single image. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 225-232, 1997.
[10] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 Conference Papers. pages 1-11, 2024.
[11] Linyi Jin, Richard Tucker, Zhengqi Li, David Fouhey, Noah Snavely, and Aleksander Holynski. Stereo4d: Learning how things move in 3d from internet stereo videos. arXiv preprint arXiv:2412.006211 2024.
[12] Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rup- precht. Cotracker3: Simpler and better point tracking by pscudo-labelling real vidcos. arXiv preprint arXiv:2410. 11831, 2024.
[13] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: is better track together. arXiv preprint arXiv:2307.07633 2023.
[14] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph,,42(4):1333 12(4))139-1, 2023.
[15] Mijcong Kim, Sconguk Sco, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendcring. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12912-12921, 2022.
[16] Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1611-1623 2021.

---

[17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[18] Vincent Leray, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mastår. In European Conference on Computer Vision, pages 71-91. Springer, 2024.
[19] Fang Li, Hao Zhang, and Narendra Ahuja. Self calibbaaig 4d novel view synthesis from monocular videos using gaussian splatting. arXiv preprint arXiv:2406.010442 2024.
[20] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf. Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13 23, 2023.
[21] Ricardo Martin- Brralla,, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7210-7219, 2021.
[22] Andreas Meuleman, Yu-Lun Liu, Chen Gao. Jia-Bin Huang. Changil Kim, Min H Kim, and Johannes Kopf. Progressively optimized local radiance fields for robust view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 16539-16548, 2023.
[23] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, Communications of the ACM, 65(1):99-106, 2021.
[24] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5480-5490, 2022.
[25] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Scitz, and Ricardo Martin-Brualla, Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[26] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla and Steven M Seitz. Hypemerf: A nighee-diimensonnal representation for topologi- cally varying neural radiance fields. arXiv preprint arXiv:2106.1322 2021.
[27] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[28] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeléez, Alex Sorkine-Horuung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1044.00675. 2017.
[29] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2021.
[30] Frano Rajič, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu. Segment anything meets point tracking. arXiv preprint 23Xi:2307.01197 2023.
[31] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-datasct transfor. IEEE transactions on pattern analysis and muchine intelligence, 44(3):1623-163) 2020.
[32] Johannes L Schonberger and Jan-Michael Frahm. Structure-fros motion revisited. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4 4104-4113, 2016
[33] Marek Simonik. Record3d point cloud animation and streaming, 2019.
[34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint 0ttl::1409.55533 2014.
[35] Cameron Smith, David Charatan, Ayush Tewari, and Vincent Sitzmann. Flowmap: High-quality camera poses, intrinsics, and depth via gradient descent. arXiv preprint arXiv:2404. 15259, 2024.
[36] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d - slam systems. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 573-554 583. IEEE, 2012.
12

---

[37] Matthew Tancik. Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi. Terrance Wang, Alexander Kristoffersen. Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. Neristudio: A modular framework for neural radiance field development. ACM SIGGRAPH 2023 Conference Proceedings, pages 1-12, 2023.
[38] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 402-419. Springer, 2020.
[39] Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 1051615581666 2021.
[40] Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint 2408 16061, 2024.
[41] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vodaldi, Christian Rupprecht, and David Novotny, Vggt: Visual geometry grounded transformer. arXiv preprint arXiv:2503.116// 2025.
[42] Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfi:: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21686-21697, 2024.
[43] Peng Wang, Lingji Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 10689, 2021.
[44] Qianqian Wang. Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. arXiv preprint 2rXiv:2501.22333 2025.
[45] Shizun Wang, Xingyi Yang. Qiuhong Shen, Zhenxiang Jiang, and Xinchao Wang. Gflow: Recovering 4d world from monocular video. arXiv preprint ar arXiv:2405. 18426, 2024.
[46] Shuzhe Wang, Vincent Leroy. Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697-20709, 2024.
[47] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 20310-20322 2024.
[48] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nert nerf: Point- based neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5438 5448, 2022.
[49] Chi Yan, Delin Qu, Dan Xu, Bin Zhao, Zhigang Wang, Dong Wang, and Xuclong Li. Gs-slam: Dense visual slam with 3d gaussian splatting. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19595-19604, 2024.
[50] Zhiwen Yan, Chen Li. and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8285-8295, 2023.
[51] Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[52] Jianing Yang, Alexander Sax, Kevin J Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. arXiv preprint Xiv::22001. 13928, 2025.
[53] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high- fighlfidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20331-20341, 2024.
[54] Lior Yariv, Jiatao Gu, Youi Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advancex in Neural Information Processing Systems, 34:4805 4815, 2021.
[55] Hao Zhang, Fang Li, Samyak Rawlekar, and Narendra Ahuja. Learning implicit representation for reconstructing articulated objects. arXiv preprint arXiv:2401.000 240.009900080 2024.
13

---

[56] Junyi Zhang, Charles Herrmann. Junhwa Hur. Varun Jampani. Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hlsuan Yang. Monst3r: A simple approach for estimating geometry the presence of motion. arXiv preprint urXiv:2410.03825, 2024.
[57] Qiang Zhang, Seung- Hwan Baek. Szymon Rusinkiewicz. and Felix Heide. Differentiable point-based radiance fields for efficient view synthesis. In SIGGRAPH Asia 2022 Conference Papers, pages 1-12, 2022.
[58] Zhoutong Zhang. Forrester Cole, Zhengqi Li, Michael Ruhinstein, Noah Snavely, and William T Freeman. Structure and motion from casual videos. In European Conference Computer Vision, pages 20-37. Springer, 2022.
[59] Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfm: Exploiting dense point trajectories for localizing moving cameras the wild. In Eumppan Conference Computer Vision, pages 523 542. Springer, 2022.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We used bold text to highlight the main claims made the abstract and
introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Due to the space limit, We put the limitation in the Appendix and set up reference to it in the conclusion and limitation section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
14

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We put the loss in the method section and show the derivation of Cauchy loaitt---oo-likkhhhooo in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We present all experiment setup and implementation details in the main paper, and put the datasets and evaluation metrics in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
15

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
We will release the code upon the acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//iips.cc/publl wttpcccpppppccceeeeee..ee..ccltt
wtt.c//pppp.gguddeeeeeeeeeeeeeee guwdes/ http///oodubissiiniil. sosoooooicssi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We show all these details in the implementation details section in the main paper and the datasets section in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
16

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We present these in the experiment section.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We present this in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics / http://www....pppuu.//.ccctaauas
Answer: [Yes]
Justification: Our research conducted in the paper conforms in every respect, with the
NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
17

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss them in the first paragraph of the main paper and the conclusion section of the main paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited every paper and work once mentioned them.
18

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pataasetsspapessiiiii paperswi thcode. com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We use and cite the existing public datasets in this work. Other assets including related code/model will be released upon acceptance.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer:
[NA]
Justification:
This paper does not include the things mentioned in the question.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not non non- stanstandard components.
involve LLMs as any important, original,
Please refer to our LLM policy for what should or should not be described.
20

---

Appendix
Due to the space limit, show more discussions, results, and details here.
A Dynamic Scene Optimization
Preliminary. As an alternative to NeRF [23], which has a lengthy runtime, 3DGS [4] recently introduced way to learn static scene representations in terms of explicit 3D Gaussian ellipsoids. Unlike the implicit representations of NeRF stored as the weights in the Convolutional Neural Network (CNN), 3DGS [14] uses explicit representations in 3D world coordinates and performs differential Gaussian rasterization on GPUs using CUDA which significantly speeds up computational efficiency. Each 3D Gaussian ellipsoid 9 (x) is parameterized by its (1) Gaussian center X' E RR: (2) quaternion factor E R4; (3) opacity a E I;; (4) scaling factor E Rs; and (5) color C E I (k denotes degrees of freedom), and represented by:
G(x) = -//
(9)
E' =WEW" = rssss" T
(10)
where E is the 3D covariance matrix in the world space, W and J are the view transformation matrix and the Jacobian matrix of the affine transformation parts, respectively, of the projective
transformation, and E' is the covariance matrix in the camera coordinates. The color is rendered by:
C(p) = Σ cRaaII axs)
(11)
KEK
where Cp and an represent the spherical harmonic (SH) coefficient and the density this point.
4D GS. To take advantage its optimization efficiency, we use 4DGS 47 to learn dynamic scene representations. With different camera estimates and the same scene optimization method [47], use NVS performance to evaluate the accuracy of camera parameter estimates. In 4DGS 47], the NVS performance depends how well the canonical representations and deformation representations optimized. The canonical (refers to 'mean' as in the previous work 14 53 47) representations G learned by canonical Gaussian field to optimize the mean (canonical) position X' E R33 color E g opacity E IR, quaternion factor E Raa scaling factor E R and the deformation representations. F are optimized using deformation field [47] learn the offsets AG, supervised by an LI loss between images and renderings. Since the color and opacity of the Gaussian ellipsoids do not change over time, the deformed attributes consist of (X" r',,'' (X' AX, r + Ar, 8 As).
More details can be found in 4DGS 47
B Derivation of Cauchy Negative-log--ikellood
The Cauchy loss function is derived from Equation (12) to Equation (4) Since we use the Cauchy probability density function (PDF) to model the uncertainty of calibration points penli we want to maximize the likelihood of the Cauchy PDF:
[(z;
r T>0 0
(12)
Equivalently, we minimize the negative-log-likelloos of [(;; Lo. r), to define the loss function::
NLL(2; 2o, r) log(s(2;; I')
log(mT)) + log(1 +
3o y2)
= llog( (I++ (I-10))
(13)
r
(x xz)2
= log n + log
r
21

---

where log т and zy denote a constant term and the ground truth which can be omitted. Thus, our objective is as follows:
+ (ar mo
min 20, r) = min lTg 2.r
(14)
2.F
C
Datasets
To demonstrate our performance on a broader range of scenarios, we have conducted extensive experiments across five public datasets- NeRF-DS [50], DAVIS (28), iPhone 77]. MPI-Sintel [2], and TUM-dynamics [36]. These videos contain different camera and object motion patterns, and different texture levels. The lengths of the videos range from about 50 to 900. Regarding the train/test split of the NVS evaluation, for every 2 adjacent frames, we take the first frame for training and the second frame for testing. For the setup of camera pose evaluation, we follow Cut3r [4] and Monst3r [56] in
the experiments on TUM-dynamics and evaluate all videos in MPI-Sintel 2
NeRF-DS. NeRF-DS (50] dataset includes seven long monocular videos (400-800 frames) of different dynamic, real-world indoor scenarios with little blur. Each video has at least one specular moving object against mix of low-texture and high-texture backgrounds. NeRF-DS [50] exhibits large scene and camera movements, so the frames have some blur. The GT motion masks provided are human-labeled and the camera parameters are estimated by COL mask Like previous works (50 533 we take the highest resolution images available (480 x 270) as the RGB input in all experiments.
DAVIS. DAVIS (8 dataset contains 40 short monocular videos that capture different dynamic scenes in the wild. Each video has 50-100 frames, including at least one dynamic object. The GT motion
masks are also provided as in NeRF-DS [50]. However, like 20
45
58
59
exclude some
videos using fixed cameras, changeable focal lengths, etc. Different from others 20 45
58 3 which
only show experiments of about 10 videos in DAVIS (28). conduct experiments on 21 videos
containing large camera and object movements. We utilize the RGB frames with the resolution of 854 480 as input.
iPhone. The iPhone 7 dataset is an extremely challenging dataset (180-475 frames) with significant camera rotations and translations, and rapid movements of objects. There are 14 monocular videos including indoor and outdoor scenes and no GT motion mask is provided. It would also be difficult to insert motion masks for this dataset because there is no clear boundary between the moving and stable regions within any frame. They represent real-world casually recorded videos. We conduct experiments all of them. The frame size is 720 960. These videos are recorded by the Record3D (33 app on iPhone which uses LiDAR sensors to obtain metric depth for camera estimation. In our comparisons with the camera estimates provided by Record3D, we also take the Record3D app as one of the baselines and compare with it.
MPI-Sintel. MPI-Sintel [2] is synthetic dataset provided GT camera parameters. It has 18 short videos (about 50 frames) in total containing large object movement. In some cases, the moving objects cover most of the screen. Most of the existing works (20 56 3) select 14 videos for evaluation, but in this paper, we evaluate the methods among all the videos. The synthetic MPI-Sintel dataset exhibits domain gaps compared to real-world scenarios, which is considered be one of the reasons why some existing methods [58 [59] perform well on MPI-Sintel, but do not work efficiently on other real-world datasets. In experiments, take the frames with default sizes as the input the method, while keeping the default resizing setup of the other methods.
TUM-dynamics. TUM-dynamics (36] dataset contains 8 long real-world blurry videos recording the dynamic indoor scenes provided with GT camera parameters. However, although the videos this dataset are indoor scenes, each video features a significant depth of field. TUM-dynamies dataset also contains large camera movement and rapid object movement. We follow the experimental setup of MonST3R [56] on this dataset, which is sampling the first 90 frames with the temporal stride of to save compute.
D Evaluation Metrics
As discussed in section 4 of the main paper, we directly conduct camera pose evaluation against the GT on MPI-Sintel and TUM-dynamics [36], using the standard metries: ATE, RPE trans, and RPE rot. Besides, regarding NeRF-DS [50). DAVIS (28), and iPhone [7] datasets which are not provided
22

---

with GT camera parameters, we conduct NVS evaluation with standard metrics: PSNR, SSIM, and LPIPS. We also employ time evaluation demonstrate the superior time efficiency of our method.
D.1 PSNR & SSIM& & LPIPS
PSNR. PSNR is measure of the ratio between the maximum possible power of signal and the power of corrupting noise that affects the fidelity of its representation. PSNR commonly used to compare the qualities of the original and the rendered images, and is obtained from the Mean Square Error (MSE) between the original and the rendered images:
MAX
PNSR 10 log10
(15)
MSE Rens@ ImagecT)
where MAX is the maximum pixel value of the image.
SSIM. SSIM measures the similarity between two images based on structural information. Its evaluation involves luminance, contrast, and structure. Compared to PSNR, SSIM is intended to match human perception more closely. The SSIM values range from -1 I, where denotes perfect. It is given as:
(2 + (20 + 02) SSIM (03 + 13 co) + o 02)
(16)
where and y two images, Max By and o². og the corresponding averages and variances of and y. Dzy represents the covariance and y, and ej and C2 denote the regularization terms.
LPIPS, LPIPS measures perceptual similarity in terms of features of deep neural networks, such as pre-trained VGG 34 or AlexNet [7]. It compares feature activations of image patches. Like [4
53 47
[9].
we here use the VGG-based networks.
D.2
ATE & RPE trans & RPE rot
ATE. ATE quantífies the difference between the actual trajectory and the estimated trajectory of robot or camera over time, offering global measure of error along the entire path. It is calculated by aligning the estimated trajectory with the ground truth and then measuring the Euclidean distance between each corresponding point on the two trajectories.
RPE trans. RPE trans quantifies the error in the translational component between consecutive poses or over a fixed time/distance interval. Unlike ATE, which assesses the overall trajectory, RPE Trans emphasizes the local accuracy of the motion estimation by evaluating how well the system preserves the relative motion between two points time or space.
RPE rot. RPE rot quantifies the error in the orientation component between the estimated poses and the ground truth. This metric is computed by measuring the difference orientation over short sequences, and it is typically expressed in angular units, such as degrees or radians.
E
More Results
E.1
Quantitative Results
E.1.1
Runtime
We report the detailed runtime comparisons on the NeRF-DS 500 DAVIS 28
and iPhone 7
datasets, each containing over 50 frames, where runtime differences become more pronounced. Specifically, in Table 9 and Table I0, the runtime of our method is the shortest. In addition, in Table 11 on the NeRF-DS 50 dataset, the runtime of COL w/ mask and COL wio mask on Plate video shorter than of ours. This is because COL mask and COL wio mask fail on this video, leading to quick convergence to the local minima. Such a conclusion can also supported by qualitative results of fig. in the main paper.
23

---

Table 9: Quantitative Runtime Results DAVIS [28]. Cam - camera optimization time;
Cam+Scene - overall (camera+scene) optimization time; - hour; m - minute. We mark the shortest time in bold. Our method is the most efficient without any failure.
Ours
COL w/o mask
casualSAM
Method
Cam
Cam+Scene
Cam
Cam+Scene
Cam
Cam+Scene
Camel
1.57m
25.28m
40m
71m
24m
46m
Bear
3.15m
1.08h
56m
88m
20m
39m
Breakdance-ffare
1.7.3m
0.92h
FAIL
15m
34m
Car-roundabout
4.97m
21.95m
10m
41m
18m
46m
Car-shadow
0.93m
17.88m
5m
31m
10m
36m
Car-turn
2.97m
17.35m
FAIL
27m
49m
Cows
2.85m
22.50m
73m
84m
26m
48m
Dog
1.60m
11.70m
10m
53m
12m
31m
Dog-agility
0.67m
26.63m
FAIL
5m
31m
Goat
2.10m
18.95m
107m
124m
22m
44m
Hike
4.20m
31.83m
FAIL
20m
42m
Horsejump-high
1.97m
21.20m
5m
33m
10m
31m
Lucia
1.97m
22.75m
44m
65m
16m
36m
Motorbike
2.08m
18.77m
6m
31m
9m
32m
Parkour
9.07m
26.65m
16m
37m
27m
48m
Rollerblade
1.25m
17.08m
FAIL
27m
Tennis
3.47m
17.03m
9m
30m
17m
38m
Train
1.90m
18.22m
32m
57m
19m
44m
Mean
2.68m
21.02m
31m
56m
17m
39m
Table 10: Quantitative Runtime Results on iPhone [7]. Cam > camera optimization time:
Cam+Scene overall (camera+scene) optimization time; h hour; m minute. We mark shortest time bold. Our method is the most efficient.
Ours
COL
casualSAM
Method
Cam
Cam+Scene
Cam
Cam+ Sam+Scene
Cam
Cam+Scene
Apple
33m
47m
10.95h
11.25h
6.80h
7.32h
Paper-windmill
15m
27m
8.18h
8.70h
3.50h
3.97h
Space-out
23m
69m
4,42h
4.72h
5.87h
6.30h
Backpack
7m
25m
1,83h
2.12h
1.58h
2h
Block
27m
42m
10h
10.45h
6.50h
Creeper
27m
45m
16.03h
16.45h
4.38h
4.82h
Handwavy
15m
30m
4,62h
5.1h
3.30h
3.72h
Haru-sit
10m
25m
0.77h
1.18h
1.25h
1.72h
Mochi-high-five
6m
18m
0.67h
1.03h
1.53h
1.93h
Pillow
21m
36m
19.18h
19.70h
3.70h
4,20h
Spin
30m
44m
20h
20.58h
5.50h
6h
Sriracha-tree
15m
27m
4,58h
4.88h
3.22h
3.68h
Toddy
31m
46m
19h
19.71h
6.78h
7.28h
Wheel
27m
46m
6m
13.80h
3.67h
4.17h
Mean
20m
38m
9.53h
9,97h
4.07h
4.53h

---

Table 11: Quantitative runtime results on NeRF-DS 150 Cam - camera optimization time; Cam+Scene - overall (cameratscene) optimization time: - hour; in - minute. We mark the shortest time bold. we show only Cam+Scene of RoDynRF 20 due to its joint optimization the camera and scene. is supervised by additional GT priors. COL w//mask and COL w/oosk faster than us on Plate because they fail on this video, leading to quick convergence to the local minima, which can be supported by qualitative results of fig. X in the main paper. Among all, our method is the most efficient.
Method
Ours
COL makk
COL mlem
casualSAM
RoDynRIF
Cam+Scene
Cam
Cam+Scene
Cam
Cam+Scene
Cam
Cam+Scene
Cam+Scene
Bell
1.05h
1.20h
2.50h
2.72h
3.00h
3.25h
16.5h
16.8h
28.6h
0.95h
1.0hh
2.00h
2.17h
2.55h
2.72h
14.87h
15.08h
33,6h
Basin
0.75h
0.92h
1.42h
1.62h
1.60h
1.85h
9.88h
10.67h
33.8h
Plate
0.53h
0.68h
0.42h
0.60h
0.50h
0.87h
4.67h
4,98h
25.6h
Press
0.68h
0.82h
0.85h
1.05h
0.90h
1.08h
6.28h
6,60h
28.5h
Cup
1.02h
1.15h
2.37h
2.58h
2.57h
2.73h
13.50h
13.78h
28.8h
Sieve
0.78h
0.92h
1.15h
1.35h
1.58h
1.77h
7.83h
8.13h
28.3h
Mcan
0.83h
0.97h
1.52h
1.73h
1.82h
2.03h
10.50h
10.80h
29.6h
E.2
Qualitative Results
E.2.1
Trajectories from Patch-wise Tracking Filters as Pseudo-supervision
In Figure TO. we show the trajectory comparisons on the NeRF-DS (50) dataset as samples. As discussed the 3rd paragraph in section [ and section of the main paper, patch-wise tracking filters can establish robust and maximally sparse hinge-like relations as accurate pseudo-supervision avoiding noisy and inaccurate tracking trajectories. Figure [0] is easy to see our proposed method avoids the inaccurate ones in the low-texture regions (walls), and meanwhile, adaptively adds new reliable trajectories when the number of left trajectories each frame is less than B.
E.2.2 NVS
We show more RGB and depth rendering results on NeRF-DS [50], DAVIS 28 and iPhone v dataset in Figure TT Figure [2 Figure [3] Figure [4] Figure T5| Figure T6 It is easy to see that the RGB and depth rendering results of our method are better than other RGB-only supervised approaches. In addition, the performance of our method also comparable with that of the LiDAR-brun R-baedd Record3D app.
Figure 17 and Figure
E.2.3 Optimized 3D Gaussian Fields
Since the iPhone 7 dataset the most challenging dataset with large camera and object movements. we show more visualizations of optimized 3D Gaussian fields in Figure 20 Figure 21 and Figure 19 Such comparisons demonstrate that our camera estimates enable superior reconstruction of 3D Gaussian fields compared to other RGB-only supervised approaches. Moreover, the reconstructed fields using our estimates are comparable to, or even surpass, those obtained with the LiDAR-base Record3D [3 app, particularly in scenes with significant motion.
25

---

o patch-wise filters
w patch-wise filters
w/o patch-wise filters
w/ patch-wise filters
Figure 10: Trajectory Comparisons on the NeRF-DS 50 Dataset. In each scenario, top row - Fo: bottom It is easy to see our proposed method avoids the inaccurate trajectories in the low-texture regions, whereas the trajectories of the points in the low-texture regions tracked by raw CoTracker [13 extremely unreliable.
F217: w/o patch-wise filters raw CoTracker 113; w/ patch-wise filters Ours.
26

---

RGB Frame (Backpack)
RGB Frame (Block)
RGB Frame (Creeper)
RGB Frame (Handwavy)
Our method
Our method
Our method
Our method
PSNR21122
PSNR 28
21
PSNR 28 02
S5iMo..64
SSIM
S
SSIM
LPISS
LPIPS:0
Record30
Record30
Record30
Record30
PSNF SsiMMo.55
PSNR 72
2SNR221 D0
PSNR
6SIM-0.71
SSIM-O
LPIPSSO 40
LPIPS
O
LPIPS:0
COL
COL
COL
COL
PSNR
PSNR
PSNR:18
SsiM .99
SSIM 60
SSIM0 45
SSIMO.61
LPIPS:0.5 54
LPIPS 9 49
O
CasuBISAM
CaSUBISAM
casuaiSAM
CAS
PSNF
PSNR-18 10
PSNR:16.4 6.40
PSNR 20 87
SsiMo.33
SSIM
SSIM:0.29
SSIM 68
0.
0.
LPIPS
Figure 11: More Qualitative NVS Results on iPhone 77 Part 1. Our renderings exhibit higher
fidelity and more accurate geometry compared to other RGB-only supervised methods. Besides, our performance is comparable with, or even better than, the ones of the LiDAR-based Record3D app.
27

---

RGB
RGB
RGB (Srtracha)
RGB (Pllow)
(Mochi-..ive)
(Spin)
Our method
Our method
Our method
Our method
PSNR:34.56 5e
PSNR244 24 81
periiaa 49
PSNR-20.88
SSIM-0
seinoo
ssiM
SSIM
PPPSS:0.2
Record30
Record30
Record30
Record30
PSNR
PSNR 85
DeNnisL 15
2SNR:20 88
SSiM
LIPS
LPIPS:0.41
COL
COL
COL
COL.
PSNR-22.47
SSNR
25
SSIM:O.77 79
SRIMO
SSlMoo
SSIM 46
LPIPS:0.38
LPIP'S 0.020 28
PIPS:
CaSuaISAM
CAS
casuaiSAM
casueiSAM
PSNR
PSNR-19.3 19:33
BA 20
PSNR:16.0 0.5M
SSIM10.84
SPIMP
SSIM.
0
OIPS
O 55
Figure 12: More Qualitative NVS Results on iPhone 77| Part 2. Our renderings exhibit higher
fidelity and more accurate geometry compared to other RGB-only supervised methods. Besides, our performance is comparable with, or even better than, the ones of the LDAR-based Record3D app.
28

---

(Apll)
(Teckyy)
(Hanu-
Our
Our
method
Our
33
PIPSI
1.
Recordzo
Record30
Record.0
RecerdDD
3ecord
20.5
EN
6SIM
3
COL
COE MT
COL
XOL
PSN
NILTES
OSIME
B3
M0
PIPS 60
LIFS 0.
CARUBEAM
SAM
PENEAI CO
PE
ONR E8
EN DO RSIM- na LPIES
LF
LPIFS
Figure 13: More Qualitative NVS Results iPhone 7 Part 3. Our renderings exhibit higher
fidelity and more accurate geometry compared to other RGB-only supervised methods. Besides, our performance comparable with, or even better than, the ones of the LiDAR-based Record3D app.
RGB
RGB
RGB
eetee ewz Joba
RGB
AM
Ours
Ours
OurS
III Ours
ewz foto
Ours
LP
TR 7T
LPIPS:0
COL 11/ ONTTURSA
COL M/ OmIIEsA
MZ omnae
MI oommak
COL Ay amnakk
FAIL
FAIL
7770500700
P4 SSIM LP
PSNR:11 SSIMAO.4 42 LPIPS0.70
CasualSAM
casualSAM
casualSAM
casualSAM
casualSAM
Camel
Break.-flare
Dog-agility
Car-shadow
Figure 14: More Qualitative NVS Results on DAVIS 28 Part 1. Our renderings exhibit higher fidelity compared to other RGB-only supervised methods.
29

---

RGB
RGB
RGB
RGB
RGB
Ours
Ours
Ours
Ours
Ours
PENR AUBS SEIUE LPIPS PSNRIAASS 35 SSIM LPISS,OTT PRAREES SSIM S.78 PPPP 24 PENTTOA DA SSIMOD LIPPSO
PSNA 54 SSIM
COL
M/ OiTuEsA
COL MY OTIIEEA
COL 9J Comaaak
COL M/ ormesk
FAIL
PINPLTS.7E SSINN LIPSI
PSHET SSIMO PSNAAT
LP
PSURA RR SSIM LIIFS
casualSAM
casualSAM
casualSAM
casualSAM
casualSAM
LPISS PANAT
LPIPS
LPIPS:0.56
Goat
Hike
Horsejump-high
Lucia
Motorbike
Figure 15: More Qualitative NVS Results on DAVIS (28 Part 2. Our renderings exhibit higher fidelity compared to other RGB-only supervised methods.
RGB
RGB
RGB
RGB
Ours
Ours
Ours
Ours
PSNR-25 SSIMM0 76 PIPSS0022 25 SSINEO S6 PIPSS:0.99
PSNR220 18 M-0.- 56 LPIPST
PSNR .5 SSIM:0.7 75 LPIPS:0.16
COL
omaas
COL
ommas
COL
omaak
COL
omask
FAIL
FAIL
MM.0.19 1PPPSSO666
PSNR 17.94 SSIMO..50
M.00 LPPISS0049
casualSAM
casualSAM
casualSAM
casualSAM
PSNR:118 19.83 SSIMO0.0 39 LPIPS:0.52
PSNR SSIM 46 LPIPSP0 49
PSNR:18.04 SS 4. LPIPS:0:53
19NR SSIMO0.57
LPIPS:0.31
Cows
Dog
Parkour
Rollernlade
Figure 16: More Qualitative NVS Results on DAVIS 28 fidelity compared to other RGB-only supervised methods.
Part 3. Our renderings exhibit higher
RGB
Ours
COL w/ /omask
casualSAM
Horse .low
FAIL
16 SSIM LPIPS:0.58
PSNR 14 SSIM.0.3 38 LPIPS:0.50
Figure 17: More Qualitative NVS Results on DAVIS (28) Part 4. Our renderings exhibit higher
fidelity compared to other RGB-only supervised methods.
30

---

RGB Frame
Frame (Press)
RGB Frame (Sieve)
(As)
Our method SE
Our method SEIM
Our method
UPIPEO 10
CPPS 12
LPISG 69
OOMM
COL My
94
COL /0/ Tmane
S0MM..9 LPIPKO
COL //
UPPSS 12
CPPSO 10
COL 1/
COL soy
COL
OSMG
UMPSO 15
CPPS
LPIPAO
casualSAM
casualSAM
casualSAM
5SMM.
UPPSS 44
AP
UPIP5C-33 39
RGB Frame
Frame (Cup) Our method
(Bell)
Our method
SSIMG 91
SSMB BE
UPIPS 15
LP 11
COL M/
COL M/
BSMM
DE B4
UPIP8.0.17 17
LP
PENNRIS SSIM
COL
60
COL My
LPIP8
LPFP8.13
PSNRCILTT SSIM S4
casualSAM PENRZ1 SEMM 68
CAS
LPISS 55
APPSO
Figure 18: More Qualitative NVS Results on NeRF-DS (50). Our renderings exhibit higher fidelity compared to other RGB-only supervised methods.
31

---

RGB Frame
Ours
Record3D
(d) COL /mmma
(e) casuaiSAM
Figure 19: Optimized 3D Gaussian Fields iPhone 7 Part 1. Our reconstructed 3D Gaussian Fields are more geometrically accurate compared to the ones of other RGB-onI, supervised methods, which demonstrates our camera estimates are more accurate. Besides, our performance is comparable with or even better than, the ones of the IDAR-based Record3D app.
32

---

(a) RGB Frame
Ours
(c) Record3D
COL w/ //emann
casualSAM
Figure 20: Optimized 3D Gaussian Fields on iPhone 7 Part 2. Our reconstructed 3D Gaussian Fields are more geometrically accurate compared to the ones of other RGB-on) supervised methods, which demonstrates our camera estimates are more accurate. Besides, our performance is comparable with, or even better than, the ones of the LiDAR- based Record3D app.
33

---

(a) RGB Frame
(b) Ours
(c) Record3D
(d) COL /ommank
(e) casualSAM
Figure 21: Optimized 3D Gaussian Fields on iPhone 77 Part 3. Our reconstructed 3D Gaussian Fields are more geometrically accurate compared to the ones of other RGB-on supervised methods, which demonstrates our camera estimates are more accurate. Besides, our performance is comparable with, or even better than, the ones of the LiDAR-ban based Record3D app.
34


Reviews:
[{"review_id": "Y9uJrMRfHu", "reviewer": "Reviewer_nBqP", "summary": "This paper solves an important problem in dynamic scenes “solely supervised by RGB video”. To solve this problem, three key components are proposed as the main contributions of this paper.", "questions": "1. How did you determine w=12/24 and τ_var=0.1? Can you provide sensitivity analysis for these parameters? What happens when there are insufficient high-texture patches?\n\n2. You mention the method fails with large moving objects. Can you quantify this - what percentage of moving pixels causes failure? Is there a way to predict these failure cases?\n\n3. Have you tested sequences longer than 900 frames ? What’s about the stability ? For example, 2000 frames, since COLMAP failed occasionally.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. Well-structured methodology: The paper presents a comprehensive approach with three well-motivated components that address specific challenges in RGB-only camera parameter optimization for dynamic scenes.\n\n2. Extensive experiments: The authors conduct thorough experiments across 5 datasets with both quantitative metrics and qualitative visualizations,  I appreciate authors' hard work.\n\n3. Efficiency improvements: The method achieves substantial runtime reductions (e.g., 55% of the second-fastest method on NeRF-DS) while maintaining or improving accuracy, which is practically valuable.\n\n4. Ablation studies: The paper provides detailed ablation studies validating each component's contribution to the overall performance.\n\n**Weakness:**\n\n1. It seems all the words on the figures are blured, this reduces readability and aesthetics. May the authors make the words on figures clearly ? For example, change all of the figures to vector format.\n\n2. The claim of being the \"first RGB-only supervised\" method (Contribution 1, line 61) contradicts Table 1, which clearly shows existing RGB-only methods like Robust-CVD and casualSAM. This should be corrected to accurately position the work's contributions.\n\n3. The patch-wise tracking filters appear to be incremental improvements over existing point tracking methods (CoTracker). I think there should be deeper analysis like:\n\n- How patch size $w$ affects performance across different scene types\n- Sensitivity analysis of the texture threshold $τ_{var}$\n\n4. I encourage authors present limitation in the maintext. I think there should be more discussion when and why the method fails.", "was_revised": true, "final_justification": "Overall, I hold a positive evaluation of this paper, and the rebuttal phase has successfully addressed all of my concerns. I agree with reviewer 79BF's point that \"The overall novelty looks fairly standard containing steps used in SfM methods,\" but this paper indeed tackles a rather challenging problem: \"RGB-only supervised camera parameter optimization in dynamic scenes.\" The authors have conducted thorough comparisons with previous work and demonstrated the superiority of their method.\n\nWhat I am not entirely satisfied with in this paper is that the numbers in the figures are too blurry (due to being saved in non-vector format), which causes some reading difficulties and diminishes the overall aesthetic quality of the paper. Therefore, I maintain my weak accept recommendation and suggest that the authors revise the figures.", "rebuttal": "# Response to Reviewer nBqP:\nThank you for your acknowledgment of our paper. We sincerely appreciate your time and feedback, and address your concerns below.\n\n## About Weakness 1, ‘The blurry words in the figures’\nThanks for your suggestions. After re-checking, we found that some figures were not correctly saved as vector graphics. Since the article cannot be modified at this stage, we plan to correct it in the camera-ready version.\n\n## About Weakness 2, ‘Words’ contradictory’\nThank you for pointing out this misunderstanding. What we meant is that our method is the first RGB-only supervised approach for dynamic scenes that is both accurate and efficient, as existing RGB-only methods are either slow or inaccurate. We will remove the word “first” in the camera-ready version.\n\n## About Weakness 3, ‘How patch size affects performance across different scene types?’\nThank you for raising this question. **We would like to clarify that although patch size can influence performance, the scene type is not a factor in this process; thus, we do not adjust patch size based on scene types.** Instead, as noted in Line 200 of our main paper, the patch size $w$ is determined solely by the frame size. \n\nSpecifically, when the input aspect ratio (long/short side) exceeds 1.5, we consider the image elongated. Camera motion in such cases causes obvious uneven scene changes across axes, so we adopt a smaller patch size to capture more valid high-texture patches. Otherwise, we use a larger patch size. The value of $w$ is chosen as a trade-off between memory cost and efficiency, **but it should definitely be adjusted for extremely small or large frame sizes**. We will add such discussions to the camera-ready version. Further sensitivity analysis on $w$ is provided in our response to **About Question 1, ‘Sensitivity analysis on $w$ and $\\tau_{var}$’**.\n\n## About Weakness 3, ‘Sensitivity analysis of the texture threshold’\nThank you for your advice. Please see our subsequent answers to **About Question 1, ‘Sensitivity analysis on $w$ and $\\tau$’**, for a detailed discussion about the sensitivity analysis of the texture threshold.\n\n## About Weakness 4, ‘The position of limitations’\nThank you for your suggestions. We will move the limitation section to the main paper in the camera-ready version.\n\n## About Weakness 4, ‘More discussion on when and why the method fails?’\nThank you for your advice. We should discuss more in the camera-ready version. As shown in Fig. 21 (Appendix, L586–589), our method fails when videos are dominated by large, moving humans and dragons. We attribute it to two main reasons: 1) high outlier percentage; 2) extreme outlier patterns where large objects sweep and cover the screen. These factors hinder robust, sparse hinge-like relations extracted as accurate pseudo-supervision. More details can be seen in our answers to **About Question 2, ‘What percentage of moving pixels causes failure? ..xxx.. failure cases?’**.\n\n## About Question 1, ‘How did you determine $w$=12/24 and $τ_{var}$=0.1?’\nThank you for your question. Please refer to our answers to **About Weakness 3, ‘How patch size affects performance across different scene types?’** for a thorough analysis of $w$.\n\n**We sincerely apologize for the typo in the paper. The value “$τ_{var}$=0.1” should be corrected to “$τ_{var}$=0.01”.** We will correct this typo in the camera-ready version. \n\nWe empirically set $\\tau_{var}=0.01$ and use this fixed value for all videos. Eq. 1 in main paper selects relatively high-texture patches by keeping those whose intensity variance exceeds $0.01$ $\\times$ $\\sigma*$ ($\\sigma*$ is the maximum patch variance in a frame). This removes textureless regions (e.g., plain backgrounds, white walls; see Fig. 9 in Appendix) while retaining useful information, as only patches with variance below 1% of the maximum are discarded.\n\n## About Question 1, ‘Sensitivity analysis on $w$ and $\\tau_{var}$’\n\nThank you for asking this question. We conduct the sensitivity analysis on these two parameters, as shown in the subsequent two tables with detailed discussions. We plan to add the following two tables to the ablation study section of our main paper in the camera-ready version to make our paper more complete.\n\n\n### Table 1: Sensitivity Analysis on Patch Size $w$ on NeRF-DS dataset\n| Scene Optimization | Camera Optimization           | PSNR  | SSIM   | LPIPS  |\n|--------------------|-------------------------------|-------|--------|--------|\n| 4DGS               | ours($w$=12, default)     | 33.55 | 0.9381 | 0.1182 |\n|  4DGS               |   ours($w$=16)               | 32.23 | 0.9085 | 0.1411 |\n|  4DGS               |    ours($w$=24)              | 24.71 | 0.7722 | 0.3084 |\n\n---------------------------------------------------------\n\n### Table 2: Sensitivity Analysis on Variance Threshold $\\tau_{var}$ on NeRF-DS dataset\n| Scene Optimization | Camera Optimization                 | PSNR                                   | SSIM                                   | LPIPS                                  |\n|--------------------|-------------------------------------|----------------------------------------|----------------------------------------|----------------------------------------|\n| 4DGS          | Ours($\\tau_{var}$ = 0.01, default)        | 33.55                                  | 0.9381                                 | 0.1182                                |\n|   4DGS        | Ours($\\tau_{var}$ = 0.04)                        | 32.45                   | 0.9136                | 0.1367               |\n|    4DGS       | Ours($\\tau_{var}$ = 0.08)                          | 32.14                   | 0.9016                | 0.1408                           |\n|   4DGS       | Ours($\\tau_{var}$ = 0.16)                          | Fail                                                       |\n\nTables 1 and 2 show that our model is more sensitive to $w$ than $\\tau_{var}$. Such ablation studies also demonstrate our good value definition on these two parameters. However, too large $\\tau_{var}$ makes the model fail due to a lack of useful information.\n\n## About Question 1, ‘What happens when there are insufficient high-texture patches?’\nThank you for asking this question. To clarify, what we mean by ‘high-texture patches’ is the **relatively** high-texture patches. We should make it clearer in the camera-ready version. Like our answers to **About Question 1, ‘How did you determine w=12/24 and τ_var=0.1?’** above, in each frame, we only discard the relatively low-texture patches whose intensity variance is less than 1% of the maximum intensity variance. This efficiently avoids losing useful high-texture patches. However, we might share limitations with others in certain extreme scenarios, such as videos that predominantly feature plain walls with super-limited texture. We will leave it as our future research topic.\n\n## About Question 2, ‘What percentage of moving pixels causes failure? Is there a way to predict these failure cases?’\nThank you for asking this question. **We would like to clarify that there is no strict threshold for the percentage of moving pixels that leads to failure.** Because the model’s ability to handle outliers is also affected by the appearance pattern of outliers. Table 3 below compares the performance, outlier percentage, and appearance pattern in all failure cases and one sampled success case. We plan to add this table to the main paper of the camera-ready version. Since the MPI-Sintel dataset does not provide any GT motion mask, we manually annotate the motion masks. \n\n### Table 3: Failure Case Analysis by Outlier Percentage and Appearance Pattern\n| Case Type            | Scene       | Percentage of outliers | ATE   | RPE trans | Appearance Pattern                                              |\n|----------------------|-------------|------------------------|-------|-----------|-----------------------------------------------------------------|\n| Failure case        | cave_2      | 35.14%                 | 1.609 | 0.226     | Object briefly sweeps across the frame, temporarily covering it |\n|  Failure case  | cave_4      | 39.36%                 | 0.125 | 0.035     | Object persistently dominates most of the frame.               |\n|  Failure case  | market_5    | 34.31%                 | 2.027 | 0.617     | Object persistently dominates most of the frame.               |\n|  Failure case  | temple_3    | 31.92%                 | 0.782 | 0.1467    | Object briefly sweeps across the frame, temporarily covering it |\n| Sampled Success | ambush_6    | 44.30%                 | 0.080 | 0.129     | Object persistently dominates about half of the frame.         |\n\nAs shown in the above table and Figure 21 of the Appendix, the model’s performance is not strictly determined by the proportion of outliers: in ambush_6, the model performs well even with a large number of outliers, while in others, performance degrades despite a small proportion of outliers. The appearance pattern also affects the performance of the model. \n\nCurrently, there is no quantitative indicator available to predict whether the method will succeed or fail. But we highly suggest forecasting based on the appearance pattern.\n\n## About Question 3, ‘Testing on extremely long videos (>2000 frames)’\nThank you for your questions. To our knowledge, existing test datasets and prior methods do not provide extremely long videos (>2000 frames) nor report experiments under such conditions. So we select two sequences exceeding 2000 frames (footlab_3rd and scene_j716_3rd) from the Point Odyssey dataset. However, even after trimming the sequence to 1500 frames, our model still encounters RAM capacity limitations on our setup, where all experiments are conducted on a single NVIDIA A100 GPU (40 GB) with 256 GB of system RAM. The longest video in our current experiments contains 881 frames (bell in NeRF-DS). We plan to investigate methods for reducing GPU and RAM consumption as part of our future work."}, {"review_id": "RFuWfVXVhB", "reviewer": "Reviewer_79BF", "summary": "This paper proposes a SfM method, which estimates camera trajectory from an RGB video of the dynamic scenes. Unlike COLMAP and its successors, this approach doesn’t require any supervision and estimates camera intrinsics and extrinsic only from the RGB frames of the video. The method introduces three contributions: 1) patch-wise keypoint filter, which removes dynamic points and extracts stable points from the entire sequence, which will be used as a supervision signal; 2) Outlier handling mechanism integrated in the camera pose optimization procedure focusing on suppressing points coming from the dynamic objects; 3) Two step optimization strategy , which enhances the speed of the optimization.  The method is evaluated on several datasets demonstrating improved performance and greater efficiency compared to the   related approaches.", "questions": "$\\Gamma_{raw}$ seems to be important parameter and its choice has not been experimentally analyzed in the main paper. It would be good to perform such analysis and backup the claims made in the two stage optimization procedure.\n\nThe qualitative results presenting camera trajectories in Fig.6 are good, but I would appreciate to see more examples in different scenes, especially, somehow ranked by the level of difficulties, e.g. from small to large numbers of moving outliers(low dynamic to high dynamic scenes).\n\nIt is strongly recommended to include and discuss failure cases both quantitatively and qualitatively (examples of recovered trajectories).", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The main strength of the proposed method is their computational efficiency achieved due to their two step optimization strategy. Moreover, paper achieves reasonable performance given that it doesn’t use any GT supervision. The experimental evaluation is very detailed and well done. The quality of the writing could be improved.\n\nWeaknesses: The overall novelty looks fairly standard containing steps used in SfM methods. The choice of the steps is reasonable, but in spite the ablation study it is hard to assess how is each step critical in different dynamic scenes. For example, it would be good to comment how many correct point trajectories have been extracted after the first step in different dynamic scenes, starting from a low to high amounts of moving outliers. Further, the question is what amount of outliers can be handled by the proposed outlier handling strategy. The current proposed outlier handling strategy is similar to standard reweighed least square procedure combined with the robust estimators, e.g. Tukey estimator. It would be good to compare the performance with other robust estimators. \nIn the experimental evaluation it is evident that the methods with mask supervision or other GT supervision signals perform better then the proposed method. The question is how would the proposed method perform with such supervisions. It would be interesting to assess if the proposed optimization and outlier handling strategy would outperform the other methods if GT supervision was used.", "was_revised": true, "final_justification": "I checked the authors' replies to my questions and I appreciate their answers and effort they put in the explanations. I also checked other reviews and answers to them. Authors are strongly encouraged to include ablations on parameter choice and include a discussion about failure cases. I am happy about the rebuttal and will increase the rating to borderline accept.", "rebuttal": "# Response to Reviewer 79BF:\nThank you for your acknowledgment of our paper. We sincerely appreciate your time and feedback, and address your concerns below. As your concerns were not explicitly numbered or listed, we summarize them as follows.\n\n## About Weakness 1, ‘how many correct point trajectories..xxx..starting from a low to high amounts of moving outliers.’\nThank you for your question. We believe there must be a misunderstanding, and we should make it clear in the camera-ready version with the following clarifications.\n\nFirstly, we do not define point trajectories by correctness. Both inlier and outlier trajectories can be correct or incorrect, and these datasets lack GT optical flow or tracking labels, so such a definition would be meaningless. Instead, we classify trajectories into inliers and outliers, which directly motivates our proposed outlier-aware joint optimization.\n\nSecondly, the number of moving outliers does not affect trajectory accuracy. Our patch-wise tracking filters are built on CoTracker, which can robustly track trajectories regardless of the number of moving points, as discussed in the CoTracker paper.\nFor further analysis on our model’s capacity to handle different amounts of outliers, please see our next response.\n\n## About Weakness 2, ‘What amount of outliers can be handled?’ \n\nThank you for the question. By “amount of outliers,” we assume you mean the percentage of outliers among all trajectories. **We want to clarify that model performance depends not only on this percentage but also on the appearance pattern of outliers.** To illustrate, we provide a table comparing performance, outlier percentage, and appearance patterns across failure cases and one success case (motion masks for MPI-Sintel are manually annotated). As shown in Table 1 (below) and Fig. 21 (Appendix), a high outlier percentage (e.g., Ambush_6) can still yield good results, while lower percentages can fail when the appearance pattern is extreme. Thus, there is no strict threshold for the outlier percentage that leads to failure.\n\n### Table 1: Failure Case Analysis by Outlier Percentage and Appearance Pattern on MPI-Sintel dataset.\n| Case Type            | Scene       | Outliers Percentage | ATE   | RPE trans | Appearance Pattern                                              |\n|----------------------|-------------|------------------------|-------|-----------|-----------------------------------------------------------------|\n| Failure  | cave_2      | 35.14%                 | 1.609 | 0.226     | Cover / Sweep across the frame    |\n|  Failure  | cave_4      | 39.36%                 | 0.125 | 0.035     | Dominate screen across the frame   |\n|  Failure  | market_5    | 34.31%                 | 2.027 | 0.617     | Dominate screen across the frame    |\n|  Failure | temple_3    | 31.92%                 | 0.782 | 0.1467    | Cover / Sweep across the frame |\n| Sampled Success | ambush_6    | 44.30%                 | 0.080 | 0.129     | Dominate half screen across the frame         |\n\n## About Weakness 2, ‘Why not the Tukey estimator? Other estimator?’ \nThank you for asking this question. To our knowledge, we believe what you mean by ‘the Tukey estimator’ is Tukey’s biweight estimator. In the weighting function of Tukey’s biweight estimator,\n$$\nw(u) =\n\\begin{cases}\n\\left(1 - \\left(\\frac{u}{c}\\right)^2\\right)^2, & \\text{if } |u| < c \\\\\\\\\n0, & \\text{if } |u| \\geq c\n\\end{cases}\n$$\nTukey’s biweight completely discards residuals beyond the cutoff $c$, a hard threshold that must be tuned for each video. In contrast, our Cauchy weighting learns all parameters, allowing our outlier-aware joint optimization to generalize across videos.\n\nWe are also happy to discuss other estimators to explore potential improvements on our current proposed method. Please feel free to let us know. \n\n## About Weakness 3, ‘How would the proposed method perform with more GT supervisions?’\nThank you for your question. More GT supervision (depth, camera pose, motion masks, 3D point clouds, camera intrinsic) will change the architecture of our pipeline, which is specifically designed for RGB-only supervision. So under such circumstances, our model needs to be carefully redesigned. But, theoretically, like most other models, we believe our method will perform better with more GT supervision. We might leave this as a future research path.\n\nFurthermore, we want to reclarify that our biggest contribution is our capability to **accurately and efficiently** estimate the camera parameters in dynamic scenes **solely supervised by a single RGB video, without any other GT supervision**. \n\n## About Question 1, ‘Analysis on $\\Gamma^{raw}$'\nThank you for your suggestion. As shown in Table  7 of the main paper, replacing our two-stage strategy with a randomly initialized and learnable $\\Gamma^{raw}$ leads to a clear performance drop. \nTo address the request for a deeper analysis, we conducted additional ablations on $\\Gamma^{raw}$ (see table below). \n'**Meaningless**' indicates that when $\\Gamma^{raw}$ is frozen in Stage 1, other value settings behave the same as setting $\\Gamma^{raw}=1$, so ablations 2 and 3 duplicate the settings of ablation 1 and the full model.\n\n### Table 2: Analysis on $\\Gamma^{raw}$ on NeRF-DS dataset. (Stage 1: 200 iterations; Stage 2: 50 iterations.)\n| Model              | Stage 1 Initialization                 | Stage 2 Initialization                 | PSNR  | SSIM   | LPIPS  |\n|--------------------|----------------------------------------|----------------------------------------|-------|--------|--------|\n| Full model         | $\\Gamma^{raw}$ = 1 (freeze)                 | $\\Gamma^{raw}$ = $\\mathbb{E}^{ACP}_{stage1}$ (learn)                  | 33.55 | 0.9381 | 0.1182 |\n| More ablation 1    | $\\Gamma^{raw}$ = 1 (freeze)                 | $\\Gamma^{raw}$ = 1 (learn)                  | 31.81 | 0.9073 | 0.1403 |\n| Existing Ablation  | $\\Gamma^{raw}$ = Random (learn)             | -  (learn)                    | 25.95 | 0.8100 | 0.2668 |\n| More ablation 2    | $\\Gamma^{raw}$ = Random (freeze)            | -  (learn)             |          |    **Meaningless**    |        |\n| More ablation 3    | $\\Gamma^{raw}$ = Random (freeze)            | $\\Gamma^{raw}$ = $\\mathbb{E}^{ACP}_{stage1}$ (learn)                  |       |   **Meaningless** |     |        |\n\nSuch comparisons show that our initialization strategy on $\\Gamma^{raw}$ in different stages is very important; otherwise, the performance drops dramatically.\n\n## About Question 2, ‘more examples in different scenes, ranked by the level of difficulties, e.g., ..xxx... (low dynamic to high dynamic scenes)’\n\nThank you for your questions. To clarify, the difficulty of camera evaluation depends on multiple factors (outliers, camera motion, appearance pattern, scene dynamics, etc.). Analysis on parts of them is in our answers to **About Weakness 2, ‘What amount of outliers can be handled?’**. To our knowledge, there are no explicit quantitative or qualitative metrics to rank it. We will clarify this in the camera-ready version.\n\nUnlike the related methods (Table 1 in the main paper), mostly supported by experiments on 30 - 50 videos, our results are supported by experiments on 5 popular public camera evaluation datasets covering about 70 videos across different difficulty levels. To make this clearer, we summarize the dataset characteristics (Section C of the Appendix) in the following table and will include it in the camera-ready version.\n\n### Table 3: Dataset Features\n| Dataset       | Scene Dynamics Level | Scene Dynamics Pattern                                    | Camera Motion Level  | Camera Motion Pattern                                                                 |\n|---------------|----------------------|-----------------------------------------------------------|-----------------------|---------------------------------------------------------------------------------------|\n| NeRF-DS       | Large                | Shaking movement of reflective oject             | Large                 | Circle-like movement w/ large rotation change                 |\n| DAVIS         | Large                | Regular movement of daily common objects  | Large                 | Curve-like movement w/ proper rotation change              |\n| MPI-Sintel    | Large & Complex      | Cartoon characters’ irregular movement                   | Large                 | Straight-line-like movement w/ large rotation change          |\n| iPhone        | Large & Complex      | Irregular movement of daily common objects               | Large & Complex       | Fluctuating movement w/ large rotation change         |\n| TUM-Dyn  | Large & Complex      | Irregular movement of humans                              | Large & Complex       | Fluctuating movement with large rotation change      |\n\nDue to the character limit, please refer to our response to **Reviewer  9gnd, About Weakness 3, ‘Performance on static scene dataset,’** for a further investigation of the generalization ability of our method. Thank you.\n\n## About Question 3, ‘More qualitative and quantitative analysis on failure cases’\nThank you for your advice. We will add the following discussions to the paper in the camera-ready version. As shown in Fig. 21 (Appendix, L586–589), our method fails when videos are dominated by large, fast-moving objects (e.g., humans, dragons). We attribute it to two main reasons: (1) a high proportion of outliers and (2) extreme outlier patterns where large objects sweep and cover the screen. These factors hinder robust, sparse hinge-like relations extracted for accurate pseudo-supervision. More details about the reasons can be seen in our answers to **About Weakness 2, ‘What amount of outliers can be handled?’**."}, {"review_id": "yTSXmhH2lj", "reviewer": "Reviewer_izms", "summary": "The paper proposes an RGB-only camera-parameter optimisation pipeline for casually captured dynamic-scene videos.\nKey contributions are: 1) Patch-wise Tracking Filters, 2) Outlier-aware Joint Optimisation 3) A Two-Stage Optimisation Strategy.\nExtensive experiments on 5 public datasets (NeRF-DS, DAVIS, iPhone, MPI-Sintel, TUM-dynamics) show better accuracy and up to 12× faster runtime than previous RGB-only baselines (casualSAM, COLMAP w/o mask) and, on some videos, even LiDAR-assisted Record3D. Ablations confirm each design choice.", "questions": "You state in line 273 that the performance is “nearly independent” of the PT backbone (Table 8).  Please add results with a much weaker tracker to support the claim, or consider improving the statement for rigorous discussion.", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n（1）Well-designed optimisation objective; thorough ablations and evaluation on five datasets.\n（2）Method is laid out step-by-step with helpful figures; loss derivations in appendix; datasets & metrics clearly listed.\n（3）Addressing a practical bottleneck for 4D reconstruction; runtime improvement is compelling for real use.\n（4）Novel coupling of sparse hinge-like tracks with Cauchy uncertainty in bundle adjustment; two-stage soft-plus trick is neat\n\nWeaknesses\n\nAll designs are well motivated, and the overall presentation is of high quality. Minor weaknesses, like the Abrupt introduction of COLMAP and visual odometry at the beginning of the paper, which confuses my understanding of the target problems, are suggested to be improved.", "was_revised": true, "final_justification": "I have no further questions. The author’s explanation makes sense and does not affect my original rating.", "rebuttal": "# Response to Reviewer izms:\nThank you for your acknowledgment. We really appreciate your time and feedback, and address your concerns below.\n\n## About Weakness 1, ‘Abrupt introduction on COLMAP and Visual Odometry’\nThank you for your suggestions on paper writing. We plan to add more background information at the beginning of the paper, to help us make the introduction of visual odometry, and the state-of-the-art COLMAP more fluent.\n\n\n## About Question 1, ‘Results with a much weaker tracker’\n\nThank you for your question. As stated in Lines 141–143 of the main paper, although our proposed patch-wise tracking filters do an iterative process until there are $B$ tracked points in each frame, such filters do $\\underline{not}$ conduct any $\\underline{repeated}$ refinement or optimization of point tracking on the $\\underline{same}$ point trajectory. This is also the main reason why our runtime grows almost linearly as frame count increases, as shown in Figure 4 of the main paper. Thus, the pre-trained PT model on which our proposed filters are based must be a **feed-forward prediction model, outputting point locations and visibility**; we will clarify this further in the camera-ready version.\n\nIn Table 8 of the main paper, we show ablation experiments on 2 different popular pre-trained PT models, CoTracker and CoTracker3. The results show that our proposed method is nearly independent of building on the particular PT models, as long as these requirements are met.\n\nBesides, Table 1 below summarizes all baseline PT methods reported in the CoTracker and CoTracker3 papers. CoTracker and CoTracker3 are the only two methods satisfying our requirements. Thus, other PT models cannot be incorporated in our proposed method due to architectural incompatibility.\n\n### Table 1: Properties of all baseline methods reported in the CoTracker and CoTracker3.\n| Method                      | Type                          | Output Format                      |\n|-----------------------------|-------------------------------|------------------------------------|\n| TAP-Net            | Feed-forward                  | Point locations only               |\n| OmniMotion            | Optimization-based            | Point locations only               |\n| PIPs               | Iterative (sliding window)    | Point locations only               |\n| MFT               | Feed-forward                  | Point locations only               |\n| PIPs++           | Iterative refinement          | Point locations only               |\n| TAPIR            | Recurrent                     | Point locations + Visibility   |\n| **CoTracker**         | **Feed-forward**              | **Point locations + Visibility**   |\n| TAPTR                | Recurrent/transformer         | Point locations + Visibility       |\n| LocoTrack           | Recurrent                     | Point locations + Visibility       |\n| **CoTracker3**        | **Feed-forward**              | **Point locations + Visibility**   |\n| BootsTAPIR         | Recurrent                     | Point locations + Visibility       |\n\nWe are also happy to discuss any other proper pre-trained PT models to explore potential improvements on our current proposed method. Please feel free to let us know."}, {"review_id": "BrmQ4d08Wx", "reviewer": "Reviewer_9gnd", "summary": "This paper addresses the challenging task of estimating camera parameters in dynamic scenes. Unlike previous methods that rely on ground-truth labels or require ground-truth masks of moving objects, the proposed approach is built entirely on a point tracking model and performs joint optimization based solely on the extracted point tracks. From my perspective, this method can be viewed as a form of bundle adjustment that is robust to dynamic objects. The results are impressive on real-world videos, even surpassing some supervised methods and commercial software such as that used on iPhones.", "questions": "Do other point trackers work as well?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1. The results are impressive—the proposed method outperforms many supervised approaches on real-world videos, despite relying solely on RGB inputs without any ground-truth supervision. \n\n2. The paper provides a clear and structured comparison with existing methods, particularly by categorizing them based on the type of supervision they use. This helps position the contribution of the proposed method within the broader literature.\n\n\nWeakness:\n\n1. While the technical content is sufficient, the paper is not easy to read. It would benefit from a more global introduction and broader background context, rather than focusing almost entirely on the method itself. This would make the paper more accessible and easier to follow.\n\n2. A key component of the proposed method—the point tracking (PT) model—is not clearly introduced or described. In particular, the paper lacks details about the PT model’s input, output, and working mechanism. Since this module is crucial for enabling joint optimization from RGB inputs, a brief explanation—perhaps as a preliminary section—would be helpful. As a reader, it was unclear how joint optimization could be performed using only raw RGB pixels until it became evident that the method relies on extracted point correspondences.\n\n3. The proposed method targets dynamic videos, but it should also apply to static scenes. I am curious how it performs on standard static video benchmarks such as 7-Scenes, Tanks and Temples, or ScanNet. While it is not necessarily expected to outperform methods specifically designed for static scenes, such a comparison would serve as a useful reference and help assess the method’s generalizability.\n\n4. It would be helpful to include the number of frames per scene when reporting runtime comparisons in Table 9 and other related tables. This would provide better context for evaluating the computational efficiency across different scenarios.", "was_revised": true, "final_justification": "My concerns have been addressed in the rebuttal, and I have decided to maintain my original rating.", "rebuttal": "# Response to Reviewer 9gnd:\nThank you for your acknowledgment of our work. We highly appreciate your time and feedback, and address your concerns below.\n\n## About Weakness 1, ‘Broader background context for a more readable paper’\nThank you for your valuable suggestions. We appreciate your recognition of the technical content in our paper. As the current submission cannot be modified, we plan to enhance the background information in the camera-ready version to facilitate a clearer understanding of our work. In particular, we will provide a more comprehensive introduction to the research field at the beginning of our paper. Furthermore, regarding the pre-trained PT model that serves as the foundation for our proposed patch-wise tracking filters, we will include a more detailed explanation in the main paper—or, if permitted by the page limit, in a dedicated preliminary section.\n\n## About Weakness 2, ‘Introduction about the pre-trained PT model’\nThank you for pointing it out. We plan to add a more detailed explanation of the pre-trained PT model in the main paper in the camera-ready version. The default pre-trained PT model on which our proposed method is built is CoTracker. CoTracker is a transformer-based model achieving dense 2D point tracking in long video sequences. Unlike other related works, it jointly tracks the 2D points so that the dependencies on each other can help improve the accuracy and robustness of the model. During inference, the vanilla CoTracker allows a set of 2D query points and an RGB video as inputs and directly outputs the 2D location and visibility of each 2D query point on each frame. Although it significantly improved the PT performance compared with the existing works, it still suffers from the tracking location drift across frames, especially when one point disappears from the camera frustum and reappears in the field of view, as shown in Figure 9 in the Appendix. Such observations motivate us to propose the patch-wise tracking filters to extract only the robust, reliable, and sparse tracking trajectories as our pseudo-supervision.\n\n## About Weakness 3, ‘Performance on static scene dataset’\nThank you for your suggestions. We agree with your words that, 'although it is not necessarily expected to outperform methods specifically designed for static scenes, such a comparison would serve as a useful reference and help assess the method’s generalizability'. We conducted additional experiments on the Tanks and Temples dataset (Table 1 below). Due to the character limit, for other baselines, please refer to Table 2 in CF‑3DGS. Compared with CF‑3DGS (SOTA static-scene method that uses GT intrinsics), although our approach does not surpass the SOTA methods specialized for static scenes with extra supervision, such low errors can also support our generalizability.\n\n\n### Table 1: Camera Pose Evaluation on static-scene Tanks and Temples dataset (ATE$\\downarrow$/RPT trans$\\downarrow$/RPE rot$\\downarrow$)\n| Scenes | Ballroom | Barn | Church | Family |\n|--------|----------|------|--------|--------|\n| Ours   |  0.005/0.003/0.526  | 0.0188/0.006/0.915 |  0.037/0.003/0.301  |  0.021/0.002/0.440  |\n\n| Scenes  | Francis | Horse | Ignatius | Museum |\n|---------|---------|-------|----------|--------|\n| Ours | 0.063/0.010/0.883  |  0.012/0.007/0.823  |  0.028/0.008/0.628  |  0.008/0.020/0.551  |\n\nWe summarize below the reasons why meaningful comparisons with existing works on the 7‑Scenes and ScanNet datasets are not feasible:\n1) Access to the ScanNet dataset requires an application process and approval, which takes additional time. \n2) After recheck, we found that the existing 7‑Scenes evaluations (e.g., DUSt3R / Fast3R) are based on unordered images and pairwise pose metrics. In contrast, our method fundamentally relies on long, continuous video sequences and temporal consistency. Since the official protocol merges multiple short sequences into unordered inputs and does not provide a single long sequence, it is not possible to conduct a fair or meaningful comparison on this dataset.\n\n\n## About Weakness 4, ‘include the number of frames per scene when reporting runtime comparisons in Table 9 and other related tables’\n\nThank you for your helpful suggestions. Since the experimental section of our paper has already demonstrated the high efficiency property of our method, and due to the space limits of rebuttal, we only present here the runtime of our method on different datasets along with the corresponding number of frames, as shown in the following Tables 2, 3, and 4. We plan to incorporate these additional results into the runtime tables in the camera-ready version to make it more complete.\n\n### Table 2: Quantitative Runtime Results on DAVIS with Corresponding Frame Count\n| Metric  | Camel | Bear | Breakdance-flare | Car-roundabout | Car-shadow | Car-turn | Cows | Dog | Dog-agility |\n|---------|-------|------|------------------|----------------|------------|----------|------|-----|-------------|\n| Cam (min) | 1.57  | 3.15 | 1.73             | 4.97           | 0.93       | 2.97     | 2.85 | 1.60| 0.67        |\n| Frames  | 90    | 82   | 71               | 75             | 40         | 80       | 104  | 60  | 25          |\n\n| Metric  | Goat | Hike | Horsejump-high | Lucia | Motorbike | Parkour | Rollerblade | Tennis | Train |\n|---------|------|------|----------------|-------|-----------|---------|-------------|--------|-------|\n| Cam (min) | 2.10 | 4.20 | 1.97           | 1.97  | 2.08      | 9.07    | 1.25        | 3.47   | 1.90  |\n| Frames  | 90   | 80   | 50             | 70    | 43        | 100     | 35          | 70     | 80    |\n\n### Table 3: Quantitative Runtime Results on iPhone with Corresponding Frame Count\n| Metric  | Apple | Paper-windmill | Space-out | Backpack | Block | Creeper | Handwavy |\n|---------|-------|----------------|-----------|----------|-------|---------|----------|\n| Cam (min) | 33    | 15             | 23        | 7        | 27    | 27      | 15       |\n| Frames  | 475   | 277            | 429       | 180      | 447   | 360     | 303      |\n\n| Metric  | Haru-sit | Mochi-high-five | Pillow | Spin | Sriracha-tree | Teddy | Wheel |\n|---------|----------|-----------------|--------|------|---------------|-------|-------|\n| Cam (min) | 10       | 6               | 21     | 30   | 15            | 31    | 27    |\n| Frames  | 200      | 180             | 330    | 426  | 302           | 472   | 437   |\n\n### Table 4: Quantitative Runtime Results on NeRF-DS with Corresponding Frame Count\n| Metric  | Bell | As | Basin | Plate | Press | Cup | Sieve |\n|---------|------|----|-------|-------|-------|-----|-------|\n| Cam (h) | 1.05 | 0.95 | 0.75 | 0.53 | 0.68  | 1.02 | 0.78  |\n| Frames  | 881  | 846 | 668   | 424   | 487   | 807  | 584   |\n\n\n## About Question 1, ‘Do other point trackers work as well?’\nThank you for your question. As stated in Lines 141–143 of the main paper, although our proposed patch-wise tracking filters do an iterative process until there are $B$ tracked points in each frame, such filters do $\\underline{not}$ conduct any $\\underline{repeated}$ refinement or optimization of point tracking on the $\\underline{same}$ tracking trajectory. This is the main reason why our runtime grows almost linearly with frame count, as shown in Figure 4 of the main paper. Thus, the pre-trained PT model on which our proposed filters are based must be a **feed-forward prediction model, outputting point locations and visibility**; we will clarify this further in the camera-ready version.\n\nIn Table 8 of the main paper, we show ablation experiments on 2 different popular pre-trained PT models, CoTracker and CoTracker3. The results show that our proposed method is nearly independent of building on the particular PT models, as long as these requirements are met.\n\nBesides, Table 5 below summarizes all baseline PT methods reported in the CoTracker and CoTracker3 papers. CoTracker and CoTracker3 are the only two methods satisfying our requirements. Thus, other PT models cannot be incorporated in our proposed method due to architectural incompatibility.\n\n### Table 5: Properties of All Baseline Methods Reported in CoTracker and CoTracker3.\n| Method                      | Type                          | Output Format                      |\n|-----------------------------|-------------------------------|------------------------------------|\n| TAP-Net            | Feed-forward                  | Point locations only               |\n| OmniMotion            | Optimization-based            | Point locations only               |\n| PIPs               | Iterative (sliding window)    | Point locations only               |\n| MFT               | Feed-forward                  | Point locations only               |\n| PIPs++           | Iterative refinement          | Point locations only               |\n| TAPIR            | Recurrent                     | Point locations + Visibility   |\n| **CoTracker**         | **Feed-forward**              | **Point locations + Visibility**   |\n| TAPTR                | Recurrent/transformer         | Point locations + Visibility       |\n| LocoTrack           | Recurrent                     | Point locations + Visibility       |\n| **CoTracker3**        | **Feed-forward**              | **Point locations + Visibility**   |\n| BootsTAPIR         | Recurrent                     | Point locations + Visibility       |\n\nWe are also happy to discuss any other proper pre-trained PT models to explore potential improvements on our current proposed method. Please feel free to let us know."}]

Author rebuttal:

Meta-review: This paper introduces an RGB-only supervised method for camera parameter estimation in dynamic scenes, combining patch-wise tracking filters, outlier-aware optimization, and a two-stage strategy. Reviewers praised the clear methodology, thorough ablations, and strong runtime and accuracy across ~70 videos, with some noting novelty is moderate and presentation could improve. The rebuttal addressed concerns by clarifying claims, adding sensitivity and static-scene experiments, and committing to improved figures and explanations. Overall, the work is solid, practical, and broadly relevant. I recommend acceptance.
Decision: Accept (spotlight)