LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation models
Authors: Ziqi Lu
Heng Yang
Danfei Xu
Boyi Li
Boris Ivanovic
Marco Pavone
Yue Wang
Venue: ICLR 2025 Spotlight (2025)
Keywords: 3D foundation model
model specialization
robust optimization
low rank adaptation
self-supervised learning
Primary area: foundation or frontier models, including LLMs
TL;DR: An efficient test-time self-calibration pipeline to specialize a 3D foundation model to a target scene in 5 min on a single GPU.

Abstract: Emerging 3D geometric foundation models, such as DUSt3R, offer a promising approach for in-the-wild 3D vision tasks.
However, due to the high-dimensional nature of the problem space and scarcity of high-quality 3D data,
these pre-trained models still struggle to generalize to many challenging circumstances,
such as limited view overlap or low lighting.
To address this, we propose LoRA3D, an efficient self-calibration pipeline to *specialize* the pre-trained models to target scenes using their own multi-view predictions.
Taking sparse RGB images as input, we leverage robust optimization techniques to refine multi-view predictions and align them into a global coordinate frame.
In particular, we incorporate prediction confidence into the geometric optimization process, 
automatically re-weighting the confidence to better reflect point estimation accuracy. 
We use the calibrated confidence to generate high-quality pseudo labels for the calibrating views and fine-tune the models using low-rank adaptation (LoRA) on the pseudo-labeled data.
Our method does not require any external priors or manual labels. It completes the self-calibration process on a **single standard GPU within just 5 minutes**.
Each low-rank adapter requires only **18MB** of storage. 
We evaluated our method on **more than 160 scenes** from the Replica, TUM and Waymo Open datasets,
achieving up to **88\% performance improvement** on 3D reconstruction, multi-view pose estimation and novel-view rendering.
For more details, please visit our project page at https://520xyxyzq.github.io/lora3d/.

Published as a conference paper at ICLR 2025
LORA3D: LOW-RANK SELF-CALIBRATION OF 3D GEOMETRIC FOUNDATION MODELS
Ziqi Lu Heng Yang! Danfei Xu Boyi Li Boris Ivanovic', Marco Pavone
Yue Wangl.
NVIDIA Research, Massachusetts Institute of Technology. HHarvard University,
+Georgia Institute Technology. UUiverssiyy of California, Berkeley, Stanford University,
UUniversity of Southem Califoriia
z siqi lu@mit. {hengy, danfeix, boyil, pavone@stanform edu, yue. que.weusc. edu
ABSTRACT
Emerging 3D) geometric foundation models, such as DUS13R (Wang et al. 2024), offer a promising approach for in-the-wild 3D vision tasks. However, due the high-dimensional nature of the problem space and scarcity of high-quality 3D data, these pre-trained models still struggle to generalize many challenging cir- cumstances, such as limited view overlap or low lighting. To address this, we pro- pose LoRA3D, an efficient self-calibration pipeline to specialize the pre-trained models to target scenes using their own multi-view predictions. Taking sparse RGB images as input, we leverage robust optimization techniques to refine multi- view predictions and align them into global coordinate frame. In particular, incorporate prediction confidence into the geometric optimization process, auto- matically re-weighting the confidence better reflect point estimation accuracy. We use the calibrated confidence to generate high-quality pseudo labels for the calibrating views and low-rank adaptation (LoRA) fine-tune the models on the pseudo-labeled data. Our method does not require any external priors or man- ual labels. It completes the self-calibration process on single standard GPU within just 5 minutes. Each low-rank adapter requires only 18MB of storage. We evaluated our method on more than 160 scenes from the Replica, TUM and Waymo Open datasets, achieving up to 88 % performance improvement on 3D reconstruction, multi-view pose estimation and novel-view rendering. For more details, please visit our project page
INTRODUCTION
Pre-trained DUS13
Pre-trained DUS13R
+
+
3D reconstruction
Novel-view synthesis
Sparse RGB imuges
Sparse images
Self-Calibrate
Self-Caliben Calibrate
Calibrated DUSI3R
Calibrated DUSt3R
3D reconstruction
Novel-view synthesis
Figure 1: Given sparse RGB images, our self-calibration pipeline efficiently specializes a pre-trained 3D foundation model to target scene improve its performance for variety of 3D vision tasks.

---

Published as a conference paper at ICLR 2025
Recently, many 3D geometric foundation models have emerged potential solution for in-the- wild 3D computer vision tasks such as 3D reconstruction. camera pose estimation and novel view rendering (Wang al. 2024 Barroso-Laguna et al. 2024 Leroy et al. 2024 Hong et al. 2023). These models, typically enabled by large scale l'ransformer pre-training, can quickly establish cross- view correspondences and directly regress 3D scene geometry from sparse RGB images. They generalize to broad range of data and exhibit strong zero-shot performance on novel tasks.
However, the performance of these pre-trained models can falter under challenging cireumstances. For instance, as highlighted in the upper left sub-figure in Fig. DUSt3R's pairwise reconstruction aceuracy significantly degrades under low visual overlaps, where certain regions are observed from only single viewpoint. This decline is rooted in the inherent complexity of 3D geometric inference task, which requires much larger-scale data to fully represent the distribution of real-world data. Unfortunately, the difficulty of annotating in-the-wild 3D data has led to the shortage of high-quality training datasets, limiting the performance of the pre-trained models.
To mitigate the problem, propose efficient self-calibration pipeline (Fig. 2e).2.... taking only sparse RGB images to specialize pre-trained 3D foundation models to the target scene. Our method requires no manual labeling, camera calibration, external priors. We only leverage the multi-view consis- tency 3D point positions to refine and select pre-trained models' predictions for pseudo labeling. To ensure the pseudo label accuracy, develop robust global optimization method to align and refine multi-view predictions while calibrating the prediction confidence. The calibrated confidence strongly correlates with pseudo-label accuracy, allowing to select high-confidence data for LoRA fine-tuning of the pre-trained model. Our method is tested on 161 test scenes for a variety of 3D vision tasks. It is able finish the self-calibration process within 5 minutes a single GPU and deliver performance improvements of up to 88%. The major contributions of our work include (1) the ceff-calibrttonn pipeline, (2) the robust global optimization method, and (3) the efficient LoRA fine-tuning strategy for DUSt3R self-calibration.
(b) Robust global oplimization
Predict
DUS:B
Spurse calib. images
Merge
Calibration image pairs
LoRA adapter
Point maps Confidence maps
Refined Point maps Confidence maps
Calibrated
(d)
(c) Confidence- based
LoRA
Fine tuning
Pseudo labeling
Pseudo-labeled data
Figure 2: Overview of our self-calibration pipeline. (a) Predict: We pair sparse input RGB images and use the pre-trained 3D foundation model to predict per-pair point maps and confidence maps. (b) Robust Global Optimization: We apply robust optimization techniques to concurrently refine multi-view point predictions and calibrate prediction confidence.
(c) Confidence-Baan Based Pseudo- Labeling: Refined point maps with high calibrated confidence are used to generate pseudo-labels on calibration views. (d) LoRA Fine-Tuning: Using the pscudo-labeled data, efficiently fine- tune the pre-trained model with LoRA. While the figure illustrates our method using DUSt3R, our approach generalizes other 3D foundation models.
RELATED WORK
2.1 FOUNDATION MODEL SPECIALIZATION
Foundation model specialization through fine-tuning or adaptation has become the standard ap- proach to customizing pre-traine pre-trained foundation models for specific tasks domains. Various meth- ods have been developed for the specialization of large language models (Brown, 2020 2020) Gururangan et al. 2020), vision-language models (Liu al. 2024b a), and vision foundation models (Hu al.
2

---

Published as a conference paper at ICLR 2025
2023 Yue al. 2024 These approaches typically employ parameter-efficiee adaptation tech- niques (Hu et al. 2021, Dettmers et al.] 2024 He et al. 2022) to adapt the pre-trained models in either supervised unsupervised fashion. However, few works have explored the specialization of 3D geometric foundation models. MASr3R (eeroy et al. 2024) and Spanner31) [Wang & Agapito (2024) fine-tuned DUST3R (Wang al., 2024 to re-purpose for image matching and incremental reconstruction respectively. [Jiang et al. (2024) applied self-training to scale up large reconstruction models (Hong al.) 2023) with real-world images. However, most of these methods still rely on vast
amounts of labeled data. In contrast, our method uses only sparse RGB images for self-calibration and requires no ground truth labels.
2.2 SELF-SUPERVISED GEOMETRIC PERCEPTION
Self-supervised learning has been successfully applied to range of geometric perception tasks, in- cluding monocular depth prediction (Godard et al. 2019), optical flow prediction (Liu et al. 2019).
camera pose estimation (Yang al.) 2021). and cantly enhancing the performance of pre-trained geometrie models. Among these, Yang et al. (202T) particularly relevant our approach, it utilizes robust optimization techniques generate ge- ometric pseudo-labels for model fine-tuning. However, this method and most others are tailored adapt smaller-scale pre-trained models for specific tasks. In this work, we extend the pseudo- labeling strategy for self-supervised learning to 3D foundation models. Leveraging the versatility of these models, we can improve their performance on various 3D vision tasks.
(Zhou et 2017). signifi-
PRELIMINARIES
Our pipeline is primarily tested on DUSt3R (Wang al. 2024). Below, we provide key details about DUSt3R give readers the necessary context for understanding our contributions.
3.1 DUST3R
As shown in Fig, 2 DUSt3R takes an RGB image pair (I,, I5) as input and directly regresses the pixel-wise point maps and confidence maps:
(X Ch)) (X Ci+) DUSBRR(T', P')
Here, X X E "xxxxx x3 are the point maps for view i and view j. both expressed the camera coordinate frame of view i, and regressed up unknown scale. Their corresponding confidence maps are denoted as C Ch E H
3.2 RECOVERING CAMERA PARAMETERS
The camera intrinsics can be recovered from the predicted point maps in Eq. [ Assuming a pinhole camera model with square pixels and principal points at image centers, the camera i's focal length fi can be estimated by solving the following optimization problem using Weiszfeld algorithm:
HW
S = arg min
Σ P
(16.
S(Xis X A2 p.o>
(2)
f
p=1
where (uh, x/)) (up W/2, up
2/2) represents the re-centered image coordinates for pixel p.
The relative camera poses estimated by comparing the predictions for image pair (T,, I, and (I,, I,). With point maps X and X can apply Procrustes alignment (Luo & Hancock 1999) to estimate the relative pose Trj E SE(3) from camera to j and the point map scale o,
(Tiji ord)" = arg min
Σ Epbtg
C PiisTis x
(3)
where we omit the homogenization of point maps for brevity.
See App. A.1 for details the training loss of DUS:3
3

---

Published as a conference paper at ICLR 2025
3.3 MULTI-VIES POINT MAP ALIGNMENT
Given multiple images {/1 I2,.... .......Iy I captured in a 3D scene, the multi-view DUSt3R-predicter point maps are aligned form global point cloud. Different from bundle adjustment, this align- ment is formalized as an DD-3I-projection G(V, E), in which the vertices represent the images and the edges & represent all image pairs with visual overlaps.
optimization problem over connectivity graph
To initialize the optimization parameters, the highest-confidenee spanning tree is extracted from the graph. Anchoring the most confident image pair at the origin, the initial estimates of focal lengths, point map scales and relative poses, as derived from Eq. 23 are propagated along the tree edges to all N views, yielding initial focal lengths {Sili = I,..., N), point maps (x" E R x$) and image-pair scales (a E [R] and poses (T(3.) E SE(3)). all expressed in unified global coordinate frame.
These initial estimates are further refined by minimizing the 3D-3D projection error between the global point maps X and the transformed predicted point maps:
HW
TT) = arg min x.T (NJ)EE vE(i.) p=1
ΣE
C
x
x
(6d)) x
(4)
Note that the global point maps X₂ can be further re- re-parameterized via depth back- pnok-prooection:
x = T.K, DarKK"Dppupp... Up, 1) TT, Dp
fo (ulp o/p 1)
(5)
where K, and T. represent the intrinsics and extrinsics for view and Dp is the depth value for pixel p. The optimization problem can therefore be reformulated as:
(T,o,
arg min T. (5:5)
ΣΣΣ
[T.
De (ulp Jo
(u.p u 1)T (007700 x
(6)
Here, the per-image-pair poses ((5)) and per-image poses T, represent the same transformations but parameterized separately to allow for additional optimization flexibility. The optimization is
solved by few hundred steps of standard gradient descent. To avoid trivial optimum of g(.d) = 0, (1,))) = 1 enforced during the optimization.
METHODOLOGY
We aim to adapt 3D geometric foundation model, such DUSt3R (Wang et al. 2024). to target scene using a sparse set of uncalibrated RGB images {T,, Je,..., IN]. The goal to enhance the pre-trained model's performance on test images {IN+1, IN+2..... IN+M] from the same scene. Our approach generates compact LoRA adapters, which integrate with the pre-trained model to produce scene-calibrated model.
4.1 SELF-CALIBRATION PIPELINE
Fig. 2 shows our self-calibration pipeline. We start by using the pre-trained DUSt3R, as in Eq. [ to predict point and confidence maps for all calibration image pairs. In challenging conditions, such as under limited camera view overlap, DUSt3R's predictions may include errors and outliers, and the prediction confidence may not precisely reflect the prediction accuracy (See Fig. for an example of overconfident prediction). For this reason, directly relying on predicted confidence for pseudo label selection may hurt the model performance (see Sec. (a.b))
However, each 3D point in the scene is co-observed by many camera view pairs. We could leverage accurate DUSt3R predictions from well-conditioned, e.g. high-visual-overle view pairs to refine and identify inaccurate point map predictions. We therefore develop a robust multi-view point map alignment method (Sec. (.2) to (1) optimize the point map and (2) calibrate the prediction confi- dence. We then use the refined point maps and calibrated confidence pseudo-label the calibration images [ (Sec. [43). after which we fine-tune the pre- pre-trained DUSt3R model using LoRA on the pseudo-labeles data (Sec. 4.
4

---

Published as a conference paper at ICLR 2025
4.2 ROBUST MULTI- VIEW POINT MAP ALIGNMENT WITH CONFIDENCE CALIBRATION
We develop a robust multi-view point map alignment method by incorporating the prediction con- fidence into the global optimization in Eq. 6 Specifically, we re-parameterize the confidence C in Eq. as an optimizable weight term up to automatically tune each point prediction 's con- tribution the optimization. While the predicted confidence can be imprecise in challenging cases, it still remains informative for prediction accuracy. Thus we intend to introduce regularization term that encourages the weights to remain close to the predicted confidence, and also avoid trivial solutions.
2.5
After
Prediction error
1.0
0.5
0.0
1
X
Confidence
(a) Image pair
(b) Pred. confidence
(c) Error map
(d) Pred. error v.s. Confidence
Figure 3: Pre-trained DUSt3R's (b) prediction confidence and (c) error map (a) example image pair: In cases of limited visual overlap, DUSt3R may produce overconfident predictions (*)). Our robust multi-view alignment method effectively reduces this overconfidence, maintaining high confidence for accurate predictions (+, x) and low confidence for outlier predictions (+).
We found our objective to be surprisingly aligned with Geman-McClun robust M-estimator (Geman al. 1992), which essentially uses a regularization term (an outlier process in robust optimization terminology) to encourage weights to be close to unity in least-squares optimizations. Inspired by this, we designed our regularization term to follow similar structure. The optimization Eq. therefore reformulated as:
arg min T. T.J.D.W (i.5)
ΣΣΣ
w |lep"11 + ulVuss up
where ex (u, th,, /) - - ((.))pp x represents the pixel-wise residual error and a constant sapee--prrmeter to control the regularization strength.
Rather than updating the weights in the joint optimization loss Eq. 7 via gradient back-propagation, we draw inspiration from the iterative re-weighted least squares approach (Rao Kreutz-Delgado 1999) for robust M-estimation, to derive closed-form weight update rule for fast confidence weighting:
" C /11 |ee 11))²²
(8)
(a) Calibrated confidence (b) Point estimation emror (c) Pixels to pseudo-label
(d) Pseudo labels
Figure 4: Pseudo-labeling with (a) calibrated confidence, which is good measure of the (b) point estimation accuracy. We select higgh-calibratee--oofddeedd point predictions as pseudo labels (d) for DUSt3R finetuning.
5

---

Published as a conference paper at ICLR 2025
With this update rule, still solve the original optimization problem (Eq.6) while periodically applying the weight updates. As demonstrated in Appendix A.2. this is equivalent to solving the joint optimization.
The weight update rule can understood as follows: point predictions with lower residual errors, meaning those that dence similar to the predicted value. In contrast, point predictions that are inconsistent across views will have their confidence significantly reduced. This method effectively minimizes confidence for overly confident predictions, as illustrated in Fig. [ ensuring that confidence becomes more closely correlated with point estimation accuracy and provides better guidance for global optimization and pseudo-labeling
consistent with predictions from other image pairs, will maintain confi-
4.3 PSEUDO LABELING WITH CALIBRATED CONFIDENCE
use the calibrated confidence and optimized point maps for confidence-based pseudo-labeling. To compute pseudo labels for the calibration image pairs, need transform the global optimiza- tion results from Eq. to local image-pail coordinate frame. Following Eq. [5] we back-project the optimized depth maps Dp 3D and transform the points to the image-pair coordinate frame. We then threshold the point estimations with a confidence cutoff Wentol and retain the high-confidence pseudo labels. The pseudo labeling rule can be summarized as:
D (1 (// 1)",
X = T
where E {plu > Wemtot))
We experimentally found that setting Weutoll = 1.5 works effectively for most test scenes.
Note that our method is naturally robust to dynamic elements in the scene (See Tab. 3 This is because the dynamic points break the multi-view consistency assumption and will be filtered out by pseudo labeling with calibrated confidence.
4.4 FINE-TUNING WITH LORA
On the and the same pre-training loss (as Eq. [2). LoRA freezes the pretrained model weights and injects trainable rank decomposition matrices into layers of Transformer architecture, greatly reducing the
data, fine-tune the pre-trained DUSt3R with LoRA (Hu al.) 2021
number of trainable parameters. This (I) improves the runtime- and memory-efficiency of self-
calibration and (2) reduces the catastrophic forgetting of the pre-training data.
Avg. Pred. Error [m]
Pretrrined
0.14
igkv. r-4) (alv, r=161
lokv, r-64) + iq, Output Head
0.12
LoRA (ig.. r-8)
LoRA (3iv, rw32]
LoRA lqv. r-16)
Rull
0.10
0.08
10 Number of Trainable Parameters [Million]
50
Figure 5: What is the best DUSt3R fine-tuning strategy? We plot the mean prediction errors test images against the number of trainable parameters for various fine-tuning options on an example test scene (Replica "officel")). We found adapting all attention weights with rank-16 LoRA (i.e. *) achieves the best trade-off between performance and efficiency on most test scenes.
To find the optimal DUSt3R fine-tuning strategy, we conducted extensive experiments to compare different fine-tuning options across multiple test scenes. Please see Fig. 5 for an example, where we plot the test errors (defined in Sec. 5) against trainable parameter counts for different LoRA and direct fine-tuning strategies. We found adapting all attention weights with rank-16 16 LoRA often leads to the best trade-off between performance and efficiency. It reduces the number of trainable parameters by more than 99% and has on-par performance with directly fine-tuning the attention or all weights.
Using rank- 16 LoRA, fine-tuning 10 calibration images converges in under 3.5 minutes with batch size of 2. Peak GPU memory usage during fine-tuning stays under 20GB, enabling the process run single standard GPU, And each LoRA adapter require less than 18MB of disk storage.

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
We evaluated our method on 161 test scenes for the tasks of 3D reconstruction, multi-view camera parameter estimation and novel view rendering.
Datasets We tested our method on all available test scenes from the Replica (Straub et al., 2019) and Waymo Open Dataset (Sun et al.) 2020), as well as on three test scenes from the TUM RGBD dataset (Schubert et al.) 2018) that are most frequently tested in literature. This amounts to total of 161 test scenes, all of which are distinct from the DUSt3R pre-training scenes.
The Replica dataset comprises eight indoor scenes, each containing 2000 RGB-D images rendered by |Sucar et al. (202T). For each scene, the first 1000 RGB images serve as the calibration split and the remaining the test split. The depth images are not utilized during calibaaii::: they are used solely to compute the ground truth point maps. We randomly sample 10 images from the calibration split the calibration images.
The Waymo Open Dataset has in total 150 test data segments. In each segment, only forward-looking camera images are adopted, where the first 100 form the calibration split and the remaining ~100 images belong to the test split. We sample 10 images from the calibration split for self-calibration.
Please refer App. A.4 for details about the data splits and tasks for the TUM RGBD dataset.
Tasks On the Replica dataset, we evaluate our method for painwise and multi-view reconstruction tasks. For pairwise reconstruction, we sample 100 image pairs with visual overlaps from the test split as test images. For multi-view reconstruction, we sample 10 views from the test split.
On the Waymo dataset, evaluate our method for the tasks of multi-view camera parameter esti- mation and novel view rendering. For novel view rendering, we use InstantSplat (Fan et al. 2024), which adopts DUSt3R DUSt3R-predicted point cloud and camera parameters as initialization, to train 3D Gaussian Splatting (3DGS) models and render novel-view images. From the test split, we select every 10th images (i.e. Oth, 10th, 20th, ...) for camera pose estimation evaluation and InstantSplat training. Images at indices 5, 15, 25, ... are used as novel views to evaluate the InstantSplat renders. Note that although our method robust to dynamic environments, InstantSplat relies on the static world assumption to train 3DGS. We therefore selected segments 10084, 10649, and 10802 that are mostly static from the first 10 test segments for the novel view rendering evaluation.
Baselines The two most important comparison models for our self-calibrated DUS(3R (DUSI3R- Self-Calib) are the pre-trained DUSt3 DDSS3R-PPetreee and the fine-tuned DUSt3R on ground truth point maps of calibration image pairs (DUSBR--GT-FT) The ground-truth fine-tuned model is considered as the upper limit of ours, serving as an oracle model.
Both methods are evaluated in most tests, with the exception of the TUM dataset, where no ground truth depth is available. In this case, we use noisy depth measurements for fine-tuning, referred to as On the Waymo dataset, we use the high-quality Lidar point clouds for ground- truth-based fine-tuning. For a fair comparison, the training hyperparameters for ground-truth and depth-based fine-tuning are kept consistent with those used in our method.
For multi-view stereo reconstruction, we also use COLMAP (Schonbbbbrrr & Frahm 2016 Schönberger et al. 2016). FlowMap (Smith al. 2024) and MAS13R (Duisterhof al., 2024) as baselines, all of which perform dense reconstructions with un-calibrated images.
COLMAP (MVS) is standard SfM and MVS pipeline for which we adopt the default setups.
FlowMap is a differentiable SfM model for RGB videos. It relies on optical flow and point tracking algorithms to bootstrap its scene parameter optimization process.
MASt3R re- re-purposes DUSt3R for image matching. Beyond 3D point regression, it establishes accu- rate cross-view correspondences, and leverages both 3D-3D and 2D-3D correspondences for global point map alignment. We adopt the pre-trained MASt3R with default
For fair comparison in multi-view reconstruction, we use the same global optimization method for DUSt3R and its variants (as detailed in Sec. 3.3). and retain all points for evaluation without applying confidence- based filtering to DUSt3R and MASt3R reconstructions.
Unless specified otherwise, all random sampling use seed=0 for re-producibility.
7

---

Published as a conference paper at ICLR 2025
For multi-view camera parameter estimation, use COLMAP, RelPose++ (Lnn al. 2024
PoseDiffusion (Wang et al., 2023). RayDiffusion (Zhang et al. 2024). FlowMap (Smith et al.
2024 and MASB3 (Duisterhof et al. 2024) the additional baselines.
RelPose++ + uses a pairwise scoring network and a multi-view reasoning transformer to predict multi- view camera poses.
PoseDiffusion develops a diffusion- -based bundle adjustment method to estimate multi- -iew camera parameters. A geometry guidedd sampling (GGS) scheme is applied to enforce epipolar constraints across views. We adopt the GGS-enabled PoseDiffusion with the CO3Dv2 checkpoint.
RayDiffusion re-parameterize cameras as rays and applies ray diffuser network denoise camera rays and recover camera parameters.
All three methods above pre-trained on domain-speciff data without further fine-tuning.
For novel view rendering, we use the pre-trained, self-calibrated and finetuned DUSt3R for In- stantSplat's 3DGS initialization, and the different variants for InstantSplat are refered In- stantSplat Faan et al., 2024), InstantSplat-Self-Cahhh InstantSplat-GTT-r
Evaluation Metries For DUS13R pairwise reconstruction, we the average point prediction error as the evaluation metric. This is the average Euclidean distance calculated between the predicted and ground truth point maps within local image-pair coordinate frames, with predicted maps normalized and realed to align with ground truth.
We assess multi-view stereo reconstructions based on accuracy and completeness relative to the ground truth Replica mesh models (Straub et al., 2019). Accuracy measures the average distance of reconstructed points to their nearest mesh points, while completeness measures the average distance of mesh points to their nearest reconstructed points. Following Zhu et al. (2022). we exclude mesh parts invisible to the test images.
The multi-view camera parameter estimations are evaluated with the absolute trajectory error (ATE) and the average focal length estimation error (AFE).
The quality of novel view renders is assessed with Peak Signal-to-Non Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Putch Similarity (LPIPS).
5.1 RESULTS
For the Replica dataset, we report pairwise and multi-view reconstruction results in Tab. and Tab. Compared to the pre-trained DUS13R, reconstructs models that are up to 61% % more accurate and 41% % more complete. As Fig. 6 shows, our approach is particularly effective at reducing outlier point predictions, thanks the multi-view consistent pseudo labels The remaining performance gap compared to the fine-tuned DUSt3R is attributed to differences in data size and label accuracy.
method reduces point prediction errors by up to 38 and
Since cameras in Replica are mostly facing inward, it is casier for COLMAP and MASt3R to estab- lish accurate rross-view feature matches, resulting better accuracy and completeness on certain test scenes, even surpassing the fine-tuned DUSt3R. On the other hand, FlowMap struggles due to the discontinuous calibration images, which disrupt the optical flow and point tracking relies on.
Table 1:
Quantitative evaluation of pairwise reconstructions on the Replica dataset. We report
the average point prediction errors (cm) for direct DUSt3R predictions.
Methods
office0
officel
office2
office3
office4
room0
room1
room2
DUSE3R- Pretrriin
14.29
11.02
14.03
15.44
14.96
13.11
27.99
16.82
DUSt3R-Self-Cait
8.84
9.38
11.05
14.41
13.92
13.02
19.88
13.65
GT-F
7.12
7.95
10.55
12.88
12.29
9.27
17.40
12.58
The multi-view camera parameter estimation results on the Waymo dataset are presented Tab. 3
Compared to the pre-trained DUSt3R, our method reduces camera trajectory estimation errors by up to 88% and focal length estimation errors by up to 79%. Out of 1,50 total test scenes (detailed in
Please check out A.9 for qualitative comparison of the reconstructions with the ground truth mesh model.
8

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative evaluation of multi-view reconstructions on the Replica dataset. We report the accuracy (Ace. [em] 1) and completeness (Comp. [em] 4) of 3D reconstructions against the ground truth meshes. Please refer to App.Tah, for the remaining results omitted due to space limit.
ofliceO
oflicel
oflice?
oflice3
ofliced
Methods
Acc.
Comp.
Acc.
Comp.
Acc.
Comp.
Acc.
Comp.
Acc.
Comp.
DUSI3R-Pretre Pretrain
5.22
6.57
12.97
15.89
DUSE3R- Self-Callb
4.43
6.08
3.56
5.48
4.75
6.89
6.60
11.00
7.81
12.22
DUSER-GT-FT
3.51
5.29
3.26
5.53
3993
6.72
4.02
7.42
5.53
11.25
COLMAP (dense)
2.61
4.87
194.16
5.51
162.53
6.42
FlowMap
51.78
152.05
142.81
107.17
24.16
189.86
19.58
248.16
15.34
153.64
MASL3
4.69
6.05
3.92
4.87
4.09
7.39
7.17
9.42
4.87
11.52
Before self-calibration (d) After self-calibration
(a) Before self-calibratton (b) After self-calibration
Figure 6: Qualitative results the Replica (a.b) and TUM (c,d) datasets. After DUSt3R self- calibration, we observe much fewer outlier points in the reconstruction the Replica scene "of- fice0". On the TUM scene "fr2 xyz", the green and red frustums represent the estimated and ground truth cameras respectively. The camera pose estimates are made more accurate by self-calibration.
App-A.1 A.11). 1 our approach successfully improves camera parameter estimation results on 116 scenes. Most failures occur when the test vehicle remains mostly static for the test images (e.g. segment- 10488). This degenerate case expect the model to predict zero relative pose across views and fails to distinguish different methods.
By comparison, COLMAP fails, and MAS(3R shows degraded performance on Waymo due to the presence of dynamic objects and the larger baselines between forward-facing cameras, which make feature matching more difficult. FlowMap still struggles due the abrupt visual changes across views. RelPose++, PoseDiffusion and RayDiffusion, without domain- or scene-specific training, fails to provide accurate estimates the
data.
The novel view rendering results on Waymo are presented in Tab. 4 and Fig. 7 Our method effec- tively reduces floating artifacts in the optimized 3DGS, resulting in quality improvements of up to 0.97 dB in PSNR, 0.09 in SSIM, and 0.04 in LPIPS.
The quantitative results the TUM dataset are presented and analyzed in App.
A.4
Table Quantitative evaluation of camera parameter estimates on Waymo Open Dataset. We report the absolute trajectory error (ATE (m) 1) and average focal error (AFE (%) J) for test camera views. Please see App. I. for full evaluation results on 150 test scenes.
Methods
segment-10084
segment-10144
segment-
1964e
segment-
10802
segment-
10980
ATE
AFE
ATE
APE
ATE
AFE
ATE
AFB
ATE
AFE
DUSCiR-iettain
0.79
2.19
0.84
3,08
0.95
2.84
L.60
0.30
DUSL3R-SA Saaek-Sf--Caaa Calib
0.37
0.61
0.25
2.14
0.49
2,54
0.35
1.08
0.09
0.69
DUSFR-GGT-F
0.20
0.17
0.17
1.34
0.29
1.73
COLMAP
Fail
Fail
Fail
Pail
Fail
Fail
Fail
Flowmup
0.31
3.97
66.62
1.80
36.44
13.74
21.16
0.44
65.17
0.66
PoseDiffusion
49.18
RayDiffusion
17.34
85.65
16.91
80.69
18.59
85.09
12.77
81.44
19.12
85.00
RelPose++
14.80
16.20
13.69
12.92
MASIER
2.85
11.87
1.35
24.92
0.65
20.53
1.26
24.75
1.61
6.59
Table 4: Quantitative evaluation of novel view renders on the Waymo open dataset
Segment-100 10084
Segment-106449
10649
Segment 10802
Methods
PSNR
SSIM +
LPIPS
PSNR
SSIM f
LPIPS J
PSNR
SSIM
LPIPS J
InstantSplat
21.45
0.67
0.33
22.45
0.72
0.30
25.94
0.79
0.24
InstantSplat-Sel--Call
22.42
0.76
0.29
22.81
0.77
0.27
26.36
0.81
0.22
Ststa

---

Published as a conference paper at ICLR 2025
(a) Before self-calibration (b) After self-calibration
(c) Before self-calibration (d) After self-calibration
Figure 7: Novel view renders by InstantSplat (Fan et al. calibration on Waymo Open Dataset.
2024 before and after DUS:3R self-
11.5
Avg. Pred. Err. [m]
pseudo-label label w/ uncalib. cont.
4.6
pseudo-label w/ uncalib. cont.
Pred. Err.
11.0
10.5
4.2
9.5
9.0
Ours (pseudo-label w/ calib. conf.)
4.0
Ours (pseudo-label w/ calib. conf.)
Confidence cutoff
Confidence cutoff
(a) Scene: Replica office0
(b) Scene: Waymo test segment 10084
Avg. Pred. Err. [cm
Pretrrinedd
Pretrained
Err.
6-
Avg.
2
10
25
Random seed
Number of calibration images
(c) Scene: Replica office0
(d) Scene: Waymo test segment- 10149
Figure 8: Ablation study: (a, b) Pseudo-labeling with un-calibrated confidence hurt the model per- formance. (c) Our method maintains consistent performance across varying random seeds used for calibration image sampling. (d) Our method's performance improves with more calibration images and saturates after around 10.
5.2 MAST3R SELF-CALIBRATION
Our pipeline is not limited to the specialization of DUSt3R. We show in A.5 applies to MASt3R.
th5 that the same idea
5.3 ABLATION STUDY
Un-calibrated confidence for pseudo labeling Directly using the prediction confidence for pseudo- labeling could harm the model performance. As Fig. 8 [a, b) shows, thresholding refined point pre- dictions with un-calibrated confidence leads to consistent under-performance of the self-calibrated model, regardless of the confidence cutoff value.
Varying random seed Our method works with casually captured RGB images and does not rely on carefully selected calibration images to succeed. As shown in Fig. g (c), our method maintains consistent performance across varying random seeds used for calibration image sampling.
The number calibration images As shown Fig. (d), using few calibration images (e.g.. fewer than 10) limits our method's performance due insufficient number of view pairs enforce multi-view consistency in global optimization and limited training data size. We also observe that the performance typically saturates after around 10 calibration images.
The size of calibration split& & Multi-scene concurrent self-calibration See
A.6 & A.7
for details.
CONCLUSION
Our celf-calibration pipeline specializes 3D geometric foundation models to target scenes in a highly time- and memory- -efficienf manner. It boosts pre-trained model performance by up 88% across diverse datasets and 3D vision tasks. However, in certain cases, the self-calibrated model still falls short of competing methods due to the inherent difficulty of 3D geometric inference.
10

---

Published as a conference paper at ICLR 2025
7 ETHIC STATEMENT
Our work utilizes publicly available datasets that adhere to strict ethical guidelines. These datasets ensure that personally identifiable information, such as human faces and vehicle license plates, is either blurred or anonymized to safeguard privacy. Our work does not engage with human subjects or introduce concerns regarding fairesss or potential misuse. We are fully committed to maintaining ethical integrity throughout the development and application of our methods.
8 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we have taken the following steps: (1) A detailed expla- nation of our method in Sec. along with implementation details provided in Appendix (2) All datasets used in our experiments are publicly accessible. (3) Comprehensive experimental results
for 161 test cases included in Sec. and further detailed in Appendix A.4 A.10 and A.11
REFERENCES
Axel Barroso-Laguna Sowmya Munukutla, Victor Adrian Prisacariu, and Eric Brachmann. Match- ing 2d images in 3d: Metric relative pose from metric correspondences. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4863, 2024.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Bardienus Duisterhof, Lojze Zust, Philippe Weinzaepfel, Vincent Leroy, Yohann Cabon, and Jerome Revaud. Mast3r-sfm: fully-integrated solution for unconstrained structure-from from-motion. arXiv preprint arXiv:2409. 010.4499999555 /9152, 2024.
Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang. Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds. arXiv preprint 2rXiv:2403.20300 2024.
Stuart Geman, Donald E McClure, and Donald Geman. A nonlinear filter for film restoration and other problems in image processing. CVGIP: Graphical models and image processing, 54(4): 281-289, 1992.
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self- supervised monocular depth estimation. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pp. 3828-3838, 2019.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004. 10964, 2020.
Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter-efficient fine-tuning for vision transformers. arXiv preprint arXiv:2203. 16329, 2022.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311 1..0233, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09683 2021.
Xinrong Hu, Xiaowei Xu, and Yiyu Shi. How to efficiently adapt large segmentation model (sam) to medical images. arXiv preprint arXiv:2306. 13731, 2023.
Hanwen Jiang, Qixing Huang, and Georgios Pavlakos. Real3d: Scaling up large reconstruction models with real-world images. arXiv preprint wxXv:22006.08479, 2024.

---

Published as a conference paper at ICLR 2025
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. arXiv preprint arXiv:2406..9976, 2024.
Amy Lin, Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose++: Recovering 6d poses from sparse-view observations. In 2024 International Conference on 3D Vision (3DV), pp- 106-115. IEEE, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu. Selflow: Self-supervised learning of optical flow. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 4571 -4780, 2019.
Bin Luo and Edwin R Hancock. Procrustes alignment with the em algorithm. In International Conference on Computer Analysis of Images and Patterns, pp. 623-631. Springer, 1999.
Bhaskar D Rao and Kenneth Kreutz-Delgado. An affine scaling methodology for best basis selec- tion. IEEE Transactions on signal processing, 47(1):187-200, 1999.
Johannes I. Schonberger and Jan-Michael Frahm. Stracture-from-mmttio revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104-4113, 2016.
Johannes L. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part III 14, pp. 501-518. Springer, 2016.
David Schubert, Thore Goll, Nikolaus Demmel, Vladyslav Usenko, Jörg Stückler, and Daniel Cre- mers. The tum vi benchmark for evaluating visual-inertial odometry. In 2018 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 1680. IEEE, 2018
Cameron Smith, David Charatan, Ayush Tewari, and Vincent Sitzmann. Flowmap: High-quality camera poses, intrinsics, and depth via gradient descent. arXiv preprint arXiv:2404. 2004::244.552222 15259, 2024.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint 19X6:1990600797,,, 2019.
Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. imap: Implicit mapping and po- sitioning in real-time. Proceedings of the IEEE/CVF international conference on computer vision, pp. 6229-6238, 2021.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEF/CVF conference on com- puter vision and pattern recognition, pp.
2020.
Sheng Wan, Tung- Yu Wu, Wing Wong, and Chen-Yi Lee. Confnet: predict with confidence. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 2921-2925. IEEE, 2018.
Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint arXiv:2408. 16061, 2024.
Jannyuan Wang, Christian Rupprecht, and David Novotny. Posediffusion: Solving pose estimation via diffusion aided bundle adjustment. In Proceedings the Computer Vision pp. 9773-9783, 2023.
the IEEE/CVF International Conference
12

---

Published as a conference paper at ICLR 2025
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 20697-207099 2024.
Heng Yang, Wei Dong, Luca Carlone, and Vladlen Koltun. Self-supervised geometric perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14350-14361, 2021.
Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, and Jan Eric Lenssen. Improving 2d feature representations by 3d-aware fine-tuning, arXiv preprint arXiv:2407.20229, 2024.
Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tul- siani. Cameras rays: Pose estimation via ray diffusion. arXiv preprint arXiv:2402.1488 2024.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1851-1858, 2017.
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Os- wald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12786- 2022.
A APPENDIX
A.I DUST3R TRAINING LOSS
DUS(3R adopts ID-regression-basce training objective, which computes the pixel-wise Euclidean distance between the predicted and ground truth point maps X " over pixels piu p C {1... H) x {1... W} where ground truth is available. For example, the regression loss for valid pixel E p on image from image pair (i,j) is computed as
(v p) = xp= -
1 v
1 x
where v E {i,J}
(10)
Here, z and are normalization factors used to resolve the scale ambiguity between predicted and ground truth point maps. They are defined as the average distance of all valid points image pair (I, 12) to the origin:
2 2(x',, X
Σ PEP
| +
Σ | PEPS
(11)
p
+
Pj
The final training loss for DUSt3R is the confidence-awee loss from Wan et al. 2018), defined as:
Leret Σ3Σ
C (u. a log Cp.
(12)
(L (E(L) PEP
This loss enables the model to learn confidence predictions that are correlated with the regression accuracy. The second term acts regularization component, where is hyperparameter that controls the strength of the regularization.
A.2 DERIVATION OF WEIGHT UPDATE RULE
Starting from the joint optimization Eq. them from the joint optimization:
we first minimize over the weight terms w to climinate
aT DD arg min
ΣΣΣ
min w 1e
+
H(V
(13)
T.o.,D, (6.f)
my
13

---

Published as a conference paper at ICLR 2025
In order to find the global minimizers up *, we take the gradient 9u of the above objective function with respect to w
fw 11 + 1
Cp /up
(14)
We observe that the gradient yu continuous and monotonic for w > O. Also, if ws + 0, then || + > 0. Therefore, there exists an unique global
Iw + -0,, and if wo > +oc then Jv - |ا minimizer wa at which the gradient foe evaluates to zero. Setting Iw 0, we can solve for wp as
w = C2 I1,"">1//))
-
(15)
This gives us the same weight update rule in Eq. During global optimization, we alternate be- tween the gradient descent step to optimize geometric parameters: T. J, D and the weight update step to set the weight terms to their global minimizers. This helps us down-weight the confidence for overconfident point predictions, and make the optimization more robust on challenging circum- stances.
A.3 IMPLEMENTATION DETAILS
Our pipeline is implemented with PyTorch and our experiments are conducted on a NVIDIA 3090 GPU.
For robust global point map alignment, set the regularization coefficient μ to 0.01. We minimize the optimization loss by running 300 steps of gradient descent using the Adam optimizer with a learning rate of 0.01, applying the closed-form weight update Eq. 8 every 10th gradient descent step. Additionally, we exclude points with prediction confidence below 0.5 by setting their weights to zero, preventing them from participating in the optimization process.
For confidence- pseudo labeling, we use cofidence threshold of 1.5 for all test scenes.
For LoRA fine-tuning, we resize all calibration images the pre- pre-training resolution of (512, 384). During fine-tuning, we optimize the LoRA weights over 10 epochs (without warmup) using the AdamW optimizer with batch size of A cosine decay learning rate scheduler employed, with base learning rate of 0.001 and a minimum learning rate of 0.00001 for most test cases.
A.4 EVALUATION RESULTS ON TUM RGBD DATASET
We evaluated our method for multi-view camera parameter estimation on TUM test scenes: "frl _dekk", "fr2_xyz" and "offccffice the TUM dataset. In each scene, the first 500 RGB im- ages are reserved as the calibration split, and the remaining 92/2897/2015 images form the test split. At test time, we sample 10 images from the calibration split for self-calibration and 10 images from the test split to assess pose estimates.
Table 5: Quantitative evaluation of camera parameter estimates on the TUM RGBD dataset.
fr1 desk
fr2 xyz
fr3 office
Methods
ATE (cm)
AFE (%)
ATE (em)
AFE (%)
ATE (em)
AFE (%)
Pretrain
0.91
8.02
3.89
14.84
3.28
1.95
MUSt3R-Self-Calin
0.62
8.32
1.24
7.67
3.10
1.81
DUSt3R-Depth Calth
0.68
7.63
1.23
4.71
4.12
1.78
COLMAP
0,51
3.87
0.97
4.71
RelPose ++
10.05
34.59
96.91
PoseDiffusion
11.90
27.18
34.32
37.30
92.61
1.28
RayDiffusion
10.95
72.30
35.63
12.12
100.56
3.52
FlowMap
8.03
39.58
14.30
84.21
32.67
24.95
MASt3R
0.56
3.32
1.74
10.88
0.92
1.24
The results of multi-view camera parameter estimation on TUM are reported in Tab. 5 Fig. (c, d). Our self-calibrated DUSt3R achieves up 68% more accurate camera pose estimates and up
14

---

Published as a conference paper at ICLR 2025
to 48% more accurate focal length estimates compared to the pre-trained baseline. The slight drop in focal length estimation accuracy for the "fri_desk" sequence is likely due to overfitting on the calibration images.
Using depth measurements as the supervision signal for DUSt3R fine-tuning appears to be unre- liable. DUSt3R-Depth-Cail under-perform, method in "frl desk" and even falls short of the pre-trained model in "fr3 _officfe".. This is the result of the noise and outliers present in the depth images.
Similar to Replica, COLMAP and MASt3R tends to perform well on the inward-facing cameras in TUM. RelPose++, PoseDiffusion, RayDiffusion and FlowMap fail to provide reasonable estimates of camera parameters.
A.5 MAST3R SELF-CALIBRATION
(a) Segment- 10084: Before self calibraainatoonn
(b) Segment 10084: After self-calibration
(c) Segment 10149: Before self- cafcbbbbratiin
(d) Segment- 10149: After self-calibration
Figure 9: Qualitative results of MASt3R self-calibration on the Waymo test segments 10084 and 10149. After self-calibration. MASt3R produces more accurate camera pose estimates. For segment-10084, the road's center lines exhibit better alignment along straight path. For segment- 10149, the roadside parking lines align more accurately along straight line.
We show that our method generalizes to MASt3R (Leroy et al. 2024 Duisterhof et al. 2024 self- calibration.
MASt3R builds upon DUS13R and re-purposes it for more precise image matching. In addition to pairwise point and confidence predictions, MASt3R also generates dense local feature maps to establish 2D feature correspondences across images. Similar to DUSt3 MAS13R employs global optimization procedure to align and optimize the local per-pair point predictions within global coordinate frame. Leveraging both 3D-3D and 2D-3D correspondences. MASt3R first minimizes a 3D projection loss as in Eq. 6 to optimize the image pair seales and camera poses, followed by minimizing 2D re-projection loss optimize the camera focal lengths and depth maps.
For MASt3R celf-calibration, we also utilize the global optimization results to generate pseudo labels. For simplicity, directly apply all point predictions across all calibration views for pseudo
15

---

Published as a conference paper at ICLR 2025
labeling, without employing confidence-based filtering. We maintain the of rank-16 LoRA and the same pre-training loss for fine-tuning MASt3R.
Table 6: Quantitative evaluation for MAS13R self-calibration the Replica "officeO", TUM
"fr2 xyz" and Waymo test segments 10084 and 10149.
Scene
Replica officel
TUM xy2
Waymo
10084
Waymo 2g.-1
Metric
[em]
Comp.
ATF[cm]
AFE[S]
ATF[m]
AFF[S]]
ATR[m]
AFE[
PrStanPretrain
6.03
10.88
1.35
24.91
MASH3 Self-Calib
4.61
6.02
1.60
10.48
1.21
16.71
1.19
22.97
MASGR-GT-FT
4.25
5.83
1.05
9.87
121
24.30
We evaluated our pipeline on four test scenes: Replica "office0", TUM "fr2_xyz", and Waymo test segment- 10084 and regment-1049) The self-calibrated MASt3R was assessed on multi-view re- construction and multi-view camera parameter estimation using the same evaluation metrics against the pre-trained baseline and the ground-truth fine-tuned model. The results are reported in Tab. and Fig.
We observe improved performance of MASt3R across various tasks and datasets compared to the pre-traue pre-trained model, though the gains are mostly marginal, likely due to the presence of unfiltered noise and outliers in the pseudo-labeled data. In future work, plan calibrate MASt3R by pseudo labeling.
incorporating calibrated-confidence
A.6 ABLATION STUDY ON THE SIZE OF CALIBRATION SPLIT
In addition the number of calibration images, the size of the calibration split plays important role in self-calibration because it affects the diversity of viewpoints among the calibration images. As shown Fig. I0 when we limit the calibration split to only the first 10 50 images in test scene, we observe decrease in the performance of the self-calibrated model.
This decline occurs primarily because the sampled calibration images from the smaller splits share very similar viewpoints. As a result, DUS(3R predictions across view pairs are of similar quality, and global optimization therefore offers limited improvements in point estimation accuracy. Reduced training set diversity and limited scene observation further contribute to the performance decrease.
Therefore, we recommend capturing calibration images from different viewpoints ensure effective self-calibration.
Avg. Prea Fr: [cm]
14
Pretrained
Avg. Pred. Err. [m]
FreErained
12
11
10
10
50
100
200
600
1000
10
20
60
100
0.5%
2.5%
5% of calibration split
10%
30%
50%
5%
10%
30%
50%
calibration split
(a) Scene: Replica officel)
(b) Scene: Waymo test segment- 1anccnt--1144 10149
Figure 10: Ablation Study: Reducing the calibration split size (e.g., using only the first 0.5% of images) may adversely impact self-calibration performance.
A.7 CONCURRENT MULTI-SCENE CELF-CALLIBATION
Table 7: Quantitative evaluation of multi-scene concurrent self-calibration
Training Data
office0 (Ours)
All Replica
All except officeO
Avg. Point Pred. Err. [cm]
8.83
9.64
11.64
Accuracy [em]
4.43
4.24
6.11
Completeness [cm]
6.08
7.13
8.74
We tested our pipeline with multi-scene concurrent celf-calibration. Specifically, we fine-tuned the pre-trained DUS13R using pseudo-labeled data from all Replica test scenes and evaluated the self-
16

---

Published as a conference paper at ICLR 2025
calibrated DUSt3R the office0 test set. We also experimented with excluding officeO's pseudo-labeled data from the training data during fine-tuning.
The performance of DUSt3R, self-calibrated on different pscudo labeled data, presented in Tab. [7] We observe comparable performance between concurrent and scene-specific training. How- ever, excluding the target scene data led decline in performance. This highlights the importance of including target scene images for model self-calibration.
A.8 USE DUST3R TO INITIALIZE COLMAP?
As discussed in Sec. 3.3 the vanilla DUS(3R (i.e., the DUSt3R-Pretrain baseline) uses novel 3D- DD-projection-bas based global optimization to align the DUS:3R-predicter point maps across views. This natually leads the question: Can we use the commonly adopted 2D-3D -a-----jeettionbasseee optimization to align these point maps? Specifically, is it feasible to use DUSt3R predictions to initialize COLMAP's bundle adjustment (Schonberger & Frahm, to create a stronger baseline?
The short answer is no.
Theoretically, DUSt3R is trained to predict 3D point maps but not explicit 2D-2D or 2D-3D corre- spondences. While we could attempt to extract reciprocal pixel-level 2D-2D matches by performing nearest-neighhor matching the 3D point map space (as discussed Sec. 3.3 of Wang et al. (2024). )
these matches are limited because (I) they only at the pixel level and (2) they can be incomplete due to potential violations of mutual consistency. Directly using these matches for COLMAP bundle adjustment can lead to inferior reconstruction quality.
Experimentally, we attempted to use 2D-2D correspondences extracted from DUSt3R predic- tions to initialize COLMAP. We retrieved the matched pixels' globally aligned point map predictions and computed their median 3D positions to initialize the 2D-3D correspondences for COLMAP's bundle adjustment. As reported in Tab. 8 and Fig. 1T. this approach, termed DUSÉRR-COLMAP, significantly underperforms our baselines such as DUSt3R-Pretras and MASt3R.
Table 8: Using DUStR to initialize COLMAP (i.e. CULMAR- significantly underper- forms other baselines.
office0
segment- 10084
Methods
Acc.[cm] 4
Comp.fcm) +
ATR[m] 4
AFE[%]
DUSt3R- Pretrain
5.22
6.78
0.79
2.79
DUSBR--OLLNE
44.36
56.66
5.59
89.51
COLMAP (dense)
2.61
89.87
Fail
Fail
MAS(3R
4.69
6.05
2,85
11.87
(a) BUSt3R-Pretrain
(b) DUSS3R-
Figure 1I:
Using DUSI3R to initialize COLMAP (i.e. DUS(3R-CLLMAP significantly under- performs the DUSC3R-initiaized 3D-projection-basee global alignment method (i.e. the DUS(3R- Pretrain baseline).
This observation is consistent with findings in the literature. For example, Leroy et al. (0024) ports (in Tab. 1) that using matches extracted from DUSt3R predictions for map-free localization significantly underperforms MASt3R, which is directly trained for image matching.
17

---

Published as a conference paper at ICLR 2025
A.9 QUALITATIVE COMPARISON AGAINST GROUND TRUTH MESH
We directly compare DUSt3R's multi-view reconstructions with the ground truth mesh to identify
and visualize the source of errors.
The qualitative results in Fig. 6 (a, b) demonstrate that our self-calibration method effectively re- duces outlier points in DUS reconstructions. However, the outlier points are not the only source of error for the pre-trained DUSI3R model, despite the otherwise visually impressive reconstruction.
In Fig. 12 we directly compare the reconstructions against the ground truth mesh model for the same Replica office0 scene. Beyond the visually obvious improvements highlighted in the green boxes, the self-call calibbabbdt model also more accurately captures the geometry of the scene. As a result, after multi-view aligmment, the calibrated DUSt3R's reconstruction aligns more closely with the ground truth various regions, as indicated the red boxes, while the pre-trained DUSt3R's reconstruction exhibits misalignments in these areas.
(a) Before self-calibration
(b) After self-calibration
Figure 12: Qualitative comparison of DUS:3R multi-view reconstructions before and after self- calibration against the ground truth mesh model in the Replica office0 scene (top view for the same reconstructions in Fig. 6 (a,b)). The ground truth mesh, with parts invisible to test cam- era views culled, is displayed with real vertex colors, while the reconstructions are visualized us- ing heatmap-like colors. Besides the visually obvious improvements in the green boxes, the self- calibrated model also better captures the overall geometry of the scene, demonstrating better align- ments with the ground truth various regions, indicated in the red boxes.
A. REMAINING EXPERIMENTAL RESULTS ON THE REPLICA DATASET
Table 9: Table (continued): Quantitative evaluation multi-view dense reconstructions on the
Replica dataset
roomO
room
room2
Methods
Acc.
Comp.
Acc.
Comp.
Acc.
Comp.
DUSt3R-Pretraii
6.97
9.97
10.54
13.13
7.79
10.92
6,83
9.86
8.88
11.54
5.80
9.10
DUSF3R-GT-FT
4.59
8.15
8.64
11.35
3.35
7.34
COLMAP (dense)
6.30
95.02
5.54
292.35
4.88
120.52
FlowMap
66.71
226.22
96.53
140.33
37.29
274.98
MASSR
4.07
8.71
12.63
11.52
3.40
8.43
A.11 FULL EXPERIMENTAL RESULTS ON WAYMO OPEN DATASET
18

---

Published as a conference paper at ICLR 2025
Table 10: Full quantitative evaluation results of multi-view pose estimation on Waymo Open Dataset (Part I)
Methods
segment- 10084
segment- 10149
sogmen--10161
segment-IO44D
segmen--
ATE())
AFE(S)
ATE())
AFE(S)
ATE(m)
AFE(S)
ATE(m)
AFE(S)
ATE(())
AFE(S)
PuStaR-Pretrain
0.79
2.19
0.84
3.08
0.38
0.34
0.02
0.15
0.00
0.55
DUS(3R- BUSS3R-Self-Calie Self-Calib
0.37
0.61
0.25
2.14
0.38
0.52
0.02
0.15
0.00
0.85
DUSGR GT
0.20
0.17
0.17
1.54
0.35
0.46
0.02
0.15
0.00
0.41
Flowzap
0.31
3.97
66.62
1.80
4.74
7,02
2.84
0.00
15.05
0.00
PoseDiffusion
19.43
25.07
16.76
49.18
7.65
13.85
0.03
10.84
0.00
1.78
RayDiffisson
17.34
85.65
16.91
80.69
6.40
83.73
0.09
84.44
0.00
84.49
RelPose++
14.80
16.20
4.06
0.08
0.00
MASI3
2.85
11.87
1.35
24.92
0.36
12.96
0.00
7.65
0.00
22.87
segment- 10504
segment- 10534
segment- 10649
segment-10649
segmenn-T0802
segment- (0940
Methods
ATE(m)
AFF(S)
ATB(m)
AFE(S))
ATF(m)
AFE(%)
ATT(m)
AFE(S))
ATB(m)
AFF(%)
DUSIR Pretrain
0.29
2.19
0.74
0.84
0.95
2.84
0.35
1.60
0.01
0.74
DUSS:R-Self-Cali
0.38
1.79
0.67
0.76
0.49
2,54
0.35
1.08
0.02
0.94
DUSGR GT.FT
1.04
0.67
1.73
0.55
0.02
Flownup
66.65
1.48
14.85
6.24
36.44
13.74
21.16
0.44
1.88
0,06
PoseDiffusio
16.38
2,54
13.31
1.42
20.19
2.26
13.61
23.74
1.43
32,79
RayDiffusion
17.03
85.26
10.73
84.73
85.09
81.44
2.38
RelPose++
16.88
13.24
13.69
12.92
2.31
9.91
20.53
24.75
Methods
segment-
segment-
10998
segment-
segment
segment-
ATE(m)
AFF(%)
ATE(m)
AFE(%)
ATB(m)
AFE(%)
ATE(m)
AFF(%)
ATB(m)
AFE(%)
DUSER Pretrain
0.80
1.19
0.51
0.66
0.69
1.43
0.50
1.28
0.40
2.09
DUSE3R-Self-Ca Self-Calib
0.09
0.69
0.50
0.64
0.69
1.42
0.34
0.74
0.17
1,29
DUS3R-GT-FT
0.13
0.49
0.80
0.63
0.71
1,46
0.37
0.83
0.10
0.60
Flowmup
65.17
0.66
34,91
8.07
12.88
6.72
12.43
0.59
66.10
2.23
PoseDiffusion
18.19
31,04
11.51
5,42
12.09
8.91
13.74
15.61
11.11
8.02
RayDiffusion
19.12
83.00
13.80
83.88
15.08
84.71
15.23
84.22
8.92
83.25
RelPoset+
13.55
14.07
13.30
MASt3R
1.61
6.59
0.57
2.48
1.11
8.72
0.51
25.26
0.46
39.65
segment. I 1867
segment: 1 1933
segment- I 1987
segment I 12056
segment 12153
Methods
ATE(m)
APE(S)
ATl(m)
AFE(3)
All/m)
AFE(5)
All(m)
AFE(S))
Alli(a))
AFE(S))
DustaR-Peetrin
0.20
1.41
0.00
0.39
1.30
0.52
1.38
0.35
0.97
DUSER Self-Calib
0.18
1.77
0.00
0.85
0.33
1.28
0.32
2.32
0.35
1.00
DUSFR--TTFT
4277
2.28
0.00
0.51
0.33
1,28
0.70
0.51
Flowmup
10.73
2.62
59.68
0.00
39.77
3.15
48.50
3.39
14.49
2.40
PoseDiffusion
2.44
5.20
0.00
15.37
11.22
11.04
9.41
34,37
7.24
15.39
RayDiffusion
11.74
84.31
85.94
6.46
RelPose++
2.39
0.00
11:01
16.00
2.92
MASt3
1.67
10.26
0.00
14.03
3.91
5.05
2.01
19.19
0.56
11.21
Methods
segment- 12537
segment- 12555
segment- 12892
segmenn-
segment- 13347
ATl5m)
AFE(S)
ATE(m)
AFE(S)
ATb(m)
AFE(%)
ATE(m)
AFE(S))
ATl5(m)
AFE(S)
Pretran
0.63
1,38
0.02
0.45
0.20
0.68
3.84
4.12
0.01
0.79
DUSS3R-SeUf-CA
0.02
0.08
0.56
4.19
0.01
DUS33-- GT-FT
0.10
1.17
0.02
0.04
0.05
1.14
1.97
2,28
0.01
0.25
Hlowmap
32.50
0.62
46.88
0.01
17.30
0.40
43.23
12.63
16.67
0.03
PoseDiffusion
37.32
0.70
RayDifTusion
13.73
86.47
0.11
83.48
6.39
83.62
33.72
83.64
1.95
84.32
RelPose++
0.07
MAStRR
0.42
2.11
0.00
4.99
0.24
15.56
1.59
24.94
0.13
7.47
Methods
segment- 13585
segment- 13732
sogment- 13748
sogment-
13763
segment-1 1378|
ATE(m)
AFE(S))
ATE(m)
AFE(S)
Alb(m)
AFE(5)
ATb(m)
AFE(SS)
ATl5mm
AFE(5)
DUSUR Preiruin
1,93
8.13
0.00
1.59
DUSL3R DUSL3R-SelfCaale Self-Calib
0.47
0.39
6.82
3.90
0.00
3.20
0.09
0.32
1.73
1.73
BUS3R-GGTFT
0.36
0.42
3.51
1.59
0.00
1.27
0.12
0.45
1.69
1.68
Flowmap
21.94
10.93
0.00
0.70
1.20
PoseDiiTusion
11.12
12.22
41.83
19.75
0.00
4.26
7.52
11.11
13.90
21.86
RayDiffusion
10.40
84.53
55.53
81.32
0.00
86.28
11.27
82.18
18.50
84.38
RelPosc++
13,53
46.03
0.00
10.67
16.10
MASI3R
0.37
14.36
2.02
45.88
0.00
13.12
0.56
16.25
0.58
18.27
segment- 13787
segment- 1.790
segment- I 13849
segiment: 13887
segmment 13944
Methods
ATF(m)
AFF(S)
ATF((m)
AFF(S))
ATF(mm)
AFF(S))
ATF(())
AFF(())
ATF((m
AFF(S)
DUSER Pretrain
1.91
1.88
4.48
4.15
0.27
0.99
1.24
2.56
0.03
0.23
DUSL3R CellC
3.34
2.14
0.04
DUS3R-GGT-F
0.24
0.46
1.87
1.24
0.29
0.69
0.23
1.71
0.03
0.27
Flownup
65.92
1.95
67.17
19.31
14.12
1.41
22.24
0.39
7.22
0.03
PoseDiffusion
16.78
17.17
9.37
RayDiffusion
5.54
82.41
41.05
85,28
17.90
83,32
18.03
83.37
3.20
83,80
RelPose++
9.65
49.12
11:05
16.24
2.51
MASI3R
2.27
8.68
41.31
28.24
0.91
15.96
0.60
20.04
0.22
25.37
19

---

Published as a conference paper at ICLR 2025
Table 11: Full quantitative evaluation results of multi-view pose estimation on Waymo Open Dataset (Part 2)
Methods
sogment-[4177
segment-144E
sogment-L4438
segment- I4477
segment-14550
ATE(()
AFE(S)
ATE(m)
AFE(S)
ATT((m)
AFE(S)
ATE(m)
AFE(S)
ATE(())
AFE(S)
PustaR-Pretrain
0.22
1.71
0.82
0.96
0.03
0.52
0.35
1.44
0.32
1.14
DUSC3R-Self-aall Self-Calib
0.09
0.26
0.83
0.99
0.03
0.55
0.20
1.13
0.26
0.70
DUSGR GT
0.05
0.48
0.88
0.65
0.02
0.37
0.11
1,02
0.12
0.38
HlowMap
10.78
0.21
34.62
8.03
67.15
0.47
67.30
0.55
20.05
1.06
PoseDiffusion
9.39
3.32
10.25
7.19
1.41
5.46
3.95
26.92
10.92
11,94
RayDiffisson
11.52
86.31
9.25
84,93
1.32
86.42
5.58
85.69
10.38
87.09
RelPosc++
8.78
13.53
1.43
7.85
MASI3
0.14
33.95
0.13
8,70
0.01
6,21
0.19
0.92
0.20
17,51
Methods
segment- 14631
146443
segment-1472
14737
segmenn-14918
segment- 15272
ATE(m)
AFF(S)
ATB(m)
AFE(S))
ATF(m)
AFR(%)
ATE(m)
AFE(S))
ATE(m)
AFF(S)
DUSIR Pretrain
0.02
0.38
0.02
0.05
0.05
0.50
0.64
2.07
0.13
1.54
MUSS3R-Self-Call
0.02
0.12
0.03
0,26
0.04
0.19
0.51
1.31
0.14
1.67
DUSGR GT.FT
0.01
0.78
1,05
FlowMap
54,43
0.02
34.54
0.01
62.51
0.02
12.51
0.13
65.77
0.67
PoseDiffusion
0.40
10.31
0.20
4.60
0.48
58.43
8.80
14.48
6.27
1.69
RayDiffusion
0.51
80.16
0.70
85.80
86.42
83.64
RelPose++
0.37
0.40
0.99
9.97
7.54
7.18
17.16
Methods
segment-
segment- 15410
segment- 15739
segment
segment
ATE(m)
AFF(%)
ATE(m)
AFE(%)
ATF(m)
AFE(%)
ATE(m)
AFF(%)
ATB(m)
AFE(%)
DUSER Pretrain
0.38
1.43
0.00
0.16
0.03
0.29
0.43
1.51
0.01
0.48
DUSER--Sef--CH Self-Calib
0.35
1.67
0.00
0.20
0.03
0.16
0.18
0.95
0.01
0.75
DUS3R-GT-FT
0.22
1,21
0.00
0.25
0.03
0.51
0.12
0,44
0.01
0.61
FlowMap
19.65
1.07
35.24
0.00
50.23
0.02
7.26
2.27
16.66
0.04
PoseDiffusion
11,95
12.85
0.00
12.67
1.65
7,88
15.60
29.29
0.22
39,76
RayDiffusion
14.86
84.60
0.00
84.32
1.50
84.64
18.44
85.29
0.22
83.85
RelPoset+
14.61
MASt3R
0.14
5,21
0.00
7.27
0.02
1.18
0.54
23,49
0.01
12.81
Methods
segment. 16062
segment: 16367
segment- 16418
segment
16645
segment 16721
ATB(m)
AFE(3))
All(m)
AFE(S)
All/m)
AFE(%)
ATI(m)
AFE(TS)
Alli(a))
AFE(S)
DustaR-Peetrin
14.93
3.35
0.52
1.55
0.12
0.41
0.89
3,36
3.91
2,13
DUSIR Self-Calib
3.23
3.09
0.31
0.99
0.05
0.36
0.47
1.71
7,73
4.40
DUSFT--GT-F
1,26
0.05
0.05
0.57
0.91
2,56
FlowMaa
64.20
5.49
3.85
1.23
6.02
0.36
63,50
4.06
64.96
11,18
PoseDiffusion
43.39
33.25
12.32
4.96
8.39
7,83
28.05
59.33
43.68
4.94
RayDiffusion
46.45
13.82
7.24
RelPose++
23.36
9.12
9.41
29.81
47.79
MASt3R
4.16
36.81
0.65
10.62
0.28
21.33
2.38
29.15
12.67
46.19
Methods
segment- 16743
1ggmen-16942
segment- 16951
segmeen-
17030
segment-1
TERGGRE-TOOSS 7752
ATT5m)
APE(S)
ATl(m)
AFE(S)
ATb(m)
AFE(%)
ATE(m)
AFE(S))
ATl5(m)
AFE(%)
Pretran
0.42
0.92
0.00
0.70
0.09
0.48
0.53
1.96
0.24
0.47
DUS(3R- Self--Cl
0.00
0.53
1.58
DUS:3R-C GT-FT
0.12
0.90
0.00
0.92
0.22
1,05
0.62
1.38
0.18
0.38
MlowMap
3.49
0.39
23.33
0.00
20.71
0.41
66.69
1.29
24.01
0.79
PoseDiffusion
3.47
18.55
RayDifTusion
6.38
82.81
0.00
83.65
3.96
84.95
12.34
86.11
8.17
85.18
RelPose++
0.00
3.13
MAStRR
0.35
17.33
0.00
27.89
0.14
23,55
0.47
37,95
0.24
16.97
Methods
segment- I7136
sogment- 17174
sogment- 17212
sogment- -
17262
sogment-1733
ATE(m)
AFE(S))
ATE(m)
AFE(S)
AlE(m)
AFE(S)
ATE(a)
AFE(SE)
ATlamm)
AFE(S)
DUSUR Preiruin
0.04
1.08
2.78
2.66
DUSL3R DUSL3R-Self-Cale Self-Calib
0.00
0.67
0.04
0.54
0.75
1.36
0.46
0.68
1.95
3.08
DUSØR-GTTFT
0.00
0.69
0.03
0.79
0.58
1.03
0.14
0.20
0.63
1,20
FlowMap
13.73
0.00
26.35
0.01
16.44
3.00
14.32
2.08
64.86
3.68
PoseDDifusion
0.00
15.44
0.77
22.26
18.75
10.02
18.89
28.26
40.69
0.61
RayDiffusion
83.01
0.87
86.80
15.09
84.30
16.71
86.99
41.94
87.07
RelPosc++
0.00
0.73
15.50
18.62
38.25
MASI3R
0.00
21.45
0.02
4.85
2.83
21.88
2.09
23.93
3.24
16.06
Methods
segment-1178e
segment- 17595
segment- 1 17652
segimeen: 17756
segiment 17792
ATF(mm)
AFF(S))
ATF((m)
AFF(S))
ATF(m)
AFF(S)
ATF(())
AFF(9))
ATF(()
AFF(S)
DUSER Pretrain
0.08
0.31
2.03
3.07
0.00
0.39
0.17
0.43
1.18
4.16
AUSL3R-Sell-Cabl
0.00
0.39
0.41
DUS3R-GT-F
0.88
0.25
0.27
1.37
0.00
0.86
0.69
0.78
0.41
2.76
FlowMap
67.21
0.16
64.83
5.51
18.89
0.00
65.36
5,34
65.86
0.96
Pisiusiiusion
21.37
32.09
9.27
RayDiffusion
1.68
84.45
17.35
86.20
0.00
84,55
10.82
83.95
19.01
84.28
RelPose++
1.68
13.92
0.00
12.56
19.40
MASI3R
0.25
2.91
1.57
5.10
0.00
28.83
1.61
13.92
1.51
32.49
20

---

Published as a conference paper at ICLR 2025
Table 12: Full quantitative evaluation results of multi-view pose estimation on Waymo Open Dataset (Part 3)
Methods
sogment-L7883
segment-
sogment-19963
segmenn-22188
segment-22577
ATE(m)
AFE(S)
ATE(m)
AFE(S)
ATT((m)
AFE(S)
ATE(m)
AFE(S)
ATE(())
AFE(S)
PustaR-Pretrain
0.44
1.10
0.02
0.72
0.61
1.07
0.94
2.63
0.22
0.61
DUS(3R- DUSC3R-Self-Cali Self-Calib
0.40
0.86
0.02
0.86
0.48
0.80
0.52
2.91
0.14
0.50
DUSGR GT
0.15
1.75
0.02
0.51
0.47
0.69
3.42
2,06
0.18
0.61
HlowMap
11.01
0.71
11.56
0.47
15.73
1,32
31.67
26.62
6.07
0.49
PoseDiffusion
2.14
17.48
2.04
2.62
11.46
23.42
29.46
17.82
5.45
6.25
RayDiffusion
10.42
83.72
1.75
86.20
12.22
85.41
22.59
85.63
12.11
84.58
RelPosc++
6.71
2.26
11,66
18.93
13.62
MASL3R
0.19
29.94
0.10
4.77
0.34
31.45
1.25
32.51
0.18
13.95
Methods
23gmemr236322
segment-2577
25741
segmeni-228899
4egmeent220122
segment-27005
ATE(m)
AFF(%)
ATB(m)
AFE(S))
ATE(m)
AFE(%)
ATT(m)
AFE(S))
ATB(m)
AFF(S)
DUSIR Pretrain
0.00
0.10
0.25
0.42
0.01
1.42
3.91
3.57
0.18
1,06
MUSS3R-Self-Call
0.00
0.06
0.21
0.35
0.01
1.36
2.11
2,97
0.08
0.79
DUSGR GT.TT
1.65
0.08
FlowMap
18.27
0.00
66.80
0.24
64.95
0.00
32.14
1,85
24.32
7.17
PoseDiffusion
0.00
19.19
10.14
15.66
0.02
48.88
23.48
23,
15.16
8.79
RayDiffusion
0.00
0.02
83.55
82.83
17.32
RelPose++
0.00
7.39
0.02
22.62
9.57
23.21
Methods
segment
segment 27951
segment-
segment-
segment
ATE(m)
AFE(%)
ATE(m)
AFE(%)
ATB(m)
AFE(%)
ATE(m)
AFE(%)
ATB(m)
AFE(%)
DUSER Pretrain
0.71
1.25
0.47
0.86
0.55
0.86
0.89
1.04
0.97
0,93
DUSE3R-Self-Ca Self-Calib
0.37
0.44
0.38
0.98
0.56
0.81
0.90
1.04
0.90
0.83
DUS3R-GT-FT
0.28
0,46
0.89
0.88
0.37
0.82
0.85
1.00
0.12
1.16
FlowMap
22.32
1.01
0.89
1.98
28.60
1.14
1.67
0.89
28.52
2.71
PoseDiffusion
14.82
16.66
22.07
50.30
11.30
2,71
10.64
5.15
16.49
28:52
RayDiffusion
16.48
86.66
28.07
81.87
10.34
83.21
13.85
84.43
15.03
83.45
RelPoset+
MASt3R
0.90
23.19
0.91
21.71
0.21
10:55
0.68
16.65
0.45
20.77
Methods
segment. 31225
segment 32758
segment-
sagment-12285
33285
segment 13418
segment 34004
ATE(m)
AFE(S))
ATl(m)
AFE(S)
All/m)
AFE(%)
ATl5mm)
APE(T))
Alli(a))
AFE(S)
DUStaR-Pretrinn
0.35
0.63
0.27
0.02
0.00
11.10
4.69
3.13
3,97
DUSIR Self-Calib
0.34
0.75
0.27
0.50
0.02
0.42
2.24
7.65
3.92
1.94
DUSFT--GT-F
0.23
0.02
0.75
4.25
2.16
4.62
FlowMap
21.33
1.57
35,86
0.75
1.21
0.02
3.56
4.89
34.54
7,30
PoseDifTusion
7.91
11.04
10.16
28.77
0.09
5.18
49.91
7.05
49.33
51.08
RayDiffusion
10.50
86.37
85.77
45.54
RelPose++
6.94
4.63
0.13
31.38
35.44
MASt3R
1.12
11.81
0.76
5.75
0.01
17.75
1.13
26.28
4.56
23.23
Methods
segmen--34500
3485
segmen--35106
5cgment-33445
ATT5m)
AFE(S))
ATl(m)
AFE(S)
ATB(m)
AFE(%)
ATE(m)
AFE(S)
ATls(m)
AFE(S)
DUstaR-Pretrain
3.48
2.32
0.16
0.80
2.38
3.22
0.06
0.35
0.13
0.69
DUS(3R- SEE-A---El--aaiit
3.30
0.36
0.31
DUS:3R-C GT-FT
1.42
1.51
0.12
0.63
2.63
2.77
0.04
0,27
0.24
0.56
FlowMap
16.36
2.13
18.08
0.07
67.26
10.43
17.20
0.11
38.77
5.01
PoseDiffusinn
8.82
1.47
9.53
RayDifTusion
35,30
85.89
6.96
83.16
26.80
85.35
3.76
85.23
9.93
85.83
RelPose++
4.37
25.92
MAStRR
1.82
29.80
0.19
11.07
4.25
14.31
0.07
8.89
0.65
11.08
Methods
segment-3655
sogment-39847
segment-HOUsI
-0379
5ogment-H0456
ATE(m)
AFE(5))
ATE(m)
AFE(S)
AlE(m)
AFE(%)
ATI(m)
AFE(SE)
ATb5mm
AFE(%)
DUSUR
19.23
1.03
3,82
DUSL3R DUSL3R-SelfCaali Self-Calib
0.35
1,52
2.24
3.72
0.98
1,82
0.94
2,58
8.01
2,78
DUS3R-GTTF
0.40
0.80
1.78
1.75
0.84
1.79
3.55
3.67
23.53
3.77
FlowMap
26.32
1,05
34.85
67.37
2.43
17.48
4.15
2.10
2.93
PoseDDifusion
22.26
24.81
44.65
13.89
18.08
16.18
30.71
1.73
21.00
53.26
RayDiffusion
24.32
84.01
41.13
85.27
14.01
86.18
86.04
25.66
RelPosc++
22.72
38.22
15:42
40.81
27.25
MASI3R
0.74
16.21
23.44
23.22
0.25
36.46
36.94
33.64
0.99
25.27
segment 4054D
segment 41409
segment- 45934
segment-45934
segiment 46325
segmment 49166
Methods
ATF(m)
AFF(S)
ATF((m)
AFF(S))
ATF(mm)
AFF(S))
ATF(m)
AFF(())
ATF(m)
AFF(S)
DUSER Pretrain
0.48
1.78
0.20
1.99
0.80
2.09
0.02
0.76
0.01
0.73
DUSL3R CellC
0.09
0.49
0.01
DUS GF-F
0.12
1.00
0.05
1.21
0.33
0.69
0.01
0.55
0.01
0.75
FlowMap
11.73
0.58
66.69
3.09
34.70
0.43
2.78
0.01
17.43
0.00
Pisfusifusiin
7.54
10.57
RayDiffusion
8.84
85.22
10.13
84.64
12.16
84.32
1.17
85
0.01
87.31
RelPose++
9.62
8.69
10.43
1.33
0.01
MASI3R
0.60
22.08
0.47
31.08
0.64
13.56
0.01
10.42
0.00
3,76
21

---

Published as a conference paper at ICLR 2025
Table 13: Full quantitative evaluation results of multi-view pose estimation on Waymo Open
Dataset (Part
Methods
sogmenn-50269
segment-50HGO
sogmen--51547
scgmeen--54445
sogment-55855
ATE(())
AFE(S)
ATE())
AFE(S)
ATE(m)
AFE(S)
ATE(m)
AFE(S)
ATE(1))
AFE(S)
PuStaR-Pretrain
0.02
1.40
0.72
0.92
0.46
0.78
0.00
0.79
LU8
3.25
DUS(3R-Self-aal Self-Calib
0.02
1.38
0.63
0.74
0.49
0,98
0.00
1.25
0.48
0.51
DUSGR GT
0.03
0.72
0.23
0.49
0.20
0.94
0.00
0.48
0.14
0.95
HlowMap
0.11
0.06
0.43
0.36
7.09
3,33
0.75
0.00
64.80
2,95
PoseDiffusion
0.20
36.80
12.33
9.39
15.11
5.49
0.00
21.16
18.87
36,92
RayDiffusion
1.07
82.00
12.55
83.03
17.47
84.13
0.00
85.85
19.73
84.64
RelPosc++
0.91
4.96
17.53
0.00
17.98
MASI3
0.06
6.43
0.16
16.06
2.23
21,26
0.00
6.52
0.14
16.11
Methods
segmenr-56382
564800
segment-56633
50gment-57613
segment-S8804
ATE(m)
AFF(%)
ATB(m)
AFE(S))
ATE(m)
AFE(%)
ATT(m)
AFE(S))
ATB(m)
AFF(S)
DUSIR Pretrain
0.12
0,82
30.44
4.34
1.35
1,09
0.03
0.07
0.53
1,87
MUSS3R-Self-Call
0.17
1.08
12.56
8.02
1.35
0.86
0.02
0.15
0.09
0.26
DUSGR GT.TT
2.82
1.49
0,92
0.11
FlowMap
6.23
1,00
63,96
8.24
65.94
5,93
44,05
0.01
1.65
0.41
PoseDiffusion
2.67
2,83
40.82
20.78
7.03
16.44
1.90
44,77
18.16
14,25
RayDiffusion
3.86
40.23
8.04
87.57
82.73
RelPose++
4.89
40.78
7.44
1.65
10.05
16.29
Methods
segment-Cee 59279
segment- 59934
segment 60792
segment
segment 61743
ATE(m)
AFE(%)
ATB(m)
AFE(%)
ATB(m)
AFE(%)
ATE(m)
AFF(%)
ATB(m)
AFE(%)
DUSER Pretrain
1.31
2.01
0.37
1,20
0.12
0.96
0.28
1.35
0.59
0.79
DUSE3R-Self-Ca Self-Calib
0.61
1.62
0.27
1.11
0.18
0.51
0.27
1.29
0.48
0.59
DUS3R-GT-FT
0.45
1,23
0.31
0.64
0.17
0.55
0.27
1.24
0.32
0.97
FlowMap
63.48
2.74
26.45
1.25
22.54
0.41
0.71
2.39
16.09
0.87
PoseDiffusion
15.22
13.01
13.24
6.33
7.29
18.23
11.56
18:07
13.57
12.09
RayDiffusion
13.96
83.50
13.53
84.17
9.01
86.26
12.11
85.22
12.58
84.62
RelPoset+
7.70
MASt3R
0.44
12.35
0.80
15.63
0.38
12.02
0.35
15.46
0.28
20.69
segment 62287
segment 62595
segment. 62783
Negment 65030
segment 68423
Methods
ATE(m)
APE(S))
ATl(m)
AFE(S)
AlE/m)
AFE(%)
ATE(m)
AFE(T))
Alli(am)
AFE(S)
DUStaR-Pretrann
0.07
0.28
0.81
2.52
0.87
1.64
0.50
0.68
0.76
1.21
DUSIBR- Self-Calib
0.08
0.51
0.57
1.33
0.54
0.38
0.49
0.59
0.48
0.74
DUSFT--GT-F
0.13
0.19
0.56
0.31
0.45
FlowMaa
12.08
0.03
34,90
1.51
62.22
0.70
16.36
1.63
65.16
12.63
PoseDifTusion
1.62
2,05
14.08
11.95
15.71
15.11
12.13
18.27
14.71
0.39
RayDiffusion
19.96
14.12
$1.05
RelPose++
2.40
13.77
16.00
13.27
13.07
MASt3R
0.03
11.91
0.28
18.00
1.21
4.88
1.59
7.40
1.55
11.60
Methods
6862
6222
segment-
segmenn-
72478
segment-74335
ATl5m)
APE(S)
ATl(m)
AFE(S))
ATB(m)
AFE(%)
ATE(m)
AFE(S)
ATIS(m))
AFE(S)
DUetrRnPretrrin
0.02
0.82
0.44
0.70
1.36
1.73
0.12
0.54
0.78
1.15
DUS(3R- SeLLA---el--aabb
0.87
0.31
0.79
DUS:3R-C GT-FT
0.02
0,56
0.42
0.27
0.70
0.52
0.10
0.34
0.76
1.39
MlowMap
48.93
0.01
50.22
0.31
3.72
2.21
13.31
0.37
32.77
0.98
PoseDiffusion
10.73
4.67
RayDifTusion
0.88
84.37
10.80
84,93
26.08
85,30
6.97
84.42
13.61
80.67
RelPose++
10.72
MAStRR
0.02
20.55
1.86
7,86
1.90
15.61
0.47
23.87
0.16
16.75
Methods
segment- 7S11
sogment-78443
segment- 78551
sogment-
78860
sogment-79235
ATE(m)
AFE(S))
ATE(m)
AFE(3)
ATE(m)
AFE(%)
ATb(a)
AFE(SE)
ATb5mm
AFE(%)
DUSUR
4.83
0.79
1.32
0.42
DUSL3R DUSL3R-Self-Cali Self-Calib
0.61
0.75
4.58
7.77
0.46
1.01
0.24
1,85
0.38
0.47
DUSØR-GGTFF
0.44
0.29
2.89
3.13
0.25
0.43
0.18
1.11
0.24
0,50
FlowMap
63.77
11.82
27.20
13.79
6.82
1.87
1.39
38.47
4.77
PoseDDifusion
14.40
61.44
42.60
22.56
21.04
64.68
6.77
9.84
9.60
2.92
RayDiffusion
21.01
82,06
43.07
84.64
22.52
83.45
6.23
84.33
10.61
84.56
RelPosc++
20.70
42.96
21.81
3.17
8.36
MASI3R
1.07
28.91
25.31
21.59
2.34
26.05
2.20
13.76
0.94
9.35
Methods
segment 80858
segment- 81973
segment 82293
segiment 82491
segmment 85664
ATF(m)
AFF(S)
ATF((m)
AFF(S))
ATF(m)
AFF(S))
ATF(m)
AFF((%)
ATF((m
AFF(S)
DUSER Pretrain
0.92
1.39
0.01
1.08
0.05
1.08
3.79
3.24
0.43
2,07
DUSL3R-Sel--Co CellC
0.01
2.19
DUS GF-F
0.68
1.15
0.01
0.64
0.03
0.62
0.26
1.06
0.12
0.48
FlowMap
51.33
2,02
26.73
0.01
12.76
0.64
22.68
9.36
60.06
0.56
Pisiusiiusion
12.26
RayDiffusion
21.98
82.51
0.02
85,90
1.37
86.79
25.74
84,38
10.92
85.28
RelPose++
22.75
0.02
1.28
16.99
13.20
MASI3R
1.49
23.72
0.01
10.38
0.10
9.91
0.69
25.66
0.46
14.95
22

---

Published as a conference paper at ICLR 2025
Table 14: Full quantitative evaluation results of multi-view pose estimation on Waymo Open Dataset (Part 5)
Methods
segment-81222
sogment-86840
segment-8G8SS
segment- -920
segment-89955
ATE(m)
AFE(%)
ATE(m)
AFE(%)
ATB(m)
AFE(%)
ATE(m)
AFE(%)
ATE(m)
AFE(%)
DUS:3R- Pretrain
0.47
0.64
0.06
0.93
0.03
0.79
0.02
0.17
2.47
3.29
DUS(3R- SclfCal-
0.31
0.06
0.03
0.62
0.15
2.88
3.06
DUSAR--TT-F
0.24
0.75
0.05
0.59
0.05
0.63
0.03
0.21
2.46
3.61
Flo
63.21
4.62
11.42
0.23
17.80
0.08
18.83
0.19
54.40
13.61
PoscDiffusion
3.50
20.01
RayDiifussin
6.76
83.67
3.94
86.41
0.90
85.31
3.02
83.67
24.47
84.09
RelPose++
MASröR
0.63
6.81
0.09
10.78
0.08
28.73
0.42
4.88
2.29
24.34
segment991500
segment- 93509
9355
segment 95847
4ogment-95847
19gment-98068
Methods
ATE(m)
AFE(%)
ATB(m)
AFE(%)
ATE/m)
AFE(%)
ATE(m))
AFE(%)
ATE(m)
AFE(%)
DuettR-Pretrun
0.07
0.61
0,08
0.07
DUSL3R-Self-CE
0.07
1.26
0.30
0.41
0.05
0.90
0.06
0.32
0.06
0.53
DUS:3R-GTFF
0.14
0.62
0.18
0.76
0.12
0.80
0.06
0.94
0.05
0.30
Mlp
61.68
0.48
0.63
0.54
34.08
1.77
24.59
0.11
19.04
0.40
PoseDiffusion
2.88
62.13
16.87
13.16
2.36
49.54
5.72
5.45
5.33
23.84
RayDillusion
16.92
87.45
RelPosc++
4.82
17.12
1.99
5.27
7.02
MASGR
0.04
11.13
0.97
23.28
0.04
0.87
0.29
30.49
0.42
35.19
23


Reviews:
[{"review_id": "biZvZoWOnd", "reviewer": "Reviewer_YtdK", "summary": "The paper proposes an approach for calibration 3D foundation model, particularly DUSt3R and MASt3R. Given a small sample of images, the paper extracts the most multi-view consistent 3D points (through a robust global optimisation and confidence calibration) and then feeds them to a LoRA-based fine-tune process.", "questions": "-", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is very well written and easy to understand.\n\nI very much appreciate the core calibration framework text section and underlying method, in that is it technically principled, produces strong state-of-the-art improvements, and is presented in an intuitive manor. This is a great example of how ML/CV papers should be written.\n\nIt is the first time I see LoRA used in the context of a 3D foundation model. I expect this to help greatly increase the potential impact of the paper.\n\nThe overall approach produces significantly improved results over the DUST3R baseline, and the small sizes of the LoRA could potentially lead to practical applications.", "weaknesses": "I find little weakness with the paper. Mainly, I would have liked to see more experiments on MASt3R in the body of the main work, and perhaps an inclusion of the calibrated-confidence-based pseudo labelling in the MASt3R integration. This is perhaps the only point that stops me from recommending a strong accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "smIVjGYwiT", "reviewer": "Reviewer_XhbG", "summary": "This paper presents an effective and straightforward calibration method based on LoRA to fine-tune a 3D foundation model for specialization in a specific scene, achieving a notable performance increase of over 80%. Extensive experiments conducted across multiple datasets and downstream tasks, along with comparisons to various baselines, further validate the model's performance.", "questions": "Please refer to the questions outlined in the weaknesses section, which includes an ablation study on the portion of data used for calibration, details on how the scenes were selected, and clarification on the visually sensible improvements observed.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This task is quite intriguing, presenting an efficient and useful technique for calibrating pre-trained 3D models for specific 3D tasks. \nThe concept of calibration used to be employed in the field of uncertainty estimation to enhance output quality. Bringing calibration into a 3D foundation model presents a novel approach for \"making foundation models\" perform more effectively or align better with new or specialized tasks.\n\nThe techniques sound and are easy to follow. Despite some poor observations (e.g., low light), there are still sufficient quality observations from the same scene that can provide valuable constraints and useful signals to supervise the calibration process. \n\nThe focus on confidence is both interesting and beneficial, as it addresses ambiguity and enhances accuracy. Confidence metrics effectively aid in selecting precise components for fine-tuning during experiments, even facilitating dynamic scenarios.\n\nThe use of M-estimation for optimization, as opposed to traditional gradient descent, is another noteworthy aspect.\n\nIt is encouraging to see further extensions to Mask3R, which demonstrate the generalizability of this method.\n\nThe paper provides good details for reproducibility.\n\nThe experiments are extensive, covering various 3D downstream tasks and comparing results with state-of-the-art methods in each category. The results indicate that close to “fine-tuning” level performance can be achieved with minimal calibration.", "weaknesses": "The system is designed to adapt to a single scene, which appears to be a limitation of the DUSt3R approach.\n\nTo evaluate performance, the methodology requires half of the test split data (1,000 images out of 2,000), which may limit practical applications. I am curious about the results if only 5%, 10%, or 30% of the data were available for calibration. Furthermore, it remains unclear what portion of the calibrated images comes from the TUM and Waymo datasets.\n\nFurther details are needed regarding the selection of the 161 test scenes. Is this selection consistent with common practices in previously published baselines?\n\nFrom Figure 6, it is evident that performance before calibration is already visually impressive, with significant improvements primarily noted in the corner regions. While the reported metrics are impressive, it remains unclear how the calibration contributes to more visually sensible performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lEPJ5rFi20", "reviewer": "Reviewer_wgpq", "summary": "This submission proposes a method to finetune large 3D models such as Duster on individual scenes. The finetuned parameters include camera intrinsics, extrinsics, point maps, and scales. To avoid finetuning a large 3D model directly, the method proposes to finetune them with LoRA. As a result, the proposed method improves per-scene reconstruction quality and camera estimation accuracy in an efficient way.\n\nIn terms of weaknesses, my primary concerns are: 1) missing a straightforward global optimisation baseline that is also initialised from Duster predictions, and 2) unconvincing comparison with COLMAP and similar works.\n\nOverall, the paper offers an efficient approach to fine-tuning Duster for scene-specific applications, demonstrating notable gains in reconstruction and camera estimation accuracy. However, the unique advantages of this pipeline are not fully clear, especially given the missing global optimization baseline and the somewhat limited comparisons with similar works like COLMAP.\n\nMy initial recommendation leans toward a borderline reject, but I am open to adjusting my rating based on further discussion.", "questions": "1. In Fig 3 and Fig 4, how do Duster predicted confidence maps look like?\n2. Fig 3d: this fig shows calibrated confidences are smaller than predictions. Two questions here:\n    1) Is it always desirable (better) to have a smaller calibrated confidence compared to Duster predicted confidence?\n    2) is there any example where the calibrated confidence is larger than Duster predicted confidence, but still leads to good finetuning results in terms of self-calibration and 3d reconstruction accuracy?\n3. Writing (not affecting rating):\n    1) L181, \"... the global point maps can be further re-parameterized via back-propagation\" -- What does “re-parameterized via back-propagation\" mean?\n    2) L222, \"second regularization term\" -- it seems like there is only one reg term?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents an efficient method to finetune Duster for individual scenes, showing considerable improvements in terms of reconstruction and camera estimation accuracy.", "weaknesses": "***Missing baseline.***\nA major weakness to me is missing a straightforward baseline, which is running a global optimisation that is similar to classical global bundle adjustment for all camera intrinsics, extrinsics, point maps, and scales, given initialisations from Dusters predictions. This baseline would also avoid finetuning Duster weights. I am curious about a fair comparison with this baseline, in terms of optimisation time, and per-scene optimisation accuracy.\n\n***The comparison with COLMAP and similar works is not convincing.***\nThe proposed DUSt3R-Self-Calib always has an initial solution from Duster. It would be more fair to other works if they started from Duster results too.\n\n***Writing (not affecting rating).***\n* The text between Sec 5 and Sec 5.1 should be better sectioned and formatted. Currently, tasks and baselines are mixed together.\n* In Eq. 7, the notation \"W\" is reused. It's already used for image size HW.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uwc5qiMQTp", "reviewer": "Reviewer_3dh5", "summary": "This paper presents a method for self-calibrating 3D geometric models. The authors utilize a pre-trained model to generate predicted point maps and their corresponding confidence scores for input images. To automatically identify and mitigate overconfident predictions that may adversely affect fine-tuning performance, they re-parameterize the confidence term as an optimizable weight. This approach effectively reduces the influence of multi-view inconsistent predictions during the optimization process. Ultimately, the refined point maps and calibrated confidence scores serve as pseudo-labels for fine-tuning the pre-trained DUSt3R model using low-rank adaptation (LoRA) techniques. The authors achieve performance enhancements while maintaining an acceptable increase in computational costs.", "questions": "1)\tDoes Figure 3 solely visualize the prediction confidence and error map of the original DUSt3R method? The caption for Figure 3 should be clarified to reduce ambiguity. Figures 3 and 4 are complementary but they are on separate pages. Readers cannot quickly grasp the points of Figure 3 (a) (b) (c).\n2)\tMinor issues: missing references in Line 817.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)\tThe paper has good writing and is easy to follow.\n2)\tThe authors present a simple but effective pipeline to take advantage of the LoRA technique to improve performance of the 3D foundation model in a self-calibration manner.", "weaknesses": "The paper does not present major weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)