RESfM: Robust Deep Equivariant Structure from Motion
Authors: Fadi Khatib
Yoni Kasten
Dror Moran
Meirav Galun
Ronen Basri
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Reconstruction
Outlier Removal
Structure from Motion
Camera Pose Estimation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Robust Deep Equivariant Structure from Motion

Abstract: Multiview Structure from Motion is a fundamental and challenging computer vision problem. A recent deep-based approach utilized matrix equivariant architectures for simultaneous recovery of camera pose and 3D scene structure from large image collections. That work, however, made the unrealistic assumption that the point tracks given as input are almost clean of outliers. Here, we propose an architecture suited to dealing with outliers by adding a multiview inlier/outlier classification module that respects the model equivariance and by utilizing a robust bundle adjustment step. Experiments demonstrate that our method can be applied successfully in realistic settings that include large image collections and point tracks extracted with common heuristics that include many outliers, achieving state-of-the-art accuracies in almost all runs, superior to existing deep-based methods and on-par with leading classical (non-deep) sequential and global methods.

Published as a conference paper at ICLR 2025
RESFM: ROBUST DEEP EQUIVARIANT STRUCTURE FROM MOTION
Fadi Khatib Yoni Kasten² Weizmann Institute of Science
Dror Moran' NVIDIA
Meirav Galun'
Ronen Basri!
ABSTRACT
Multiview Structure from Motion is a fundamental and challenging computer vision problem. A recent deep-based approach utilized matrix equivariant architectures for simultaneous recovery of camera pose and 3D scene structure from large image collections. That work, however, made the unrealistic assumption that the point tracks given as input are almost clean of outliers. Here, we propose an architecture suited to dealing with outliers by adding multiview inlier/outlier classification module that respects the model equivariance and by utilizing robust bundle adjustment step. Experiments demonstrate that our method can be applied successfully in realistic settings that include large image collections and point tracks extracted with common heuristics that include many outliers, achieving state- of-the-art accuracies in almost all runs, superior to existing deep-based methods and on-par with leading classical (non-deep) sequential and global methods.
INTRODUCTION
Simultancous recovery of camera pose and 3D structure from large image collections, commonly termed Multiview Structure from Motion (SfM), is longstanding fundamental problem in computer vision with applications in augmented and virtual reality, robot manipulation, and more. Classical, as well as some recent deep-based SIM techniques, rely on extracting point tracks, i.e., point matches across multiple frames, solve for camera poses and structure. However, existing heuristics for point track extraction and chaining often return erroneous (outlier) point matches due to significant viewpoint, illumination differences, and repetitive scene structures, adversely affecting the perfor- mance of SfM methods. Designing robust methods that can effectively overcome the effect of such outliers, therefore, has been an active thread of research in the past several decades.
A promising approach to SfM is based on generalizations of projective factorization. This approach uses the observation that if we stack the tracked points in matrix, which we denote by M, and assuming M has no missing entries and is error-free, then there exists a set of scale factors such that scaling each tracked point in M will make it rank 4. Enforcing this constraint allows us to recover both the 3D positions of the points that generate the tracks and the poses of the observing cameras (Sturm) & Triggs 1996). However, prior algorithms required M to be complete and error-free; they were sensitive initialization and were applied only in uncalibrated settings, producing reconstruction up to global projective ambiguity.
A deep network architecture for projective factorization recently proposed in (Moran al. 2021). Their network uses a sets-of-sets architecture (Hartford et al. (2018), which is permutation equivariant to both the rows and columns of the point track matrix. Brynte al. (2023) proposed related architecture that uses atffaattnnion blocks to improve runtime further. Both these networks showed promising results on image collections acquired with single camera. However, as we show this paper (see also Figure]] IT they fail to handle collections of internet photos, primarily because they were not designed to remove outlier matches.
In this paper, we aim to construct robust network for projective factorization that can handle outliers in realistic settings. To this end, we return to (Moran al. 2021)'s 's architecture and enrich it with an outlier removal module. Our module is integrated into the equivariant architecture, allowing it to identify outliers if their motion is inconsistent with other points in the same image or their motion is "The project page:
https: hitps://robust-eee equ vari annttantteee sfm. gi thub. io/

---

Published as a conference paper at ICLR 2025
(a) Our recovery
(b) ESFM recovery
(c) GASFM recovery
Figure 1: 3D reconstruction and camera pose recovery in the presence of outliers. The figure shows reconstruction results (point clouds) and camera poses (in red) obtained with our method (left) and existing deep-hased methods including ESFM (Moran et al. 2021) (middle), and GASPM (Brynte et al. 2023 for scene 0185 (MegaDepth dataset, 30% outliers). It is evident that our method copes well with outheers in contrast to these existing methods.
odds with the motion of other points in their track. We train this module with cross-entropy loss, using labels that are inferred automatically using COLMAP reconstructions, allowing for cross-scene generalization. We further utilize final bundle adjustment (BA) step that robust classification errors. Experiments show that our network improves over the methods of Moran et al. (202T): Brynte
al. (2023) in almost all runs, obtaining oftte-of-the-ar accuracies and runtimes comparable the leading classical methods, including COLMAP, Theia, and GLOMAP (Schönberger & Frahm 2016
Sweeney et al. 2015 Pan al. 2024).
In summary, our contributions are:
1. A deep network for robust multiview SIM, utilizing a sets-of-sets permutation equivariant architecture.
2. A trainable architecture allows for cross-scenc and cross-dataset generalization.
3. Our method is applied to large collections (hundreds of images) of uncontrolled internet photos. This, the best of our knowledge, is the first deep method for simultancous recovery of structure and camera pose that handles such challenging inputs.
4. Our method achieves highly accurate recovery of camera pose and structure, superior to existing deep methods and on par both in accuracy and speed with state-of-the-art classical methods.
5. A benchmark with point tracks and pseudo ground truth computed with COLMAP on the MegaDepth and IDSFM datasets.
2 RELATED WORK
The recovery of camera pose and 3D structure from image collections has been a central subject of research in computer vision in the past several decades, leading to multiple breakthroughs that enabled accurate reconstructions from hundreds and even thousands of unordered images (Agarwal al. 2011 I Schönberger & Frahm 2016 SSnavlly et al. 2006 Wu 2013). In the typical sequential pipeline point tracks are first extracted from the input images. Camera pose and 3D structure are next computed for two images and then updated by processing the remaining images one by one. This pipeline yields highly accurate recovery of pose and structure but can be slow when applied to large image collections. Alternative global approaches attempt to compute camera poses by process of "averaging" the relative rotations and translations estimated from pairwise essential matrices (Martinec & Pajdla 2007 Özyeşil et al. 2017 Kasten et al. 2019). The recent Theia (Sweeney et al. 2015) and GLOMAP (Pan et al.) 2024). in particular, were shown to yield accurate recovery. Related to our approach the projective factorization (PF) method (Sturm & Triggs 1996 Dai et al. 2010 Lin et al., 2017). which is based the observation that the full track matrix is derived from rank four matrix by per-point scale factor. Classical PF algorithms, however, are limited to uncalibrated settings and address neither missing entries nor outliers.
2

---

Published as a conference paper at ICLR 2025
Recent deep-learnngg techniques attempt improve these pipelines by exploiting priors the input images, camera settings, and 3D structures learned with these networks. Existing methods attempt improve keypoint detection and matching (Lindenberger
al. 2021 Sun et al. [202] DeTone et al. 2018 Lindenberger et al., 2023 Sarlin et al. 2020). produce point tracks (He et al.
2023), formulate differentiable alternatives to RANSAC (Yi al. 2018 /Zang al. 2019 Zhaa
ct al. 2021 Sun et al. 2020), or directly infer the relative orientation and location for pairs of images
(Khatib et al.
2024 Laskar al. 2017 Cai et al. 2021 Rockwell et al.
2022 Arnold et al. 2022).
Recent work proposed end-to-end methods for camera pose estimation. Works such as RelPose (Zhang et al., 2022) and its successor, RelPose++ (Lin et al. 2023), harness energy-based -baeed models to recover camera poses from inputs that include the relative rotations between images. Similarly, SparsePose (Sinha et al., 2023) learns to regress initial camera poses which are then refined iteratively. PoseDiffusion (Wang et al.] 2023b) employs diffusion model to refine camera poses. /hang et al. (2024) improves on this by tocusing on purifying the camera rays. These works, however, are only applied to small collections of images (typically <30) and are commonly applied in object-centric scenarios, e.g., as in (Reizenstein et al. (02TT. Recent learnable SIM pipelines such as VGGSIM (Wang et al., (023a),, DUST3R (Wang et al., 2023c), and MAST3R (Leroy et al.] 2024 still
limited to handling only a small number of images, while Ace-Zero (Brachmann et al. (2024) and
FlowMap (Smith al. 2024) applicable video sequences with constant illumination.
Motivated by projective factorization schemes Moran et al. (2021) presented a trainable network for simultaneous camera pose and 3D structure recovery. Arranging the input point tracks in matrix, an equivariant network to row and column permutations, i.e., sets-of-sets architecture, is used to regress the camera poses and the 3D point cloud coordinates. The network trained with unsupervised data by optimizing a reprojection loss and is applied at inference to novel scenes not seen in training. Additional fine-tuning and Bundle Adjustment (BA) are applied to attain sub-pixel reprojection errors. This approach achieved highly accurate pose and structure recovery on large image collections (with several hundreds of images) of outdoor urban scenes. However, has a significant limitation as requires the point tracks matrix, to be almost free of outliers. This greatly restricts its applicability in realistic settings.
GASFM (Brynte et al., 2023) replaces the set-of-sets architecture in (Moran et al. 2021 with graph attention network for increased expressiveness, enabling them to avoid fine-tuning, thereby reducing inference runtime compared to (Moran al. 2021) without compromising the performance. Chen et al. (2024) applies (Moran et al. (021)'ss 's network (with some modifications) to acrial images captured with GPS information under roughly constant camera orientation. Neither Brynte al. (2023) nor Chen et al. (2024) demonstrate results on point tracks contaminated by a significant
portion of outliers.
Our approach extends and improves over the work of Moran et al. (2021) by integrating a multiview in- lier/outlier classification module that respects and exploits the equivariant structure of the network and by introducing a robust BA scheme. This allows us to work with realistic point tracks contaminated with outliers obtained with standard heuristics and still achieve high-accuracy performance.
3 METHOD
3.1 PROBLEM FORMULATION
We assume a stationary scene viewed by m cameras with unknown poses. We obtaín as input a (sparse) point track tensor M that includes 2D observations of n 3D points viewed by partial sets of cameras. We further assume in this work that the cameras are internally calibrated. A camera matrix therefore is expressed in the form of P. [R.]t.]. where R2 E 50(3) is rotation matrix. With this notation, the camera is placed at the position - R t,. We denote by X, E I the j'th 3D scene point and by X,y E IR.2 its observed position in the 7th image. The set Tj = {xi,,, Xigh with C2 (úr, i..) } C [m] represents the j'thh track, associated with the j'th 3D point. These tracks are generally constructed in preprocessing step by employing heuristics and, therefore, are contaminated by small displacement errors (noisy measurements) and outliers.
We arrange the tracks Ti,.... T, in the columns of the measurement point track tensor M. The tensor M is of size m
so the rows of M correspond to the cameras, and its columns correspond
3

---

Published as a conference paper at ICLR 2025
tracks
06) K, 011e
3D points Head Hpix
Input imazes
alter
Permutation Equivariant feature encoder E
Cameras Head
Robust
Haus
Bundle- Adjustment
Outlier Head
Hauttirr
10 pee prodeced ourders or Hiturrra reteet oo soie
Figure 2: Network architecture. Our network comprises four main steps: (1) Given point track tensor (left), an cquivariant feature encoder module (light blue) produces latent feature representation. (2) The outlier head (gray) applies inlier/outlier classification. Predicted outliers are then removed from the input track, and the encoder is fine-tuned for 1000 epochs. (3) 3D point locations and camera poses are predicted by the point head (yellow) and the camera head (green), respectively. (4) The predicted camera poses and the point cloud finally optimized by robust bundle adjustment (orange).
the unknown 3D points whose contaminated projections are given. We set (MaJ1, May2) = Xiy i Cj and otherwise leave Mi11 and May2 empty.
We aim to recover the pose parameters of the m camera matrices and the 3D positions of the n feature points. Additionally, we seek to classify each track point as either an inlier an outlier. The output of our network will comprise three matrices: P E Tmmx7 for the set of cameras matrices, X E TRax3 for the 3D structure, and O R for the track point classification results. We represent camera pose by a rotation quaternion q, E I2 and a translation vector t, E R Classification results in O represented by scores in [0, 1), where X,j with O.j 0.6 is considered outlier. (The threshold 0.6 was determined by a hyperparameter search.) Note that we classify individual points as outliers, while other points in the same track may be classified inliers.
3.2 NETWORK ARCHITECTURE
We use architecture that is equivariant to row and column permutations of the tensor M, equivalent respectively independent permutations of the cameras and the 3D points. It comprises four modules: permutation equivariant feature encoder, followed by inlier/outlier classification module, pose, and structure regression modules (see Fig. 2
Permutation equivariant feature encoder. This module takes input the tensor M Rmxxx and outputs a latent representation xxxx where = 256. It comprises three permutation equivariant layers with linear maps interspersed with pointwise non-linear ReLU function. A linear equivariant map is the core of permutation equivariant networks. In our case, permutation equivariance should apply to the rows (cameras) and columns (3D points) of the tensor M. Hartford et al.) (2018) showed that the space of linear maps that keep the permutation equivariant property of a single channel (i.e., the input and the output are matrices) spanned by the identity, the row sums, the column sums, and the matrix sum. This can be applied to multiple input and output channels from a feature space with d channels mxxxd to feature space with d' channels IR mmxxxd in the following way:
Ma + W. L(M)); = WiM, + W Maj + W3
Ma + b,
(I)
1=1
k= 1=1
where My E Rdd represents the vector of entries of M at the (i.j) position. W. E Rd'xd for
i 1,.... 4, and b udd are learnable parameters. We follow Moran et al. (2021) and replace sums by averages over the non-empty entries of M, yielding invariance to the number of observed projections.
Inlier/Outlier classification module. Given the latent representation mxxxd provided by the equivariant feature encoder, this module. Houtters employs three MLP layers followed by sigmoid function. The output is a matrix O e [0, 1]mxX* expressing the probability of each entry being an outlier. Note that this module respects the equivariance property, allowing our model to classify track
4

---

Published as a conference paper at ICLR 2025
points based their relations to other points of the same track as well other points within the same image.
Pose and structure regression modules. Utilizing the latent representation Rxxux from the cquivariant feature encoder, the pose and structure heads, Heams and Hps. consist of three MLP layers each. The first head, Heams- processes the pooled average of the features along the columns Rmxd, where each camera's s features are encoded into a vector in R and outputs a tensor in T=xx This tensor encapsulates the camera's translation vector (first three values) and orientation (last four values, represented as quaternion), collectively forming the camera's projection matrix. The second head, Hpu. processes the pooled average of features along the rows "xd where each scene point encoded by a vector in Rd, and outputs a tensor in R=x3,, representing the coordinates of the scene points.
3.3 LOss FUNCTION
Our loss function combines two terms
L = Coutiers reCrprojcctionn
(2)
where the hyperparameter a 10 balances the two terms and determined by hyperparameter search. The outlier classification loss, Loutiees, employs a Binary Cross-Entropy (BCE) loss. The second part of the loss Lepprooettion an unsupervised objective, and it includes reprojection and hinge loss terms. The reprojection loss aims to minimize the error between projected scene points and their detected positions in the image, similar to the objective in bundle adjustment. A hinge loss is also used to prevent the prediction of non- non-positive depth values, thus ensuring a physically plausible scene reconstruction.
m n
Creprojection
EtySuj.
1=1
where p = プラ [Ty] the number of measured projections, and Eij E {0, 1 is indicating whether point X, is detected image 4.
Tij+ hijs
P.X; 2 P. < h
86)
and
Tij (
P!
x
x
and hyj max(0, h - P X2). where h is a small constant (in our setting, 0.0001). hit therefore is the hinge loss for the depth value of X, in image I,
3.4 TRAINING
During training, processed all training scenes sequentially for each epoch. For each scene, randomly selected subset of 10%-20% of the images. We employed validation set for early stopping, selecting the checkpoint with minimal error. More details can be found in Appendix A.
3.5 INFERENCE
After training our network, we the network unseen scenes. We first apply inlier/outlier classification predictions to remove outliers, and then fine-tune the network for the tested scene. This yields predictions for camera poses and 3D point locations. Having those predictions, we apply our robust Bundle Adjustment procedure.
Fine-tuning. At this step, we leverage our inlier/outlier classification prediction and remove the points predicted outliers from the point tracks tensor M (using similar to (Moran et al., 2021), we fine-tune the network for the tested scene by minimizing the unsupervised reprojection loss equation for 1000 epochs.
the threshold 0.6). Subsequently,
5

---

Published as a conference paper at ICLR 2025
Robust Bundle Adjustment. As common in SfM pipelines, post-processing step of applying bundle adjustment necessary to achieve sub-pixel reconstruction accuracy. Standard BA was found to be ineffective as it was unable to handle the remaining outliers. We therefore utilize Robust Bundle Adjustment process:
1. Apply bundle adjustment (BA) to the predicted cameras and predicted 3D points.
2. Remove 3D points with projection error higher than 5 pixels and remove 3D points viewed in fewer than cameras.
3. If the view graph of the point tracks becomes unconnected due to the removals, then we take the largest component and discard point tracks that originate from the removed images.
4. Triangulate the remaining point tracks and apply a second round of (standard) BA.
More details on the robust BA stage can be found in Appendix A.
EXPERIMENTS
4.1 DATASETS
Our network trained on scenes from the MegaDepth dataset (Li & Snavely 2018). It is then tested on both novel scenes from the MegaDepth dataset and in cross- generalization tests the 1DSfM dataset (Wilson & Snavely 2014), Strecha (Strecha et al. 2008). and BlendedMVS (Yao) al. 2020). These datasets offer diverse range of real-world scenes, which are instrumental in assessing the robustness and versatility of our proposed architecture across different environments and challenges.
The point tracks, which our network takes as imput, are constructed by concatenating pairwise matches between images within each scene. These matches are obtained by applying RANSAC SIFT matches, ensuring a robust selection of correspondences by mitigating the influence of outliers. The detailed procedure for constructing these point tracks, including parameter settings and algorithmic choices, is provided in Appendix
MegaDepth (Li & Snavely, 2018). The MegaDepth dataset comprises 196 different outdoor scenes, each populated with internet photos showcasing popular landmarks around the globe. To facilitate a comprehensive evaluation, we divide the dataset into two groups based on the number of images per scene: (1) scenes with fewer than 1000 images, and (2) scenes with more than 1000 images. From the first group, we randomly sampled 27 scenes to serve as our training dataset, along with four scenes designated for validation purposes. For the test set, randomly selected 14 scenes from the first group. Moreover, from each scene in the second group, we randomly sampled 300 images represent a condensed version of the scene. These samples form a part of the test dataset, introducing a significant challenge for Structure from Motion (SfM) methods due to the reduced number of images representing vast and complex scenes. In Table T the scenes above the middle rule belong to Group 1 of scenes with fewer than 1000 images, while the scenes below the rule belong to Group
IDSFM (Wilson & Snavely 2014). The IDSFM dataset is renowned for its collection of diverse scenes reconstructed from community photo collections. It includes variety of urban locations, making it valuable resource for evaluating Structure from Motion (SfM) and multi-view stereo algorithms. For our purposes, the dataset offers a challenging yet realistic setting to evaluate our architecture's is effectiveness in dealing with large-scale reconstructions. Specifically, we train the models on MegaDepth dataset and assess its generalization to IDSFM dataset.
Strecha (Strecha et al. 2008). The Strecha dataset is widely used for benchmarking 3D reconstruc- tion algorithms and consists of five outdoor scenes. provides high-resolution images alongside ground-trutt truth data acquired with LIDAR system. However, each scene includes only small (S 30) number images acquired with a single camera. We perform our tests on four of these scenes.
BlendedMIVS (Yao et al.) 2020). The BlendedMVS is synthetic dataset built by reconstructing textured meshes and rendering them into color images and depth maps, which are blended with the original inputs to create realistic data, thereby generating ground truth for camera poses.

---

Published as a conference paper at ICLR 2025
Ground truth camera poses. Challenging datasets such MegaDepth and IDSFM lack ground truth. Therefore. as is common in the field Jiang et al. (2013): Wilson & Snavely (2014); Cui
& Tan (2015): Ozyesil & Singer (2015); Brynte et al. (2023); Wang et al. (202aa): Zhang al. (2024), we utilize COLMAP, a state-of-the-ar incremental Structure from Motion (ShM) method, to establish ground truth camera poses for the scenes in those datasets. COLMAP stands one of the
most popular solutions in the field due to its robust performance in reconstructing 3D models from unordered image collections. We apply COLMAP directly to the scene images obtain the ground truth poses. Additionally, we show results with the smaller datasets Strecha (Strecha et al. (2088) and BlendedMVS Yao et al. 2020) for which ground truth camera poses are available.
4.2 BASELINES
ESFM (Moran et al. 2021). We compare our method to this equivariant SIM method where, for fairness, we replace BA with our robust BA. The model is trained on MegaDepth dataset and evaluated both on MegaDepth and IDSFM. We conduct two distinct evaluations: one in which ESFM trained and tested on the original point tracks, which include outliers (denoted as ESFM), and another evaluation where the model is trained and tested on the same point tracks, but in which outliers have been removed (this version denoted as ESFM*). By testing ESFM on the original contaminated tracks, we evaluate the method's 's ability to handle realistic settings. A comparison with outlier-free tracks provides an empirical upper bound for our method.
GASFM (Brynte et al. ). We compare our method to GASFM, which replaces the set-of-sets architecture in ESFM (Moran et al. (021) with a graph attention network. This modification increases expressiveness, enabling them avoid fine- tuning and thereby reduce inference runtime compared to the traditional SfM method (Schönberger Frahm 2016). without compromising performance when tested the outlier-free Olsson's 's dataset (Olsson & Enqvist 201T). We trained and tested GASFM on the original MegaDepth point tracks (which include outhiers) and also tested it on the IDSFM
dataset scenes.
To allow a fair comparison, in both ESFM and GASFM, the inference step is followed by 1000 epochs of fine-tuning.
VGGSIM WWag et al., 2024) is a differentiable, end-to-end trainable SfM pipeline, simplifying some of its components streamline the process while maintaining accurate 3D reconstruction capabilities.
MAS13R (Leroy et al., 2024) is a pipeline for SIM that merges pairwise pointmap predictions through an optimization- based global alignment procedure to handle image collections effectively.
Theia (Sweeney al. 2015). A widely recognized global SfM pipeline that begins by estimating camera rotations using rotation averaging, followed by translation averaging to estimate camera positions. It concludes with global triangulation to reconstruct the 3D point cloud and a final bundle adjustment, similar to other global SfM techniques.
GLOMAP (Pan et al. 2024). GLOMAP is a newly proposed global SfM pipeline that addresses the limitations of previous global methods, which were considered efficient but less robust than incremental approaches. Instead of relying on separate translation averaging and point triangulation, GLOMAP combines them into single global positioning step, optimizing both camera positions and 3D structure simultaneously.
4.3 METRICS AND EVALUATION
We evaluate our results using camera position and orientation errors. Specifically, after performing a similarity alignment per scene, we compare our camera orientation predictions with the ground truth ones by measuring angular differences in degrees. Similarly, we measure differences between our predicted and ground truth camera locations. For a fair comparison, both our method and all the baseline methods (except VGGSÉM, which is applied directly to the input images) were run with the same set of point tracks. For all methods, we apply a final post-processing step of our robust bundle adjustment. When highlighting the best results in the tables, do not compare to ESFM
7

---

Published as a conference paper at ICLR 2025
Table I: MegaDepth experiment. For each scene, we show the number input images (denoted N.) and the fraction of outliers. For each model, we show the number of images used for reconstruction (denoted N.) and mean values of the rotation (in degrees) and translation errors. (Above the middle rule are Group scenes with <1000 images; below Group 2 scenes with >100 images, subsampled to 300 for testing,) Winning results are marked in bold and underlined. Yellow represents the best result among the deep-based based algorithms and green among the classical algorithms.
Same
N₂
Outlievs
Ours
BSTM
GASFM
Theia
GLOMAP
DSFM
Nr
Rot
Trans
N-
Rot
Truss
N-
Roi
TTrat
Nr
Rot
Trans
N;
Rot
Trans
N,-
Rot
Trans
0238
522
44.6
283
2.61
1.325
11.49
1.088
76
9.39
0,969
600
1.21
499
7.
512
3.06
0060
528
41.6
503
0.29
0.029
303
14.92
2.167
258
18.48
2.736
525
0.85
0.124
522
0.11
0.048
524
0,01
0,005
0197
870
40.7
667
4.22
0.353
281
9.62
0.980
12:23
1.678
855
1.16
0.237
0.48
0.129
825
0.41
0,050
0694
763
40.1
537
3.77
0.750
93
14.91
1.772
359
2.27
0.322
742
0.75
0.160
717
0.88
3.907
643
4.55
1,018
571
38.8
346
1.25
0.389
22.04
2.712
274
2.756
554
5.83
2.216
568
7.46
599
0.12
0081
635
31.3
506
0:64
0.058
568
4.40
0.558
574
3.05
0.245
632
0.57
0.372
614
0.08
0100
596
21,00
1.004
0076
55%
524
0.37
0.094
0.655
0.11
0.015
540
0.78
0.130
0.17
0.042
547
0.04
0.006
0185
168
30.0
350
0.06
0.010
261
1.76
0.271
254
1:36
0.164
365
0.41
0.004
365
0.16
0.051
130
38.49
2,478
00:88
512
24.2
474
4.69
0.178
469
4.00
0.235
481
1.44
0.073
507
0.41
0. 105
506
0.13
0.224
508
0.09
0.006
00224
155
23.0
300
2.03
0.
0.356
271
9.08
1.320
900
9.50
2,546
9.55
0.56
0.219
339
0.15
104
343
2.78
0.568
0223
214
17.0
204
3.76
0.510
191
11:97
2.272
194
13:69
2.649
212
3.34
0.519
21.4
1.
0.275
211
0.09
0.017
5016
26
0.2
28
0.11
0.016
28
0.09
0.015
28
0.00
0.015
28
0.10
0.061
28
0. 0.08
0.046
26
0.04
0.009
00:66
440
14.6
399
0.95
0.043
1.68
0.082
425
0.45
0.019
434
0.25
0.112
4.00
0.03
0.007
53
37.09
1,387
0099
299
47.4
190
3.53
0.709
0.862
128
8.13
1,116
297
3.28
0,664
235
0.15
0.005
243
6.22
1,075
1001
285
433.9
251
1.70
243
4.40
1,846
261
3.26
1.143
276
7.97
4.014
28.1
4.56
3.817
250
0.12
0.065
0231
296
246
0.84
0.065
214
0.090
209
0.91
0.058
286
1.57
0.322
279
0.73
0.134
264
0.56
0,022
0411
299
29.9
273
0.18
0.020
188
15:39
1.650
232
2.95
0.304
203
0.39
0.196
269
0.19
0.148
288
0.08
0.013
205
0.20
0.044
0.18
1.13
0.65
0112
299
25.8
284
0.28
050
255
1.55
0.403
27M
1.79
0.478
20.
2.51
0,60%
295
0.1.
0.301
155
21.00
3,770
0147
208
0.522
10.89
6.36
6.75
3.23
0.215
01:48
287
24.6
197
0.60
0.99
206
1.64
0.153
209
1.73
0.135
275
133.98
1.558
28.
22.73
2.646
249
0.94
0.063
0446
208
265
0:004
0.193
1.71
1.23
0.20
0022
297
21.2
27.
0:20
0:39
241
1.38
0.184
263
0.54
0.062
2106
0.58
0,160
281
0.22
0.087
289
0.30
0.646
208
0.36
0.908
0.25
1.27
0.008
0015
284
20.6
215
1.04
0.167
142
5:00
0.920
149
14:02
2.105
244
2.21
0,559
27.4
0:228
0.95
165
4.51
0.941
0155
295
293
0.68
0.138
294
0.34
294
0.77
205
0.33
0496
297
19.2
251
0.35
O. 0.55
281
3.59
0.311
273
0.68
0.061
285
1.40
0.550
291
0.44
0.303
290
0.25
0.008
1589
299
290
0.14
0.131
2.71
0.508
265
0.82
299
0.07
0.041
296
0,40
0012
200
16.3
267
0.40
0. 027
291
5:20
0.327
294
0.88
0.114
129
1.04
0,318
295
0.51
0.121
294
0.47
0.044
0104
284
0.29
0.029
220
24.40
2.174
228
4.50
265
17.05
250
19:59
200
0019
299
15.4
250
0.06
0.008
267
9.34
0.329
293
2.34
0.113
271
0.81
0.250
206
0.02
0.025
296
4,90
0.180
0063
293
14.5
262
0.46
262
0.456
257
0.44
0.040
266
0.92
2858
0.32
01:30
285
14.4
192
0.20
0.023
192
1.46
0.054
194
2.27
0.077
187
1.20
0.344
281
2.00
0.909
282
1.59
0.179
0080
284
12.9
189
0.59
0.096
137
0.325
1.39
2.18
0.104
278
2.62
0.868
289
1.52
0.237
163
1.92
0.173
0240
295
11.9
270
3.33
0.265
274
1.69
0.170
272
2.54
0.227
278
1.31
0,471
204
0.39
0.195
296
0.27
0.111
0007
290
11.7
172
0.91
0.041
260
38.74
2.254
280
1.87
0.101
277
1.24
0.174
200
0.19
0.085
286
1,50
0,364
Meas
379
25.6
296
1.29
0.169
239
6.77
0.790
267
4.53
0.630
346
2.42
0.556
353
2.51
0.662
319
4.45
0.443
4.4
RESULTS
Our results for the MegaDepth and IDSFM test scenes are shown in Tables and 2 respectively. Each table lists the number of input images (N.), the fraction of outlier track points, and our results compared the baseline methods. For each method, we provide the number of registered cameras (N₂) and mean camera orientation and position errors. It can be seen that our method outperforms both ESFM and GASFM in almost all runs, achieving accurate results that are close to what is obtained with ESFM on clean tracks (ESFM*). Our results are also on par with state-of- -of-th--rrrrrr classical methods, including Theia and GLOMAP, often yielding superior translation recovery (but handling fewer images). This demonstrates the utility of our method in realistic settings.
We further tested our method on the smaller Strecha and BlendedMVS datasets which include ground truth measurements. Our method is more accurate than the deep-based VGGSIM and MASt3R (that cannot run on the larger datasets) and is on par with the classical methods, see Tables and 4
Qualitative results. Figure [ shows example of 3D reconstructions and camera parameters obtained using our method compared to those obtained with ESFM and GASFM. These results clearly demonstrate that our method produces superior 3D reconstructions, effectively handling outliers in contrast to the other baselines. Additional qualitative results are provided in Appendix A
Runtime and resources. Table 5 compares the runtimes of our method with the classical COLMAP, Theia, and GLOMAP when applied to the point tracks generated in our preprocessing stage. Our approach is significantly faster than COLMAP and GLOMAP but is slower than Theia We note that
8

---

Published as a conference paper at ICLR 2025
Table 2: IDSFM experiment. For each scene, we show the number of input images (denoted N.) and the fraction of outliers. For each model, we show the number of images used for reconstruction (denoted N,)) and mean values the rotation (in degrees) and translation errors. Winning results are marked in bold and underlined. Yollow represents the best result among the deep-based algorithms and green among the classical algorithms.
Scrne
N₂
Outlurek
Ouns
ESRM
GASFM
Thds
GLOMAP
ESFM
Tlans
Ket
Trans
Trans
Rot
Trans
TTras
Alatio
0.315
4.53
0.919
0.460
1.433
657
2.45
1,520
526
bills Island
227
25.1
21.4
0.82
0.122
196
21.13
2,053
195
21.91
2,081
213
5.01
1.527
21.00
0.58
0.155
230
0.35
0,033
Madrd Metsopolis
19.56
1.946
2.305
1.22
290
2.664
Mortnal Notro Dame
31.7
940
2.82
0.352
909
10:13
1.733
1.557
144
0.80
0.211
414
0.000
Notre Dame
549
35.6
017
1.2
0.231
057
1
0.236
899
1.76
0.232
314
3,70
0.826
644
2.73
0. 389
526
0.72
0.011
NYC
0.667
Piance Depaloo
88.1
249
2.20
0.188
10.77
0.997
195
1.063
331
0.80
0.188
4,31
0.644
Towes of Londos
467
27.0
94
0.07
0.026
196
22.02
2,239
152
32.35
2,306
448
6.61
1,189
460
0.81
0.138
213
13.08
0.704
Caihasin
Yorkminster
433
29.0
3011
14.56
1,408
215
10.03
0.037
251
13.01
0.846
350
8:35
1.91
418
0.05
0.336
389
12,63
0.9
Table 3: Strecha experiment. For each scene, we present the number of input images (denoted N₂) and the fraction of outliers. For each model, we show the number of images used for reconstruction (denoted N,) and the mean values of the rotation errrr (in degrees). translation error (in meters) and runtime (in seconds). The best results are marked in bold and the second best are underlined.
Some
Out
X,
Sut
Tian
Yane
X.
Ret
MASKEK
Tram
Tixe:
X.
Iox. Tran
VIDGS04
Tax:
X,
Ret
Thdia
Tian:
Tix:
X
Tixa
Tixc
X:
Rei
Tren
Tix
P10
10
45
10
U225
2.
19
0442
0065
15
10
1165
0036
103
10
DICE
09
10
8,223
560
0.187
0.005
125
0033
154
1.5
14
0363
3107
1092
0.05
22.0
0.861
0.00
531
Bam Samas P25
25
34
was
84
25
0366
0167
25
1158
199
194
25
0826
9186
24
25
(64
25
0.138
0.01
76.6
Maan
15
27
138
6027
wass
139
8488
0004
345
135
8178
198
153
153
0.85
14
135
16.8
13.9
0.333
332
Table 4: BlendedMVS experiment. For each scene, we present the number of input images (denoted N.) and the fraction of outliers. For each model, we show the number of images used for reconstruction (denoted N,) and the mean values of the rotation error (in degrees), translation error, and runtime (in seconds). The best results marked in bold and the second best are underlined.
Sone
x,
ous
Rot
Oaze
Tisza
Tix
Box
MASIR
Tune
Tue
X
Rut
VGGSIM Than
Taxe
X.
Rut Tin
Thii
Tixe
X.
Rot
COLMAP
Tius
Tiu
Ruc
GLOMAP
Trams
Tine 198
sorao 29
20
noas
0.01
518
35
008 annus
75
75
ILBSS
106
C1877
1971
117
0227
30
0065
0.63
0,4622
67
65
36
66
0007
0.5007
66
0927
0045
815
65
0.72
0.174
S
65 0.09
0.008
21 66
0.664
0,4022
126
66
0.088
6.001
392
the fine-tuning phase is the most time-consuming part of our pipeline. furthher compares the utilization of resources of the deep-based methods. Our method utilizes slightly more parameters than (Moran al., 202T) and is two orders of magnitude smaller than (Brynte et al. 2023). In terms of memory usage and proceeding speed our method is the most efficient.
Classification performance. Table 7 (left) presents various classification metrics to assess the performance of our inlier/outlier classification module. After removing the predicted outliers, our method significantly decreases the percentage of outliers in the point tracks. These classification results enable the robust BA module to perform well and produce accurate reconstruction and camera pose estimation, yielding another significant decrease in the outlier ratio. Removing these remaining outliers is challenging since these outlier matches survived the RANSAC preprocessing
4.5 ABLATIONS
For ablations, we first examine the impact of our permutation sets-of-sets equivariant architecture. We replace our inlier/outlier classifier module with a set equivariant PointNet architecture. The input in this experiment consists of set of quadruples, (z, y, t), where (æ, y) are the keypoint coordinates, e represents camera id, and t denotes track id. As shown in Table 7 our equivariant architecture achieves superior classification accuracies, justifying the importance of using ofts-of-sets equivariance.
Next, examine the importance of the equivariant features. We compare method, which is trained end- -o-ddd to (1) the case that only the classification head is trained while the feature encoder is frozen (i.e., trained as in ESFM without outlier classification), and (2) the same architecture but

---

Published as a conference paper at ICLR 2025
Table 5: Runtime. Given the same contaminated point tracks, we compare the runtime our proposed method
to classical methods, including COLMAP, Theia, and GLOMAP.
Scone
Cuxtacts
Ixforece deei)
Fire tavigg (Ses)
Oars
Rx (Soo)
Texa (Mies)
N,
N2
(ral(MMss N, N,
COLMAN MAP
Trut (Miw))
Thets
N,
N,
Tieal
GLOMAP
N. N,/*
Alien
67%
0.301
6263
15S4
173
161
B3T
156
551
41.4
X10
551
15:4
Tii liaed
26:1
0.001
0014
66.1
355
334
75-
110
223
150
111
213
201.5
1:7
214
70.4
Menqells
331
24.4
0.001
296.1
61.2
536
341
42.1
26.1
123
12.6
1.1
335
45.3
Morma Nor Dore
445
31.3
0.011
190/2
174.5
6.1
346
56.7
35.0
447
12.5
37
422
114.6
13.5
644
32.4
Narz Dane
35.6
0.001
1001.35
2334
22.
317
23.3
126
546
7.5
11A6
534
66,0
21.3
543
25.1
NVC Libary
130
33.6
0.934
123.35
88.0
431
234
35.7
26.6
330
12.4
1.3
314
206.1
1,5
123
-6.3
Paos cell Popdle
0.002
916
100
17
341
92.6
96
354
34.9
3.0
198.
20
131
36.1
Seecrrs London
457
27.0
0.033
21146
860
39
96
1539
55.0
401
1:2
142.5
233
685
15.5
Marro Cihedda
634
11.4
0.01
610.5
9535
253
679
360
68.0
824
8.3
1123
683
41.
133
191
Yorkndeskrr
-32
20.0
0.02
370
1923
331
459
31.4
419
133
29
390
1353
141
618
261
Table 6: Resources. Performance comparison between different methods in terms of number of parameters, maximum memory usage, and processing speed (images per minute) averaged over the ddsfm scenes.
Method #Farums (mallions)
Ours
ESFM
GASPM
0.73
0,66
145.17
Memory (GBls)
25.7
Image/minutes 7
45.3
40.9
10.7
with the classification head removed (equivalent ESFM). It can be seen in Table that training the feature encoder an end-to-end schedule makes a crucial impact on the accuracy of our method.
Finally, in Appendix D we validate the importance of our Robust BA; we compare it to standard BA, showing that the accuracy is significantly improved with our Robust BA.
Table 7: Classification metric and architecture ablation. Inlicr/outlier classification accuracies and the fraction of outliers predicted with our cormutation-qquivaiiaa network (sets of sets) compared to an alternative set network (PointNet architecture).
Ours Precison (outiei))
PointNet architecture
Some
N: Outlies bijat
Recall Gilles)
Recll (cutliss)
F-sone (oulleen)
Outlions Pretsced
Cullec/ Robust BA
Rocall (Gallws)
Recall (cetlees)
Predakis (oatliess)
F-jore (orlien)
Outlicnf Predktee 1
10.5
74,3
15.7
Filx blaed
25.1
61.9
10.4
16.6
89.7
13.9
5.7
11
76.5
11.8
74.01
Mades Menopelis
111
62.6
63.1
51.7
66.3
27.9
36.9
16.
71.6
41.
K3.9
13.5
Marama thme
11/7
61.3
7044
66.4
66.1
10.3
118
71.8
119
16.
28.5
NYC Libry
56.7
N1.8
46.5
89.1
9.2
56.0
710
15.2
Papza di Topala
116
11.1
56.7
7K.1
67.5
59.1
16.3
7.5
26.
61.1
15.
084
26.5
Toweral
28.3
44.8
2.9
500
26.1
2000
50A
37.5
89.3
10.2
269
283
Mon
651.9 31.9
26.1
72.1
44.1
54.1
19.0
11.5
36.1
56.3
316
04.9
28.9
Table 8: Encoder ablation. For each scene, we show the number of input images (denoted No) and the fraction of outliers. For each model, show the number of images used for reconstruction (denoted N.) and the values of the rotation (in degrees) and translation errors.
Scese
N_.
Datters'
Bux
Chrrr
Tian
N:
Pronre Firevir
Rui
Trao
X,
No Flazoder (FST
Bus
Tran-
Alamo
571
112.6
3.86
11515
513
0.:54
457
4.55
0.3338
Mdrd
577
121
196
21.13
Libras
330
5.90
0.429
0.42
1V7
4.42
0.546
Paana Popele
196
115.1
2.20
0330
301
10.79
1.006
Tower of Londsn
407
27.11
941
0.67
0.026
141
25.91
1.000
196
33/02
2,32
Viors Calacial
924
31.4
479
1.52
0.112
523
386
0.33
551
6.3
0.533
Yoiinzkkr
432
23.0
14.44
1.49
354
15:24
1.83
215
10.03
0.636
5 CONCLUSION
We present permutation equivariant architecture for robust multiview structure from motion. By integrating a sets-of-sets equivariant inlier/outlier classification module, our proposed method copes well with point-track tensors contaminated with many outliers originating from scenes that include hundreds of images. In addition, we modified the bundle adjustment module to make it robust enough to handle classification errors. Our method successfully handles challenging datasets that include hundreds of uncontrolled internet images, achieving highly accurate recovery, superior to existing deep methods and on par with ofate-of-the-art classical methods. However, we observed cases in which method uses only subset of the input cameras. This occurs due to excess removal of predicted outliers, which might yield an unconnected viewing graph. We plan to address this limitation in future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported in part by the Israel Science Foundation, grant No. 1639/19, by the Israeli Council for Higher Education (CHE) via the Weizmann Data Science Research Center, by the MBZUAI-WIS Joint Program for Artificial Intelligence Research and by research grants from the Estates of Bernice Bernath and Marni Josephs Grossman; Joel B. Levey; Tully and Michele Plesser and the Anita James Rosen and Harry Schutzman Foundations.
REFERENCES
Sameer Agarwal, Keir Mierle, and Others. Ceres solver. http://ceres-solvee. .rg
Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome day. Communications of the ACM, 54(10):105-112 2011.
Eduardo Arnold, Jamie Wynn, Sara Vicente, Guillermo Garcia-Hernando, Áron Monszpart, Vic- tor Adrian Prisacariu, Daniyar Turmukhambetov, and Eric Brachmann. Map-free visual relocaliza- tion: Metric pose relative to a single image. In ECCV, 2022.
Eric Brachmann, Jamie Wynn, Shuai Chen, Tommaso Cavallari, Áron Monszpart, Daniyar Tur- mukhambetov, and Victor Adrian Prisacariu. Scene coordinate reconstruction: Posing of image collections via incremental learning of a relocalizer. arXiv preprint arXiv:2404. 2024.
Lucas Brynte, José Pedro Iglesias, Carl Olsson, and Fredrik Kahl. Learning drructureffrom-motti motioo with graph attention networks. arXiv preprint 2tti::200..1984 2023.
Ruojin Cai, Bharath Hariharan, Noah Snavely, and Hadar Averbuch-Elor. Extreme rotation estimation using dense correlation volumes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13566-14575, 2021.
Zequan Chen, Jianping Li, Qusheng Li, Bisheng Yang, and Zhen Dong. Deepaat: Deep automated aerial triangulation for fast uav-based mapping. 2024.
Zhaopeng Cui and Ping Tan. Global frotuurefrrommmotion by similarity averaging. In Proceedings of the IEEE International Conference on Computer Vision, pp. 864-872, 2015.
Yuchao Dai, Hongdong li, and Mingyi He. Element-wise factorization for n-view projective recon- struction. pp. 396-409, 09 2010. ISBN 118-3-642-155600 doi: 10. 10.10007/78-3-644 15561-1 1-6611112999
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest. point detection and description. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 224-236, 2018.
Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981.
Jason Hartford, Devon Graham, Kevin Leyton-Brown, and Siamak Ravanbakhsh. Deep models of interactions across sets. In International Conference on Machine Learning, pp. 1909-1918. PMLR, 2018.
Xingyi He, Jiaming Sun, Yifan Wang. Sida Peng, Qixing Huang, Hujun Bao, and Xiaowei Zhou. Detector-free structure from motion. arXiv preprint arXiv:2306. 15669, 2023.
Nianjuan Jiang, Zhaopeng Cui, and Ping Tan. A global linear method for camera pose registration.
In Proceedings of the IEEE international conference on computer vision, pp. 481-488, 2013.
Yoni Kasten, Amnon Geifman, Meirav Galun, and Ronen Basri. Algebraic characterization of essential matrices and their averaging in multiview settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5895-5903, 2019.
Fadi Khatib, Yuval Margalit, Meirav Galun, and Ronen Basri. Leveraging image matching toward end-to-end relative camera pose regression, arXiv preprint 2..::22/1.119551 2024.
11

---

Published as a conference paper at ICLR 2025
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Zakaria Laskar, laroslav Melekhov, Surya Kalia, and Juho Kannala. Camera relocalization by computing pairwise relative poses using convolutional neural network. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 929-938, 2017.
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. arXiv preprint urXiv:2406.07556 2024.
Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2041-2050, 2018.
Amy Lin, Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose++: Recovering 6d poses from sparse-view observations. arXiv preprint 23Kiv:3305.04222. 2023.
Yang Lin, Li Yang, Zhouchen Lin, Tong Lin, and Hongbin Zha. Factorization for projective and metric reconstruction via truncated nuclear norm. In 2017 International Joint Conference on Neural Networks (ICCN), pp. 470-477, 2017.
doi: 10.1 101.1//CNNN.0117796551.
Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Marc Pollefeys. Pixel-perfect structure-from from-motion with featuremetric refinement, In Proceedings of the IEEE/CVE interna- tional conference on computer vision, pp. 5987-5997, 2021.
Philipp Lindenberger, Paul-Edouart Sarlin, and Mare Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV, 2023.
David G Lowe. Distinctive image features from ccale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Daniel Martinec and Tomas Pajdla. Robust rotation and translation estimation multiview recon- struction. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2007.
Dror Moran, Hodaya Koslowsky, Yoni Kasten, Haggai Maron, Meirav Galun, and Ronen Basri. Deep permutation equivariant structure from motion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5976-5986, 2021.
Carl Olsson and Olof Enqvist. Stable structure from motion for unordered image collections. Image Analysis: 17th Scandinavian Conference, SCIA 2011, Ystad, Sweden, May 201 I. Proceedings 17, pp. 524-535. Springer, 2011.
Onur Ozyesil and Amit Singer. Robust camera location estimation by convex programming. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2674-2683, 2015.
Onur Özyeşil, Vladislav Voroninski, Ronen Basri, and Amit Singer. A survey of structure from motion Acta Numerica, 26:305-364, 2017.
Linfei Pan, Dániel Baráth, Mare Pollefeys, and Johannes L Schönberger. Global structure-from- motion revisited. In European Conference Computer Vision (ECCV), 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalía Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, begghperformance deep learning library. Advances in neural information processing systems, 32, 2019.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10901-10911, 2021.
12

---

Published as a conference paper at ICLR 2025
Chris Rockwell, Justin Johnson, and David F Fouhey. The 8-point algorithm an inductive bias for relative pose prediction by ViTs. arXiv preprint arXiv:2208.08988 08988, 2022.
Paul-Edouard Sarlin, Daniel DeTone. Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, 2020. URL htttt::t//arxttt
org/ abs 911 11763
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Samarth Sinha, Jason Y Zhang, Andrea Tagliasacchi, Igor Gilitschenski, and David B Lindell. Sparsepose: Sparse-view camera pose regression and refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21349-21359, 2023.
Cameron Smith, David Charatan, Ayush Tewari, and Vincent Sitzmann. Flowmap: High-quality camera poses, intrinsics, and depth via gradient descent. arXiv preprint arXiv:2404.1525% 2024.
Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers, pp. 835-846. 2006.
Christoph Strecha, Wolfgang Von Hansen, Luc Van Gool, Pascal Fua, and Ulrich Thoennessen. benchmarking camera calibration and multi-view stereo for high resolution imagery. In 2008 IEEE conference on computer vision and pattern recognition, pp. 1-8, leee, 2008
Peter Sturm and Bill Triggs. A factorization-basel algorithm for multi-image projective structure and motion. In European conference on computer vision, pp. 709-720. Springer, 1996.
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. CVPR, 2021.
Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, and Kwang Moo Yi. Acne: Attentive context normalization for robust connutation-equiaart learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11286-12295, 2020.
Christopher Sweeney, Tobias Hollerer, and Matthew Turk. Theia: A fast and scalable structure- from-motion library. In Proceedings of the 23rd ACM international conference Multimedia, pp. 693-696, 2015.
Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Visual geometry grounded deep structure from motion. arXiv preprint arXiv:2312.04566 2023a.
Jianyuan Wang, Christian Rupprecht, and David Novotny. Posediffusion: Solving pose estimation via diffusion-aided bundle adjustment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7773-9783, 2023b.
Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 21686-21697, 2024.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. arXiv preprint arXiv:2312.14132. 2023c.
Kyle Wilson and Noah Snavely. Robust global translations with Idsfm. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part III 13, pp. 61-75. Springer, 2014.
Changehang Wu. Towards linear-time incremental structure from motion. In 2013 International Conference on 3D Vision-3DD 2013, pp. 127-134. IEEE, 2013.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scat scale dataset for generalized multi-view stereo networks. In Proceedings of the IEFE/CVF conference computer vision and pattern recognition, pp. 7790-1799, 2020.
13

---

Published as a conference paper at ICLR 2025
Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to find good correspondences. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2676,, 2018.
Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In European Conference on Computer Vision, pp. 592-611. Springer, 2022.
Jason Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang. Deva Ramanan, and Shubham Tulsiani. Cameras rays: Pose estimation via ray diffusion. In International Conference Learning Representations (ICLR), 2024.
Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao. Learning two-view correspondences and geometry using order-aware network. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5845-5854, 2019.
Chen Zhao, Yixiao Ge, Feng Zhu, Rui Zhao, Hongsheng Li, and Mathieu Salzmann. Progressive correspondence pruning by consensus learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6464-6473, 2021.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A QUALITATIVE RESULTS
Figures
3 and show reconstruction examples with our method, compared other deep-based
methods
(a) Our recovery
(b) ESFM recovery
(c) GASFM recovery
Figure 3: Reconstruction results on scenes from the DDFM dataset. The figure shows 3D reconstructions and camera pose estimation. The triplet in cach row shows reconstruction with our method (left), ESFM (Moran) et al. 2021) (middle), and GASFM (Brynte et al. 2023) (right). The scenes are Piazza del Popolo (top row. 33.1% outliers), Montreal Notre Dame (middle row, 33% outliers), and Madrid Metropolis (bottom row, 39.4% outliers).
15

---

Published as a conference paper at ICLR 2025
(a) Our recovery
(b) ESFM recovery
(c) GASFM recovery
Figure 4: Reconstruction results on scenes from MegaDepth dataset. The figure shows 3D reconstructions and camera pose estimation. The triplet in each row shows reconstruction with our method (left), ESFM (Moran et al. 202T) (middle), and GASEM (Brynte el al. (203) (right). The scenes are 0012 (top row, 16.3% outliers), 0015 (middle row, 20.6% outliers), and 0060 (bottom row, 41.6% outliers).
16

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS
Our code and preprocessed point tracks data will made publicly available.
Framework. Our method was trained and evaluated on NVIDIA A40 GPUs (48GB of GPU Memory). We used Py PyTorch (Paszke al., 2019) as the deep learning framework and the ADAM optimizer (Kingma Ba (014) with normalized gradients.
Training. During training. for each epoch, we processed all training scenes sequentially. For each scene, we randomly selected a subset of 10%-20% of the images in the scene. We used a validation set for early stopping. The validation and test sets were evaluated by using the complete point tracks matrix. The training time of our model on the Megadepth dataset took roughly 16 hours on a single NVIDIA A40 GPU. We fix the seed to 20.
Architecture details. We use normalized point tracks Xij. as inputs to our method which are normalized using the known intrinsic parameters. The shared features encoder E has layers, each with 256 feature channels and ReLU activation. The camera head Heams, 3D point head Hpts, and the outliers classification head Houtlier, each have 3 layers with 256 channels. After each layer in E we normalize its features by subtracting their mean.
Hyper-parameete search. We tried different implementation hyper- papper-prraeterss including (1) learning rates E {le 2, le 3, le - 4). (2) network width E {128, 256, 512} for the encoder E and the heads, (3) number of layers E (2, 3, 4, 5) in these networks, and (4) threshold for outlier removal {0.4, 0.5, 0.6, 0.7, 0.8}. For the Strecha and BlendedMMS datasets, we chose a threshold of 0.8, as these datasets are captured with a single camera and therefore contain fewer outliers.
Bundle adjustment For the bundle adjustment, bundle adjustment Aggawal et al.J with the Huber loss to enhance robustness (with parameter 0.1). In each bundle adjustment round, we limited the number of iterations 300 until convergence, whichever occurred first.
employed the Ceres Solver's 's implementation of
C CONSTRUCTING POINT TRACKS
Our preprocessing step for constructing the point tracks is as follows:
1. Extracting SIFT
Lowe 2004) features for all the images.
2. Applying exhaustive pair-wise RANSAC (Fischler & Bolles 1981) matching for all image pairs.
3. Chaining each sequence of two-view matches for creating a point track.
We discard 3D point and its track if it is viewed in less than 3 cameras or if it includes an inconsistent cycle, i.e., we detect two key points in the same image for the same point track.
The above preprocess defines valid point track structure for our architecture. We next define the inliers/outliers labeling process, which assumes that every training scene has an associated COLMAP (Schönberger & Frahm, 2016) reconstruction:
1. For each track within our point track set:
(a) Identify the corresponding track in COLMAP's reconstruction output
(b) For every keypoint in the track:
Label the keypoint as an outlier if it does not appear in the corresponding COLMAP track; otherwise. label
an inlier.
2. Remove keypoints labeled as outliers from the point tracks.
3. Triangulate the 3D points from the cleaned point tracks using the ground truth camera poses obtained from COLMAP.
4. Calculate the reprojection error for the triangulated 3D points across the entire point tracks, including the ones labeled initially outliers.
5. Assign an inlier label to each keypoint whose reprojection error is below 4 pixels; keypoints exceeding this threshold are classified as outliers.
17

---

Published as a conference paper at ICLR 2025
D ADDITIONAL RESULTS
Here we show median results for the MegaDepth and IDSFM experiments (Tables 9 and T0). further validate the importance of our Robust BA compared a standard BA (Table
We
Table 9: MegaDepth experiment. The table shows the median values of the rotation (in degrees) and translation errors. (Above the middle rule are Group 1 scenes with <1000 images; below are Group 2 scenes with >100 images, subsampled 300 for testing.) Winning results are marked in bold and underlined. Yellow represents the best result among the deep-based algorithms, and green among the classical algorithms.
Soone
N₂
Cuilizees
Ours
ESFM
GASPM
Theia
GLOMAP
ESFM
Nr
Trans
N--
Ros
Tras
N-
Rot
N-
Rot
Trans
Rot
Trans
N-
0238
522
44.6
283
0.72
0.048
76
1.75
0,100
76
5.15
0.325
506
1.55
0.
499
0.22
0.048
512
0.64
0.308
0060
528
41.6
503
0.14
0.01 2.011
503
8.54
1.268
258
13.81
2.744
525
0.36
0.089
523
0:012
534
0.02
0197
670
40.7
687
2.06
0.133
261
4.14
0.153
454
6.65
0.688
855
0.77
0.118
814
0.18
0.016
825
0.31
0.008
0694
763
40.1
637
0.38
0.0
0.015
17.62
2,048
359
0.58
0.028
742
0.033
20
1:957
643
0.18
0.014
0265
571
38.6
340
0.74
20.
270
14.64
666
274
18.91
1,716
554
1.11
1.651
558
6.65
1.889
539
0.06
0.022
0083
635
31.3
596
0.15
0.009
568
0.90
0.027
574
2,48
0.099
632
0.15
0.013
614
0.04
0.007
556
16.03
0.913
0076
550
524
0.010
454
1.601
0,099
0.05
1000
0.44
0.058
0.08
0.004
0185
368
30.0
350
0.04
0.008
261
0.53
0.037
254
0.466
0048
366
0.31
0.087
365
0.11
0.012
130
20.39
2.160
0048
512
474
0.058
1.57
0,080
0.60
507
0.020
9.06
0.007
0024
356
23.0
309
0.58
0.046
271
4.64
0.187
500
0.48
0.104
3.5
0.24
0.091
339
0.07
0. 0.046
348
1.41
0.078
0023
2144
1.81
0.180
5016
28
0.2
28
0.10
0,005
28
0.07
0,004
28
0.06
000
28
0.07
0.019
28
0.04
0.016
28
0.02
0,005
0046
440
1.4.6
3999
0.78
0,028
0.38
0:017
426
0.335
0.011
0.16
0.016
440
0.002
29.18
1.331
0639
209
47.4
190
3.18
0.607
104
2.21
0.402
12M
3.95
0.419
397
1.79
0.394
255
0.00
0.028
243
2.97
0.618
1001
285
43.9
251
1.41
0.274
2.44
0.900
261
2.36
0.386
276
4.85
2.893
28:
3.29
2.645
280
0.08
0.053
0231
296
42.2
246
0.38
0.014
214
0.28
0.011
209
0.19
0.000
288
0.58
0.072
279
0.20
0.021
784
0.26
0,011
0411
299
29.9
273
0.07
0.
0.0009
188
6.21
0,338
232
1.28
0.051
293
0.19
0.079
269
0.09
0.036
288
0.05
0.007
0377
295
27.5 5
210
0.28
0.014
162
0:25
0.010
167
0.09
000
269
0.29
0.075
268
0.23
0.021
279
0.52
0.002
0102
299
25.8
284
0.07
0.007
255
0.50
0.002
278
0.37
0.025
294
1.03
0.114
293
0.04
0.013
135
13:56
2.859
0147
298
207
0.088
197
2.05
0,097
225
5.26
0.246
1.10
0000
1.75
2,056
1.34
0148
287
24.6
197
0. 54
0. 0.024
206
0.59
0.078
200
0.59
0,028
275
3.01
0.301
28.
3.09
1,301
249
0.46
0.023
0446
208
0.70
0.011
0022
207
21.2
2274
0.13
0.011
241
0.51
0.025
263
0.36
0.013
298
01.28
0.065
281
0.08
0.023
289
0.08
0.008
0327
208
0.012
7.14
0015
28:4
20.6
215
0.27
0.021
142
1.30
0.105
149
6.35
0.556
24-1
0.42
0.064
274
0.111
0.014
185
0.45
0.083
0455
2940
19.8
293
0.18
9.0
0.010
0.23
0.014
294
0.29
294
0.36
0.047
298
0.14
0.017
0,012
0496
207
19.2
281
0.18
0. 000000
280
2.81
0.197
277
0.44
0.027
285
0.61
0.080
291
0.16
0.028
203
0.23
0.009
1589
209
17.4
200
0.08
0000
0.40
0.008
283
1.00
0.16
258
0.92
0.057
290
0.03
0:007
206
0.20
0.004
0012
209
16.3
267
0.39
0.023
291
2.04
0.131
294
0.21
0.018
125
0.556
0.092
295
0.20
0.017
204
0.22
0.013
0104
284
16.2
193
0.10
000
220
11:31
0.568
228
1.30
0.051
265
8.24
0.658
280
9.85
0.550
300
0.49
0.019
0019
299
15.4
250
0..004
267
2:46
0,128
293
0.58
0.023
271
0.31
0.090
206
04
0.004
296
2:23
0.087
0063
293
14.5
262
0.26
0.013
202
1.18
0.057
237
0.40
0.020
264
0.45
0.063
258
0.17
0.017
275
0.15
0.009
0130
285
14.4
192
0..00
0.94
0.030
1,00
0.463
0.072
0.94
282
0080
284
12.9
1:39
0.27
0.010
197
0.32
0.012
1.30
1.36
0,054
278
1.84
0.335
283
1.71
0.1
163
0.32
0.011
0240
298
276
0.090
1.14
0.064
2,00
273
0.057
0.041
0007
200
11.7
172
0.23
0.010
260
25.
50
1.43
280
0.18
0.010
277
0.69
0.071
290
0.08
0.008
786
0.96
0.053
Mean
379
298
3,46
0.291
2.25
0.228
Table 10: IDSFM experiment. The table shows the median values of the rotation (in degrees), and translation errors. Winning results are marked in bold and underlined. Yellow represents the best result among the deep-based algorithms and green among the classical algorithms.
Same
Oulland
N_
Rui
Ours
Trax
X,.
Rte
ESFM
Teams
N.
GASFM
Rei
Trass
N,
Theis
Rei
Trass
...
GLOMAP
Rei
Teans
N_
ESFM
Roi
Trans
Alame
573
32.6
494
0.037
457
1.20
0,087
2.64
0.115
555
2.29
0.539
557
0.1.4
525
0.35
0,016
Idbadd
25.1
214
0.32
0.056
196
18.8K
1.554
198
19.34
1.548
213
0
212
219
0:46
0.087
0.19
0.006
Madrid Metiopolis
333
20.4
244
4.42
0.193
151
19:35
1.64]
159
21.76
1.500
3.20
0.03
0.096
290
23.31
2,154
Montreal Noue
81.7
0.056
309
4,42
0,790
422
0.168
0.009
Notre Dune
569
35.6
617
0.55
0.025
467
0.43
0.026
499
0.51
0.025
314
1.54
0.135
641
1.15
0.130
525
0.30
0,012
NYC Libnay
330
88.6
224
1.48
0.074
177
1.93
0,102
218
3.74
0.264
534
1.65
0.360
323
0.48
0.076
301
1,16
0.087
Popalo
Tower of Landen
467
27.0
94
0.48
0.012
196
13.59
0.564
152
33.19
2.330
448
3.23
0.527
468
0.42
0.071
213
4.53
0.163
Viens Osthedraa
834
31.4
479
0.48
0.016
551
1.40
0.095
558
1.67
0.044
712
9.3
0.818
N22
0.306
595
0:44
0.011
Yotimunster
20.0
215
6.15
0.302
251
390
0.60
18

---

Published as a conference paper at ICLR 2025
Table 11: Mandle-Adjustment (BA) ablation. We compare the effect of post-processing with robust vs. standard BA. For each scene, we show the number input images (denoted N₂) and the fraction of outliers. For each model, we show the number of images used for reconstruction (denoted N2)) and the mean values the rotation (in degrees) and translation errors.
Robust BA (Ours)
Standard BA)
Scene
N.
Outliers
N.
Trans
Alamo
573
32.6
484
3.66
0.515
568
10.18
1.970
Ellis Island
227
25.1
214
0.82
0.122
227
4.05
1.045
Madrid Metropolis
333
39.4
244
8.42
0.827
333
20.07
2.352
Montreal Notre Dame
448
31.7
346
2.82
0.352
447
4.52
0.563
Notre Dame
549
35.6
517
1.20
0.231
548
3,45
0.378
NYC Library
330
33.6
224
3.96
0.129
329
24.94
3.571
del Popolo
335
2.001
Tower of London
467
27.0
94
0.67
0.026
465
57.41
3.347
Cathedral
31.4
1.52
43.38
2.784
Yorkminster
432
29.0
331
14.54
1.468
431
21.17
3.660
19


Reviews:
[{"review_id": "tpWvfmAK8v", "reviewer": "Reviewer_PnnE", "summary": "The paper presents an architecture for Multiview Structure from Motion (SfM) focusing on the robust recovery of camera pose and 3D scene structure from large, uncontrolled image collections that contain numerous outliers. Traditional and deep-based SfM approaches often struggle with outlier point tracks resulting from viewpoint variations, illumination changes, and repetitive structures, significantly impacting their performance. The authors propose an enhancement to the deep network architecture proposed by Moran et al. in 2021, incorporating an outlier classification module within a permutation equivariant framework. The revised architecture also integrates a robust bundle adjustment step to correct classification errors. The method has been tested on challenging datasets like MegaDepth and 1DSFM and has shown good accuracy and competitive runtimes against both contemporary deep-learning methods and established classical techniques.", "questions": "The main question is about the Weaknesses 1 and 2, i.e., what's the main novelty of the proposed method compared to the existing literature?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper improves the existing ESFM framework by integrating a new inlier-outlier classification branch, alongside a robust structure from motion (SfM) mechanism. This approach makes sense, given that outliers represent a primary challenge for SfM methods in real-world applications. Quantitative experiments demonstrate the efficacy of these components.\n\n2. This paper conducts extensive experiments over various datasets to prove their claims.", "weaknesses": "1. The main concern about this work is the novelty of the proposed framework. Compared to ESfM, the new designs are just (1) a simple classfication branch to identify inlier/outlier, which conducts simple binary classfication, and (b) robust BA considering high projection error, point track length, and multi-step refinement. Both these two designs have been proven effective over the long time and are not the new techniques from this work. For instance, systems like COLMAP, Theia, and VGGSfM incorporate variants of robust BA. Numerous matching and tracking methodologies, including SuperGlue, LightGlue, PiPs, and Cotracker, implement classification branches to classify inlier/outlier status. While the reviewer acknowledges the effectiveness and importance of these features, there are reservations about their originality as claimed by this work.\n\n\n2. Moreover, the section of ablation study needs more thinking. The authors discussed \"the impact of our permutation sets-of-sets equivariant architecture\" and \"importance of the equivariant features\". However, as far as the reviewer understands, both these two are the benefits from the architecture of ESfM.  To validate the design choices of this paper, the authors should focus on why their error handling designs are better, instead of proving their base model is better. \n\n\n3. Concerns also arise regarding the reliability of the results. If interpreted correctly, the rotational metrics in the study are reported in degrees, not radians. Notably, in Tables 3 and 4, the reported rotational errors for some methods are as low as 0.02 or 0.03 degrees. The reviewer questions whether such marginal differences are statistically significant enough to discriminate between methods. (e.g., BlendedMVS, despite being half-synthetic, does not offer flawless ground truth)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "huSNvtY7dL", "reviewer": "Reviewer_D44J", "summary": "In this work, authors robustify the existing deep equivariant SfM method from [1] by incorporating an inlier-outlier prediction module and robust Bundle Adjustment. Specifically, deep SfM uses projective factorization on a 2D point track matrix, but the method in [1] assumes that this matrix is complete and outlier-free. Authors in this work argue that these assumptions are unrealistic. Hence, authors proposed a deep equivariant SfM that works on a large collection of internet photos and works with a matrix of point tracks that might have outliers. In their method, they first train an end-to-end outlier prediction and camera poses + 3D point prediction network from a matrix of 2D point tracks. At test time, they use \"inlier-outlier\" predictions from the pre-trained network to remove outliers and fine-tune the network for 3D pose + 3D points recovery using unsupervised losses for each scene.\n\nAuthors show impressive results across different datasets. Their method achieves the best results among deep SfM methods and achieves comparable results to the best classical SfM methods.\n\nReferences:\n\n[1] Moran, Dror, et al. \"Deep permutation equivariant structure from motion.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "questions": "Can we compare with Mast3R / Spann3R [3]? These methods are new way of doing scene reconstruction and does show promise. They are also deep \"reconstruction\" methods because they use deep networks directly to get 3D pointmaps.  This comparison with help researchers find limitations or benefits of one method over another.\n\nIs it possible to make results table a bit easy to read? Color + bold letters are not visible clearly. It is hard to find best method for each scene. Also, is there a way to get an average of some of metrics across scenes? (or over whole dataset). \n\nReferences:\n\n[3] Wang, Hengyi, and Lourdes Agapito. \"3d reconstruction with spatial memory.\" arXiv preprint arXiv:2408.16061 (2024).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Paper is well-written with a clear objective being robustifying the method proposed in [1]. Overall content of the paper is well-written, coherent, and easy to understand and follow.\n\nThe proposed inlier-outlier prediction head shows improvement compared to ESFM [1]. Results across many scenes of indoor and outdoor datasets show that the proposed method improves over [1] and achieves comparable results to the best classical SfM methods.\n\nAuthors use unsupervised reprojection losses. This means that the network can be easily fine-tuned for different scenes, removing a limitation of deep SfM networks that they cannot generalize to scenes outside their training sets.\n\nReferences:\n\n[1] Moran, Dror, et al. \"Deep permutation equivariant structure from motion.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "weaknesses": "Authors mention that they did not use Mast3R [2] as one of the baseline method because it does not work with large number of images. But, I do think Mast3R [2] is able to work with large sets of images. At least on Stretcha and BlendedMVS experiments, authors should be able to use Mast3R for full 3D reconstruction \n\nWhat is not clear from the paper is whether other methods use Robust Bundle Adjustment proposed in this work. I agree robust BA is necessary for accurate reconstruction, but the contribution of the paper is robust \"deep equivariant SfM\". For apples-to-apples comparison, authors should also compare with Standard BA or authors should apply robust BA post-processing to compared methods. I see that authors do provide ablation study for effectiveness of robust BA, but to verify the claim of deep robust sfm, it would be great to compare all the methods with either regular BA or robust BA.\n\nReferences:\n\n[2] Leroy, Vincent, Yohann Cabon, and Jérôme Revaud. \"Grounding Image Matching in 3D with MASt3R.\" arXiv preprint arXiv:2406.09756 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JXycZ7yE4W", "reviewer": "Reviewer_oVEX", "summary": "This paper studies an important problem in multi-view structure from motion, specifically aiming to recover 3D points and camera poses from input point tracks. To this end, based on an existing permutation equivariant architecture, an outlier classification module is integrated to cope with outliers. A robust Bundle Adjustment method is also proposed, based on recursive potential outlier removal. Experiments are conducted on multiple datasets, demonstrating the robustness of the proposed method against existing ones.", "questions": "Experiment-wise, why ESFM* has much larger rotation error in some cases, and why GLOMAP has good rotation predictions but not translation predictions. And it might be better to give the mean errors in the table for more straightforward comparisons.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper is well-presented in general and easy to understand, with a comprehensive literature review and extensive experiments.", "weaknesses": "The paper studies an interesting and practical problem, but the main weakness is its limited contribution. \n\n1. The proposed method is largely built on existing network architecture, the difference being only that outlier classification output channels are added. \n\n2. The proposed solution for handling outliers incurs much additional overhead in the recursive \"finetune\" process, lacks theoretical justification, comprises inliers recall, and can not reject outliers well (according to Table 7). Thus the contribution of the method is limited. The same problem also goes for the proposed robust Bundle Adjustment method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)