Robust Deep Equivariant Structure from Motion

## Abstract

Abstract Multiview Structure from Motion is a fundamental and challenging computer vision problem. A recent deep-based approach was proposed utilizing matrix equivariant architectures for the simultaneous recovery of camera pose and 3D scene structure from large image collections. This work however made the unrealistic assumption that the point tracks given as input are clean of outliers. Here we propose an architecture suited to dealing with outliers by adding an inlier/outlier classifying module that respects the model equivariance and by adding a robust bundle adjustment step. Experiments demonstrate that our method can be successfully applied in realistic settings that include large image collections and point tracks extracted with common heuristics and include many outliers.

###### Keywords:

## 1 Introduction

Simultaneous recovery of camera pose and 3D structure from large image sets, commonly termed Multiview Structure from Motion (SfM), is a longstanding fundamental problem in computer vision with applications in augmented and virtual reality, robot manipulation, and more.
Classical SfM techniques as well as some recent deep-based techniques rely on the extraction of point tracks, i.e., point matches across multiple frames, to solve for camera poses and structure. Existing heuristics for point track extraction, however, often return erroneous (*outlier*) point tracks, due to large viewpoint and
illumination differences and the presence of repetitive scene structures. Moreover, producing point tracks by connecting chains of pairwise matches further introduces outliers. Identifying these outlier track points significantly slows reconstruction down and in certain cases results in inaccurate reconstruction. Developing robust techniques that can efficiently and accurately remove outlier track points is still an outstanding challenge.

Early classical SfM techniques rely on point feature extraction and matching, e.g., using SIFT [], to construct point tracks. Sequential algorithms [, ] then incrementally update their camera poses and 3D sparse (point cloud) reconstruction by processing one camera at a time, pruning the outlier matches at every step of the algorithm. While such processing, often yields state-of-the-art results, it can be slow, particularly for large image collections.

Recent deep-based methods attracted considerable attention due to their ability to encode priors for cameras and shapes effectively. In particular, Neural Radiance Field (NeRF) methods and methods based on implicit surface reconstruction exploit such priors to obtain accurate dense (pixel resolution) reconstruction [, , , ]. Yet, many of these methods still rely on classical methods to estimate camera pose. Moreover, deep methods for multiview SfM (see review in Section 2) are typically applied to small image collections, usually $\leq$20 images [], due to both accuracy and complexity considerations. The problem of using deep networks in large multiview SFM setups, even in the sparse setting, is still unsolved.

Below we introduce a robust, deep-based technique for multiview SfM. We address multiview SfM in its classical setting and modify and extend the recent method of Moran et al. [] to filter out outlier track points. Moran et al. obtain a collection of point tracks as input, which they arrange in the columns of a matrix (tensor) whose rows represent the cameras. Their work, which is inspired by projective factorization techniques, uses a network that is equivariant to both matrix row and column permutations[] to map the point tracks to camera poses and 3D structures. Their method, however, only works on inputs in which all outliers are removed in preprocessing. This severely limits the applicability of their method.

In this paper, we extend the approach presented in [], by adding an inlier/outlier classification module that respects equivariance. In particular, our architecture allows identifying track points that behave inconsistently compared to other points of the same image (i.e., points that share the same matrix row) or are inconsistent with other members of their track (i.e., sharing the same column). Additionally, we modify the final bundle adjustment step to make it robust to classification errors. These modifications allow our model to (1) handle large image collections (hundreds of images) of novel scenes not seen in training, and (2) use point tracks obtained with standard common heuristics that include many outliers without any pre-filtering of these outliers. Experiments show that our network improves over the method of [] in almost all runs, obtaining state-of-the-art accuracies while considerably reducing run time, compared to the classical COLMAP [].

## 2 Related Work

The recovery of camera pose and 3D structure from image collections has been a central subject of research in computer vision in the past several decades, leading to multiple breakthroughs that enabled accurate reconstructions from hundreds and even thousands of unordered images [, , , , ]. In the typical *sequential pipeline* point tracks are first extracted from the input images. Camera pose and 3D structure are next computed for two images and then updated by processing the remaining images one by one. This pipeline yields highly accurate recovery of pose and structure but can be slow when applied to large image collections. Alternative global approaches attempt to solve SfM simultaneously for all images using a variety of factorization formulations [, , ].

Recent deep-learning techniques attempt to improve these pipelines by exploiting the priors on the input images, camera settings and 3D structures learned with these networks. Existing methods attempt to improve keypoint detection and matching[, , , , ], produce point tracks [] formulate differentiable alternatives to RANSAC [, , , ], or directly infer the relative orientation and location of a pair of images [, , , , ].

Recent work proposed end-to-end methods for camera pose estimation. Works such as RelPose [] and its successor, RelPose++ [], harness energy-based models to recover camera poses from input that includes the relative rotations between images. Similarly, SparsePose [] learns to regress initial camera poses which are then refined iteratively. PoseDiffusion [] employs a diffusion model to refine camera poses. [] improve this by focusing on purifying the camera rays. These works, however, are only applied to small collections of images (typically $\leq$20) and are commonly applied in object-centric scenarios, e.g., as in [].

Our approach extends the work of [], which introduced a trainable network for simultaneous camera pose and 3D structure recovery. This approach was demonstrated to achieve highly accurate pose and structure recovery on large image collections (with several hundreds of images) of outdoor urban scenes. It however requires point tracks as input in which all outliers are removed in preprocessing. This greatly restricts its applicability in realistic settings.

[] adopts a methodology motivated by projective factorization schemes. Arranging the input point tracks in a matrix, an equivariant network to row and column permutations is used to regress the camera poses and the 3D point cloud coordinates. The network is trained with unsupervised data by optimizing a reprojection loss and is applied at inference to novel scenes not seen in training. Additional fine-tuning and Bundle Adjustment (BA) are applied to attain sub-pixel reprojection errors. We improve over this work by adding an inlier/outlier classification module that respects and exploits the equivariant structure of the network and by introducing a robust BA scheme, allowing us to work with realistic point tracks that are obtained with standard heuristics.

We finally mention two concurrent extensions of [] published in Arxiv.
[] replace the set of sets architecture in [] with a graph attention network, for increased expressiveness, enabling them to avoid fine-tuning, thereby reducing inference runtime compared to [] without compromising the performance.
In [] applied []’s network (with some modifications) to aerial images captured with GPS information under roughly constant camera orientation.
Neither [, ] show results on point tracks contaminated by a significant number of outliers.

## 3 Method

Figure: Figure 1: Network architecture. Our network comprises three main modules and works in these stages (1) E, the feature encoder, which outputs a latent feature representation; (2) a module that removes predicted outliers from the input point tracks; (3) a process where the network is fine-tuned for 1000 epochs; and (4) a final stage where the predicted cameras and point cloud are optimized by bundle adjustment (BA), resulting in optimized cameras and 3D points.
Refer to caption: extracted/2404.14280v1/figs/architicture.png

Our goal is to utilize deep networks for robust multiview Structure from Motion. Given a collection of point tracks extracted from input images, we aim to simultaneously recover the unknown camera poses and the (sparse) 3D scene structure. We follow and modify the approach suggested in [] and aim to use matrix row and column premutation equivariant network to solve the problem. As existing matching heuristics generally produce many outliers, we learn to classify track points as either inliers or outliers while relying on the sets-of-sets equivariant structure suggested in [].

### 3.1 Problem formulation

We assume a stationary scene viewed by $m$ cameras with unknown poses, and we obtain as input a (sparse) point track tensor $M$, which includes the 2D observations of $n$ 3D points viewed by partial sets of cameras. We further assume in this work that the cameras are internally calibrated. A camera matrix therefore is expressed in the form of $P_{i}=[R_{i}|{\bf t}_{i}]$ where $R_{i}\in SO(3)$ is a rotation matrix. With this notation, the camera is placed at the position $-R_{i}^{T}{\bf t}_{i}$. We denote by ${\bf X}_{j}$ the $j$’th 3D scene point and by ${\bf x}_{ij}\in\mathbb{R}^{2}$ its observed position in the $i^{\prime}$th image. The set $T_{j}=\{{\bf x}_{i_{1}j},{\bf x}_{i_{2}j},...\}$ with $C_{j}=\{i_{1},i_{2},...\}\subseteq[m]$ represents the $j$’th track. These tracks are constructed in preprocessing by a heuristic and therefore are contaminated by small displacement errors and outliers.

We arrange the tracks $T_{1},\ldots,T_{n}$ in the columns of the measurement point track tensor $M$. The tensor $M$ is of size $m\times n\times 2$, so the rows of $M$ correspond to the $m$ cameras, and its columns correspond to the unknown 3D points whose contaminated projections are given. We further set $(M_{ij1},M_{ij2})={\bf x}_{ij}$ if $i\in C_{j}$ and $C_{j}$ represents the set of images in which $X_{j}$ is detected. Otherwise, $M_{ij1}$ and $M_{ij2}$ are empty.

We aim to recover the pose parameters of the $m$ camera matrices and the 3D positions of the $n$ feature points. Additionally, we seek to classify each track point as either an inlier or an outlier. The output of our network will comprise three matrices: $P\in\mathbb{R}^{m\times 7}$ for the set of cameras matrices, $X\in\mathbb{R}^{n\times 3}$ for the 3D structure, and $O\in\mathbb{R}^{m\times n}$ for the track point classification results. Here every camera in $P$ is represented by a rotation quaternion $\mathbf{q}_{i}\in\mathbb{R}^{4}$ and a translation vector $\mathbf{t}_{i}\in\mathbb{R}^{3}$. Classification results in $O$ are represented by scores in $[0,1]$, where $\tilde{\bf x}_{ij}$ with $O_{ij}\geq 0.6$ is considered an outlier.

### 3.2 Network architecture

Our architecture is equivariant to both row and column permutations of $M$, as in []. It comprises four modules (see Fig. 1): an equivariant feature encoder followed by an outlier removal, pose, and structure prediction modules.

Equivariant feature encoder. This module takes as input the tensor $M\in\mathbb{R}^{m\times n\times 2}$ and outputs a latent representation in $\mathbb{R}^{m\times n\times d}$, where $d=256$. It comprises 3 permutation equivariant layers interspersed with pointwise non-linear ReLU function, to process and encode the features.
An equivariant layer. A linear equivariant map is the core of permutation equivariant networks. In our case, the permutation equivariance should hold on rows (cameras) and columns (3D points) of the tensor $M$. In [], it was shown that the space of linear maps that keep the permutation equivariant property of a single channel (i.e., the input and the output are matrices) is spanned by the identity, the row sums, the column sums, and the matrix sum. In [], it was applied to multiple input and output channels from a feature space with $d$ channels $\mathbb{R}^{m\times n\times d}$ to a feature space with $d^{\prime}$ channels $\mathbb{R}^{m\times n\times d^{\prime}}$ in the following way:

$$ $\begin{split}L(\tilde{M})_{ij}=&W_{1}\tilde{M}_{ij}+W_{2}\sum_{k=1}^{m}\tilde{ M}_{kj}+\\ &W_{3}\sum_{l=1}^{n}\tilde{M}_{il}+W_{4}\sum_{k=1}^{m}\sum_{l=1}^{n}\tilde{M}_ {kl}+\mathbf{b},\end{split}$ (1) $$

where for the input tensor $\tilde{M}_{ij}$ is a vector in $\mathbb{R}^{d}$, $W_{i}\in\mathbb{R}^{d^{\prime}\times d}$ for $i=1,...,4$, and $\mathbf{b}\in\mathbb{R}^{d^{\prime}}$ are learnable parameters. We follow [] and replace sums by averages over the non-empty entries of $M$, yielding invariance to unobserved projections.

Inlier/Outlier classification module. Given the latent representation $\mathbb{R}^{m\times n\times d}$ provided by the equivariant feature encoder, this module, $H_{\text{outlier}}$, employs 3 MLP layers followed by a sigmoid function. The output is a matrix $O\in[0,1]^{m\times n}$ expressing the probability of each entry being an outlier. Note that this module respects the equivariance property.

Pose and structure regression modules. Utilizing the latent representation $\mathbb{R}^{m\times n\times d}$ from the equivariant feature encoder, the pose and structure heads, $H_{\text{cams}}$ and $H_{\text{pts}}$, consist of 3 MLP layers each. The first, $H_{\text{cams}}$, processes the pooled average of the features along the columns $\mathbb{R}^{m\times d}$, where each camera’s features are encoded into a vector in $\mathbb{R}^{d}$, and outputs a tensor in $\mathbb{R}^{m\times 7}$. This tensor encapsulates the camera’s translation vector (first three values) and orientation (last four values, represented as a quaternion), collectively forming the camera’s projection matrix. The second head, $H_{\text{pts}}$, processes the pooled average of features along the rows $\mathbb{R}^{n\times d}$, where each scene point is encoded into a vector in $\mathbb{R}^{d}$, and outputs a tensor in $\mathbb{R}^{n\times 3}$, representing the coordinates of the scene points.

### 3.3 Inference

After training our network (see Section 4 for details), we test the network on unseen scenes. We first apply inference and then use fine-tuning followed by robust Bundle Adjustment.

Fine-tuning.
At this step, we leverage our inlier/outlier classification prediction and remove the points predicted as outliers from the point tracks tensor $M$ (using the threshold $0.6$). Subsequently, similar to [], we fine-tune the network by minimizing the unsupervised reprojection loss for 1000 epochs.

Robust Bundle Adjustment.
As common in SfM pipelines, a post-processing step of applying bundle adjustment is necessary to achieve sub-pixel reconstruction accuracy. [] proposed first applying bundle adjustment to the predicted cameras and 3D point cloud. Next, they use the optimized camera poses from the previous step to triangulate the scene’s point cloud and subsequently reapply bundle adjustment. However, this approach was found to be non-robust in realistic scenarios involving many outliers. Therefore, propose a Robust Bundle Adjustment Module:

- 1.
Apply bundle adjustment (BA) to the predicted cameras and predicted 3D points.
- 2.
Remove 3D points with projection error higher than 5 pixels and remove 3D points viewed in fewer than 3 cameras.
- 3.
If the view graph of the point tracks becomes unconnected due to the removals, then we take the largest component and discard point tracks from the removed images.
- 4.
Triangulate the remaining point tracks and apply a second round of (standard) BA.
- 5.
Bring back the point tracks removed in step 2, triangulate them with the optimized camera poses, and apply a third and final round of BA. This step succeeds in restoring many of the 3D points.

More details on the BA stage can be found in the appendix.

### 3.4 Loss Function

Our loss function combines two terms:

$$ $\displaystyle\mathcal{L}=\mathcal{L}_{\text{outliers}}+\alpha\mathcal{L}_{ \text{reprojection}}$ (2) $$

where the hyperparameter $\alpha$ balances the two terms.
The outlier classification loss, $\mathcal{L}_{\text{outliers}}$, employs a Binary Cross-Entropy (BCE) loss. $\mathcal{L}_{\text{reprojection}}$, which is adopted from [], includes a reprojection and a hinge loss terms. The reprojection loss aims to minimize the error between projected scene points and their detected positions in the image, akin to the objective in bundle adjustment. Additionally, a hinge loss is used to prevent the prediction of non-positive depth values, thus ensuring a physically plausible scene reconstruction.

$$ ${\cal L}_{\text{reprojection}}=\frac{1}{p}\sum_{i=1}^{m}\sum_{j=1}^{n}\xi_{ij} s_{ij},$ (3) $$

Where $p=\sum_{j=1}^{n}|T_{j}|$ the number of measured projections, and $\xi_{ij}\in\{0,1\}$ is indicating whether point $\mathbf{X}_{j}$ is detected in image $I_{i}$.

$$ $\displaystyle s_{ij}=\begin{cases}r_{ij},~{}~{}&P_{i}^{3}\mathbf{X}_{j}\geq h \\ h_{ij},~{}~{}&P_{i}^{3}\mathbf{X}_{j}<h,\end{cases}$ $$

And

$$ $r_{ij}=\left\lVert\left(\mathbf{x}_{ij}^{1}-\frac{P_{i}^{1}\mathbf{X}_{j}}{P_{ i}^{3}\mathbf{X}_{j}},\mathbf{x}_{ij}^{2}-\frac{P_{i}^{2}\mathbf{X}_{j}}{P_{i} ^{3}\mathbf{X}_{j}}\right)\right\rVert,$ $$

And $h_{ij}=\max(0,h-P_{i}^{3}\mathbf{X}_{j})$, where $h>0$ is a small constant (in our setting, $h=0.0001$). $h_{ij}$ therefore is the hinge loss for the depth value of $\mathbf{X}_{j}$ in image $I_{i}$.

## 4 Experiments

### 4.1 Datasets

Our network is trained on scenes from the MegaDepth dataset [] and tested on novel scenes from the MegaDepth dataset as well as the 1DSfM dataset []. These datasets offer a diverse range of real-world scenes, which are instrumental in assessing the robustness and versatility of our proposed architecture across different environments and challenges.

The point tracks, which our networks take as input, are constructed by concatenating pairwise matches between images within each scene. These matches are obtained by applying RANSAC to SIFT matches, ensuring a robust selection of correspondences by mitigating the influence of outliers. The detailed procedure for constructing these point tracks, including parameter settings and algorithmic choices, is provided in the appendix.

MegaDepth. The MegaDepth dataset [] comprises 196 different outdoor scenes, each populated with internet photos showcasing popular landmarks around the globe. To facilitate a comprehensive evaluation, we divide the dataset into two groups based on the number of images per scene: (1) scenes with fewer than 1000 images, and (2) scenes with more than 1000 images. From the first group, we randomly sampled 27 scenes to serve as our training dataset, along with four scenes designated for validation purposes. For the test set, we randomly selected 14 scenes from the first group. Moreover, from each scene in the second group, we randomly sampled images to represent a condensed version of the scene. These samples form a part of the test dataset, introducing a significant challenge for Structure from Motion (SfM) methods due to the reduced number of images representing vast and complex scenes. In tables  2 to  3, scenes above the middle rule belong to Group 1 with fewer than 1000 images, while scenes below belong to Group 2, from which we subsampled 300 images for the test set.

1DSFM. The 1DSFM dataset [] is renowned for its collection of diverse scenes reconstructed from community photo collections. It includes a variety of urban locations, making it a valuable resource for evaluating Structure from Motion (SfM) and multi-view stereo algorithms. For our purposes, the dataset offers a challenging yet realistic setting to assess our architecture’s effectiveness in dealing with large-scale reconstructions.

### 4.2 Baselines

COLMAP []. Recognized as the state-of-the-art incremental Structure from Motion (SfM) method, COLMAP stands as one of the most popular solutions in the field. It demonstrates robust performance in reconstructing 3D models from unordered image collections. Notably, we utilize COLMAP to establish ground truth camera poses for the scenes in our datasets by applying it directly to the scene images. To evaluate the performance of our approach relative to COLMAP, we compare the results obtained from COLMAP with those achieved using the point tracks generated by our preprocessing stage.

ESFM [].
We compare our approach to the original equivariant SfM method [] where for fairness we replace BA with our robust BA. We conduct two distinct evaluations: one in which ESFM is trained and tested on the original point tracks, which are contaminated with outliers (denoted as $ESFM$), and another where it is trained and tested on outlier-free point tracks (denoted as $ESFM^{*}$). By comparing ESFM on the original, contaminated tracks we demonstrate our method’s ability to handle realistic settings. Comparison to the clean tracks provides an empirical upper bound for our method.

### 4.3 Metrics and Evaluation

We measure accuracy with the average reprojection error, measured in pixels as follows

$$ $\frac{1}{p}\sum_{i=1}^{m}\sum_{j=1}^{n}\xi_{ij}\left\lVert\left(\mathbf{x}_{ij }^{1}-\frac{P_{i}^{1}\mathbf{X}_{j}}{P_{i}^{3}\mathbf{X}_{j}},\mathbf{x}_{ij}^ {2}-\frac{P_{i}^{2}\mathbf{X}_{j}}{P_{i}^{3}\mathbf{X}_{j}}\right)\right\rVert.$ $$

For notation see Sec. 3.4.
We further evaluate our predictions of camera parameters. Specifically, we compare our camera orientation predictions with the ground truth ones by measuring angular differences in degrees. Similarly, we measure differences between our predicted and ground truth camera locations (up to a global scale for the scene). For a fair comparison, both our method and all the baseline methods were run with the same set of point tracks. For all methods, we apply a final post-processing step of our robust bundle adjustment. When highlighting the best result in the tables, for a fair comparison, we only compare to ESFM.

Table: Table 1: MegaDepth experiment. The table shows the mean and median camera rotation error (degrees). (Above the middle rule are Group 1 scenes with <1000 images; below are Group 2 scenes with >1000 images, subsampled to 300 for testing.)

Table: Table 2: MegaDepth experiment. The table shows the mean and median camera location error. (Above the middle rule are Group 1 scenes with <1000 images; below are Group 2 scenes with >1000 images, subsampled to 300 for testing.)

Table: Table 3: Megadepth experiment. The table shows the mean reprojection error (pixels). (Above the middle rule are Group 1 scenes with <1000 images; below are Group 2 scenes with >1000 images, subsampled to 300 for testing.)

Table: Table 4: 1DSFM experiment. The table shows the mean and median camera rotation error (degrees).

Table: Table 5: 1DSFM experiment. The table shows the mean and median camera location error.

Table: Table 6: 1DSFM experiment. The table shows the mean reprojection error.

Figure: (a) 0007_300 (11.7% outliers)
Refer to caption: extracted/2404.14280v1/figs/megadepth/0007.png

Figure: (a) Ellis Island (25.1% outliers)
Refer to caption: extracted/2404.14280v1/figs/1dsfm/Ellis_Island.png

### 4.4 Results

Our results for the Megadepth and 1DSFM test scenes are detailed in Tables 2 to 3 and Tables 5 to 6, respectively. For both datasets, we present camera location errors (Tables 2 and 5), camera rotation errors (Tables 1 and 4), and reprojection errors (Tables  3 and  6). Each table lists the number of input images ($N_{c}$), the fraction of outliers, our results, and comparisons with ESFM on original and cleaned tracks (denoted ESFM^∗), as well as COLMAP results using our tracks. For each method, we detail the number of images used and the mean and median errors for camera rotation and location, with reprojection errors reported as means.

It can be seen that our method outperforms ESFM in almost all runs, achieving accurate results that are close to what is obtained with ESFM on clean tracks and with Colmap with our original tracks. This demonstrate the utility of our method in realistic settings.

Visual results.
Figures 2 and 3 show the 3D reconstructions and camera parameters obtained using our method compared to those obtained with $ESFM$[]. The comparison clearly demonstrates that our method produces superior 3D reconstructions. A limitation of our method can be observed in Figure 3(b), where a significant number of images is excluded from the output due to a large number of false positive predictions (inliers that are predicted as outliers). Addressing this limitation presents an avenue for future research.

Runtime.
Table 7 compares the runtimes of our method with Colmap. Our approach is significantly faster, though the fine-tuning phase is the most time-consuming part of our pipeline.

Table: Table 7: Runtime (in seconds, except for Total, in mins) of our proposed method for Euclidean Reconstruction on Various Test Scenes, compared to COLMAP.

Classification performance.
Table 8 presents various classification metrics to assess the performance of our Outlier Removal module. Our method significantly decreases the percentage of outliers in the point tracks after removing the predicted outliers. However, it suffers from a relatively high number of false positive predictions, where inliers are incorrectly identified as outliers.

Table: Table 8: Classification performance, we show various classification metrics to assess the performance of our Outlier Removal module.

## 5 Conclusion

We present a robust solver based on permutation equivariant architecture for the structure from motion problem. Our proposed method can cope with a point track tensor contaminated with many outliers. In addition, we modified the bundle adjustment module to make it robust enough to handle classification errors. Our method achieves high accuracy while reducing the time significantly compared to classical methods. However, our method suffers from limitations we will address in future work. Among the limitations of the technique are that it often fails to consider the entire set of input images and also struggles with a very high outlier ratio.

### Acknowledgments

Research at the Weizmann Institute was partially supported by the Israel Science Foundation, grant No. 1639/19, by the Israeli Council for Higher Education (CHE) via the Weizmann Data Science Research Center, by the MBZUAI-WIS Joint Program for Artificial Intelligence Research and from the Estates of Tully and Michele Plesser and the Anita James Rosen and Harry Schutzman Foundations.

## Appendix

## A Implementation details

Our code and preprocessed point tracks data will be made publicly available.
Framework. Our method was trained and evaluated on NVIDIA A40 GPUs. We used PyTorch [] as the deep learning framework and the ADAM optimizer [] with normalized gradients.

Training.
During training, for each epoch, we processed all training scenes sequentially. For each scene, we randomly selected a subset of 10%-20% of the images in the scene. We used a validation set for early stopping. More specifically, we selected the checkpoint with the minimal reprojection error after Bundle Adjustment. The validation and test sets were evaluated by using the complete point tracks matrix. The training time of our model on the Megadepth dataset took roughly 16 hours on a single NVIDIA A40 GPU.

Architecture details.
We use normalized point tracks $\mathbf{x}_{ij}$, as inputs to our method which are normalized using the known intrinsic parameters. The shared features encoder $E$ has 3 layers, each with 256 feature channels and ReLU activation. The camera head $H_{\mathrm{cams}}$, 3D point head $H_{\mathrm{pts}}$, and the outliers classification head $H_{\mathrm{outlier}}$, each have 3 layers with 256 channels. After each layer in $E$ we normalize its features by subtracting their mean.

Hyper-parameter search.
We tried different implementation hyper-parameters including (1) learning rates $\in\{1e-2,1e-3,1e-4\}$, (2) network width $\in\{128,256,512\}$ for the encoder $E$ and the heads, (3) number of layers $\in\{2,3,4,5\}$ in these networks, and (4) threshold for outlier removal $\in\{0.4,0.5,0.6,0.7\}$.

Bundle adjustment
For the bundle adjustment, we employed the Ceres Solver’s implementation of bundle adjustment [] with the Huber loss to enhance robustness (with parameter 0.1). In each bundle adjustment round, we limited the number of iterations to 300 or until convergence, whichever occurred first.

## B Constructing point tracks

Our preprocessing step for constructing the point tracks is as follows:

- 1.
Extracting SIFT [] features for all the images.
- 2.
Applying exhaustive pair-wise RANSAC [] matching for all image pairs.
- 3.
Chaining each sequence of two-view matches for creating a point track.

We discard a 3D point and its track if it is viewed in less than 3 cameras, or if it includes an inconsistent cycle i.e. we detect two key points in the same image for the same point track.

The above preprocess, defines a valid point track structure for our architecture. We next define the inliers/outliers labeling process which assumes that every training scene has an associated COLMAP[] reconstruction:

- 1.
For each track within our point track set:
(a)
Identify the corresponding track in COLMAP’s reconstruction output
(b)
For every keypoint in the track:
•
Label the keypoint as an outlier if it does not appear in the corresponding COLMAP track ; otherwise, label it as an inlier.
- 2.
Remove keypoints labeled as outliers from the point tracks.
- 3.
Triangulate the 3D points from the cleaned point tracks using the ground truth camera poses, obtained from COLMAP.
- 4.
Calculate the reprojection error for the triangulated 3D points across the entire point tracks, including the ones labeled initially as outliers.
- 5.
Assign an inlier label to each keypoint whose reprojection error is below 4 pixels; keypoints exceeding this threshold are classified as outliers.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The main question is about the Weaknesses 1 and 2, i.e., what's the main novelty of the proposed method compared to the existing literature?
soundness: 3
strengths: 1. This paper improves the existing ESFM framework by integrating a new inlier-outlier classification branch, alongside a robust structure from motion (SfM) mechanism. This approach makes sense, given that outliers represent a primary challenge for SfM methods in real-world applications. Quantitative experiments demonstrate the efficacy of these components.

2. This paper conducts extensive experiments over various datasets to prove their claims.
summary: The paper presents an architecture for Multiview Structure from Motion (SfM) focusing on the robust recovery of camera pose and 3D scene structure from large, uncontrolled image collections that contain numerous outliers. Traditional and deep-based SfM approaches often struggle with outlier point tracks resulting from viewpoint variations, illumination changes, and repetitive structures, significantly impacting their performance. The authors propose an enhancement to the deep network architecture proposed by Moran et al. in 2021, incorporating an outlier classification module within a permutation equivariant framework. The revised architecture also integrates a robust bundle adjustment step to correct classification errors. The method has been tested on challenging datasets like MegaDepth and 1DSFM and has shown good accuracy and competitive runtimes against both contemporary deep-learning methods and established classical techniques.
weaknesses: 1. The main concern about this work is the novelty of the proposed framework. Compared to ESfM, the new designs are just (1) a simple classfication branch to identify inlier/outlier, which conducts simple binary classfication, and (b) robust BA considering high projection error, point track length, and multi-step refinement. Both these two designs have been proven effective over the long time and are not the new techniques from this work. For instance, systems like COLMAP, Theia, and VGGSfM incorporate variants of robust BA. Numerous matching and tracking methodologies, including SuperGlue, LightGlue, PiPs, and Cotracker, implement classification branches to classify inlier/outlier status. While the reviewer acknowledges the effectiveness and importance of these features, there are reservations about their originality as claimed by this work.


2. Moreover, the section of ablation study needs more thinking. The authors discussed "the impact of our permutation sets-of-sets equivariant architecture" and "importance of the equivariant features". However, as far as the reviewer understands, both these two are the benefits from the architecture of ESfM.  To validate the design choices of this paper, the authors should focus on why their error handling designs are better, instead of proving their base model is better. 


3. Concerns also arise regarding the reliability of the results. If interpreted correctly, the rotational metrics in the study are reported in degrees, not radians. Notably, in Tables 3 and 4, the reported rotational errors for some methods are as low as 0.02 or 0.03 degrees. The reviewer questions whether such marginal differences are statistically significant enough to discriminate between methods. (e.g., BlendedMVS, despite being half-synthetic, does not offer flawless ground truth)

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Can we compare with Mast3R / Spann3R [3]? These methods are new way of doing scene reconstruction and does show promise. They are also deep "reconstruction" methods because they use deep networks directly to get 3D pointmaps.  This comparison with help researchers find limitations or benefits of one method over another.

Is it possible to make results table a bit easy to read? Color + bold letters are not visible clearly. It is hard to find best method for each scene. Also, is there a way to get an average of some of metrics across scenes? (or over whole dataset). 

References:

[3] Wang, Hengyi, and Lourdes Agapito. "3d reconstruction with spatial memory." arXiv preprint arXiv:2408.16061 (2024).
soundness: 3
strengths: Paper is well-written with a clear objective being robustifying the method proposed in [1]. Overall content of the paper is well-written, coherent, and easy to understand and follow.

The proposed inlier-outlier prediction head shows improvement compared to ESFM [1]. Results across many scenes of indoor and outdoor datasets show that the proposed method improves over [1] and achieves comparable results to the best classical SfM methods.

Authors use unsupervised reprojection losses. This means that the network can be easily fine-tuned for different scenes, removing a limitation of deep SfM networks that they cannot generalize to scenes outside their training sets.

References:

[1] Moran, Dror, et al. "Deep permutation equivariant structure from motion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
summary: In this work, authors robustify the existing deep equivariant SfM method from [1] by incorporating an inlier-outlier prediction module and robust Bundle Adjustment. Specifically, deep SfM uses projective factorization on a 2D point track matrix, but the method in [1] assumes that this matrix is complete and outlier-free. Authors in this work argue that these assumptions are unrealistic. Hence, authors proposed a deep equivariant SfM that works on a large collection of internet photos and works with a matrix of point tracks that might have outliers. In their method, they first train an end-to-end outlier prediction and camera poses + 3D point prediction network from a matrix of 2D point tracks. At test time, they use "inlier-outlier" predictions from the pre-trained network to remove outliers and fine-tune the network for 3D pose + 3D points recovery using unsupervised losses for each scene.

Authors show impressive results across different datasets. Their method achieves the best results among deep SfM methods and achieves comparable results to the best classical SfM methods.

References:

[1] Moran, Dror, et al. "Deep permutation equivariant structure from motion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
weaknesses: Authors mention that they did not use Mast3R [2] as one of the baseline method because it does not work with large number of images. But, I do think Mast3R [2] is able to work with large sets of images. At least on Stretcha and BlendedMVS experiments, authors should be able to use Mast3R for full 3D reconstruction 

What is not clear from the paper is whether other methods use Robust Bundle Adjustment proposed in this work. I agree robust BA is necessary for accurate reconstruction, but the contribution of the paper is robust "deep equivariant SfM". For apples-to-apples comparison, authors should also compare with Standard BA or authors should apply robust BA post-processing to compared methods. I see that authors do provide ablation study for effectiveness of robust BA, but to verify the claim of deep robust sfm, it would be great to compare all the methods with either regular BA or robust BA.

References:

[2] Leroy, Vincent, Yohann Cabon, and Jérôme Revaud. "Grounding Image Matching in 3D with MASt3R." arXiv preprint arXiv:2406.09756 (2024).

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: Experiment-wise, why ESFM* has much larger rotation error in some cases, and why GLOMAP has good rotation predictions but not translation predictions. And it might be better to give the mean errors in the table for more straightforward comparisons.
soundness: 2
strengths: The paper is well-presented in general and easy to understand, with a comprehensive literature review and extensive experiments.
summary: This paper studies an important problem in multi-view structure from motion, specifically aiming to recover 3D points and camera poses from input point tracks. To this end, based on an existing permutation equivariant architecture, an outlier classification module is integrated to cope with outliers. A robust Bundle Adjustment method is also proposed, based on recursive potential outlier removal. Experiments are conducted on multiple datasets, demonstrating the robustness of the proposed method against existing ones.
weaknesses: The paper studies an interesting and practical problem, but the main weakness is its limited contribution. 

1. The proposed method is largely built on existing network architecture, the difference being only that outlier classification output channels are added. 

2. The proposed solution for handling outliers incurs much additional overhead in the recursive "finetune" process, lacks theoretical justification, comprises inliers recall, and can not reject outliers well (according to Table 7). Thus the contribution of the method is limited. The same problem also goes for the proposed robust Bundle Adjustment method.

Meta-review: ## Summary
The paper presents a new architecture for Multiview Structure from Motion (SfM) that improves the recovery of camera pose and 3D scene structure from large, uncontrolled image collections with outliers. The architecture incorporates an outlier classification module and a robust bundle adjustment step, achieving good accuracy and competitive runtimes against contemporary deep-learning methods and classical techniques.

##Strengths
* Integrates a new inlier-outlier classification branch and a robust structure from motion (SfM) mechanism. 
* Extensive experiments conducted over various datasets to prove claims, demonstrating the efficacy of these components. 
* The proposed inlier-outlier prediction head shows improvement compared to ESFM.
* Uses unsupervised reprojection losses for easy fine-tuning for different scenes.
* The paper is well-presented with a comprehensive literature review and extensive experiments.

## Weaknesses
* The ablation study section needs more consideration, as it should focus on why error handling designs are better than proving the base model is better.
* Concerns arise regarding the reliability of the results, as the rotational metrics are reported in degrees, not radians.
* The authors did not use Mast3R [2] as a baseline method due to its limitations in working with large number of images.
* The paper's contribution is largely built on existing network architecture, with the addition of outlier classification output channels.
* The proposed solution for handling outliers incurs additional overhead, lacks theoretical justification, comprises inliers recall, and can not reject outliers well.

## Conclusions

Based on the reviews and author's feedback, the paper shows an interinting approach and should be accepted as most of the concerns were addressed by the authors.

Decision: Accept (Poster)