Title: GRAPH NEURAL NETWORKS FOR MULTI-IMAGE MATCHING

Abstract: In geometric computer vision applications, multi-image feature matching gives more accurate and robust solutions compared to simple two-image matching. In this work, we formulate multi-image matching as a graph embedding problem, then use a Graph Neural Network to learn an appropriate embedding function for aligning image features. We use cycle consistency to train our network in an unsupervised fashion, since ground truth correspondence can be difficult or expensive to acquire. Geometric consistency losses are added to aid training, though unlike optimization based methods no geometric information is necessary at inference time. To the best of our knowledge, no other works have used graph neural networks for multi-image feature matching. Our experiments show that our method is competitive with other optimization based approaches.

1. INTRODUCTION:

Feature matching is an essential part of Structure from Motion and many geometric computer vision applications. The goal in multi-image feature matching is to take 2D feature positions from three or more images and find which ones correspond to the same point in the 3D scene. Methods such as SIFT feature matching (Lowe, 2004) combined with RANSAC (Fischler & Bolles, 1981) have been the standard for decades. However RANSAC-based approaches are limited to matching pairs of images, which can lead to global inconsistencies in the matching. Other works, such as Wang et al. (2018) , have shown improvement in performance by optimizing cycle consistency, i.e. enforcing the pairwise feature matches to be globally consistent.

However, these multi-view consistency algorithms struggle in distributed and noisy settings. Having image features suited for this task would help improve performance, and deep learning has revolutionized how image features are computed (Yi et al., 2016) . In this paper, we want to leverage the power of deep representations in order to compute feature descriptors that are robust across multiple views.

Unfortunately, there are obstacles to applying multi-view constraints directly to deep learning. Multi-view constraints are formulated in terms of sparse features, which traditional convolutional neural nets are not designed to handle. Thus we will need a new architecture to handle such constraints. More fundamentally, deep neural networks need large amounts of labeled data to train. Consequently unsupervised training is a more practical approach. In the absence of direct supervision, the additional signal of geometric constraints can help disambiguate visually similar features and reject outliers. Thus incorporating such constraints is important in training a network to solve this task.

In this work, we address these concerns using Graph Neural Networks (GNNs). The proposed method works directly on the graph of correspondences between the image features, which is agnostic to how the correspondences were computed, thus allowing the algorithm to work in a broad class of environments. To the best of our knowledge this work is the first to apply deep learning to the multi-view feature matching problem. We use an unsupervised loss, the cycle consistency loss, to train the network. We use geometric consistency losses to aid training, though no geometric information is used at inference time. Although our network is simple, it shows promising results compared to baselines which optimize for cycle-consistency without learned embeddings, using a matrix factorization loss (Zhou et al., 2015b; Leonardos et al., 2017) . Furthermore, since inference requires only a single forward pass over the neural network, our approach is faster to achieve Figure 1 : An illustration of the approach of this work. The Graph Neural Neural Network (GNN) (Battaglia et al., 2018) takes as input the graph of matches and then outputs a low rank embedding of the adjacency matrix of the graph. The GNN operates on an embedding over the vertices of the graph. In the figure, the GNN vertex embeddings are represented by different colors. The final embedding is used to construct a pairwise similarity matrix, which we train to be a low dimensional cycle-consistent representation of the graph adjacency matrix, thus pruning the erroneous matches. We train the network using a reconstruction loss on the similarity matrix with the noisy adjacency matrix, and thus do not need ground truth matches. In addition, we can use geometric consistency information, such as epipolar constraints, to assist training the network. comparable accuracy than methods which must solve an optimization problem every time. We perform experiments on the Rome16K dataset (Li et al., 2010) to test the effectiveness of our method compared to optimization based methods. Our contributions in this work are:

• We use a novel architecture to address the multi-image feature matching problem using GNNs with graph embeddings. • We introduce an unsupervised multi-view cycle consistency loss that does not require labeled correspondences to train. • We demonstrate the effectiveness of geometric consistency losses in improving training.

2. RELATED WORK:



2.1. FEATURE MATCHING:

Image feature matching has a rich history of research in computer vision. Much work has be done using hand-crafted feature descriptors such as SIFT (Lowe, 2004) , SURF (Bay et al., 2006) , BRIEF (Calonder et al., 2012) , or ORB (Mur-Artal et al., 2015) . RANSAC Fischler & Bolles (1981) is the most widely used robust estimation technique to filter out outliers from the matches. The combination of RANSAC and hand-crafted feature descriptors has constituted the bulk of the matching literature for the last 40 years. More recently Suh et al. (2015) and Hu et al. (2016) have shown graph matching of the features can be added for more robust matches between images.

2.2. MULTI-IMAGE MATCHING:

Multi-image matching has traditionally been done using optimization based methods minimizing a cycle consistency based loss (see Section 3.3). Pachauri et al. (2013) and Arrigoni et al. (2017) use the eigenvectors of the matching matrix to obtain a low dimensional embedding. However, the assumption of low Gaussian noise is not realistic. Zhou et al. (2015b) and ? use more sophisticated optimization techniques on the matching matrix and thus produce more robust solutions. Leonardos et al. (2017) implement a distributed optimization scheme to solve for cycle consistency. Swoboda et al. (2019) implement a convex relaxation of the low dimensional embedding problem. Shi et al. (2016) use tensor power iterations to solve the matching problem, also taking into account the intraimage matching graph. As an alternative to optimization based techniques, Tron et al. (2017) used density based clustering techniques to compute multi-image correspondence. Fathian et al. (2019) use a similar technique but formulates it as a generalized Rayleigh quotient problem to achieve better results. Moving away from feature matching, Zach et al. (2010) uses cycle-consistency-like constraints on pose graphs quite effectively. To the best of our knowledge, we are the first to use graph neural networks for multi-image matching.

2.3. DEEP LEARNING FOR MATCHING:

Previous attempts to improve image matching techniques using machine learning have focused on learning the descriptors given ground truth correspondence from curated datasets such as Zagoruyko & Komodakis (2015) ; Yi et al. (2016); and Brachmann et al. (2017) . Rocco et al. (2018) use a dense matching tensor with a weakly supervised loss to learn good features for matching. Zhou et al. (2019) has shown that these learned representations have limited ability for improving SfM tasks.

There are other methods to build correspondences such as Choy et al. ( 2016), but they only handle two-view constraints and require dense correspondences. Most similar to our work, Zhang & Lee (2019) use graph neural networks to do intra-image feature processing before doing two image inter-image feature similarity. However, their work only trains for pairs of image, while ours is explicitly trained for multi-image matching. Also similarly, Yi et al. (2018) attempts to improve correspondences by learning match probabilities for two-view RANSAC for greater robustness and speed. Like us Zhu et al. (2017) use cycle consistency in their loss; however their method is for image generation and is restricted to pairwise cycle consistency. Our method can be applied to 3 or more images. Hartmann et al. (2017) learns multi-image matching but requires heavy supervision from 3D object reconstructions, which can be difficult or expensive to obtain. Zhou et al. (2015a) , unlike our method, uses dense correspondence, but uses cycle consistency to find semantic matches accross multiple views. Suwajanakorn et al. (2018) find 3 dimensional latent keypoints, trained using ground truth rotation and translation. However, their method is restricted to a limited number of object categories, which is different from the SfM setting we are considering here.

2.4. GRAPH NEURAL NETWORKS:

Graph neural networks, true to their name, are deep neural networks operating in graph domains. Multi-image matching is novel application of them, as they are more typically used for applications such as citation networks or recommendation systems. They have received more attention recently (Bronstein et al., 2017; Defferrard et al., 2016; Kipf & Welling, 2017; Scarselli et al., 2009; Gama et al., 2018b; a; Battaglia et al., 2018) . The first methods to learn neural networks over graphs were the so-called Spectral methods. They used the eigenvectors of the graph Laplacian to compute convolutions, which can learn graph specific convolution kernels (as in Bruna et al. (2013) ), but require an a-priori known graph structure. Newer non-spectral methods do not require a-priori knowledge, as seen in Bronstein et al. ( 2017); Kipf & Welling (2017) ; Scarselli et al. (2009); and Gama et al. (2018a) . Most of these methods use polynomials of the graph Laplacian to compute neighborhood averages. Gama et al. (2018b; a) formalize this notion and generalize it beyond the use of the graph Laplacian. To improve performance, more sophisticated aggregation techniques and global information passing can be used as discussed in Battaglia et al. (2018) . The closest work on Graph Neural Networks to ours is Kipf & Welling (2016) , which uses GNNs to reconstruct adjacency matrices, though not in a geometric context.

3. METHOD:

Our goal is to learn optimal features that capture multiple image views by filtering out noisy feature matches. The input to our algorithm is a set of features and noisy correspondences, and the output is a new set of features where the pairwise similarities of these features correspond to the true matches. An outline of our approach can be seen in figure 1 . We do this by training the new set of feature embeddings to be cycle consistent. We formulate this problem in terms of the correspondence graph of the features. Graphs G = (V, E) have a set of vertices V and of directed edges E ⊆ V × V. On the left we have the raw outputs, which are difficult to interpret. In the center, we rotated the features to best match the ground truth for a more interpretable visualization (see the end of Section 3.3).

On the right, we have the ground truth embeddings, given as indicator vectors for which feature in the world the points correspond to. For the optimally rotated embedding we can see that the true embedding structure is recovered (with some noise).

For a vertex v ∈ V we use N h (v) to denote the h-hop neighbors of v, with the superscript left out for 1-hop neighbors. Similarly E(v) is used to denote the edges associated with v. To denote the vertices connected to an edge e ∈ E we write e(v 1 , v 2 ).

3.1. CORRESPONDENCE GRAPH:

We assume there is an initial set of feature matches represented as a graph G = (V, E), with an associated adjacency matrix A. The graph is constructed from putative correspondences of image features across images, typically constructed using feature descriptor distance (e.g. SIFT feature distance). While there are many interesting methods for computing these putative correspondences (Suh et al., 2015; Yi et al., 2018) , we do not explore them in this work. Typically putative correspondences are matched probabilistically, meaning a feature in one image matches to many features in another. The ambiguity in the matches could come from repeated structures in the scene, insufficiently informative low-level feature descriptors, or just an error in the matching algorithm. Filtering out these noisy matches is our primary learning goal.

Each vertex of the graph v ∈ V is an image feature, corresponding to some ground truth 3D point p(v). Each edge e = (v 1 , v 2 ) ∈ E is a potential correspondence. Associated with each vertex v is an embedding f v ∈ R m , which can include the visual feature descriptor, position, scale, orientation, etc. Similarly, each edge e has an associated feature f e ∈ R p (in this work, initially just the weight of the feature association). We use these features as the initialization for our learning algorithm.

In the absence of noise or outliers, this graph would have a connected component for each visible point in the world, all mutually disjoint. Without noise, vertices v would only match with other vertices v ′ that correspond to the same 3D point in the scene. Since features in this case represent unique locations in the scene, no points in the same image would have edges e between them.

Mathematically, this can be expressed as e = (v 1 , v 2 ) ∈ E =⇒ P (v 1 ) = P (v 2 ). In the noisy case we expect this structure to be corrupted, i.e. there are some edges e = (v 1 , v 2 ) ∈ E such that P (v 1 ) = P (v 2 ). Thus we need to prune the erroneous edges.

However, standard CNNs cannot operate on this general graph structure. Thus we cannot use standard convolutional nets to learn features for this task. Instead we use graph networks to learn feature representations on this space, which we describe in the next section.

3.2. GRAPH NEURAL NETWORKS FOR FEATURE MATCHING:

As input to our method we are given a graph G = (V, E) with the features described in Section 3.1: f v ∀v ∈ V and f e ∀e ∈ E. As with any neural network, GNNs have layered outputs. We describe the output of layer k as f

(k) v ∈ R m k ∀v ∈ V and f (k) e ∈ R p k ∀e ∈ E, with the initial embeddings denoted f (0) v = f v and f (0) e

= f e . To aid future analysis, we will represent the features as matrices, denoting the vertex embedding matrix as F (k)

V and the edge embedding matrix as

F (k) E .

If a superscript is not specified then it refers to the final output of the network.

First we describe older methods of GNNs to give context, then we describe the method we use in this work. Many older methods assume we have the adjacency matrix A of the graph known a-priori Bruna et al. (2013) , and can encode graph convolutions using the eigenvectors of A (these are known as spectral methods). However, we do not have this luxury, as the correspondence structure changes from image set to image set, and thus we use non-spectral Graph Neural Networks. Newer models use non-spectral methods, which often ultimately amount to transforming each node with learned weights then averaging each node's representation with its neighbors, known as a message pass (Kipf & Welling, 2017; Defferrard et al., 2016; Gama et al., 2018b; a) . Some works such as Gama et al. use pooling operations on the vertices to make the graph smaller and thus aid computation, but as we need labels on every vertex of the original graph, we cannot use this. Most GNNs used in these works can be expressed mathematically as: k) are all learned, with no learning done on the edge weights f e(v,v ′ ) . Note that this is just averages over h-hop neighborhoods, where the weights on the edges remain static through the computation. Given that we are trying to prune edges, we add features over edges to learn which ones to prune and which to keep such as in Scarselli et al. (2009) .

EQUATION

In this work we use the method and implementation described in Battaglia et al. (2018) . Battaglia et al. (2018) uses message passes between node features as well as edge features, which the model can use to prune unnecessary or erroneous edges. Therefore there is intermediate processing on the edges before information is passed to the vertices.

Mathematically, this is expressed as:

f (k+1) e(v1,v2) = σ a (k) + U (k) 0 f (k) e + U (k) 1 f (k) v1 + U (k) 2 f (k) v2

(1)

EQUATION

Here the learned weights are denoted W and U , and the biases a (k) and b (k) . In Battaglia et al.

(2018), they allow for more sophisticated aggregation functions, but in this work we simply use the mean function. Each one of these steps we refer to here as a message pass, and it is analogously equivalent to an iteration in a distributed graph based optimization method. Between each of the message passes, we further process the features using MLPs.

3.3. CYCLE CONSISTENCY:

Let M be the noiseless set of matches between our features, with M ij being the partial permutation representing the matches between image i and image j. If the pairwise matches are globally consistent, then for all i, j, k:

EQUATION

3) In other words, the matches between two images stay the same no matter what path is taken to get there. This constraint is known as cycle consistency, and has been used in a number of works to optimize for global consistency Zhou et al. (2015b) ; ?); Leonardos et al. (2017) . Stated in this form, there are O(n 3 ) cycle consistency constraints to check. A more elegant way to represent cycle consistency is to first create a 'universe' of features that all images match to (see figure 2a ). Then, one can match the i th set of features to the universe using a ground-truth matching matrix X i . Then the cycle consistency constraint becomes:

M ij = X i X ⊤ j (4)

This reduces the number of our constraints from O(n 3 ) to O(n 2 ). This was shown to be equivalent to the original definition of cycle consistency (equation 3) in Huang & Guibas (2013) . We try to learn vertex embeddings F V to approximate X -in other words the final embedding should be an encoding of the universe of features. As we do not have the ground truth matches M , we approximate it using the noisy adjacency matrix A of our correspondence graph. Thus our loss would be

L(A, F V ) = D(A, F V F ⊤ V )

(5) Here D could be an L 2 loss, L 1 loss, or many others. In this work, we use the L 1 loss. Note that because of this formulation, we can determine our embeddings only up to a rotation, as

F V R(F V R) ⊤ = F V RR ⊤ F ⊤ V = F V F ⊤ V

Thus when visualizing embeddings, we rotate them to make them more interpretable (see figure 2b ).

3.4. GEOMETRIC CONSISTENCY LOSS:

In order to use geometric information, more traditional optimization based methods require the geometric information at inference time, while with learning approaches we can use it to speed up training while not needing it at inference time. Thus geometric consistency losses are one distinct advantage of our method over more traditional optimization based approaches. We use the epipolar constraint, the simplest way to add a geometric consistency loss. The epipolar constraint describes how the positions of features in different images corresponding to the same point should be related. An illustration of this is provided in figure 3a , showing how this loss can help reject erroneous points. Given a relative pose (R ij , T ij ) between two cameras i and j (transforms j to i) the epipolar constraint on corresponding feature locations X i and X j :

X ⊤ i [T ij ] × R ij X j = 0.

In this work we use the two pose epipolar constraint (Tron & Daniilidis, 2014) :

X ⊤ i R ⊤ i [T j -T i ] × R j X j = 0 (6)

The constraint assumes that the X k are calibrated (i.e. the camera intrinsics are known). Given our vertex embeddings matrix f v , we can formulate a loss between all cameras i and j (the vertices associated with camera i denoted V(i)):

L ij,geom (F V ) = v∈V(i),u∈V(j) (f v • f u ) X ⊤ v R ⊤ i [T j -T i ] × R j X u (7)

For our purposes, since we use low rank embeddings F V , the loss would read (where c(k) is the appropriate camera for point index k):

L geom (F V ) = tr(G ⊤ F V F ⊤ V ) = k,l (F V ) k • (F V ) l (G) kl (8) (G) kl = X ⊤ k R ⊤ c(k) [T c(l) -T c(k) ] × R c(l) X l 4 EXPERIMENTS 4.1 SYNTHETIC GRAPH DATASET

We first test our method on synthetically generated data as a simple proof of concept. As these were simpler datasets, we the simpler edge-feature free model of (Kipf & Welling, 2017) . To generate the data, we generate p points, each with its own randomly generated descriptor. To create the graph, we generate random permutation matrices, with a noise applied to it after it is generated. We initialize the input descriptors using the synthetically generated ground truth descriptor, plus some added Gaussian noise. No geometric losses were added during training for these experiments. However, the method was robust in testing with different noise functions and parameters. The normalized noisy input descriptors are our baseline -they correlate with the true values but do not preserve the structure well. However, the GNN recovered the true structure very well, as shown in Table 1 , showing the appropriate edge similarities. With this simple test on synthetic data passed, we now move to more challenging datasets.

4.2. ROME 16K GRAPH DATASET:

We use the Rome16K dataset (Li et al., 2010) to test our algorithm in real world settings. Rome16K consists of 16 thousand images of various historical sites in Rome extracted from Flickr, along with the 3D structure of the sites provided by bundle adjustment. While not a standard dataset to test cycle consistency, most standard datasets have tens or hundreds images, not enough to train a GNN on. Rome16K is typically used to test bundle adjustment methods. Therefore, to use our method, we extract 6-tuples and 10-tuples of images with overlap of 80 points or more to test our algorithm, with the points established as corresponding in the given bundle adjustment output. For the initial embedding we use the original 128 dimensional SIFT descriptors, normalized to have unit L 2 norm, AUC ROC (Graffiti)

Figure 4 : Plot of the losses of the baselines at different iteration numbers. The line shows the mean of the graph while the translucent coloring shows the 25 th to 75 th percentiles. The ROC AUC curves remain fairly consistent while the L1 loss goes noticibly down after more iterations. Our method compares to 35-45 iterations of MatchALS, while only having 8 message passes. PGDDS performs better than us in L 1 but we perform similarly in the ROC AUC metric. These results still hold even when we change domains to the Graffiti dataset (see 4.2.1).

the calibrated x-y position, the orientation, and log scale of the SIFT feature. To calibrate we use the focal lengths provided in Rome16K's data, and we assume the image center is in the center of the image (as none is provided by Rome16K). To construct the graph, we take each feature as a vertex and create edges to the 5 nearest SIFT descriptors for the other images.

For these experiments we train with the L 1 norm and geometric consistency losses. We evaluate on a test set using the ground truth adjacency matrix, which we compute from the bundle adjustment given by the Rome16K dataset. However, we do not train with the ground truth adjacency matrix, only with a noisy version of the adjacency matrix. We also add the geometric loss (8) which helps improve testing performance (see figure 3b ). We use the L 1 and ROC AUC metrics to measure performance. For this method to work, we need the dimension of the embedding to be at least the number of unique points in the scene. Picking the correct number is difficult a-priori, and is a problem with all cycle consistency based methods. Here we use the ground truth dimension of the embedding to test both our method and the baselines.

The network was implemented using the code provided by Battaglia et al. ( 2018) using Tensorflow 1.11 (Abadi et al., 2015) . Our network has 16 layers, with 8 message passing operations placed every other layer. All layers were simple Multi-layer Perceptrons, with no batch norm. The network was trained with the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 10 -4 , with an exponentially decaying learning rate. We incorporate skip connections between the input, 6 th , and 12 th layers (all possible pairs).

We compare our method to spectral and optimization based baselines with different maximum iteration cutoffs. Figure 4 illustrates this by plotting the means of various metrics and their 25 th and 75 th percentiles, with table 2 giving the exact numbers. Our network, though only using 8 message passes, has comparable accuracy to MatchALS (Zhou et al., 2015b) (Zhou et al., 2015b) method, but does not perform as well as 50 iterations. We perform better than a simple eigenvalue based method (Pachauri et al., 2013) . Note that we perform much better in L 1 performance rather than L 2 , as we optimized the network weights using an L 1 loss.

4.2.1. GRAFFITI DATASET:

We run our trained model on the more Graffiti Dataset from the Affine Covariant Regions dataset (formatted to be able to be input to our model properly). The Graffiti Dataset is the most common benchmark used in feature matching algorithms (e.g. Leonardos et al. (2017) ; Zhou et al. (2015b) ).

The results are shown in figure 4 in the rightmost figure. As the graffiti dataset is very small (only 6 views total), we were not able to train on it. We randomly permute the intra-image order of the features to add some variance -by design the GNN outputs the same result each time, while the optimization methods have a very small amount of variance. The transferred results of Graffiti are similar to the test error of Rome16K -smaller L 1 error and comparable ROC error. This shows that the GNNs trained in Rome16K generalize similarly to the optimization based methods.

5. CONCLUSION:

We have shown a novel method for training feature matching using GNNs, using an unsupervised cycle consistency loss and geometric consistency losses. We have demonstrated end-to-end trainable GNNs have comparable performance the traditional optimization-based baselines. For future work, we will investigate robust losses for better outlier rejection, and using higher order geometric constraints, such as the tri-focal tensor, as additional loss terms. With this new architecture, we have the capability of training multi-image matching pipelines end to end, thus allowing us to train for image features explicitly for this task. We can extend this to distributed settings where we can train for matching images from multiple distributed agents.

A MORE DETAIL ON GNN ARCHITECTURE:

All experiments were run with a 12 layer GNN with the ReLU nonlinearity and skip connections. The feature vector lengths were 32, 64, 128, 256, 512, 512, 512, 512, 512, 512, 1024, 1024, with skip connections between layers 1 and 6, 6 and 12, and 1 and 12. All were trained with the Adam optimizer (Kingma & Ba, 2014) and a learning rate 10 -4 The network was implemented in Tensorflow (Abadi et al., 2015) , version 1.11.

B COMPUTING GEOMETRIC LOSS:

To compute the Geometric Loss during training we use the given rotations and translations from Rome16K to compute the relative poses. In practice, one could just use the relative poses computed. We build up an intermediate Essential Matrix representations P, Q ∈ R 3pv×3 , with v being the number of views and p being the number of points. The matrix P is defined by the block matrix representation P i = [T c(i) ] × R c(i) X i . The matrix Q is defined by the block matrix representation Q i = R c(i) X i . We build the Essential matrices using E = Q ⊤ P + P ⊤ Q, which has 3 by 3 blocks

E ij = (R c(i) X i ) ⊤ [T c(j) ] × R c(j) X j + ([T c(i) ] × R c(i) X i ) ⊤ R c(j) X j = X ⊤ i R ⊤ c(i) [T c(j) ] × R c(j) X j -X ⊤ i R ⊤ c(i) [T c(i) ] × R c(j) X j = X ⊤ i R ⊤ c(i) [T c(j) -T c(i)

] × R c(j) X j Note that this is the same as equation 6 and thus we have our relative pariwise constraints.

:



Figures and Tables:

Figure2: (a) An illustration of the idea of the universe of features. Each feature in each image corresponds to a 3D point in the scene. We can construct cycle consistent embeddings of the features by mapping each one to the one-hot vector of its corresponding 3D point. While there can be many features, there are fewer 3D points and thus this corresponds to a low rank factorization of the correspondence matrix. Best viewed in color. (b) Visualization of the learned embeddings. On the left we have the raw outputs, which are difficult to interpret. In the center, we rotated the features to best match the ground truth for a more interpretable visualization (see the end of Section 3.3). On the right, we have the ground truth embeddings, given as indicator vectors for which feature in the world the points correspond to. For the optimally rotated embedding we can see that the true embedding structure is recovered (with some noise).

Figure 3: (a) Errors are computed via absolute distance from the epipolar line, as expressed by Equation 6 via the epipolar constraint. The epipolar line is the line of projection of the feature in the first image, projected into to the second. The distance to this line on the second image indicates how likely that point is to correspond geometrically to the original feature. There can be false positives along the projected line, as shown by the square feature in the figure, but other points will be eliminated, such as the hexagonal feature. (b) Training curves with and without Geometric Training loss, described in 8. The geometric training loss improves testing performance. Note how training with geometric consistency losses decreases the convergence time of the network. Best viewed in color.

Results for unsupervised training on synthetic data under various noise conditions. The table plots out the weights (mean and standard deviation) of the edges reconstructed by our model, for true positive matches and true negative ones. This shows under various noise conditions that our architecture can still recover the original connectivity structure of the matching graph.

Table content: <table><tr><td>Method</td><td colspan="2">Same Point Similarities Different Point Similarities</td></tr><tr><td>Ideal</td><td>1.000 ± 0.0000</td><td>0.0000 ± 0.0000</td></tr><tr><td>Initialization Baseline</td><td>0.511 ± 0.0168</td><td>0.2560 ± 0.2060</td></tr><tr><td>5 Views, Noiseless</td><td>1.000 ± 0.0004</td><td>0.1220 ± 0.1670</td></tr><tr><td>6 Views, Added Noise</td><td>0.984 ± 0.0031</td><td>0.0746 ± 0.1570</td></tr><tr><td>3 Views, 5% Outliers</td><td>0.929 ± 0.1790</td><td>0.1410 ± 0.1480</td></tr><tr><td>3 Views, 10% Outliers</td><td>0.927 ± 0.1790</td><td>0.1400 ± 0.1510</td></tr></table>

Results on Rome16K Correspondence graphs, showing the mean and standard deviation of the L 1 and L 2 . Our method was not trained on ground truth correspondences but using unsupervised methods and geometric side losses. Thus we test against ground truth correspondence graph adjacency matrices computed from the bundle adjustment output. Our method performs better than 35 iteration of the MatchALS

Table content: <table><tr><td>Method (6 Views)</td><td>L 1</td><td>L 2</td><td>Area under ROC</td><td>Time (sec)</td></tr><tr><td colspan="3">MatchALS 15 Iterations 0.101 ± 0.008 0.022 ± 0.004</td><td>0.918 ± 0.073</td><td>0.074 ± 0.008</td></tr><tr><td colspan="3">MatchALS 35 Iterations 0.046 ± 0.016 0.010 ± 0.005</td><td>0.910 ± 0.072</td><td>0.139 ± 0.041</td></tr><tr><td colspan="3">MatchALS 50 Iterations 0.029 ± 0.017 0.008 ± 0.005</td><td>0.905 ± 0.068</td><td>0.260 ± 0.048</td></tr><tr><td>PGDDS0 15 Iterations</td><td colspan="2">0.017 ± 0.002 0.007 ± 0.001</td><td>0.918 ± 0.087</td><td>0.796 ± 0.147</td></tr><tr><td>PGDDS0 25 Iterations</td><td colspan="2">0.016 ± 0.002 0.007 ± 0.002</td><td>0.919 ± 0.087</td><td>1.670 ± 0.328</td></tr><tr><td>PGDDS0 50 Iterations</td><td colspan="2">0.015 ± 0.002 0.006 ± 0.002</td><td>0.920 ± 0.087</td><td>3.363 ± 0.528</td></tr><tr><td>Spectral</td><td colspan="2">0.073 ± 0.006 0.027 ± 0.003</td><td>0.921 ± 0.083</td><td>0.036 ± 0.005</td></tr><tr><td>GNN (ours)</td><td colspan="2">0.044 ± 0.005 0.031 ± 0.005</td><td>0.872 ± 0.081</td><td>0.765 ± 0.046</td></tr><tr><td>Method (10 Views)</td><td>L 1</td><td>L 2</td><td>Area under ROC</td><td>Time (sec)</td></tr><tr><td colspan="3">MatchALS 15 Iterations 0.114 ± 0.008 0.028 ± 0.004</td><td>0.915 ± 0.051</td><td>0.142 ± 0.009</td></tr><tr><td colspan="3">MatchALS 35 Iterations 0.065 ± 0.009 0.013 ± 0.003</td><td>0.907 ± 0.053</td><td>0.355 ± 0.073</td></tr><tr><td colspan="3">MatchALS 50 Iterations 0.045 ± 0.012 0.011 ± 0.004</td><td>0.914 ± 0.051</td><td>0.455 ± 0.022</td></tr><tr><td>PGDDS 15 Iterations</td><td colspan="2">0.017 ± 0.001 0.008 ± 0.001</td><td>0.903 ± 0.061</td><td>1.225 ± 0.159</td></tr><tr><td>PGDDS 25 Iterations</td><td colspan="2">0.016 ± 0.001 0.007 ± 0.001</td><td>0.904 ± 0.061</td><td>2.637 ± 0.357</td></tr><tr><td>PGDDS 50 Iterations</td><td colspan="2">0.016 ± 0.001 0.007 ± 0.001</td><td>0.905 ± 0.061</td><td>6.116 ± 1.009</td></tr><tr><td>Spectral</td><td colspan="2">0.073 ± 0.005 0.029 ± 0.002</td><td>0.912 ± 0.057</td><td>0.081 ± 0.021</td></tr><tr><td>GNN (ours)</td><td colspan="2">0.053 ± 0.006 0.035 ± 0.005</td><td>0.872 ± 0.061</td><td>2.438 ± 0.070</td></tr></table>

Peer review (iclr2020):
The paper presents graph neural network approach for learning multi-view feature similarity. The input data is local feature descriptors (SIFT), keypoint location, orientation and scale. The objective is to learn embedding such that features corresponding to the same 3d location will have similar embeddings, while different - far away. Such embedding distance matrix is called *feature universe* in the paper. Instead of ground truth correspondence matrix, authors use smth called *noisy adjacency matrix*, although it was not clear to me, what does it mean precisely. Loss is also augmented with epipolar distance reprojection error. Training and testing is performed on Rome16k dataset. Overall I find paper hard to follow and weak on experiment side. Comments and questions: 1) There is no proper (or any) traditional baseline, which is: match SIFT features, apply cross-consistency and SNN ratio thresholds, run RANSAC, throw away all the inconsistent things. See COLMAP, Bundler or any other Visual SfM/MVS pipeline. Moreover, Hartman et.al. method is cited, but not compared to, because it needs *3d reconstruction* for supervision. Here is my second objection. 2) Paper states that the method is unsupervised. Yet, it is based on known scene geometry (R and t) , which is typically obtained via 3d reconstruction pipeline. Once you do it, you actually have ground truth correspondences, a lot of them. I don`t understand, why not use them. 3) Experimental validation is also weird. Method was trained on Rome16k and tested on it as well. No other datasets were used besides Graffity sequence (see below). The metric is L1 norm on adjecency matrix instead of some. 4) Regarding *feature universe*. It is clear, that one cannot fit (or can?) the full n_features x n_3d_points matrix into GPU memory, as it should be super huge matrix. No details were given on how such problems are tackled. 5) The paper, unfortunately, has a number of side claims, which are false and actually irrelevant to the paper core. Just to list a few: *deep learning has revolutionized how image features are computed (Yi et al., 2016).* It has not. SIFT is still quite the gold standard: (To Learn or Not to Learn: Visual Localization from Essential Matrices, https://arxiv.org/abs/1908.01293, https://image-matching-workshop.github.io), besides that cited LIFT method was not used in practice because of being super slow. - *More fundamentally, deep neural networks need large amounts of labeled data to train. In the case of multi-image feature matching, one would need hand-labeled point correspondences between images, which can be difficult and expensive to obtain.*. Nobody hand-labels correspondences. Instead, one uses runs 3d reconstruction pipeline with densification like COLMAP to obtain dense depth map, what where one can get multiview correspondences (e.g., MegaDepth: Learning Single-View Depth Prediction from Internet Photos https://arxiv.org/abs/1804.00607) - *Typically putative correspondences are matched probabilistically, meaning a feature in one image matches to many features in another. The ambiguity in the matches could come from repeated structures in the scene, insuffi- ciently informative low-level feature descriptors, or just an error in the matching algorithm. Filtering out these noisy matches is our primary learning goal.*. Typically, one-to-many matches are just thrown away (e.g from Bundler SfM paper *Modeling the World from Internet Photo Collections*, Sec.4.2: *If a track contains more than one keypoint in the same image, it is deemed inconsistent. We keep consistent tracks containing at least two keypoints for the next phase of the reconstruction procedure*). 6) Citations are sometimes weird. E.g. part of OxfordAffine dataset (http://www.robots.ox.ac.uk/~vgg/research/affine/) is referred as Graffity without any reference at all to the dataset itself (???), but with references to two irrelevant works which are testing on it. Why benchmark sycle consistency of such a small dataset of a flat surfaces? Then one could use Fountain sequence, at least, which has some non-planar structures on it. Minor Comments: - Table 1 is hardly readable because of scientific notation used. **** After rebuttal update. I am increasing my score a bit, but still think that paper is not is the shape for publishing. Method itself might be good, but evaluation is still bad, and what is worse - authors haven`t even tried to improve it. >At the end of the review period it will be revealed that we have 20 years of publication history in structure from motion and visual odometry. This, unfortunately, does not help the current paper. >We do in fact test on data not in our training set: we never trained on Graffiti. Graffity is still only 6 (six!!!) images, which are related by homography. One could show results, at least for HSequences (118 * 6) images or add other dataset. Since, it is not training, it could be done quite fast. > However, the focus of our paper is a novel feature representation, the formulation using a GNN framework, and the self-supervised losses. Overall, our paper is a novel approach to learning feature representations, a topic of great interest to the ICLR community, rather than a new structure from motion system that has to prove its superior performance over the current state of the art. I completely agree with it and would be happy to accept such paper to any kind of workshop.