Graph Neural Networks For Multi-Image Matching
Authors: Stephen Phillips
Kostas Daniilidis
Venue:  (2020)
Keywords: Graph Neural Networks
Multi-image Matching

Abstract: In geometric computer vision applications, multi-image feature matching gives more accurate and robust solutions compared to simple two-image matching. In this work, we formulate multi-image matching as a graph embedding problem, then use a Graph Neural Network to learn an appropriate embedding function for aligning image features. We use cycle consistency to train our network in an unsupervised fashion, since ground truth correspondence can be difficult or expensive to acquire. Geometric consistency losses are added to aid training, though unlike optimization based methods no geometric information is necessary at inference time. To the best of our knowledge, no other works have used graph neural networks for multi-image feature matching. Our experiments show that our method is competitive with other optimization based approaches.

Under review as a conference paper at ICLR 2020
GRAPH NEURAL NETWORKS FOR MULTI-IMAGE MATCHING
Anonymous authors Paper under double-blind review
ABSTRACT
In geometric computer vision applications, multi-image feature matching gives more accurate and robust solutions compared to simple two-image matching. In this work, we formulate multi-image matching as graph embedding problem, then use Graph Neural Network to learn appropriate embedding function for aligning image features. We use cycle consistency to train our network in an unsu- pervised fashion, since ground truth correspondence can be difficult or expensive to acquire. Geometric consistency losses are added to aid training, though unlike optimization based methods geometric information is necessary at inference time. To the best of our knowledge, no other works have used graph neural net- works for multi-image feature matching. Our experiments show that our method is competitive with other optimization based approaches.
INTRODUCTION
Feature matching is an essential part of Structure from Motion and many geometric computer vision applications. The goal in multi-image feature matching is to take 2D feature positions from three or more images and find which ones correspond to the same point in the 3D scene. Methods such as SIFT feature matching (Lowe, 2004) combined with RANSAC (Fischler & Bolles, 1981) have been the standard for decades. However RANSAC-based approaches are limited to matching pairs of images, which can lead to global inconsistencies in the matching. Other works, such as Wang et al. (2018), have shown improvement in performance by optimizing cycle consistency, i.e. enforcing the pairwise feature matches to be globally consistent.
However, these multi-view consistency algorithms struggle in distributed and noisy settings. Having image features suited for this task would help improve performance, and deep learning has revolu- tionized how image features are computed (Yi et al., 2016). In this paper, we want to leverage the power of deep representations in order to compute feature descriptors that are robust across multiple views.
Unfortunately, there obstacles to applying multi-view constraints directly to deep learning. Multi-view constraints are formulated in terms of sparse features, which traditional convolutional neural nets not designed handle. Thus will need new architecture to handle such straints. More fundamentally, deep neural networks need large amounts of labeled data to train. Consequently unsupervised training is more practical approach. In the absence of direct supervi- sion, the additional signal of geometric constraints can help disambiguate visually similar features and reject outliers. Thus incorporating such constraints important in training a network solve this task.
In this work, we address these concerns using Graph Neural Networks (GNNs). The proposed method works directly on the graph of correspondences between the image features, which is ag- nostic to how the correspondences were computed, thus allowing the algorithm work broad class of environments. To the best of our knowledge this work is the first to apply deep learning to the multi-view feature matching problem. We use an unsupervised loss, the cycle consistency loss, to train the network. We use geometric consistency losses to aid training, though no geometric information is used at inference time. Although our network is simple, it shows promising results compared to baselines which optimize for cycle-consistency without learned embeddings, using matrix factorization loss (Zhou et al., 2015b; Leonardos et al., 2017). Furthermore, since infer- ence requires only single forward pass the neural network, approach is faster achieve

---

Under review as a conference paper at ICLR 2020
I
Graph Neural Network
Embedding
Reature
Fy
Neisy Adjaceny Matrix
Embeedding Similaaity
Aij
FvinE
Geometric Consisteney
Information
Dist
PuJi
Loss
An illustration of the approach of this work. The Graph Neural Neural Network (GNN) (Battaglia et al., 2018) takes input the graph of matches and then outputs low rank embedding of the adjacency matrix of the graph. The GNN operates on an embedding over the vertices of the graph. In the figure, the GNN vertex embeddings are represented by different colors. The final embedding is used to construct pairwise similarity matrix, which we train to be low dimensional cycle-consistent representation the graph adjacency matrix, thus pruning the erroneous matches. We train the network using reconstruction loss on the similarity matrix with the noisy adjacency matrix, and thus do not need ground truth matches. In addition, we can use geometric consistency information, such as epipolar constraints, to assist training the network.
Figure 1:
comparable accuracy than methods which must solve optimization problem every time. We per- form experiments on the Rome16K dataset (Li et al., 2010) to test the effectiveness of our method compared to optimization based methods. Our contributions in this work are:
We use a novel architecture to address the multi-image feature matching problem using GNNs with graph embeddings.
We introduce an unsupervised multi-view cycle consistency loss that does not require la- beled correspondences train.
We demonstrate the effectiveness of geometric consistency losses in improving training.
2
RELATED WORK
2.1
FEATURE MATCHING
Image feature matching has a rich history of research in computer vision. Much work has done using hand-crafted feature descriptors such as SIFT (Lowe, 2004), SURF (Bay al., 2006), BRIEF (Calonder et al., 2012), or ORB (Mur-Artal et al., 2015). RANSAC Fischler & Bolles (1981) the widely used robust estimation technique filter out outliers from the matches. The combi- nation of RANSAC and hand-crafted feature descriptors has constituted the bulk of the matching literature for the last 40 years. More recently Suh et al. (2015) and Hu et al. (2016) have shown graph matching of the features can be added for more robust matches between images.
2.2 MULTI-IMAGE MATCHING
Multi-image matching has traditionally been done using optimization based methods minimizing cycle consistency based loss (see Section 3.3). Pachauri et al. (2013) and Arrigoni al. (2017) use the eigenvectors of the matching matrix to obtain a low dimensional embedding. However, the sophisticated
assumption of low Gaussian noise is not realistic. Zhou et al. (2015b) and ? optimization techniques on the matching matrix and thus produce more robust solutions. Leonardos al. (2017) implement distributed optimization scheme solve for cycle consistency. Swoboda
2

---

Under review as a conference paper at ICLR 2020
et al. (2019) implement convex relaxation of the low dimensional embedding problem. Shi et al. (2016) use tensor power iterations to solve the matching problem, also taking into account the intra- image matching graph. As alternative to optimization based techniques, Tron et al. (2017) used density based clustering techniques to compute multi-image correspondence. Fathian et al. (2019) use similar technique but formulates it as generalized Rayleigh quotient problem to achieve better results. Moving away from feature matching, Zach et al. (2010) uses cycle-consisteney constraints on pose graphs quite effectively. To the best of our knowledge, we are the first to use graph neural networks for multi-image matching.
2.3 DEEP LEARNING FOR MATCHING
Previous attempts improve image matching techniques using machine learning have focused on learning the descriptors given ground truth correspondence from curated datasets such as Zagoruyko & Komodakis (2015); Yi al. (2016); and Brachmann al. (2017). Rocco et al. (2018) use dense matching tensor with weakly supervised loss to learn good features for matching. Zhou et al. (2019) has shown that these learned representations have limited ability for improving SIM tasks.
There are other methods to build correspondences such as Choy et al. (2016), but they only handle two-view constraints and require dense correspondences. Most similar to our work, Zhang & Lee (2019) use graph neural networks to do intra-image feature processing before doing two image inter-image feature similarity. However, their work only trains for pairs of image, while ours is explicitly trained for multi-image matching. Also similarly, Yi et al. (2018) attempts to improve correspondences by learning match probabilities for two-view RANSAC for greater robustness and speed. Like us Zhu et al. (2017) use cycle consistency in their loss; however their method is for image generation and is restricted to pairwise cycle consistency. Our method can be applied to 3 or more images. Hartmann al. (2017) learns multi-image matching but requires heavy supervision from 3D object reconstructions, which can be difficult or expensive to obtain. Zhou et al. (2015a), unlike our method, uses dense correspondence, but uses cycle consistency to find semantic matches accross multiple views. Suwajanakorn et al. (2018) find 3 dimensional latent keypoints, trained using ground truth rotation and translation. However, their method is restricted to a limited number of object categories, which is different from the SfM setting we are considering here.
2.4 GRAPH NEURAL NETWORKS
Graph neural networks, true their name, are deep neural networks operating in graph domains. Multi-image matching is novel application of them, as they are more typically used for applications such citation networks or recommendation systems. They have received more attention recently (Bronstein et al., 2017; Defferrard et al., 2016; Kipf & Welling, 2017; Scarselli et al., 2009; Gama al., 2018b;a; Battaglia et al., 2018). The first methods to learn neural networks over graphs were the so-called Spectral methods. They used the eigenvectors of the graph Laplacian to compute con- volutions, which can learn graph specific convolution kernels (as Bruna al. (2013)), but require an a-priori known graph structure. Newer non-spectral methods do not require a-priori knowledge, as seen in Bronstein et al. (2017); Kipf & Welling (2017); Scarselli et al. (2009); and Gama et al. (2018a). Most of these methods use polynomials of the graph Laplacian to compute neighborhood averages. Gama et al. (2018b;a) formalize this notion and generalize it beyond the use of the graph Laplacian. To improve performance, more sophisticated aggregation techniques and global informa- tion passing can be used as discussed in Battaglia et al. (2018). The closest work on Graph Neural Networks to ours is Kipf & Welling (2016), which uses GNNs to reconstruct adjacency matrices, though not in a geometric context.
METHOD
Our goal is to learn optimal features that capture multiple image views by filtering out noisy feature matches. The input to our algorithm is a set of features and noisy correspondences, and the output is a new set of features where the pairwise similarities of these features correspond to the true matches. An outline of approach embeddings to be cycle consistent. We formulate this problem in terms of the correspondence graph of the features. Graphs (V,E) have set of vertices y and of directed edges & C V x V.
be seen in figure 1. We do this by training the new set of feature
3

---

Under review as a conference paper at ICLR 2020
Original Eimbedding
Optimally Rotated Embedding
Ground Truth Embedding
u =
E
I
Is
L
I
(a)
(b)
Figure 2: (a) An illustration of the idea of the universe of features. Each feature in each image corresponds to 3D point in the scene. We construct cycle consistent embeddings of the features by mapping each one to the one-hot vector of its corresponding 3D point. While there can be many features, there are fewer 3D points and thus this corresponds to a low rank factorization of the correspondence matrix. Best viewed in color. (b) Visualization of the learned embeddings. On the left we have the raw outputs, which are difficult to interpret. In the center, we rotated the features to best match the ground truth for a more interpretable visualization (see the end of Section 3.3). On the right, we have the ground truth embeddings, given as indicator vectors for which feature in the world the points correspond to. For the optimally rotated embedding we can see that the true embedding structure recovered (with some noise).
For vertex E V we use ^ to denote the h-hop neighbors of with the superscript left out for 1- 1-hop neighbors. Similarly E(v) is used to denote the edges associated with v. To denote the vertices connected to an edge & we write e(U1,22).
3.1 CORRESPONDENCE GRAPII
We assume there is initial set of feature matches represented graph G (V,E), with an associated adjacency matrix A. The graph is constructed from putative correspondences of image features across images, typically constructed using feature descriptor distance (c.g. SIFT feature distance). While there are many interesting methods for computing these putative correspondences (Suh et al., 2015; Yi al., 2018), do not explore them in this work. Typically putative corre- spondences are matched probabilistically, meaning a feature in one image matches to many features in another. The ambiguity in the matches could come from repeated structures the scene, insuffi- ciently informative low-level feature descriptors, or just an error in the matching algorithm. Filtering these noisy matches is our primary learning goal.
Each vertex of the graph E V is an image feature, corresponding to some ground truth 3D point p(v). Each edge e (V1. 02)
a potential correspondence. Associated with each vertex is an embedding f. E IRm which can include the visual feature descriptor, position, scale, orientation, etc. Similarly, each edge e has an associated feature f. R (in this work, initially just the weight of the feature association). We use these features the initialization for our learning algorithm.
In the absence of noise or outliers, this graph would have connected component for each visible point in the world, all mutually disjoint. Without noise, vertices v would only match with other vertices o' that correspond to the same 3D point the scene. Since features in this case represent unique locations in the scene, points in the same image would have edges between them. Mathematically, this can be expressed as e = (V1, 02) E & = P(r)) P(vz). In the noisy case expect this structure to be corrupted, i.e. there some edges (U1, U2) & such that P(v1) + ((22). Thus we need to prune the erroneous edges.
However, standard CNNs cannot operate this general graph structure. Thus cannot stan- dard convolutional nets to learn features for this task. Instead we use graph networks to learn feature representations this space, which descrihe in the next section.

---

Under review as a conference paper at ICLR 2020
3.2 GRAPH NEURAL NETWORKS FOR FEATURE MATCHING
As input to our method we are given graph G E) with the features described in Section 3.1: So Ve V and fe Ve E. As with any neural network, GNNs have layered outputs. We describe the output of layer & as {(k) E Rm Vo E V and () E P Ve E &, with the initial embeddings denoted fo f. and fe = fe. To aid future analysis, we will represent the features as matrices, denoting the vertex embedding matrix as F( and the edge embedding matrix as F If a superscript is not specified then it refers to the final output of the network.
First we describe older methods of GNNs give context, then we describe the method we use in this work. Many older methods assume we have the adjacency matrix A of the graph known a-priori Bruna et (2013), and can encode graph convolutions using the eigenvectors of A (these are known as spectral methods). However, from image set to image set, and thus we use non-spectral Graph Neural Networks. Newer models use non-spectral methods, which often ultimately amount to transforming each node with learned weights then averaging each node's 's representation with its neighbors, known as a message pass (Kipf & Welling, 2017; Defferrard al., 2016; Gama al., 2018b;a). Some works such Gama et al. use pooling operations on the vertices to make the graph smaller and thus aid computation, but need labels on every vertex of the original graph, we cannot use this. Most GNNs used in these works can be expressed mathematically as:
do not have this luxury, as the correspondence structure changes
H
f(k+1) =a 600 (00++
(k)
Σ Σ
Setowr Wh
h=0 h=O ('Eh(())
The weights/biases W ((()) are all learned, with no learning done on the edge weights fe(a,w") Note that this is just averages over h-hopp neighborhoods, where the weights on the edges remain static through the computation. Given that we are trying to prune edges, we add features over edges to learn which ones to prune and which to keep such as in Scarselli et al. (2009).
In this work we use the method and implementation described in Battaglia et al. (2018). Battaglia et al. (2018) uses message passes between node features as well as edge features, which the model can use to prune unnecessary or erroneous edges. Therefore there is intermediate processing on the edges before information is passed to the vertices.
Mathematically, this expressed as:
flk+1)
(4) + +U, (()))((())))) +U²( f(x) + U p(())
(1)
f(++ ((k) + (x( p(())
Σ ( f(k+1)
(2)
(EE(v)
Here the learned weights are denoted W and U, and the biases (k) and ((0). In Battaglia et al. (2018), they allow for more sophisticated aggregation functions, but in this work we simply use the mean function. Each one of these steps we refer to here as a message pass, and it is analogously equivalent to an iteration in distributed graph based optimization method. Between each of the message passes, we further process the features using MLPs.
3.3 CYCLE CONSISTENCY
Let M be the noiseless set of matches between our features, with Mij being the partial permuta- tion representing the matches between image i and image j. If the pairwise matches are globally consistent, then for all i,j.
M, = M
(3)
In other words, the matches between two images stay the same no matter what path taken to get there. This constraint is known cycle consistency, and has been used in number of works to optimize for global consistency Zhou et al. (2015b); ?) Leonardos et al. (2017). Stated this form, there are (n²² cycle consistency constraints check. A more elegant way represent cycle
5

---

Under review as a conference paper at ICLR 2020
Test Lios Wowd
011
000
7010
mendan Vhowo)
R12 T12
Geometric Loss from feature
Large Loss
to feature
3.00
(a) Illustrated here an example of how the geometric loss com- puted for one feature.
(b)
Figure 3: (a) Errors computed via absolute distance from the epipolar line, expressed by Equation 6 via the epipolar constraint. The epipolar line is the line of projection of the feature in the first image, projected into the second. The distance this line on the second image indicates how likely that point is to correspond geometrically to the original feature. There can be false positives along the projected line, as shown by the square feature in the figure, but other points will be eliminated, such as the hexagonal feature. (b) Training curves with and without Geometric Training loss, described The geometric training loss improves testing performance. Note how training with geometric consistency losses decreases the convergence time of the network. Best viewed color.
consistency is to first create a 'universe' of features that all images match to (see figure 2a). Then, can match the ith set of features to the universe using ground-truth matching matrix X, Then the cycle consistency constraint becomes:
M. = X
(4)
This reduces the number of our constraints from O ) to O(n²). This was shown to be equivalent to the original definition of cycle consistency (equation 3) in Huang & Guibas (2013). We try to
learn vertex embeddings Fy to approximate X in other words the final embedding should be an encoding of the universe features. As we do not have the ground truth matches M, we approximate it using the noisy adjacency matrix A of correspondence graph. Thus our loss would be
(5)
Here D could be L2 loss, L1 loss. many others. In this work, use the L1 loss. Note that because of this formulation, we can determine our embeddings only up to a rotation, as Fy R(Fy R) Fy RR F, Fy FJ Thus when visualizing embeddings, rotate them to make them more interpretable (see figure 2b).
3.4 GEOMETRIC CONSISTENCY LOss
In order to use geometric information, more traditional optimization based methods require the geometric information at inference time, while with learning approaches we can use it to speed up training while not needing at inference time. Thus geometric consistency losses are one distinct advantage of our method over more traditional optimization based approaches. We use the epipolar constraint, the simplest way to add a geometric consistency loss. The epipolar constraint describes how the positions of features in different images corresponding to the same point should be related. An illustration of this is provided in figure 3a, showing how this loss can help reject erroncous points. Given relative pose (Rej, Tıy) between two cameras and (transforms, to the epipolar

---

Under review as a conference paper at ICLR 2020
Method
Same Point Similarities
Different Point Similarities
Ideal
1.000 0.0000
0.0000 + 0.0000
Initialization Baseline
0.511 + 0.0168
0.2560 = 0.2060
5 Views, Noiseless
1,000 E 0.0004
0.1220 + 0.1670
6 Views, Added Noise
0.984 0.0031
0.0746 + 0.1570
3 Views, 5% Outliers
0.929 = 0.1790
0,1410 = 0,1480
3 Views, 10% Outliers
0.927 + 0.1790
0.1400 + 0.1510
Table I: Results for unsupervised training on synthetic data under various noise conditions. The table plots out the weights (mean and standard deviation) of the edges reconstructed by our model, for true positive matches and true negative ones. This shows under various noise conditions that our architecture can still recover the original connectivity structure of the matching graph.
constraint on corresponding feature locations X; and X,: X, [Tyyx R, = 0. In this work we use the two pose epipolar constraint (Tron & Daniilidis, 2014):
x; R [T, R,, =0
The constraint assumes that the Xa are calibrated (i.e. the camera intrinsics are known). Given our vertex embeddings matrix fu. we can formulate loss between all cameras and (the vertices associated with camera denoted V(i)):
Lig.geom (Fv)
E (EV()),EV()
(f. (o..s))X,TR. x
For our purposes, since we use low rank embeddings Fv, the loss would read (where c(k) is the appropriate camera for point index k)):
Lycom (Fv) = tr FvFT) = Σ (F (Fv))(G))
(8)
1
(G) = X R R (Ta)) Re(
4 EXPERIMENTS
4.1 SYNTHETIC GRAPH DATASET
We first our method on synthetically generated data as simple proof of concept. As these were simpler datasets, we the simpler edge-feature free model of (Kipf & Welling, 2017). To generate the data, we generate p points, each with its own randomly generated descriptor. To create the graph, we generate random permutation matrices, with noise applied to it after it is generated. We initialize the input descriptors using the synthetically generated ground truth descriptor, plus some added Gaussian noise. No geometric losses were added during training for these experiments. However, the method was robust in testing with different noise functions and parameters. The normalized noisy imput descriptors the structure well. However, the GNN recovered the true structure very well, as shown in Table 1, showing the appropriate edge similarities. With this simple test synthetic data passed, now move to more challenging datasets.
our baseline they correlate with the true values but do not preserve
4.2 ROME 16K GRAPH DATASET
We use the Rome16K dataset (Li et al., 2010) to test our algorithm in real world settings. Rome 16K consists of 16 thousand images of various historical sites in Rome extracted from Flickr, along with the 3D structure of the sites provided by bundle adjustment. While not standard dataset to test cycle consistency, most standard datasets have tens hundreds images, not enough to train GNN on. R6mel6K is typically used to test bundle adjustment methods. Therefore, to use our method, extract 6-tuples and 10-tuples of images with overlap of 80 points more test our algorithm, with the points established as corresponding in the given bundle adjustment output. For the initial embedding use the original 128 dimensional SIFT descriptors, normalized to have unit Tag norm,

---

Under review as a conference paper at ICLR 2020
Li Virw)
11 09 siees
Li (6
NusbALs Spocint PCEDS
012
0.18
6.64
012
0.0
0.2
0.04
0.04
9.42
SCEER ABC VS991
AII:
ABE (QrurT0
0.74
0.0
3.5
6.72
0.00
0.00
6.04
0.0
6.60
Figure 4: Plot of the losses of the baselines at different iteration numbers. The line shows the mean of the graph while the translucent coloring shows the 25th to 75th percentiles. The ROC AUC curves remain fairly consistent while the L1 loss goes noticibly down after more iterations. Our method compares to 35-45 iterations of MatchALS, while only having 8 message passes. PGDDS performs better than in L,1 but we perform similarly in the ROC AUC metric. These results still hold even when we change domains to the Graffiti dataset (see 4.2.1).
the calibrated x-y position, the orientation, and log scale of the SIFT feature. To calibrate we use the focal lengths provided in Romel6K's data, and we assume the image center in the center of the image (as none is provided by Rome16K), To construct the graph, we take each feature as a vertex and create edges to the nearest SIFT descriptors for the other images.
For these experiments we train with the L, norm and geometric consistency losses. We evaluate on a test set using the ground truth adjacency matrix, which we compute from the bundle adjustment given by the Romel6K dataset. However, we do not train with the ground truth adjacency matrix, only with a noisy version of the adjacency matrix. We also add the geometric loss (8) which helps improve testing performance (see figure 3b). We use the L1 and ROC AUC metrics to measure performance. For this method to work, we need the dimension of the embedding to be at least the number of unique points in the scene. Picking the correct number is difficult a-priori, and is problem with all cycle consistency based methods. Here we use the ground truth dimension of the embedding to test both our method and the baselines.
The network was implemented using the code provided by Battaglia et al. (2018) using Tensorflow 1.11 (Abadi et al., 2015). Our network has 16 layers, with 8 message passing operations placed every other layer. All layers were simple Multi-layer Perceptrons, with no batch norm. The network was trained with the Adam optimizer (Kingma & Ba, 2014) with learning rate of 10-4, with an exponentially decaying learning rate. We incorporate skip connections between the input, 6th, and 12th layers (all possible pairs).
We compare our method to spectral and optimization based baselines with different maximum iter- ation cutoffs. Figure 4 illustrates this by plotting the means of various metrics and their 2h and 75th percentiles, with table 2 giving the exact numbers. Our network, though only using 8 message passes, has comparable accuracy to MatchALS (Zhou et al., 2015b) run 35 to 45 iterations, with an equivalent message passing step at each phase. Although method does not outperform the Pro- jected Gradient Descent- Doubly Stochastic (PGDDS) (Leonardos et al., 2017) method, we perform comparably to them in the ROC AUC metric.
8

---

Under review as a conference paper at ICLR 2020
Method (6 Views)
L1
L
Area under ROC
Time (sec)
MatchALS 15 Iterations
0.101 = 0.008
0.022 = 0.004
0.918 = 0.073
0.074 = 0.008
MatchALS 35 Iterations
0,046 0.016
0.010 = 0.005
0.910 0.072
0.139 = 0.041
MatchALS 50 Iterations
0.029 + 0.017
0.008 0.005
0.905 0.068
0.260 0.048
PGDDSO 15 Iterations
0.017 0.002
0.007 = 0.001
0.918 = 0.087
0.796 = 0.147
PGDDSO 25 Iterations
0.016 + 0.002
0.007 + 0.002
0.919 + 0.087
1.670 + 0.328
PGDDSO 50 Iterations
0.015 + 0.002
0.006 0.002
0.920 0.087
3.363 0.528
Spectral
0.073 = 0.006
0.027
= 0.003
0.921 = 0.083
0.036 = 0.005
GNN (ours)
0.044 + 0.005
0.031 = 0,005
0.872 1 0,081
0.765 = 0.046
Method (10 Views)
T,2
Area under ROC
Time (sec)
MatchALS 15 Iterations
0.114 + 0.008
0.028
0.004
0.915 0.051
0.142 0.009
MatchALS 35 Iterations
0.065 + 0.009
0.013 + 0.003
0.907 + 0.053
0.355 + 0.073
MatchALS 50 Iterations
0,045 = 0.012
0.011 = 0.004
0.914 = 0.051
0.455 = 0.022
PGDDS 15 Iterations
0.017 = 0.001
0.008 0.001
0.903 1 0.061
1.225 0.159
PGDDS 25 Iterations
0.016 = 0.001
0.007 = 0.001
0.904 = 0.061
2.637 0.357
PGDDS 50 Iterations
0.016 = 0.001
0.007 = 0.001
0.905 = 0.061
6.116 = 1.009
Spectral
0.073 = 0.005
0.029 = 0.002
0.912 = 0.057
0.081 = 0.021
GNN (ours)
0.053 + 0.006
0.035 + 0.005
0.872 + 0.061
2.438 + 0.070
Table 2: Results on Rome16K Correspondence graphs, showing the mean and standard deviation of the L1 and L2- Our method was trained on ground truth correspondences but using unsuper- vised methods and geometric side losses. Thus we test against ground truth correspondence graph adjacency matrices computed from the bundle adjustment output. Our method performs better than 35 iteration of the MatchALS (Zhou et al., 2015b) method, but does not perform as well as 50 iter- ations. We perform better than simple eigenvalue based method (Pachauri al., 2013). Note that we perform much better in Lı performance rather than L2, as we optimized the network weights using an Ln loss.
4.2.1 GRAFFITI DATASET
We run our trained model on the more Graffiti Dataset from the Affine Covariant Regions dataset (formatted to be able to be input to our model properly). The Graffiti Dataset is the most common benchmark used in feature matching algorithms (c.g. Leonardos et al. (2017); Zhou et al. (2015b)). The results are shown in figure 4 in the rightmost figure. As the graffiti dataset is very small (only views total), we were not able to train on it. We randomly permute the intra-image order of the features to add some variance by design the GNN outputs the same result each time, while the optimization methods have a very small amount of variance. The transferred results of Graffiti are similar the test error of Romel6K- smaller Lı error and comparable ROC error. This shows that the GNNs trained in Rome 6K generalize similarly to the optimization based methods.
CONCLUSION
We have shown novel method for training feature matching using GNNs, using an unsupervised cycle consistency loss and geometric consistency losses. We have demonstrated end-to-end train- able GNNs have comparable performance the traditional optimization based baselines. For future work, will investigate robust losses for better outlier rejection, and using higher order geometric constraints, such as the tri-focal tensor, as additional loss terms. With this new architecture, we have the capability of training multi-image matching pipelines end to end, thus allowing image features explicitly for this task. We can extend this to distributed settings where we can train for matching images from multiple distributed agents.
train for
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath

---

Under review as a conference paper at ICLR 2020
Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fermanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing heterogeneous systems, 2015. URL https: www.tensorfll....oommm org/ Software available from tensorflow.org.
Federica Arrigoni, Eleonora Maset, and Andrea Fusiello. Synchronization in the symmetric inverse semigroup. In International Conference on Image Analysis and Processing, pp. 70-81. Springer, 2017.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez- Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.00 01261, 2018.
Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. conference on computer vision, pp. 404-417. Springer, 2006.
European
Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. dbac-differentiable ransac for camera localization. In JEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), volume 3, 2017.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, 2017.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint 1rXiv::312. 2.600,, 2013.
Michael Calonder, Vincent Lepetit, Mustafa Ozuysal, Tomasz Trzcinski. Christoph Strecha, and Pascal Fua. Brief: Computing local binary descriptor very fast. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7):1281-1295 2012.
Christopher B Choy, Jun Young Gwak, Silvio Savarese, and Manmohan Chandraker. Universal correspondence network. In Advances in Neural Information Processing Systems, pp. 2414-2422, 2016.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pp. 3844-3852, 2016.
Kaveh Fathian, Kasra Khosoussi, Parker Lusk, Yulun Tian, and Jonathan How. Clear: consistent lifting. embedding, and alignment rectification algorithm for multi-agent data association. arXiv preprint arXiv:1902.0225: 2019.
Martin Fischler and Robert Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24
(6)5381 1981.
Fernando Gama, Antonio Marques, Geert Leus, and Alejandro Ribeiro. Convolutional neural network architectures for signals supported on graphs. IEEE Transactions Signal Processing, 1044)
Fernando Gama, Geert Leus, Antonio G Marques, and Alejandro Ribeiro. Convolutional neural networks via node-varying graph filters. In 2018 IEEE Data Science Workshop (DSW), pp. 1-5. IEEE, 2018a.
Fernando Gama, Antonio G Marques, Alejandro Ribeiro, and Geert Leus. Mimo graph filters for convolutional neural networks. arXiv preprint arXiv: (00::8803000 02247, 2018b.
Wilfried Hartmann, Silvano Galliani, Michal Havlena, Luc Van Gool, and Konrad Schindler. Learned multi-patch similarity. In Proceedings of the IEEE International Conference on Com- pufer Vision, pp. 1586-1594, 2017.
10

---

Under review as a conference paper at ICLR 2020
Nan Hu, Boris Thibert, and Leonidas Guibas. Distributable consistent multi- graph matching. arXiv preprint arXiv:1611.0799) 2016.
Qi-Xing Huang and Leonidas Guibas. Consistent shape maps via semidefinite programming. In Computer Graphics Forum, volume 32. pp. 177-186. Wiley Online Library, 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint M4Kiv:4412.69000 2014.
Thomas N Kipf and Max Welling. 16Xiv:611 2016.
Variational
graph auto-encoders.
arXiv preprint
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Spyridon Leonardos, Xiaowei Zhou, and Kostas Daniilidis. Distributed consistent data associa- tion via permutation synchronization. In 2017 IEEE International Conference on Robotics and Automation (ICRA). pp. 2645-2652 2652. IEEE, 2017.
Yunpeng Li, Noah Snavely, and Daniel P Huttenlocher. Location recognition using prioritized fea- ture matching. European conference on computer vision, pp. 791-804. Springer, 2010.
David G Lowe. Distinctive image features from ocale-invarian keypoints. International journal of computer vision, 60(2):91-110, 2004.
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: versatile and accu- rate monocular slam system. IEEE Transactions on Robotics, 31(5):1147-1166 2015.
Deepti Pachauri, Risi Kondor, and Vikas Singh. Solving the multi-way matching problem by per- mutation synchronization. In Advances in neural information processing systems, pp. 1860-1868, 2013.
Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović. Akihiko Torii, Tomas Pajdla, and Josef Sivic. Neighbourhood consensus networks. In Advances in Neural Information Processing Systems, pp. 1651-1662, 2018
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2009.
Xinchu Shi, Haibin Ling, Weiming Hu, Junliang Xing, and Yanning Zhang. Tensor power itera- tion for multi-graph matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5062-5070, 2016.
Yumin Suh, Kamil Adamczewski. and Kyoung Mu Lee. Subgraph matching using compactness prior for robust feature correspondence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5070-5078, 2015.
Supasorn Suwajanakorn, Noah Snavely, Jonathan Tompson, and Mohammad Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. In Advances in Neural Information Processing Systems, pp. 2059-2070, 2018
Paul Swoboda, Ashkan Mokarian, Christian Theobalt, Florian Bernard, el. A convex relaxation for multi-graph matching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11156-11165, 2019.
Roberto Tron and Kostas Daniilidis. On the quotient representation for the essential manifold. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1574- 1581, 2014.
Roberto Tron, Xiaowei Zhou, Carlos Esteves, and Kostas Danillidis, Fast multi-image matching via density-based clustering. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit,, pp. 4057-4066, 2017.
11

---

Under review as a conference paper at ICLR 2020
Qianqian Wang, Xiaowei Zhou, and Kostas Daniilidis. Multi-image semantic matching by mining consistent features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 685-694, 2018.
Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In European Conference on Computer Vision, pp. 467-483. Springer, 2016.
Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning find good correspondences. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), number CONF, 2018.
Christopher 7ach, Manfred Klopschitz, and Mare Pollefeys. Disambiguating visual relations using loop constraints. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1436-1433. IEEE, 2010.
Sergey Zagoruyko and Nikos Komodakis. Learning compare image patches via convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 4353 ,
Zhen Zhang and Wee Sun Lee. Deep graphical feature learning for the feature matching problem. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
Qunjie Zhou, Torsten Sattler, Mare Pollefeys, and Laura Leal-Taixe. To learn or not to learn: Visual localization from essential matrices. IEEE Conference on Computer Vision and Pattern Recogni- tion Workshop: Image Matching: Local Features and Beyond, 2019.
Tinghui Zhou, Yong Jae Lee, Stella X Yu, and Alyosha A Efros. Flowweb: Joint image set alignment by weaving consistent, pixel-wise correspondences. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1191-1200, 2015a.
Xiaowei Zhou, Menglong Zhu, and Kostas Daniilidis. Multi-image matching via fast alternating minimization. In Proceedings of the IEEE International Conference on Computer Vision, pp- 4032 4040, 2015b.
Jun-Yan Zhu Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint, 2017.
A MORE DETAIL ON GNN ARCHITECTURE
All experiments were run with a 12 layer GNN with the ReLU nonlinearity and skip connections. The feature vector lengths were 32, 64, 128, 256, 512, 512, 512, 512, 512, 512, 1024, 1024, with skip connections between layers and 6, 6 and 12, and and 12. All were trained with the Adam op- timizer (Kingma& & Ba, 2014) and a learning rate 10 4 'he network was implemented in Tensorflow (Abadi et al., 2015), version 1.
B COMPUTING GEOMETRIC Loss
To compute the Geometric Loss during training we use the given rotations and translations from Rome 66K to compute the relative poses. In practice, one could just use the relative poses computed. We build up an intermediate Essential Matrix representations P, Q E 3p3xx3 with e being the number of views and p being the number of points. The matrix P is defined by the block matrix representation P: = [Taa]x Renn The matrix Q is defined by the block matrix representation Q: = Ro(s) Xs. We build the Essential matrices using F. =Q P PT Q, which has 3 by 3 blocks
Egj (Rei) [Tanl Ren + ([Ten)l Re(y X Re) Reti X, (Tnnl - X. R ( [Te(ixx Reu
x, IT (Ta))) Te(g)x Ro
Note that this is the same as equation 6 and thus have relative pariwise constraints.
12


Reviews:
[{"review_id": "S1egIL32FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents graph neural network approach for learning multi-view feature similarity. \nThe input data is local feature descriptors (SIFT), keypoint location, orientation and scale. The objective is to learn embedding such that features corresponding to the same 3d location will have similar embeddings, while different - far away. Such embedding distance matrix is called \"feature universe\" in the paper. \nInstead of ground truth correspondence matrix, authors use smth called \"noisy adjacency matrix\", although it was not clear to me, what does it mean precisely. Loss is also augmented with epipolar distance reprojection error. \nTraining and testing is performed on Rome16k dataset. \n\nOverall I find paper hard to follow and weak on experiment side. Comments and questions:\n\n 1) There is no proper (or any) traditional baseline, which is: match SIFT features, apply cross-consistency and SNN ratio thresholds, run RANSAC, throw away all the inconsistent things. See COLMAP, Bundler or any other Visual SfM/MVS pipeline. Moreover, Hartman et.al. method is cited, but not compared to, because it needs \"3d reconstruction\" for supervision. Here is my second objection. \n \n 2) Paper states that the method is unsupervised. Yet, it is based on known scene geometry (R and t) , which is typically obtained via 3d reconstruction pipeline. Once you do it,  you actually have ground truth correspondences, a lot of them. I don`t understand, why not use them. \n \n 3) Experimental validation is also weird. Method was trained on Rome16k and tested on it as well. No other datasets were used besides Graffity sequence (see below). The metric is L1 norm on adjecency matrix instead of some.\n \n 4) Regarding \"feature universe\". It is clear, that one cannot fit (or can?) the full n_features x n_3d_points matrix into GPU memory, as it should be super huge matrix. No details were given on how such problems are tackled.  \n\n 5) The paper, unfortunately, has a number of side claims, which are false and actually irrelevant to the paper core. Just to list a few: \"deep learning has revolutionized how image features are computed (Yi et al., 2016).\" It has not. SIFT is still quite the gold standard:  (To Learn or Not to Learn: Visual Localization from Essential Matrices, https://arxiv.org/abs/1908.01293, https://image-matching-workshop.github.io), besides that cited LIFT method was not used in practice because of being super slow.\n - \"More fundamentally, deep neural networks need large amounts of labeled data to train. In the case of multi-image feature matching, one would need hand-labeled point correspondences between images, which can be difficult and expensive to obtain.\".\n  Nobody hand-labels correspondences. Instead, one uses runs 3d reconstruction pipeline with densification like COLMAP to obtain dense depth map, what where one can get multiview correspondences (e.g., MegaDepth: Learning Single-View Depth Prediction from Internet Photos https://arxiv.org/abs/1804.00607) \n  - \"Typically putative correspondences are matched probabilistically, meaning a feature in one image matches to many features\nin another. The ambiguity in the matches could come from repeated structures in the scene, insuffi-\nciently informative low-level feature descriptors, or just an error in the matching algorithm. Filtering\nout these noisy matches is our primary learning goal.\". Typically, one-to-many matches are just thrown away (e.g from Bundler SfM paper \"Modeling the World from Internet Photo Collections\", Sec.4.2: \"If a track contains more than one keypoint in the same image, it is deemed inconsistent.\nWe keep consistent tracks containing at least two keypoints for the next phase of the reconstruction procedure\").\n\n 6) Citations are sometimes weird. E.g. part of OxfordAffine dataset (http://www.robots.ox.ac.uk/~vgg/research/affine/)  is referred as Graffity without any reference at all to the dataset itself (???), but with references to two irrelevant works which are testing on it. Why benchmark sycle consistency of such a small dataset of a flat surfaces? Then one could use Fountain sequence, at least, which has some non-planar structures on it. \n \n \n Minor Comments:\n \n  - Table 1 is hardly readable because of scientific notation used.\n\n\n****\nAfter rebuttal update.\n\nI am increasing my score a bit, but still think that paper is not is the shape for publishing. Method itself might be good, but evaluation is still bad, and what is worse - authors haven`t even tried to improve it.\n\n>At the end of the review period it will be revealed that we have 20 years of publication history in structure from motion and visual odometry.  \n\nThis, unfortunately, does not help the current paper.\n\n>We do in fact test on data not in our training set: we never trained on Graffiti. \nGraffity is still only 6 (six!!!) images, which are related by homography. One could show results, at least for HSequences (118 * 6) images or add other dataset. Since, it is not training, it could be done quite fast.\n\n> However, the focus of our paper is a novel feature representation, the formulation using a GNN framework, and the self-supervised losses. Overall, our paper is a novel approach to learning feature representations, a topic of great interest to the ICLR community, rather than a new structure from motion system that has to prove its superior performance over the current state of the art.\n\nI completely agree with it and would be happy to accept such paper to any kind of workshop. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HygrfK3jKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors provide a novel approach for outlier detection of SIFT feature matchings. They construct a graph by connecting each SIFT feature to its 5 nearest neighbors initially. Then optimize a regression loss to find the matching between the 2d keypoints and 3d universal points. Hence applying cycle-consistency to figure out image matchings.\n\nTheir formulation of the problem as a GNN pruning is brilliant and widens the path for future research in the feature matching field. They also incorporate epipolar line constraints as a regularizer for their training. Experiments show effectiveness of adding the epipolar constraints.\n\nTheir experiments show that this is a promising approach, but probably requires further research to achieve state of the art results.  \n\nI believe this work is a valuable and novel method for pruning the sift feature matches. It is in an early but acceptable stage. Adding extra regularizers on F_v (to make it one-hot?) would be a promising first step. Also it has been shown that GNNs performance deteriorates with increased depth. There are recent developments in GNNs that alleviate the oversmoothing problem. Maybe switching to these architectures would enable this work to try 15 pass GNNs.\n\nQuestion: How do you tune the hyper-parameters? (learning rate, number of layers, etc)\n\nImprovement: In the experiment section explain the specifics of the geometric loss, how camera calibration, etc is calculated.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Byx4oeYW9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multi-image matching method using a GNN with cyclic and geometric losses. The authors use vertex embeddings to exploit cycle consistency constraints in an efficient manner, then the GNN is used to learn the appropriate embeddings in an unsupervised fashion. The geometric consistency loss is used to aid training in addition. Experimental evaluation shows its performance compared to MatchALS (Zhou et al., 2015) and PGDDS (Leonardos et al., 2016). \n\nI think this paper has some potential but has not matured yet. My main concerns are as follows. \n\n1) Cyclic consistency terms\nThe way of learning vertex embeddings using cycle consistency in an unsupervised manner is interesting, but I'm not sure whether we should call it real cycle consistency. The authors assume a cycle going from a vertex to its embeddings, coming back to the vertex. Given two vertices, it has the form of Eq. (4), which is called cycle consistency constraints in this paper. But, it's in effect nothing but the dot product similarity between two vertex embeddings. I agree that this can be interpreted as a type of cycle constraint, but does not involve any real cycle at the end. This needs to be justified. \n\n2) Unsupervised learning\nThe proposed, so-called, cycle constraint terms are learned using noisy adjacency matrix (as pseudo labels) from initial matches. This would be sensitive to the quality of the inial matches, so needs to be analyzed by experiments, which are not done. And, the effect of geometric consistency term is not clear at all in the experiments. Does it help? then how much? Some ablation studies are required. \n\n3) Experimental comparison\nWhile there exist many related papers on multi-image matching, the authors compared only to two methods, and the performance gain is not significant. The overall results are not convincing. See more related papers in the following. \n\nZhou et al., FlowWeb: Joint Image Set Alignment by Weaving Consistent, Pixel-Wise Correspondences, CVPR15\nSwoboda et al, A convex relaxation for multi-graph matching, CVPR2019\nShi et al., Tensor Power Iteration for Multi-Graph Matching, CVPR2016\nYan et al., Multi-Graph Matching via Affinity Optimization with Graduated Consistency Regularization, TPAMI16\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for learning multi-image matching using graph neural networks. The model is learned by making use of cycle consistency constraints and geometric consistency, and it achieves a performance that is comparable to the state of the art. While the reviewers view the proposed method interesting in general, they raised issues regarding the evaluation, which is limited in terms of both the chosen datasets and prior methods. After rounds of discussion, the reviewers reached a consensus that the submission is not mature enough to be accepted for this venue at this time. Therefore, I recommend rejecting this submission.
Decision: Reject