Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a multi-image matching method using a GNN with cyclic and geometric losses. The authors use vertex embeddings to exploit cycle consistency constraints in an efficient manner, then the GNN is used to learn the appropriate embeddings in an unsupervised fashion. The geometric consistency loss is used to aid training in addition. Experimental evaluation shows its performance compared to MatchALS (Zhou et al., 2015) and PGDDS (Leonardos et al., 2016). 

I think this paper has some potential but has not matured yet. My main concerns are as follows. 

1) Cyclic consistency terms
The way of learning vertex embeddings using cycle consistency in an unsupervised manner is interesting, but I'm not sure whether we should call it real cycle consistency. The authors assume a cycle going from a vertex to its embeddings, coming back to the vertex. Given two vertices, it has the form of Eq. (4), which is called cycle consistency constraints in this paper. But, it's in effect nothing but the dot product similarity between two vertex embeddings. I agree that this can be interpreted as a type of cycle constraint, but does not involve any real cycle at the end. This needs to be justified. 

2) Unsupervised learning
The proposed, so-called, cycle constraint terms are learned using noisy adjacency matrix (as pseudo labels) from initial matches. This would be sensitive to the quality of the inial matches, so needs to be analyzed by experiments, which are not done. And, the effect of geometric consistency term is not clear at all in the experiments. Does it help? then how much? Some ablation studies are required. 

3) Experimental comparison
While there exist many related papers on multi-image matching, the authors compared only to two methods, and the performance gain is not significant. The overall results are not convincing. See more related papers in the following. 

Zhou et al., FlowWeb: Joint Image Set Alignment by Weaving Consistent, Pixel-Wise Correspondences, CVPR15
Swoboda et al, A convex relaxation for multi-graph matching, CVPR2019
Shi et al., Tensor Power Iteration for Multi-Graph Matching, CVPR2016
Yan et al., Multi-Graph Matching via Affinity Optimization with Graduated Consistency Regularization, TPAMI16

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper presents graph neural network approach for learning multi-view feature similarity. 
The input data is local feature descriptors (SIFT), keypoint location, orientation and scale. The objective is to learn embedding such that features corresponding to the same 3d location will have similar embeddings, while different - far away. Such embedding distance matrix is called "feature universe" in the paper. 
Instead of ground truth correspondence matrix, authors use smth called "noisy adjacency matrix", although it was not clear to me, what does it mean precisely. Loss is also augmented with epipolar distance reprojection error. 
Training and testing is performed on Rome16k dataset. 

Overall I find paper hard to follow and weak on experiment side. Comments and questions:

 1) There is no proper (or any) traditional baseline, which is: match SIFT features, apply cross-consistency and SNN ratio thresholds, run RANSAC, throw away all the inconsistent things. See COLMAP, Bundler or any other Visual SfM/MVS pipeline. Moreover, Hartman et.al. method is cited, but not compared to, because it needs "3d reconstruction" for supervision. Here is my second objection. 
 
 2) Paper states that the method is unsupervised. Yet, it is based on known scene geometry (R and t) , which is typically obtained via 3d reconstruction pipeline. Once you do it,  you actually have ground truth correspondences, a lot of them. I don`t understand, why not use them. 
 
 3) Experimental validation is also weird. Method was trained on Rome16k and tested on it as well. No other datasets were used besides Graffity sequence (see below). The metric is L1 norm on adjecency matrix instead of some.
 
 4) Regarding "feature universe". It is clear, that one cannot fit (or can?) the full n_features x n_3d_points matrix into GPU memory, as it should be super huge matrix. No details were given on how such problems are tackled.  

 5) The paper, unfortunately, has a number of side claims, which are false and actually irrelevant to the paper core. Just to list a few: "deep learning has revolutionized how image features are computed (Yi et al., 2016)." It has not. SIFT is still quite the gold standard:  (To Learn or Not to Learn: Visual Localization from Essential Matrices, https://arxiv.org/abs/1908.01293, https://image-matching-workshop.github.io), besides that cited LIFT method was not used in practice because of being super slow.
 - "More fundamentally, deep neural networks need large amounts of labeled data to train. In the case of multi-image feature matching, one would need hand-labeled point correspondences between images, which can be difficult and expensive to obtain.".
  Nobody hand-labels correspondences. Instead, one uses runs 3d reconstruction pipeline with densification like COLMAP to obtain dense depth map, what where one can get multiview correspondences (e.g., MegaDepth: Learning Single-View Depth Prediction from Internet Photos https://arxiv.org/abs/1804.00607) 
  - "Typically putative correspondences are matched probabilistically, meaning a feature in one image matches to many features
in another. The ambiguity in the matches could come from repeated structures in the scene, insuffi-
ciently informative low-level feature descriptors, or just an error in the matching algorithm. Filtering
out these noisy matches is our primary learning goal.". Typically, one-to-many matches are just thrown away (e.g from Bundler SfM paper "Modeling the World from Internet Photo Collections", Sec.4.2: "If a track contains more than one keypoint in the same image, it is deemed inconsistent.
We keep consistent tracks containing at least two keypoints for the next phase of the reconstruction procedure").

 6) Citations are sometimes weird. E.g. part of OxfordAffine dataset (http://www.robots.ox.ac.uk/~vgg/research/affine/)  is referred as Graffity without any reference at all to the dataset itself (???), but with references to two irrelevant works which are testing on it. Why benchmark sycle consistency of such a small dataset of a flat surfaces? Then one could use Fountain sequence, at least, which has some non-planar structures on it. 
 
 
 Minor Comments:
 
  - Table 1 is hardly readable because of scientific notation used.


****
After rebuttal update.

I am increasing my score a bit, but still think that paper is not is the shape for publishing. Method itself might be good, but evaluation is still bad, and what is worse - authors haven`t even tried to improve it.

>At the end of the review period it will be revealed that we have 20 years of publication history in structure from motion and visual odometry.  

This, unfortunately, does not help the current paper.

>We do in fact test on data not in our training set: we never trained on Graffiti. 
Graffity is still only 6 (six!!!) images, which are related by homography. One could show results, at least for HSequences (118 * 6) images or add other dataset. Since, it is not training, it could be done quite fast.

> However, the focus of our paper is a novel feature representation, the formulation using a GNN framework, and the self-supervised losses. Overall, our paper is a novel approach to learning feature representations, a topic of great interest to the ICLR community, rather than a new structure from motion system that has to prove its superior performance over the current state of the art.

I completely agree with it and would be happy to accept such paper to any kind of workshop. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Authors provide a novel approach for outlier detection of SIFT feature matchings. They construct a graph by connecting each SIFT feature to its 5 nearest neighbors initially. Then optimize a regression loss to find the matching between the 2d keypoints and 3d universal points. Hence applying cycle-consistency to figure out image matchings.

Their formulation of the problem as a GNN pruning is brilliant and widens the path for future research in the feature matching field. They also incorporate epipolar line constraints as a regularizer for their training. Experiments show effectiveness of adding the epipolar constraints.

Their experiments show that this is a promising approach, but probably requires further research to achieve state of the art results.  

I believe this work is a valuable and novel method for pruning the sift feature matches. It is in an early but acceptable stage. Adding extra regularizers on F_v (to make it one-hot?) would be a promising first step. Also it has been shown that GNNs performance deteriorates with increased depth. There are recent developments in GNNs that alleviate the oversmoothing problem. Maybe switching to these architectures would enable this work to try 15 pass GNNs.

Question: How do you tune the hyper-parameters? (learning rate, number of layers, etc)

Improvement: In the experiment section explain the specifics of the geometric loss, how camera calibration, etc is calculated.
review_title: Official Blind Review #2

Meta-review: The paper proposes a method for learning multi-image matching using graph neural networks. The model is learned by making use of cycle consistency constraints and geometric consistency, and it achieves a performance that is comparable to the state of the art. While the reviewers view the proposed method interesting in general, they raised issues regarding the evaluation, which is limited in terms of both the chosen datasets and prior methods. After rounds of discussion, the reviewers reached a consensus that the submission is not mature enough to be accepted for this venue at this time. Therefore, I recommend rejecting this submission.
Decision: Reject