Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a surprising new method for keypoint matching across image pairs. The idea is essentially to use randomly-initialized CNNs to generate the features. The random features are first preprocessed slightly to get local peaks, and then the features which do not have mutual neighbors are discarded. Finally matches are found with a consensus mechanism similar to RANSAC, which can be split into a clustering stage (generating hypotheses) followed by a scoring stage (selecting hypotheses). The experiments show that this method is competitive with existing methods.

strength_and_weaknesses: I think this paper presents some surprising findings. I would not have expected that randomly-initialized CNNs could be made to compete so closely with models like SuperPoint and D2 and R2D2. I also appreciate the analysis on the number of CNNs, and the analysis over distance in the HPatches dataset. It's not the best method, but I learned something from reading the paper. 

clarity,_quality,_novelty_and_reproducibility: > "Our observation is that the CNN architecture ... can be regarded as visual descriptors"

Some typo here I guess. You cannot regard an architecture as "descriptors".


> for keypoint description, description and matching, 

?

> These matches enable us, e.g., 

I think this is a bad sentence. It almost sounds like "us" is the example. 

> "the aforementioned camera pose estimation is a minimal problem"

What is a "minimal" problem?

> "the feature map F is normalized to a saliency map to filter out homogeneous regions"

How does this work? I suppose the normalization here is to make the vector magnitude 1, or maybe asks for some sum to be 1, but why does that turn the vectors into saliency cues, and how does it "filter out homogeneous regions"?


> "Consequently, a descriptor can only be used to match the same type of descriptors extracted by the same CNN. Therefore, the process of keypoint extraction and matching of multiple CNNs is independent and can be deployed in parallel"

I don't really see how these two claims connect. The first claim is obvious (you can only compare features within a CNN), and the second is also obvious (you can run multiple CNNs in parallel), but the first claim does not imply the second.

> "a local softmax \alpha in each channel"

What is meant by "local" here? Is this redundantly referring to the fact that the softmax is applied per-channel, or is something else meant? The whole description in the neighborhood of this phrase could be improved. In particular, it would be really great to clarify the i,j,k notation or maybe not use it, since I don't think it's very natural for space and channels to be indexed the same way. 
summary_of_the_review: I like the paper, because it shows a surprising result. I think many people would want to see this.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to apply an ensemble of siamese CNNs to detect and describe keypoints. The parameters of these CNNS are not trained but set randomly. The keypoint descriptors produced by each siamese CNN are matched using a classical nearest neighbor with ratio test followed by a mutual nearest neighbor test. All these match candidates are filtered using a novel consensus mechanism.

The proposed approach is evaluated on 7-scenes and MegaDepth against RootSIFT, SuperPoint, D2-Net and R2D2 on three different modalities : color images, depth images and normal images.
strength_and_weaknesses: ### Strengths

1. The paper is overall well written and easy to read, except section 3.2 Consensus mechanism which is difficult to understand.
2. Employing untrained CNNs is an interesting idea

### Weaknesses

1. The consensus mechanism section 3.2 is difficult to understand.
2. The computational time is probably high compared to other methods as it requires 8 forward passes (8 siameses CNNs are employed in the experiments).
3. Concerning MegaDepth, thresholds of 10 cm and 50 cm have been defined, but if I am not mistaken, for each MegaDepth scene the 3D point cloud is obtained using SfM up to a scale factor. How did you obtain this scale factor ?
4. Why did not you run experiments on color images for MegaDepth ? (column "color" is missing both in Table 1 and 2)
clarity,_quality,_novelty_and_reproducibility: The paper is well written, except section 3.2 that is very difficult to understand. I believe adding some illustrations of the output of the MeanShift algorithm would help. The originality of the paper is somewhat limited to me as recently reviewed several papers investigating the idea of employing Neural Networks with random parameters is the related field of shape matching.
summary_of_the_review: The paper is not far from the acceptance threshold but the novelty and the performances are not sufficient (to me).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The proposed method adopts randomized CNNs ensemble and uses it as descriptor extractors and detects keypoints. Comparing to the state-of-the-art requires training specifically with supervised or self-supervised learning, the method doesn't need any of that and the author claims to have high robustness against illumination and other appearance variations.
strength_and_weaknesses: The idea is quite interesting. Basically the method seeks matching generalization with an ensemble of purely random parameters. The idea can potentially be used in some data requires matching which contains no training samples.

However, the task that authors targeted is specifically for appearance matching which is a well studied domain. The claim of missing matches with depth or nomal are failing (Table.1, Table.2, Figure.5 ...) are simply because those methods (e.g. Superpoint, D2-Net, R2D2) are not trained with those inputs. It's not very hard to swapping those inputs and re-train them.

Despite I know the idea is more like unsupervised vs supervised method, there is simply not much hard work you need to do with supervised method. For example, R2D2 is basically a self-supervised method, the training data can be generated with a set of normal or depth images. The authors need to motivate the idea more than showing the results.

Apart from that, the proposed method doesn't really work on any view point changes (Figure 8.) which is essential for image matching. The paper also missed important baseline such as DELF (multi-scale) which is also very robust for illumination changes.
clarity,_quality,_novelty_and_reproducibility: The idea is simple and clear. Should be fairly easy to reimplement.
summary_of_the_review: Overall the idea is simple and interesting. However, I believe image matching task has been well studied and there is no difficulty to adapt some state-of-the-art method onto depth or normal image for retraining. Therefore, the authors need to rethink about the task and the relation between the proposed method and the baselines. The question you might want to ask yourself: "Is there any need for using this unsupervised method in this task?" Otherwise it's just interesting but without any practical impact.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
This paper proposes a method for keypoint matching across image pairs by adopting randomized CNNs ensemble and using it as descriptor extractors and detects keypoints. The keypoint descriptors produced by each siamese CNN are matched using a classical nearest neighbor with ratio test followed by a mutual nearest neighbor test. The proposed approach is evaluated on 7-scenes and MegaDepth against RootSIFT, SuperPoint, D2-Net and R2D2 on three different modalities : color images, depth images and normal images.
# Strengths:
- The paper is overall well written and easy to read.
- Ablation study based on the number of CNNs and distance in the HPatches dataset.
# Weaknesses:
- The consensus mechanism section 3.2 is difficult to understand although it has been improved during the rebuttal phase.
- The computational time is probably high compared to other methods as it requires 8 forward passes (8 siameses CNNs are employed in the experiments).
- the task that authors targeted is specifically for appearance matching which is a well studied domain. The claim of missing matches with depth or normal are failing (Table.1, Table.2, Figure.5 ...) are simply because those methods (e.g. Superpoint, D2-Net, R2D2) are not trained with those inputs. It's not very hard to swapping those inputs and re-train them.
- Idea is more like unsupervised vs supervised method, there is simply not much hard work you need to do with supervised method. For example, R2D2 is basically a self-supervised method, the training data can be generated with a set of normal or depth images. The authors need to motivate the idea more than showing the results.

justification_for_why_not_higher_score: The paper proposes an interesting idea for key point matching, and the author feedback addressed some of the concerns from the reviewers. However, the paper should include some references about random-initilialized CNN because the computer vision literature shows that randomly weighted neural networks perform reasonably as feature extractors, and clearly add the constructive feedback from the reviewers before being accepted to ICLR.
Decision: Reject