Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method that accepts two partially overlapping images and a keypoint in one of them, and detects the location of the keypoint in the other image, regardless of whether it is visible, occluded, or outside the frame of the image. This is an interesting re-formulation of the correspondence estimation problem, the conventional formulation of which considers only keypoints that are visible in both images. Under the previous formulation, correspondence estimators are considered successful if they can declare that the correspondent of the keypoint is invisible in the target image. In addition to the capability to predict the location of invisible keypoints in isolation, the paper demonstrates that this capability is beneficial to camera pose estimation.
main_review: *Strengths*

The capability to predict the location of points that have just become occluded or exited the frame is useful in many tasks, such as perception for robots where such predictions could guide the robot’s next moves. Humans are good at this, but computer vision research has focused on the limited setting when the keypoints are visible in both images. Loss functions used for training networks for correspondence estimation only consider descriptors of visible keypoints. This is an important contribution of the paper and it goes beyond more qualitative work of predicting unseen objects based on context.

The approach builds upon the concept of Neural Reprojection Error (NRE) which was recently introduced by Germain et al. (2021). NRE does not require that the appearance of two image regions, that are hypothesized to be projections of the same 3D point, is similar. It can be viewed as an extension of the purely geometric reprojection error. Germain et al. did not consider occluded or out-of-bounds keypoints.

I consider the unified treatment of visible, occluded and out-of-bounds keypoints an advantage.

Experiments are thorough, findings are explained clearly, and limitations of the proposed method are pointed out. I consider the use of four publicly available datasets, two indoor and two outdoor, sufficient. Over one million keypoints are used in the experiments.

Generalization on additional datasets not used for training is also shown. This is important since it allows broader deployment of the algorithm.

The main limitation of NeurHal is low accuracy under favorable conditions, due to the low resolution of its output. (Improving this is left as a direction for future work.) 

The appendices contain useful additional information, experiments and implementation details. 

*Weaknesses*

It is unclear to me what the network actually learns. I speculate that it learns to warp the source image to the target given the relative pose between the two cameras and the depth of the keypoint in the source image. The output of this process is a correspondence map, which suggests that multiple warpings are considered. This is not described clearly enough. (The fact that the visibility of keypoint in the target images does not need to be labeled is clear.)

The metrics used for evaluation are somewhat arbitrary, or not sufficiently justified. Figure 3, for example, compares the results to random chance, which is a very weak baseline. Camera pose estimation is considered correct when the rotation error is under 20 degrees and the translation error is under 1.5 m. The latter is uninformative without knowing the magnitude of the translation between the two cameras. 1.5 m may be a very small or a very large error depending on the input data.


*Other Comments*

I find the term hallucination in the title marginally acceptable, but I think that it is abused in the rest of the paper. For example in: “Local feature matching methods are only able to identify the correspondent’s location when it is visible, while humans can also hallucinate its location when it is occluded or outside the field of view through geometric reasoning.” Humans do not hallucinate, they predict or estimate. 

Footnote 1 on p. 3 is unnecessary. The equation it refers to holds for any images in general configuration sharing the same intrinsic parameters (K matrix). The latter constraint can be easily relaxed.

Section 3.2: the calibration matrix K_c does not encode the boundaries of the image. Instead of a different matrix, what is needed are different boundary conditions for considering pixels to be within the image or not.

Section A.2 that discusses additional related work should be moved to the main paper. It is very relevant to the problem at hand.

Figure 8: precision on the y-axis is perplexing. The description suggests that this should be recall.


summary_of_the_review: The paper contains an important contribution as discussed above. The fact that predicting the locations of invisible points improves camera pose estimation provides further support that the method is useful in downstream tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a deep method (NeurHal) to predict visible, occluded or out-of-view keypoint matching from source images to target images. In training, a correspondence map is obtained from ground truth of camera matrix/pose, and images. In testing, the model directly outputs three categories of matchings: identified, outputpainting and inpainting. As an application of NeurHal, the method is applied to camera pose estimation and tested on ScanNet and Megadepth. The experiments shows the method improves the estimation accuracy, particularly the outputpainting correspondences. 
main_review: Strengths:
+The paper proposed an new problem: if and how neural network can predict unseen scene (keypoints) from a source-target image pair. Human is able to heuristically guess the location of outpainting correspondences from the geometry. For example, the keypoint displacement is relative camera pose and scene depth. This paper uses a deep model to accomplish this goal.
+ The paper proposed a method that is based on correspondence map and neural reprojection error [Germain 2021] with sufficient details.
+ NeurHal is tested on public datasets for precision of correspondences and camera pose estimation, showing better result than baselines.

Weakness:
- The proposed method has limited technique contribution. For example, the original NRE [Germain 2021]  is able to predict outpainting correspondences (it gives an extra category for un-matched keypoints). As a result, this paper adapts [Germain 2021] to solve this problem.
- In figure 3, the baseline (uniform correspondence map) is too weak. In figure 4, I do not find the precision for identified keypoints. It is unfair for the other methods.
- In camera pose estimation (section 4.2), the description of the baseline (correspondent to the first (light blue) method in figure 6) is not clear to me. Because some methods highly rely on good correspondences but others are not. So, it is hard to tell if the proposed method is able to improve the state-of-the-art camera pose estimation methods.   
summary_of_the_review: I tend to reject the paper because of the lack of contribution in technique and insufficient experiments. I would like to see 
1. the technique differences, compared with [Germain 2021] .
2. improve the baselines, and compare with the state-of-the-art camera pose estimation methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this manuscript, authors proposed a new problem of correspondence hallucination, in which for keypoint in the source image, its correspondence should be detected regardless if it is occluded or outside the field of view. In particular, the authors proposed a new model and training paradigm that learn to hallucinate correspondence by predicting its probability distribution of its location. Extensive experiments on both indoor and outdoor benchmarks demonstrated that proposed method  can help camera pose estimation and outperforms prior state of the art feature matching approaches.

main_review: Strengths:
(1) This work provides an interesting research direction correspondence learning, in which occluded and out of FoV correspondence can also help large-baseline camera pose estimation.
(2) Based on this insight: authors proposed a new model for hallucinating the correspondence based on attention blocks.
(3) Extensive experiments on indoor and outdoor benchmarks demonstrated that proposed model is able to hallucinate correspondence and outperform prior SoTA on wide baseline setting in terms of camera poes estimation.
(4) Author provided detailed supplementary material and trained model and source code for better supporting reproducibility.

Weakness:
(1) Authors admitted that the proposed method can only produce low resolution correspondence, which can have a negative impact on localization accuracy. 
(2) Is there any explanation why correspondence inpainting is harder for inpainting?
(3) Did authors define how to compute Kc for transforming points from image plane to correspondence plane? It seems that I am not able to find it in the main manuscript.
(4) It would be better if authors could provide more analysis and ablation study on the relationship between accuracy of camera pose estimation and distribution of hallucinated correspondence. And in which case, hallucinated correspondence will be more accurate or less accurate.
(5) Authors should also cite several other related work for feature matching:
Learning feature descriptors using camera pose supervision, ECCV 2020
Self-Supervised Geometric Perception, CVPR 2021
Patch2Pix: Epipolar-Guided Pixel-Level Correspondences, CVPR 2021

summary_of_the_review: In summary, based on the strength and weakness I mentioned, I like this paper's idea for hallucinating correspondence so that we can obtain better accuracy for camera pose estimation, although it would be better to have more analysis and ablation study for the impact of correspondence hallucination on camera pose estimation accuracy.
review_title: null

Meta-review: This paper receives positive reviews. The authors provide additional results and justifications during the rebuttal phase. All reviewers find this paper interesting and the contributions are sufficient for this conference. The area chair agrees with the reviewers and recommends it be accepted for presentation.
Decision: Accept (Poster)