Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a descriptor-free feature matching method based on LoFTR. Given a pair of input images, it uses a transformer architecture to calculate pairwise similarity between two sets of coarse features, and then optimize the fine-grained positions with fine-level features. The main differentiating factor compared with previous work is a 3d positional embedding generator that makes the transformer encoder aware of 3d information. A depth prediction head is also jointly learnt to provide depth encoding to aid the feature encoding. The proposed method achieves a large margin over the previous SoTA methods.

strength_and_weaknesses: Strengths:
- The paper is written well with many ablation studies and numerical findings. The results also show that the proposed 3d positional embedding helps.

- In the ablation study, the author illustrates the performance with and without the depth predicted, and presents a few failed approaches to make it work. Though the result probably would work against the proposed method in the paper, I’d appreciate this part which makes it clearer where it works better.

Weakness:
- The idea of 3d position embedding is not new. For example, this recent paper “Input-level Inductive Biases for 3D Reconstruction” described how to encode camera intrinsics and extrinsics into a transformer in positional encoding. As the paper mentions, PETR (Liu et al. 2022) also introduces a similar idea to the problem of object detection. Therefore, the application of a similar idea to the specific problem (using a transformer to do image matching) is not very novel.

- It’s a bit weird that the paper contributes a big chunk to depth predictors and in the end it doesn’t generalize well to many scenarios (Sec 4.3). In addition, training the depth predictor as a joint head would also make the training harder, converge slower, and increase the overhead for the model to be deployed. It’s realistic not to assume that the depth head trained on ScanNet would generalize well on KITTI. In my opinion, if you want to try to add depths, starting with a pretrained mono-depth model would be a better route. Since it doesn’t add overhead in the training process, and possibly would generalize better since it’s trained on a large corpus of data.

clarity,_quality,_novelty_and_reproducibility: - The paper is written clearly with useful figures.
- The ablation study is especially good and informational, which clearly shows the strengths and weaknesses of the proposed method.
- It should be reproducible since it’s based on an established approach with well tested data. The authors also mentioned the code will be published soon which should make reproducing the results easy.

summary_of_the_review: I recommend reject due to the following reasons:
- The proposed method should intuitively improve the results as a better positional embedding, incorporating more inductive bias such as 3d information of the input features has been demonstrated to improve the results quite well. However, the novelty is a bit hurt because this point has been shown on other domain-specific tasks. In addition, this paper is heavily based on LoFTR, the idea of using a transformer to do descriptor-free image matching.

- The emphasis on the depth predictor as a contribution is a bit problematic, as the results seem to show contradictory conclusions. I suggest using an off-the-shelf depth predictor to help with this argument.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This manuscript proposes a method, PA-LoFTE, for solving feature matching between images which generally follows a detection-description-matching three-stage pipeline. In order to achieve that, the authors utilize depth information from a depth predictor to generate 3D position embedding, then combine visual features, depth features, and 3D position embeddings with transformer.
The discussion on the necessity of 3D space information is limited, which is the major motivation of the proposed method. The main contribution is leveraging the 3D position embedding generator for feature generation. They perform an empirical study on indoor dataset and multiple tasks and conclude the superior performance of PA-LoFTR.

Overall, this paper could be a borderline paper. Given the clarification in the rebuttal, I would increase the score.

For the motivation:
1. It should be aware depth estimation the similar challenges to correspondence prediction, why depth estimation helps correspondence prediction?

Some suggestions for the experiments:
1. Compared with LoFTR, PA-LoFTR injects 3D position information into the original Coarse-Level Local Feature Transform. The proposed method (depth feature generator and 3D position embedding generator) is effective based on LoFTR, how about adopting other baselines?
2. In Table 1, how about the performance of LoFTR?
3. In Table 3, Without Depth Feature Generator performs better on the test dataset while having lower performance on the validation dataset, could you please discuss this problem? 
4. Could you please give some internal estimated depth images in the depth feature generator?

Minor comments:
1. Figure 1: Depth Feature Gnerator -> Depth Feature Generator.
strength_and_weaknesses: Please see summary
clarity,_quality,_novelty_and_reproducibility: Please see summary
summary_of_the_review: Overall, this paper could be a borderline paper. Given the clarification in the rebuttal, I would increase the score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a 3D position embedding module to a state-of-the-art feature matcher, LoFTR.

Specifically, the authors develop a 3D position embedding generator that encodes 3D point clouds for each pixel instead of encoding 2D pixel coordinates.

The proposed 3D position embedding generator requires a known relative pose from GT or from a third-party relative pose estimator.

Experiments on the Scannet and Kitti datasets show the effectiveness of the proposed method.
strength_and_weaknesses: [Strength]

1.  Using 3D position embedding to replace the original 2D pixel position embedding is interesting.
2. Results on the KITTI Stereo 2015 validation results are good.
3. Overall, this paper is easy to follow.

[Weaknesses]
The main weaknesses of this paper are experiments.

1. Since the proposed 3D position embedding module requires a known relative pose, the proposed method has limited applications. 

a) Applying the proposed method to the KITTI Stereo 2015 dataset is good. However, only results on the validation set are given. Please report the results on the testing set;
b) For the relative pose estimation task, though authors show that the proposed method can refine an estimated pose from Lofter, the performance gap between the proposed method and the original Lofter is big. To address this, I would expect:

i) Exactly following the training configuration of the original Lofter, and see whether the proposed 3D position embedding module still works, i.e., can refine an estimated pose from Lofter;

ii) Add a baseline experiment using the original 2D pixel position embedding and see whether the proposed 3D position embedding module can obtain better performance under the authors' setting;

iii) Using more recent networks (e.g., QTA) to check the proposed 3D position embedding module still works;

iv) Using the YFCC100M and IMC2021 datasets to check whether the proposed method can work on outdoor datasets.

c) For the ablation study, authors use the GT relative pose. It will make this section questionable;

d) Please add an ablation study, showing the robustness of the proposed method with respect to prior relative poses at a different level of accuracy.

2. [Minor] typos. For example, 

a) Gnerator in Figure 1, 
b) Self Attention (Inter-Image) ----->>>> Self Attention (Intra-Image)
c) Please bold matrix/vectors.



clarity,_quality,_novelty_and_reproducibility: Clarity : Good;
Quality: Fair;
Novelty: Fair;
Reproducibility: Fair;
summary_of_the_review: The main weakness of the proposed method is relying on a known relative pose, either from GT or from a third-party estimator.

The experiments fail to show that using a third-party estimator (Lofter) would make the proposed method outperforms the third-party estimator. 

Please use the same training setting as the third-party estimator.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper receives 3x rejects, the decision is clear. 
justification_for_why_not_higher_score: This paper receives 3x rejects, the decision is clear. 
justification_for_why_not_lower_score: NA
Decision: Reject