Exploring Matching Rates: From Key Point Selection to Camera Relocalization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 02:41:21

Abstract: Camera relocalization is the task of estimating camera pose within a known scene. It has important applications in the fields of Virtual Reality (VR), Augmented Reality (AR), robotics, and more within the domain of computer vision. Learning-based camera relocalizers have demonstrated leading pose accuracy, yet all current methods invariably utilize all the information within an image for pose estimation. This may offer robustness under challenging viewpoints but impacts the localization accuracy for viewpoints that are easier to localize. In this paper, we propose a method to gauge the credibility of image pose, enabling our approach to achieve more accurate localization on keyframes. Additionally, we have devised a keypoint selection method predicated on matching rate. Furthermore, we have developed a keypoint evaluation technique based on reprojection error, which estimates the scene coordinates for points within the scene that truly warrant attention, thereby enhancing the localization performance for keyframes. We also introduce a gated camera pose estimation strategy, employing an updated keypoint-based network for keyframes with higher credibility and a more robust network for difficult viewpoints. By adopting an effective curriculum learning scheme, we have achieved higher accuracy within a training span of just 20 minutes. Our method's superior performance is validated through rigorous experimentation. The code will be released.

Main paper: 
 introduction: 
 The concept of certainty is of profound importance in human endeavors, providing solace through the predictability of contemporary life. This pursuit of dependability permeates the field of camera relocalization, wherein there is a desire for pose estimation to be imbued with a similar degree of certainty. camera relocalization Figure 1 : Camera Relocalization. Camera relocalization is the task of estimating the camera pose within a known scene. Our approach endeavors to incorporate confidence measures and keypoint cues into the process of camera pose estimation, thereby providing credibility support during practical applications and further enhancing the positional accuracy of keyframes tasks necessitate the computation of a camera's 6-DoF pose relative to a pre-mapped scene when presented with query images. Training involves the ingestion of image sequences paired with their respective camera poses, with the aim of facilitating rapid and precise determination of the camera's position upon its subsequent scene encounter. This functionality is pivotal in diverse applications such as VR, AR, robotics, and autonomous navigation. Predominant research efforts in this area strive to refine the accuracy of camera pose estimations uniformly across all test images. Nonetheless, a nuanced analysis uncovers intrinsic limitations of this approach. Consequently, the ability to assess the likelihood of obtaining an accurate camera pose from an input image is equally critical as achieving high precision in the pose estimation itself. camera relocalization's evolution commenced with image retrieval, the primitive form of which, scene recognition, depended on robust image retrieval for localization. Progressive iterations in camera relocalization have focused on feature-based methods to augment precision. The introduction of random regression forests [11, 24] , capable of regressing scene coordinates, markedly enhanced camera relocalization accuracy. Deep learning frameworks [14] [15] [16] , utilizing deep networks for direct pose regression from images, have validated the proficiency of these networks in correlating imagery with their respective poses, albeit with limitations in generalization across novel viewpoints.Scene coordinate regression [2, [4] [5] [6] [7] signifies a dissection of the camera relocalization task, assigning the network the exclusive function of correlating input imagery with scene coordinates. The ACE [2] framework advances this task decomposition, with a feature backbone isolating scene feature descriptors, and scene coordinate regression dedicated to aligning these descriptors with scene coordinates. This bifurcation lessens the computational demand on scene-specific network models, thus expediting camera relocalization.We advocate for a further subdivision of the camera relocalization task. Specifically, the identification of inliers should be decoupled from the RANSAC and PnP procedures. This can be realized by employing a convolutional network to assist in keypoint evaluation. In parallel, this network can be harnessed, in conjunction with inlier data from the estimated pose, to ascertain the confidence of the camera pose estimation.To achieve our objectives, we have devised a pose estimation confidence assessment method based on the inlier rate, and on this foundation, we have developed a keypoint judgment network and training framework to further enhance the accuracy and assessment capabilities of camera relocalization. Specifically, our contributions can be summarized as follows:• We have designed a scene coordinate estimation method and a camera pose estimation framework based on a keypoint network.• We have developed a pose estimation confidence method based on the keypoint discrimination network.• Through extensive experimentation, we have demonstrated that our method outperforms the state-of-the-art methods in accuracy and validated the effectiveness of each component of our approach. 
related work: 
 The problem of camera relocalization involves recovering the 6DoF camera pose of a query image from a known scene. Current mainstream approaches include image or feature retrieval-based methods, pose regression methods, and scene coordinate regression methods. A more detailed introduction to these methods will be provided below.The implementation of image retrieval methods involves searching a query image [21] from a database of images with poses to retrieve similar images. The retrieved image with the most similar pose is then outputted. Arandjelovic et al. proposed VLAD [26] for global image feature description, enabling the retrieval of similar images. NetVLAD [1] improved upon VLAD features by using a convolutional neural network model. It introduced a learnable generalized VLAD layer, enabling a plug-and-play approach for network-based feature extraction. InLoc [25] provided a dataset with panoramic images that had variations in lighting and dynamics. It proposed a method to optimize pose estimation using synthesized virtual views. By extracting and matching feature points [10] , it established numerous matching relationships with noisy points. Robust pose estimation algorithms, such as RANSAC, significantly enhance the pose estimation accuracy. The main challenges of the above methods are the speed of database retrieval [1] and the accuracy of pose estimation. As the dataset size increases, the number of retrievals and computational complexity also increase.Methods based on random forests [24] can be applied to feature extraction and matching [13] . They use a random forest composed of regression decision trees to regress scene coordinates, which represent the coordinates of image pixels in the scene model. Method [9] based on random forests enables real-time learning of new scenes. Method [11] enhances camera relocalization in dynamic scenes with moving objects by suppressing the routing process based on the stability of features. However, random forest methods use depth information as input features.The PoseNet series [14] [15] [16] utilizes convolutional neural networks to extract features from images and directly regress camera poses.In subsequent PoseNet methods, PoseNet16 [14] improved localization results by controlling the model's confidence through uncertainty measurement and determining the correlation between input images and the scene. PoseNet17 [15] explores effective training methods and focuses on designing loss functions for direct pose regression.In DSAC [4] , a probabilistic model derived from reinforcement learning makes the optimal selection of RANSAC differentiable. This allows deep learning-based methods to be applied to camera relocalization in an end-to-end manner. DSAC++ [5] improves the network model, training methods, and data representation, further enhancing accuracy. ESAC [6] uses a hybrid expert model based on [4] to address the coverage of large datasets and ambiguity problems. DSAC* [7] improves [4] by using a better Resnet network structure and an improved training loss function, further enhancing pose estimation. AECRN [17] introduces RWEI to represent event data, enabling its effective application to scene coordinate regression.By designing an attention-based network architecture, it achieves higher accuracy in event-based scene coordinate regression. By extracting and matching feature points [10] , it established numerous matching relationships with noisy points. ACE [2] further refines the DSAC series methods [4] [5] [6] [7] by separating feature extraction from scene coordinate regression and using curriculum learning to reduce the training time of the scene coordinate regression head to around 5 minutes. This greatly reduces the deployment time of the model in new scenes. However, [2] neglects the estimation of important coordinate points. It estimates all points, neglecting both the estimation of image credibility and the consideration of highconfidence images and pixels. In this paper, we propose a method that measures the credibility of image pose estimation and guides scene coordinate estimation using key points, achieving further improvement in accuracy. 
method: 
 The previous ACE method [2] proposed a novel concept of dividing scene coordinate regression into two parts: the first part involves extracting features from the image, and the second part focuses on regressing the scene coordinates from the extracted features. This raised another question: can we integrate keypoint guidance into this process to enhance the scene coordinate regression capability of the regression head for those points critical to accurate localization in real scenes, thus achieving more precise camera relocalization?To address this, we designed a keypoint discrimination network.Before training the scene coordinate regression head, we trained a keypoint discrimination network to determine whether a particular feature is a key point in the scene that is essential for camera relocalization or scene recognition. Utilizing these keypoints, we determine the features and points to be used for scene coordinate regression learning.The selection of key points is, in fact, a highly challenging endeavor, yet it is crucial for localization tasks. This is because localization relies on the establishment of matching relationships between images, and the foundation of these relationships is the pairing of key points. The selection and matching of key points also represent the primary method by which humans and all other visual animals perform localization. A common procedure involves initially identifying points that are easily recognizable within an observation, followed by continuous tracking of these points and estimation of one's own pose. Inspired by insights from SiLK [13] , we posit that the fundamental criteria for key points should be their ease of matching and the likelihood of correct matches. Based on this, we define those key points through the highest matching rate. Let 𝑃 = {𝑝 1 , 𝑝 2 , . . . , 𝑝 𝑛 } be the set of all points in the image, and let 𝑀 (𝑝) be a function that measures the matching rate of point 𝑝. We define the key points 𝐾 as follows:EQUATIONwhere 𝜏 is a threshold value chosen based on the desired confidence level for the matches. The function 𝑀 (𝑝) is designed to reflect the ease of matching and correctness of the matches, which can be formulated as:EQUATIONHere, 𝑤 𝑖 are weights assigned based on the relative importance of matching with point 𝑝 𝑖 , and match_quality(𝑝, 𝑝 𝑖 ) is a function that returns a score representing the quality of the match between points 𝑝 and 𝑝 𝑖 . where 𝑑𝑒𝑠𝑐 (𝑝) represents the feature descriptor of point 𝑝, and 𝜎 is a scaling parameter that adjusts the sensitivity of the match quality to differences in the descriptors.EQUATIONWe utilized Colmap [22] to obtain the collection of keypoints, upon which we performed projection and optimization. The figure 4 visualizes the finalized selection of our keypoint collection. It is evident that our keypoint collection encompasses the primary and pivotal regions of the scene while disregarding areas that are deficient in texture or lack identifiable qualities.Selecting effective inliers in keypoint-guided inlier ratio estimation requires the network to determine which points should be considered as keypoints and which points generate more accurate 3D scene coordinates, leading to more correct 2D-3D correspondences. There are various criteria for determining keypoints.Traditional methods control the number of keypoints through nonmaximum suppression, while some newer convolutional neural network-based methods use matching accuracy to evaluate keypoints. However, for convolutional networks and scenes, each model's keypoints in each scene actually exhibit randomness and uncertainty. Therefore, we propose a keypoint probability evaluation method based on reprojection error, converting the reprojection error into keypoint probabilities using the following formula:EQUATION)where 𝜆 is the softness parameter that controls the tolerance level of the reprojection error. A smaller value indicates that a smaller reprojection error is required to achieve a higher keypoint probability. In this work, we set 𝜆 to 4. 𝑝 𝑖 represents the reprojection error of the scene coordinate estimation, and 𝑘 𝑖 represents the keypoint probability of that point. This formula is inspired by the sigmoid function. We further estimate the keypoint probabilities based on the trained scene coordinate estimation head. We found that after feature extraction by the convolutional network, the performance of the scene coordinate regression head in regressing scene coordinates exhibits randomness, meaning it does not always perform well at a specific point but rather shows a probabilistic behaviour. Therefore, we estimate the keypoint probabilities based on the trained scene coordinate estimation head, making our keypoint estimation method scene-and model-specific. This requires effective and efficient training of our network. Thus, we employ a simple network structure and accelerate the training of the keypoint selection network using curriculum learning. This approach enables us to achieve our goal of efficient training with minimal additional time.Our priority should be ensuring the quality of our outputs rather than maximizing the number of outputs. Currently, all vision-based camera relocalization methods aim to improve overall performance across the entire test set. However, it is not realistic to expect camera relocalization methods to perform efficiently and consistently in challenging scenarios with limited sampled data or abundant repetitive textures, considering that even humans exhibit limited generalization abilities in scenes with restricted input. This can be observed from the practical applications of existing methods in various scenes. For instance, in the 7scenes dataset, the "stairs" scene is notably more challenging for relocalization. Even a random image from a particular viewpoint in this scene could result in misjudgment.An approach from the perspective of human localization suggests continuously increasing confidence in their observations. When humans reach a certain viewpoint during the localization process and have sufficient confidence, they consider that they have accurately determined their position in the scene. Based on this, we believe that it is also necessary to evaluate the confidence of the input images and their output poses. The output pose of an image with high confidence can be considered a valid camera pose.To address this, we propose an image pose confidence estimation based on the inlier ratio. The confidence estimation of an image is calculated using the following formula:EQUATIONwhere 𝑟 𝑖 represents the pose estimation confidence of image 𝑖, 𝑁 𝑖 is the number of inliers output by the RANSAC-based PnP algorithm, and 𝑁 𝑎 is the total number of 2D-3D correspondences provided as input. When using this confidence measure for pose estimation, specific thresholds need to be set to determine the level of confidence. We consider confidence levels above 90% to be considerably confident, above 80% to be highly confident, above 60% to be moderately confident, and below 60% to be questionable. These threshold settings are based on our observations of the experimental data.When evaluating the confidence of image pose estimation using the inlier ratio, we found that existing methods perform poorly on the test set, with a negligible proportion of considerably confident estimates in the "stairs" scene. This is because current methods use all constructed 2D-3D correspondences for pose estimation indiscriminately with the selection of inliers solely determined by RANSAC. To improve the inlier ratio, we propose keypointguided inlier ratio estimation. Our approach is based on the idea that we should assess the suitability of 2D-3D correspondences for scene coordinate estimation during their construction. Therefore, we introduce an additional network branch in the relocalization framework to estimate the confidence of the scene coordinates of the network's estimated pixels. Utilizing a keypoint estimation network to estimate the probability of regressing correct inliers for each pixel, we decide whether to use that point for camera pose estimation. Consequently, our updated confidence estimation is given by:EQUATIONwhere 𝑁 𝑎𝑟 denotes the subset of 2D-3D correspondences with reliable confidence, as estimated using the keypoint network.In our study, we are required to train a total of four networks: one to obtain keypoints with the highest matching rate, one to acquire keypoints with the minimal reprojection error, and two additional networks dedicated to the initialization and further refinement of these two types of keypoints, respectively. Among all training processes, the network trained to identify keypoints based on the matching rate is the most time-consuming. This is attributed to the fact that the majority of points in an image are not keypoints, precluding targeted training. Conversely, all other training activities are confined to the regions of keypoints identified by the matching rate.We have ascertained the ground truth probabilities for keypoints at each point, denoted as 𝐺𝑇 . Consequently, within our keypoint estimation network, we are fundamentally estimating a probability mask for the extraction of keypoints. Accordingly, we employ the Binary Cross Entropy (BCE) loss as the loss function for our keypoint estimation network. This approach is applied to both the keypoint selection of the Highest Matching Rate and reprojection error:EQUATIONHere, 𝑁 is the number of samples, 𝑘 𝑖 represents the ground truth keypoint probability for the 𝑖-th feature, and k𝑖 represents the predicted keypoint probability for the 𝑖-th feature.Consequently, during the supervision process, we are limited to methods such as re-projection error for guiding the learning of scene coordinates. Specifically, the loss used for supervising is:EQUATION)where x is the 2D coordinate, y is the 3D scene coordinate, ȳ𝑖 is the GT 3D scene coordinate, h * is the GT pose, and V is the group of 3D scene coodinate satisfied with the determination of re-projection error 𝑒 𝜋 in [2] .We have designed a novel scene coordinate estimation head to accommodate more complex scene conditions. Enhancing the network depth of the scene coordinate estimation head boosts its ability to solve scene coordinates in challenging scenarios. We follow the Curriculum Training technique adopted in [2] for training our scene coordinate regression network. 
experiments: 
 Our method was implemented using PyTorch, building upon the publicly available code from ACE [2] . Here we detail our primary parameter settings. For the training of the matching-based keypoint network, a ResNet-like architecture was adopted. In training the keypoint-guided scene coordinate regression head, we fine-tuned the initialized head rather than starting the training anew, thus preserving the network's capability to regress scene coordinates for non-keypoint features. For the pose estimator, we continued to employ the method from ACE [2] , but with improvements made to the code to accommodate our approach. When employing the keypoint estimation network as a guide, we select points with an estimated probability value greater than 0.8 as keypoints, corresponding to a reprojection error of approximately 5.We conducted experiments on the 7Scenes [24] dataset, which offers a variety of small-scale indoor scenes captured with handheld devices, along with depth information and camera poses. We utilized the camera poses obtained via Structure from Motion (SfM) provided in [3] as ground truth (GT), as it has been validated in [3] that scene coordinate regression methods are more suited to pose estimation frameworks that employ a point matching mechanism akin to SfM. Our primary method of comparison was ACE [2] , yet we also compared against other scene coordinate regression approaches, including precursors to ACE and methods based on random forests. In Table 1 , we present the percentage performance of our method and others on the 7Scenes [24] dataset within 5 degrees and 5 centimeters of error. Our method is shown to further enhance relocalization performance, especially in challenging scenes such as 'stairs'.During our experimentation, we found that our method not only excels in percentage performance within the (5 • , 5𝑐𝑚) error margin but also significantly improves the precision of pose estimation. We have detailed the performance of our method compared to others within (2 • , 2𝑐𝑚) and (1 • , 1𝑐𝑚) error margins in Table 4 , respectively. The data reveal that the precision of camera pose estimates from our method has been further elevated, offering a solid guarantee for the practical application of camera relocalization technology in real-world scenarios.Table 3 : Survival Rate and Accuracy. We measure the survival rate across three different stages, defined as the ratio of pose estimates that meet our set confidence criteria, in conjunction with the accuracy of the pose estimation. We have set the confidence threshold to an inlier rate exceeding 90%. "Init." denotes the network post-initialization, utilizing all coordinate estimates for pose estimation; "Init. With K.S." refers to the post-initialization network employing keypoint selection for pose estimation; "Keypoint fine-tune with K.S." indicates the network after keypoint fine-tuning, using keypoint selection for pose estimation. Almost all pose estimates deemed to meet the confidence criteria fall within (5°,5cm), hence we only list the percentages for errors under (2°,2cm) and (1°,1cm). We validated our method for outdoor relocalization on the Cambridge Landmarks [16] with the results tabulated in Tables 2. For comparison, we replicate experiments from prior research using various methods: AS (SIFT) [20] , hLoc(SP+SG) [18] , pixLoc [19] , Go-Match [28] , HybridSC [8] for FM; PoseNet17 [15] , MS-Transformer [23] for APR; DSAC*(Full) [7] , SANet [27] , SRC [12] for SCR w/Depth; DSAC*(Full) [7] , DSAC*(Tiny) [7] , ACE [2] for SCR. The Cambridge Landmarks dataset consists of images of various historical buildings in the old town area of Cambridge, with ground truth poses obtained using the SfM [3] . Our method performed exceptionally well in some scenes, outperforming our main comparison method, ACE, in most scenarios. This is attributed to our network's enhanced capability for scene coordinate inference. During our experiments, we also observed that a significant reason for poorer performance on outdoor data was the sparsity of the dataset. Our method showed commendable performance on the training set, yet its capacity to generalize to new viewpoints was somewhat limited. Overall, our method's performance on outdoor data was superior to our main comparison method, ACE [2] .4.3.1 The Relationship Between Inlier Ratio and Error. Our method introduces a confidence-based scene coordinate estimation approach. We ponder the potential outcomes if we solely rely on the poses of images with high confidence. We have detailed the performance of our method across various scenes at different confidence levels in the tables. To validate the appropriateness of our confidence measure, we plotted a curve representing the relationship between the confidence utilized and the translational error, as illustrated in Figure 3 . For clarity, we scaled and averaged the data. The blue curve in the figure represents confidence, while the orange curve indicates the translational error. It is observable from the graph that an increase in error tends to follow a decrease in confidence. Although this correlation is not absolute, our observations suggest that there is generally an inverse relationship between inlier rate We present the inlier rate progression over three distinct phases on the chess test dataset. The green curve represents the inlier rate following initialization. The orange curve depicts the inlier rate after the keypoint estimation network has been trained and directly applied to the initialization network.and error, implying that a higher inlier rate may be associated with increased error.Frame and Key Point Guided. Enhancement of Confidence in Keypoint-Guided Networks. Our method aims to further improve the accuracy of images with high confidence; hence, we estimate scene coordinates on keypoint data and employ the inlier rate as a measure of confidence for the precision of image pose estimation. We preserve images with a confidence level of 90% and estimate their camera poses. Table 3 lists the survival rate and accuracy of our method at various stages for test images. It is evident that the survival rate of test images was quite low initially without keypoint guidance. The introduction of keypoint guidance resulted in a significant increase in the survival rate, where survival rate refers to images we consider to have sufficiently high accuracy. We also observe that the survival rate of the network, further optimized using keypoint guidance, did not decrease significantly, while the localization accuracy for the test data with high survival rates was further improved within error margins of (2 • , 2𝑐𝑚) and (1 • , 1𝑐𝑚). Moreover, within the subset of high-confidence images, we nearly achieved 100% accuracy within an error range of (5 • , 5𝑐𝑚), which validates the practical reliability of our method. Additionally, we note that in the Stairs scene, our confidence measurement method was not able to maintain a high survival rate. This could be attributed to the scene's higher difficulty and the presence of more repetitive textures, a phenomenon also observed in the Heads scene.4.3.3 Usage of Time and Computational. Our approach does not significantly increase training duration or computational resource usage under standard conditions. The total training time is approximately around 20 minutes in two Nvidia GeForce 2080 Ti GPUs, which is acceptable considering the desired higher accuracy. During testing, when the survival rate meets our satisfaction, our computational load is lower than that of ACE. This is because we use fewer but more accurate 2D-3D matching points, reducing the required iterations. However, if the survival rate is suboptimal, we may incur the cost of performing pose estimation twice: first, a confidencebased pose estimation, and then a more generalized pose estimation. When pose estimation is solely based on confidence, we use fewer 2D-3D correspondences for pose estimation.4.3.4 Limitations. We found several shortcomings that require further research and improvement. Firstly, our method faces the challenge of a low number of keypoints in difficult scenes with sparse textures. Consequently, fine-tuning the process becomes nearly infeasible, and we heavily rely on the initialized generalized network. Secondly, our approach is limited by the confidence threshold. Currently, we use a fixed confidence threshold, but we have noticed that its performance varies across different scenes. 
conclusion: 
 Measuring the confidence of pose estimation is critical in practical applications. In this paper, we present a method for assessing the credibility of image pose estimates, enabling effective evaluation of camera poses after estimation. Additionally, to improve the effective assessment of confidence and enhance localization accuracy, we have designed a keypoint evaluation method based on reprojection error. This method estimates scene coordinates for keypoints of interest, optimizing keyframe localization performance. Furthermore, in practical application, our designed gated camera pose estimation strategy, based on confidence thresholding, combines keypoint-guided networks with more generalized networks to further enhance the camera relocalization accuracy. Notably, our approach does not significantly increase training duration or volume compared to state-of-the-art methods, achieving greater accuracy within a training period of just 20 minutes. Through extensive experimental comparisons, we have demonstrated the effectiveness of our proposed method, surpassing state-of-the-art results.  
 
 Appendix:  


Decision: The paper deals with the problem of camera relocalization through scene coordinate regression, followed by a PnP algorithm for which a keypoint selection process is specifically designed. The proposal is technically sound and the comparative evaluations (enriched with the ones provided in the rebuttal) show that the proposal consistently outperforms existing state-of-the-art competitors. 

Shared by the 3 reviewers, the main weaknesses of the article rely on the clarity of the presentation for which parts need to be rewritten (also because of many spelling inconsistencies) by considering the reviewers recommendations, and on the unimodal data and application aspect which is not at the heart of the ACM MM community's priorities. Depending on the acceptance rate at the conference, this may call into question its acceptance for oral presentation, and tip him towards a poster presentation.

Reviews: review: paper_summary: paper_summary: The paper deals with the problem of camera relocalization via scene coordinate regression, followed by a PnP algorithm. The paper more specifically focuses on the keypoint selection process for the PnP algorithm. The proposed approach uses several ratios and scores to select the best keypoints most suited for localization. The paper advocates that more than the best scene representation, in the case of relocalization, the most suited keypoints should be what the network focuses on.
strength_weakness: strength_weakness: First of all, I find that the paper’s initial position is sound and should definitely be investigated. Focusing on a scene representation of suitable keypoints for pose estimation rather than global and extensive scene representation appears to be the way to go. Thus, the idea is interesting and the paper has merits as the evaluation tends to show, especially when showing improvement in localization even with a scene representation less suited for reconstruction (shown in supp. mat). 

The proposed implementation looks interesting. Indeed, exploiting the scene coordinate regression scheme both in a “global” and “local” approach shows similarities with global/local features joint use in image retrieval, which showed great promise. The proposed ratios are simple but coherent and the results seem to prove it. I also appreciated the “survival rate” evaluation. Indeed, being able to have an a priori confidence estimation on the final pose has multiple practical applications and should be highlighted.
limitations: limitations: Although the paper has merits and I find both the idea and its implementation interesting, I feel that the main problem of the paper is the paper itself. I found it very hard to understand the ins and outs of the proposed method. First, the structure of the paper and the logical progress when detailing the approach were lacking to my opinion. Second, some aspects in the way things were explained limited an easy understanding of the method and are very detrimental to the paper. I will go into more details below but bear in mind that although some answers to the questions may seem evident, in my opinion, the reason they arise is due to the lack of clarity in the paper.

In my opinion, a typical example of the lack of clarity is the third sentence of the Introduction. Although the problem of camera relocalization is well defined in the first sentence of the related work, in the introduction, a convoluted and unclear formulation is used. Furthermore this sentence shows the need for a thorough pass to check for typos and especially missing capital letters.
Staying with easy to fix issues, some small aspects should be addressed :
the query image used in the figures is not always the same (it differs in Figure 2). It could be interesting to always have the same, especially when compared to Figure 4 and the set of selected keypoints that does not show the same area of the scene,
the figures are almost never referenced in the text, except for Figure 4. This, alongside the fact that they are placed very far from the (assumed) corresponding text is quite confusing,
some figures seem to not be present (to which Figure and curves is section 4.3.1 referring to ?),
the figure 6 comes maybe a bit late in my opinion. It could be further used to explain the process and limit the confusion in the method description,
in the tables, the colors red and blue are not explained. Although it seems to represent the first and second results, those are usually highlighted with bold and underlined. It is necessary to define the color code,
keypoints is used in an undiscriminated way for 2D and 3D points and that can become confusing.

I feel that the confused aspect of the paper is also reflected by the related work section. The several parts of the section are not linked together and there is no global overview of where the proposed approach is set with regards to related approaches. Furthermore, the state of the art on retrieval-based methods is quite outdated. Only regarding the features used for retrieval, many more recent methods exist, such as this one example (Zhu, Sijie, et al. "R2former: Unified retrieval and reranking transformer for place recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.) that could lead the authors towards a more recent related work. Furthermore, RANSAC is NOT a pose estimation algorithm. It can be used alongside one to make it more robust but that is it.

On the proposed approach, if I understood correctly, the keypoint selection network uses a 3D scene reconstruction from Colmap to compare the 2D keypoints extracted in the image with supposedly important points in the scene. However, are we sure that the points suited for and kept by the SfM reconstruction are the most-suited for camera relocalization. This is assumed to be true in the method but is there any more justification ? Furthermore, does it mean that to retrain the model on another scene, a novel 3D reconstruction of the scene must be performed or is the training of the keypoint selector scene-agnostic ? 

To my opinion, the approach has merits but I feel that both a thorough refactoring and a step back to have more hindsights would greatly benefit the paper. Most importantly, I believe that introductory paragraphs outlining the reasoning (at different levels) in each part would help readers to understand the whole paper.
paper_summary: paper_summary: This paper presents a straightforward and effective method to enhance the accuracy of camera pose estimation by improving the correctness of keypoint and keyframe matches.
strength_weakness: strength_weakness: 1. The authors propose a method to evaluate image confidence, analyzing each image's likelihood of yielding a valid pose, thereby enhancing the accuracy of the captured information.
2. By leveraging different networks, the authors obtain reliable keypoints contributing to improved pose estimation precision.
3. Ablation experiments are conducted to validate the effectiveness of individual networks, demonstrating slight improvements over current state-of-the-art approaches.
limitations: limitations: 1. Although comparisons are made with the most advanced existing methods, the limited number of comparative methods in the experiments is insufficient to prove the superiority of this proposed method over others. For instance, SANet[1] and SRC[2].
[1] Yang, Luwei, et al. "Sanet: Scene agnostic network for camera localization." IEEE International Conference on Computer Vision 2019.
[2] Dong, Siyan, et al. "Visual localization via few-shot scene region classification." International Conference on 3D Vision. IEEE, 2022. 
2. Both the methods for assessing image confidence and the confidence in keypoint matches rely on threshold-based determinations, which lack generalizability and may not be adaptable to different scenarios.
3. Scenes with low texture or repetitive patterns notably affect keypoint matching. Although the authors address the scenario of stairs in repetitive textures, they overlook discussions on other instances of repetitive or low-texture contexts, such as walls and railings, highlighting the need for more comprehensive experimental evaluations.
4. The subheading in Section 4.3 should be capitalized for consistency.
5. It would benefit the authors to elaborate on how the ground-truth probabilities for each key point in the dataset are obtained.
paper_summary: paper_summary: This paper addresses the camera relocalization task, which estimates the camera pose of a query image in a known 3D scene. The proposed method extends the ACE framework to two branches and introduces a keypoint network, pose confidence estimation, and gated mechanism in the pose selection. The proposed method identifies keypoints by the highest matching rate, converts reprojection errors into keypoint probabilities, and trains the keypoint network using binary cross-entropy loss based on the keypoint probabilities. Evaluation experiments using the 7 Scenes and Cambridge Landmarks datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The idea of improving the relocalization accuracy by selecting only the features essential for camera relocalization is interesting.
2. The modules are reasonably motivated and designed, and the proposed method is technically sound.
3. The proposed method showed improvement in relocalization accuracy in most scenarios compared to the state-of-the-art method ACE.
limitations: limitations: 1. This paper claims to propose a method of measuring pose estimation confidence based on inlier ratio. However, measuring pose confidence based on the inlier ratio is part of the basic ideas of the RANSAC family and is not a novel idea.
2. The algorithm of the proposed method has some unclear parts. From the description in section 4.3.3, it can be interpreted that generalized pose estimation is performed only when the survival rate is suboptimal. However, from Figure 6 and its caption, it can be interpreted that both confidence-based pose estimation and generalized pose estimation are always performed. These seem contradictory, and it is also unclear what suboptimal represents. The authors should clearly describe these.
3. Experiments lack a comparison with ACE Ensemble [2]. Since the proposed method increases training time to 20 minutes, a comparison with this method should be considered. If this method is included in the comparison, the proposed method may not have state-of-the-art performance.
4. Experiments lack sufficient evaluation of the identifying key points proposed in section 3.1. The manuscript does not contain a specific value for the threshold τ. How does changing this threshold affect the re-localization results?
5. The effectiveness of the proposed gating mechanism in pose selection phase has not been demonstrated. Can poses with higher estimated confidence consistently achieve more accurate re-localization results?
6. In Tables 1, 3, and 4, the two decimal places are zero in most cases, which is unnatural. The authors should explain why such results were obtained.
7. Figures 1, 2, and 5-7 are not referenced from the text. Also, the authors refer to Figure 3 on line 806, but Figure 7 is probably correct. Furthermore, Table 4 is referenced before Tables 2 and 3. Figures and tables should be appropriately referenced from the text. 
8. There are several format errors in the manuscript.  For example, the initial letter should be capitalized in line 97, and a space is lacked before the word "by" in  line 246. There are also some typos, e.g., "coodinate" on line 632.
9. The manuscript contains many spelling inconsistencies. For example, a keypoint network is sometimes referred to as a keypoint judgment network, keypoint discrimination network, keypoint selection network, keypoint estimation network, keypoint classification network, or keypoint selector, which can be confusing.