Activating Self-Attention for Multi-Scene Absolute Pose Regression

Authors: miso  lee
jihwan  kim
jae-pil  heo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 06:58:50

Abstract: Multi-scene absolute pose regression addresses the demand for fast and memory-efficient camera pose estimation across various real-world environments. Nowadays, transformer-based model has been devised to regress the camera pose directly in multi-scenes. Despite its potential, transformer encoders are underutilized due to the collapsed self-attention map, having low representation capacity. This work highlights the problem and investigates it from a new perspective: distortion of query-key embedding space. Based on the statistical analysis, we reveal that queries and keys are mapped in completely different spaces while only a few keys are blended into the query region. This leads to the collapse of the self-attention map as all queries are considered similar to those few keys. Therefore, we propose simple but effective solutions to activate self-attention. Concretely, we present an auxiliary loss that aligns queries and keys, preventing the distortion of query-key space and encouraging the model to find global relations by self-attention. In addition, the fixed sinusoidal positional encoding is adopted instead of undertrained learnable one to reflect appropriate positional clues into the inputs of self-attention. As a result, our approach resolves the aforementioned problem effectively, thus outperforming existing methods in both outdoor and indoor scenes.

Main paper: 
 introduction: 
 Camera pose estimation is a fundamental and essential computer vision task, adopted in numerous applications such as augmented reality and autonomous driving. Geometric pipelines [1] [2] [3] [4] [5] [6] with 2D and 3D data have been a mainstream with high accuracy. After extracting features and matching 2D-3D correspondences, camera pose is approximated via Perspective-n-Points (PnP) algorithm and RANSAC [7] . However, there still remains several challenges for real-world applications, including high computational cost and a huge amount of 3D point cloud.Absolute Pose Regression (APR) tackles these issues by directly estimating the 6-DoF pose from a single RGB image in an end-to-end manner. First introduced by Kendall et al. [8] , subsequent APR methods [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] have been devised based on convolutional neural networks (CNN). However, they still demand multiple models and individual optimization to be applied in real-world multi-scene scenarios. In this regard, Multi-Scene Absolute Pose Regression (MS-APR) has emerged to satisfy the needs of speed and memory efficiency across multiple scenes [19] . MSTransformer [20] pioneers a streamlined one-stage MS-APR approach with transformer architecture. It leverages the transformer decoder not only to improve memory efficiency but also to enhance accuracy significantly.However, we point out that the learning capacity of transformer encoders is underutilized. As shown in Tab. 1, MSTransformer's encoder self-attention modules do not significantly improve or even degrade performance. Although we discovered low-rank, collapsed attention maps, which are known Table 1 : Ablation on encoder self-attention modules in MSTransformer [20] . We report the average of median position and orientation errors for each experiment.Outdoor [8] Indoor [27] MST [20] 1.28m/2.73 • 0.18m/7.28 • MST w/o encoder SA 1.21m/2.84 • 0.18m/7.49 • to cause gradient vanishing [21] or training instability [22] , general solutions [22] [23] [24] were not correct the problem in the case of MSTransformer. This challenge motivates us to elucidate the phenomenon from another perspective in the context of MS-APR.Therefore, this work proposes brand new analysis: distortion of query-key embedding space, where the attention score originates. We statistically substantiate that queries and keys are mapped in completely separated regions in the embedding space while only a few keys are blended into the query region. In this situation, attention maps inevitably collapse, as all queries are represented similarly to those few keys. Intuitively, it means that all image patches are just represented by only a few image patches, as illustrated in Fig. 1 . We conjecture that the task difficulty contributes to the issue; the model should extrapolate the camera pose from a single RGB image across multiple scenes.To support this, we empirically reveal that the model tends to avoid exploring self-relationships at the beginning of the training. These findings align with our additional observations that learnable positional embeddings used in self-attention are also undertrained.Nevertheless, we expect that image features incorporating global self-relations will be valuable to estimate the camera pose by capturing salient features such as long edges and corners. This work thus introduces simple but effective solutions that activate encoder self-attention modules for MS-APR. Concretely, we design an auxiliary loss which aligns the query region and the key region. By forcing all queries and keys to be mapped into the close and dense space, they are highly encouraged to interact with each other by self-attention-like putting boxers in the ring and blowing the whistle! Furthermore, we explore various positional encoding methods based on empirical evidence since current undertrained one confuses the model to estimate the camera pose with incorrect positions. Finally, fixed 2D sinusoidal encoding is adopted instead of learnable parameter-based methods [25, 26] . This enables the model to consider appropriate positional clues of each image feature during exploring self-relationships from the beginning of the training.As such, the model can obtain rich global relations from activated self-attention as shown in Fig. 1 . These relations are incorporated into image features, acquiring more informative encoder output. Extensive experiments demonstrate that our solution recovers the self-attention successfully by preventing the distortion of query-key space and keeping high capacity of self-attention map [22] . As a result, our model outperforms existing MS-APR methods in both outdoor [8] and indoor [27] scenes without additional memory during inference, upholding the original purpose of MS-APR. 
related work: 
 Absolute Pose Regression (APR) is to estimate 6-DoF camera pose directly using only an RGB image as input. First proposed by Kendall et al. [8] , subsequent works [9, 10, 13, 14, 17] have introduced advanced architectures and training methods. However, Sattler et al. [28] demonstrated that these models have a tendency to memorize training data, resulting in poor generalization when presented with novel views. In response, IRPNet [16] attempted fast APR by adopting pre-trained image retrieval model as feature extractor. On the other hand, E-PoseNet [15] substituted vanilla CNN with Group-Equivalent CNN to extract geometric-aware image features. Recently, NeRFbased models [11, 12, 29] have improved the APR performance significantly by combining existing APR models with NeRF-W [30] to capture geometric information or synthesize additional images. However, these models have the limitation of being scene-specific, making them impractical for real-world applications [20] . NeRF-based models [11, 12, 29] are particularly vulnerable since NeRF-W is not just scene-specific, but also requires lots of time for taking structure-from-motion, training, and generating additional data.Multi-Scene Absolute Pose Regression (MS-APR) aims to estimate a camera pose from multiple scenes with a single model for scalability. Initially, MSPN [19] made a feature weight database for each scene using CNN, and then fed the indexed features into scene-specific regressors for estimating camera pose in multi-scenes. However, it still had drawbacks of training time and memory inefficiency. Against the backdrop, MSTransformer [20] enabled sound one-stage MS-APR with transformer-based architecture. They proposed to use decoder queries as scene queries to make both scene classification and camera pose regression possible at once. The whole decoder outputs are firstly used to predict the scene index, and then the decoder query corresponding to the scene is used to regress the camera pose. This allows it to become more general through various scenes. Recently, additional studies [31, 32] have been proposed fine-tuning methods, but reverting scene-specific again and using additional parameters. In contrast, we propose simple but effective training methods for training MSTransformer from scratch, maintaining the original purpose of MS-APR.Self-Attention is a attention mechanism which represents input sequences by inter-relationships among all elements. Following the remarkable performance of transformer [33] [34] [35] [36] , it became widely adopted in various fields. MSTransformer [20] also used self-attention both the transformer encoder and decoder. However, recent studies [21] [22] [23] [24] [37] [38] [39] [40] have pointed out the failure of self-attention. Firstly, several studies on natural language processing [37, 38] have analyzed that self-attention map collapses to a low-rank matrix, by focusing attention scores to meaningless tokens such as CLS and SEP. In the case of vision, several works [23, 40] have addressed the issue of collapsed self-attention in transformer encoder. There are also studies [21, 22, 24, 39] which fundamentally demonstrated that self-attention maps collapse into low-rank matrices in certain conditions. Similar to these findings, self-attention modules in MSTransformer's encoders do not significantly improve or even impair performance due to the collapse. However, we found that pioneers are not very effective in the case of MS-APR, which is shown in our experiments. Therefore, we analyse the problem from a new viewpoint, distortion of query-key space and undertrained learnable positional embedding, and propose solutions that settle down the problem successfully in MS-APR.3 Preliminary Camera Pose. The location of a captured image I can be calculated from camera intrinsic and extrinsic parameters. The former is specific to the camera itself, and the latter corresponds to the pose of the camera with respect to a fixed world coordinate system. As such, the camera pose p can be represented as (t, r). Here, t ∈ R 3 the camera position (translation) and r ∈ R 4 the camera orientation (rotation). Note that r is an unit quaternion vector so p has 6-DoF.Model Architecture. MSTransformer [20] suggests the model architecture inspired by DETR [35] . It is composed of a CNN, transformer encoder-decoders, a scene classifier, and regressors. Firstly, given an image I, CNN features for the orientation f r is extracted from middle level while those for the position f t is computed from middle-high level. f t and f r are projected by 1 × 1 convolution and reshaped as ft ∈ R Nt×D and fr ∈ R Nr×D to be transformer-compatible inputs, where N t and N r are the number of tokens for the position transformer and the orientation transformer, respectively. Note that the purity is 1.0 when the query region is composed only with queries, but slightly lower than 1.0 when a small subset of keys resides in the query region. According to (a), statistical evidence supports the prevalent occurrence of the blending of a few keys into the query region across the entire dataset, both in the position and orientation transformer encoders. (b) illustrates the increasing tendency of distance between the query region and the key region in the encoder. They lean away each other even at the beginning of the training. Here, the distance between query region and key region is an average value across layers and heads.The architecture of transformer encoder-decoder follows the general transformer [33, 35] . It utilizes decoder query as scene query to make it possible for both scene classification and camera pose regression at once. There are the position and the orientation transformers, each of which has scene queries z t ∈ R M ×D and z r ∈ R M ×D where M is the number of scenes, respectively. In the encoder, there are several layers to reinforce the CNN features. Every layer of the encoder consists of a multi-head self-attention module and multi-layer perceptron (MLP) with layer normalization and residual connection.Meanwhile, the decoder is designed to not only enhance the decoder queries but also learn the relationship between encoder features and decoder queries. It has multiple layers like the encoder but additionally includes a multi-head cross-attention module in each layer. Afterwards, the scene classifier predicts the scene index m from concatenated decoder outputs z = [z t ; z r ] ∈ R M ×2D . In the end, t and r are predicted by MLP regressors from the m-th decoder outputs z t m ∈ R D and z r m ∈ R D , respectively.Self-Attention. There are three main components in attention mechanism: query Q, key K, and value V . The final output is computed as a weighted sum of V , with weights derived from the similarity score between Q and K. Formally, given the input X ∈ R N ×D with N tokens of dimension D, the h-th attention head is defined as follows:Attn h (X) := A h V h ,whereEQUATIONHere, A h is h-th attention map and H is the number of heads. When inputs to query, key, and value are all the same, it is usually called as self-attention.4 Problem AnalysisQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitation in Sec. 8. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Answer: [Yes] Justification: We include the code in the supplemental materials. Guidelines: Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
distortion of query-key embedding space: 
 We introduce our viewpoint why encoder self-attention modules in MSTransformer are not utilized as shown in Tab. 1. Our main intuition is that the model tends to avoid the self-attention mechanism [38, 24] by distorting the query-key embedding space due to the learning difficulty. On the face of it, collapsed self-attention maps may seem to be the root cause. However, we further investigate the issue, focusing on the query-key embedding space. The attention map is determined by computed similarity scores between queries and keys, as expressed in Eq. 1. In other words, the projection matrices and bias are what the model learns, and the attention map is merely a calculated consequence. As such, the collapsed attention map which induces the model's bypassing comes from the distorted query-key space, therefore we highlight it to address the problem. When collapsing, queries and keys are completely separated while a small subset of keys are blended into the query region across all layers and heads, both in the position and the orientation encoders of the baseline [20] . To gauge the statistical incidence of such cases, we first clarify the query region and the key region in the query-key embedding space. By clustering Q ∪ K through k-means using a mean vector of queries q and a mean vector of keys k as initial centers, query-dominant region Q is obtained from the cluster with center q. Then we define the purity of query region P as follows:EQUATIONIntuitively, P is 1.0 when the query region is composed only with queries, slightly lower than 1.0 when small subset of keys are blended into the query region, and about 0.5 when queries and keys are mixed together. Fig. 2 (a) illustrates the statistical results on 7Scenes [27] dataset. It shows that the phenomenon we point out is predominant throughout the whole dataset both for the position and orientation transformer encoders. Under this condition, the attention map inevitably collapses as all queries are computed to be similar to those keys.We also verify that queries and keys lean away each other from the beginning of the training as shown in Fig. 2 (b ). The model is trained to locate only a few keys close to the query region while keeping the long distance between the query region and the key region. Here, the distance is calculated using the L2 metric between their centers. These investigations imply that the model avoids the process of identifying self-relations due to the task difficulty only with the original task loss.Moreover, we discuss the importance of appropriate positional encoding for the task and the negative effects of the current positional embeddings in this section. As self-attention is a permutation-invariant mechanism, positional encoding is widely adopted to the queries and keys to account for the order of the input sequence [34] [35] [36] . It plays a vital role to utilize the encoder self-attention modules, particularly in MS-APR. To be specific, it is not only important to detect features like roofs and windows in the image, but also crucial to know where they are located in the image for accurate pose estimation. In other words, the model has to estimate the camera pose with shuffled image patches if it uses inappropriate positional embedding.Despite its importance, learnable positional embeddings in the baseline remain undertrained with deactivated self-attention modules. Fig. 3 shows the visualization results of the distances between the top-left token and other tokens, based on the fixed 2D sinusoidal positional encoding and the learnable positional embedding trained in the baseline. One can observe that distances between tokens are drastically larger if the token has different vertical or horizontal position with the top-left token, in the case of learnable positional embeddings. This makes it even harder for the model to learn geometric relationships between image features by self-attention mechanism. 4 illustrates the training pipeline with our solutions. We apply additional objectives L QKA t and L QKA r to the model to activate the self-attention modules. Specifically, queries Q and keys K interact with each other by forcing the centroid of query region q and the centroid of key region k to become closer. Here, we encode all input queries and keys with fixed 2D sinusoidal positional encoding to ensure active interaction between Q and K with reliable positional clues. 
activating self-attention for ms-apr: 
 In this section, we introduce our solutions with full objectives for training our transformer-based MS-APR model successfully by mitigating self-attention map collapse. Firstly, we employ the L2 loss to regress the camera position and orientation. Let us denote the ground-truth camera pose as (t, r) and the estimated camera pose as ( t, r). Then, the position loss and the orientation loss are defined as L t = ∥t -t∥ 2 and L r = ∥r -r ∥r∥ ∥ 2 , respectively. Here, it is required to balance two losses to make the model learn both camera position and orientation since the former is a scalar value but the latter is a unit quaternion [8, 14] . According to [14] , the final pose loss is given by:EQUATIONwhere s t and s r are the learnable parameters to adjust the uncertainty.Secondly, the model should be capable of classifying the scene from the input image to work with multi-scene dataset. Let us denote y ∈ R M as one-hot vector encoding the ground-truth scene index of the input image, and ŷ ∈ R M as the predicted probability distribution of scene classes from concatenated decoder outputs z. Then, the scene classification loss proposed in [20] is given by:EQUATIONOn top of it, we introduce an auxiliary loss to prevent the model from avoiding the self-attention as depicted in Fig. 4 . As we discussed in Sec. 4.1, it is essential to ensure that the queries and the keys do not become alienated from each other while encouraging their interaction by embedding in close proximity. To achieve this, we propose the Query-Key Alignment (QKA) loss as an additional objective. It is defined as follows:EQUATION)where L is the number of encoder layers, H is the number of heads, and ql h and kl h are the mean vectors of the queries and the keys per each encoder layer and head, respectively. By inducing queries and keys to be placed in a shared compact area, we ensure that they interact with each other in the self-attention module. Here, we encode all input queries and keys with fixed 2D sinusoidal positional encoding instead of undertrained learnable positional embedding. This guides the model to stably learn self-relationships from the beginning of the training. As a result, the accurate interaction can be accomplished by reflecting the verified correct positional information to input queries and keys.We apply our QKA loss to both the position and orientation transformer encoders, while adjusting the weight of the loss to integrate with the original task loss. Accordingly, the final auxiliary lossL aux = λ aux L QKA t + L QKA r, where λ aux is the weight of the auxiliary loss. L QKA t and L QKA r are QKA losses for the position and orientation transformer encoders, respectively. Putting all together, our full objective L = L pose + L scene + L aux .6 Experimental Results 
experimental setup: 
 Datasets. We train and evaluate the model on outdoor and indoor datasets [8, 27] , which include RGB images labeled with 6-DoF camera poses. Firstly, we use the Cambridge Landmarks which consist of six outdoor scenes scaled from 875m 2 to 5600m 2 . Each scene contains 200 to 1500 training data. We conduct the experiment on four scenes in Cambridge Landmarks, which are typically adopted for evaluating absolute pose regression [12, 15, 20] . On the other hand, we use the 7Scenes dataset which consists of seven indoor scenes scaled from 1m 2 to 18m 2 . Each scene includes from 1000 to 7000 images.Training Details. We entirely follow the configuration of the baseline [20] . We train the model with a single RTX3090 GPU, Adam optimizer with β 1 = 0.9, β 2 = 0.999, ϵ = 10 -10 , and the batch size of 8. For the 7Scenes dataset, we train the model for 30 epochs with the initial learning rate of 1 × 10 -4 , reducing the learning rate by 1/10 every 10 epochs. In the case of Cambridge Landmarks dataset, we train the model for 500 epochs with the initial learning rate of 1 × 10 -4 , reducing the learning rate by 1/10 every 200 epochs. Afterwards, we freeze the CNN and the orientation branch, then fine-tune the position branch as same as the baseline [20] . The model is trained for 60 epochs with the initial learning rate 1 × 10 -4 , reducing the learning rate by 1/10 every 20 epochs.The learnable parameters s t and s r for the original task loss are initialized as in [14] . Both for the position and orientation transformer encoder-decoder, the number of layers L is 6 and the number of heads H is 8. Lastly, we set the λ aux for our query-key alignment loss as 0.1.MS-APR Methods. We compare the model trained our solutions with other single-frame multi-scene APR models for fair comparison, i.e., excluding scene-specific methods. Tab. 2, Tab. 3, and Tab. 4 show the experimental results on four outdoor scenes in Cambridge Landmarks [8] and seven indoor scenes in 7Scenes [27] , respectively. With our solutions, the model shows better performance through all scenes, compared to other MS-APR methods. This results verify that activating self-attention has [25] 0.18m/6.97 • Rotary PE [26] 0.18m/6.94 • Fixed PE 0.17m/6.64 • the potential to become a general module, generating task-relevant features across multiple scenes, regardless of the specific features of each scene.Alternative Methods for Collapsed Self-Attention. We apply general-purpose techniques [22] [23] [24] , proved to alleviate collapsed self-attention, to the baseline to compare the effectiveness of each method and ours in MS-APR. Here, we conduct experiments with fixed positional encoding, as incorrect inputs may suppress the potential of the solutions. Tab. 5a shows that existing techniques have little or no effect in terms of recovering the self-attention for the task. Meanwhile, ours significantly improves the baseline in both conditions. We conjecture that it is required to guide the model in a more direct manner to utilize self-attention for MS-APR.Alternative Positional Encoding Methods. We conduct experiments with recent advanced learnable positional encoding methods [25, 26] with our QKA loss. As shown in Tab. 5b, even advanced methods are not suitable to the task as they mainly depend on learnable parameters. We assume that purified positional clues, i.e.not randomly initialized ones, is deemed important to stabilize the training of the embedding space in the case of MS-APR.Attention Entropy of Self-Attention Maps. To provide quantitative evidence that our solutions enhance performance of model by activating the encoder self-attention, we measure attention entropy [22] defined as follows:EQUATIONwhereEQUATIONNote that high attention entropy indicates training stability and high representation capacity of self-attention map [22] .Fig. 5 shows the attention entropy for each encoder layer in both the position and orientation transformers. As illustrated, the model employing our solutions has significantly higher attention entropy than the original baseline model across all encoder layers. These results validate that our solutions successfully induce the transformer-based model to activate the encoder self-attention, thus improving the representation quality of self-attention map.Purity Levels in Query Regions. To verify the effectiveness of our auxiliary loss on rectifying the distortion of query-key space, we measure the purity levels in query regions defined in Eq. 2. The statistical results for both the baseline and the model employing our solutions are presented in Fig. 6 . Figure 6 : The results show the change of purity of query region P, defined in Eq. 2, across 7Scenes dataset. Note that the purity is 1.0 when the query region is composed only with queries, but slightly lower than 1.0 when few keys resides in the query region. Compared to the baseline, there are no cases where only a few keys are blended into the query region with our solution.Here, the purity levels represent the proportion of the queries allocated to the query-dominant cluster.Our observations reveal that the nearly half keys belong to the query region in almost all cases, indicating that the query and key regions are highly interleaved. This demonstrates that our solutions restrain all queries from being treated as similar to only a few keys, thus preventing self-attention collapse and activating self-attention.Fig. 7 reports visualization of query-key space by t-SNE and attention results of the baseline and the model employing our solution. Note that the attention results depict the scores of the keys, averaged over the queries in the self-attention map. One can observe that the query regions and key regions of the baseline are separate. In contrast, the regions are highly aligned by our solutions, reducing the occurrence of only a few keys blended into the query region. This condition necessitates that the model reflects global interactions between queries and keys, therefore, global relations are incorporated into image features as depicted in Fig. 7 . We can see that geometric relations such as long edges are usually obtained, which are the critical points for the task. More examples are in the supplementary material.We conduct an ablation study on our solutions to verify the effectiveness of each component. As shown in Tab. 6, our QKA loss enhances the performance in both the outdoor and indoor datasets. Additionally, overall performances are further improved with fixed positional encoding, rectifying the incorrect inputs in the self-attention. This ablation study indicates that several useful self-relation can be extracted from noisy content features by self-attention, but reliable positional clues are particularly important to learn various crucial relationship for estimating more accurate camera pose. 
discussion: 
 Limitation and Future Work. The proposed auxiliary loss is designed based on the assumption that an image has many key features which are useful for estimating the camera pose. However, there Table 6 : Ablation on our solutions. We report the average of the median position/orientation errors.Outdoor [8] Indoor [27] MST [20] 1.28m/2.73 • 0.18m/7.28 could be a side effect if only a few key features are in the image; i.e., if a dynamic moving object mainly occupies the image. Developing an algorithm to decide whether global self-attention is useful for the image or not could be an interesting future work.Broader Impacts. The camera pose estimation technology with this work can be expected to have several positive social impacts, such as being utilized in programs to locate missing persons. On the other hand, it could potentially be misused for purposes such as illegal location tracking programs. 
conclusion: 
 This work has exposed that self-attention modules in transformer encoders are not activated in MS-APR. Looking beyond the collapsed self-attention maps, we have discovered the distortion of query-key embedding space; queries and keys are entirely separate while a small subset of keys resides in the query region. We have examined the predominance of the phenomenon, adducing statistical and empirical evidence. Additionally, the importance of accounting for the order of the input sequence has been addressed in the context of the task. Based on these analyses, this work has proposed simple but effective solutions to activate self-attention by rectifying the distorted query-key space with alignment loss and appropriate positional clues. As a result, the model could learn salient global features through self-attention, thus achieving state-of-the-art performance both in outdoor and indoor scenes without additional memory in inference. Our extensive experiments have demonstrated the effectiveness of proposed solutions in terms of recovering self-attention for MS-APR. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts in Sec. 8.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work uses public datasets and poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We checked the license of our baseline code and datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We entirely follow the setting of the baseline [20] . The settings are as below.Data Augmentation. Data augmentation is performed in the same way as [8] , which is also adopted in the baseline. Afterwards, brightness, contrast, and saturation are randomly jittered for photometric robustness. Test images are also resized to 256 × 256, but they are center-cropped to 224 × 224, without any photometric augmentation. Additionally, following the baseline, smaller scenes in the Cambridge Landmarks dataset [8] are oversampled during training to deal with data imbalance problem.Backbone. As did in the baseline [20] , we utilize a CNN model, the EfficientNet-B0 [41] , to extract local image features from input images. The image features f t for the position and f r for the orientation are extracted from the fourth block and the third block, respectively. Accordingly, the size of f t is 14 × 14 × 112 and the size of f r is 28 × 28 × 40, i.e., the number of transformer encoder tokens N t and N r are 196 and 784, respectively.Query-Key Spaces. In addition to the statistical and quantitative results introduced in the main paper, we visualize the query-key spaces across heads and layers. Fig. A1 reports the visualization results of query-key spaces from the baseline [20] across all even layers and heads. It qualitatively shows that the distortion of query-key space occurs in most heads in transformer encoder of the baseline. In contrast, one can observe that queries and keys properly intertwine each other with our solutions as shown in Fig. A2 . Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.† Corresponding author 


Decision: The reviewers generally agree that the manuscript is well written and is clearly motivated, that the proposed method is simple, has novelty, and is reproducible, and that the results and visualizations are compelling. An extensive rebuttal was submitted, which addressed a majority of the reviewers concerns, and also demonstrated that the proposed approach could benefit a different task (temporal action detection). After this phase, R-K2gT raised their rating to borderline-accept and the other three reviewers maintained their ratings. R-FKQN remained negative (borderline reject) but was not active during any of the discussion phases. Therefore, the AC reviewed the paper in detail, finding the review of R-aJYL to be the most compelling. The paper identifies a weakness in existing approaches, proposes a novel solution, and demonstrates a positive performance impact. Further, the AC found the weaknesses identified by R-FKQN to be insufficient reasons for rejection and adequately addressed by the authors. For example, multi-scene pose regression is an established research area and does not need to be re-motivated by the authors.  Additionally, penalizing the proposed approach for only supporting attention-based methods (i.e., transformers) is not viewed as a valid complaint, especially in the current era. As such, the AC reached an accept decision. Please take the reviewer feedback into account when preparing the camera ready version.

Reviews: review: paper_summary: paper_summary: This paper is about a improving self attention in the transformer architecture for multi-scene APR. The show that the self attention module in the SOTA transformer model for APR is actually not helping much and offer an potential explanation. The paper claims that the keys and queries end up in different spaces, such that the inner product between keys and queries is very close to zero in most cases, leading the attention to collapse to zero. The paper proposes an additional loss term that encourages the mixing of queries and keys leading to much more overlap. The addition of this term results in a noticeable improvement in APR metrics across indoor and outdoor datasets.
strength_weakness: strength_weakness: - Table 5 is difficult to parse. I feel like it could easily be represented with histograms as in Figure 2a. Similarly I feel like the histograms a fairly course. The point comes across okay but I'm not sure why such a course histogram would be used.
questions: questions: In Table 3, MST and +Ours is incorrectly bold for the position error for the Office scene.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper analyzes the collapse of self-attention map in Multi Scene Pose Transformer model and proposes two simple but effective methods: auxiliary loss and fixed 2D sinusoidal encoding to solve this problem. The improved method delivers SOTA  performance on the Multi Scene Pose Regression task.
strength_weakness: strength_weakness: 1. Since query-key distortion is studied in the transformer literature and the fixed 2D sinusoidal positional encoding is an off-the-shelf module, the methods in this paper seems to be lack of novelty. It would therefore be nice to discuss more on why APR task will lead to query-key distortion rather than saying "The model tends to avoid the self-attention mechanism due to the learning difficulty." in line 127-128. The paper reads as if it's discussing improvements to the transformer and not related to APR.

2. Ablation study in Table6 indicates that most of the performance improvements come from the using of fixed sinusoidal positional encoding, a more complete ablation study may be helpful to show the ability of the proposed loss function.

3. Section5: Activating Self-Attention for MS-APR is overly detailed, most of the information is a repetition of MS-Transformer.
questions: questions: The results in Table5 indicates that the loss function is a very hard constraint that limits the purity of query region to [0.4, 0.6), such constraint is not suitable for all situations. Do authors have comments?
limitations: limitations: The manuscript is limited in scholarship, missing references to more APR methods like [A].

[A] Sc-wls: Towards interpretable feed-forward camera re-localization, ECCV 2022
paper_summary: paper_summary: The paper investigates multi-scene absolute pose regression from a new perspective: query-key embedding space. Focusing on distortion of queries and keys, the paper solutions to activate self-attention, which includes an auxiliary loss to align queries and keys and fixed sinusoidal positional encoding. Experimental results demonstrate that the proposed method outperforms existing MS-APR methods in outdoor Cambridge Landmarks and indoor 7 Scenes.
strength_weakness: strength_weakness: 1.The motivation of Multi-Scene Absolute Pose Regression is somewhat insufficient. The paper declares that Multi-Scene Absolute Pose Regression can satisfy the needs of speed and memory efficiency across multiple scenes. However, these are not validated in the experiments, especially compared with single-scene methods. 

2.The proposed method seems to only support transformer based APR, which are limited to its further applications. 

3.Some references are missing. Although the paper focuses on Multi-Scene Absolute Pose Regression, the references about single-scene camera relocalization methods still should be discussed, including single-scene ARP methods and 2D-3D correspondence based methods. 

4.The experimental results are limited, reflected by the following aspects. 

(1)The paper only lists the localization results in comparison with transformer based Multi-Scene APR methods, but the comparisons with other single-scene state-of-the-art methods are missing, especially the speed and memory efficiency which is declared as advantages of Multi-Scene methods.

(2)From Tables 1,2, the improvements of the proposed method seem not obvious, which can not show the method superiority. More discussions are preferred.
questions: questions: 1.Currently, the 2D-3D correspondence based methods (also called coordinate regression based methods) still achieve the state-of-the-art localization performance in both static and dynamic scenes, such DSAC*, KFNet and so on. It is curious that whether the proposed method can apply to 2D-3D correspondence based methods? 

2.Does the proposed method affect the network training convergence time?
limitations: limitations: The authors have adequately addressed the limitation.
paper_summary: paper_summary: In this paper, the authors focus on improving the performance of multi-scene absolute pose regression models based on transformers. From statical analysis, the authors assume that the distortion between Q and K features in self-attention and the learnable position encoders are the reasons. Therefore, a Q-K alignment loss and the fixed-pose encoding method are adopted and experiments demonstrate the efficacy on indoor 7scenes and CambridgeLandmarks datasets.
strength_weakness: strength_weakness: The proposed method is easy to follow as I mentioned before, so I don’t have many concerns. Several minor concerns are as follows. 

1.	Contribution. I can understand this paper starts from analyzing the application of the absolute pose regression, but the key technique about the distortion of Q-K features in self-attention mechanism comes from previous works [19]. As this is the major contribution of this paper, I am not very sure if it is enough. Although an additional Q-K alignment strategy is proposed as an additional contribution, its improvement over prior strategies such as [19] is not significant (0.02m and 0.17 deg). These improves may come from a suitable learning rate or more proper hyper-parameters balancing different losses.

2.	Results. Overall, the improvements against the baseline method MSTransformer [17] on indoor (0.02m, 0.64deg) and outdoor (0.09m, 0.44deg) datasets are not significant, which further degrades the contribution of this paper.
questions: questions: Please see the weakness.
limitations: limitations: Yes.

Comments: title: post rebuttal
Response Uploaded
Response Uploaded
BA
Response Uploaded
comment: Thanks for the updating. I don't have any other concerns. I stick to my initial rating.
Thank you for taking the time and effort to review our rebuttal.

We have uploaded our responses according to the reviewer's request.

With only one hour left for discussion, we kindly ask you to verify our responses.

Your verification will help us improve the overall quality and clarity of our work.

If we have satisfactorily addressed your concerns, we would appreciate a positive reassessment.
Thank you for taking the time and effort to review our rebuttal.

We have uploaded our responses according to the reviewer's request.

With only one hour left for discussion, we kindly ask you to verify our responses.

Your verification will help us improve the overall quality and clarity of our work.

If we have satisfactorily addressed your concerns, we would appreciate a positive reassessment.
We appreciate your positive feedback on the value of our insights in APR and how it echoes other methods like [A].

We agree that a proper literature discussion is crucial for helping the APR community converge on useful scientific conclusions.

To address this, we will include a comprehensive discussion, along with references to [A] and other relevant works, in our revised version.
I have updated the recommendation to BA.

This insight has value in APR, echoing insights from other methods like [A] that attending to useful regions help APR. However, a proper literature discussion is needed to help the APR community converge to useful scientific conclusions.

[A] Sc-wls: Towards interpretable feed-forward camera re-localization, ECCV 2022
Thank you for all the valuable feedback.

We have uploaded the responses according to the reviewers' requests on the global rebuttal page and individual comments.

Today is the only day left for discussion, so please verify our responses, and if we have addressed your concerns satisfactorily, we kindly request a positive reassessment.