Activating Self-Attention for Multi-Scene Absolute Pose Regression
Authors: Miso Lee
Jihwan Kim
Jae-Pil Heo
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-Scene Absolute Pose Regression
Transformer
Attention Collapse
Primary area: machine_vision
TL;DR: Unleashing the potential of self-attention by rectifying the distortion of query-key embedding space

Abstract: Multi-scene absolute pose regression addresses the demand for fast and memory-efficient camera pose estimation across various real-world environments. Nowadays, transformer-based model has been devised to regress the camera pose directly in multi-scenes. Despite its potential, transformer encoders are underutilized due to the collapsed self-attention map, having low representation capacity. This work highlights the problem and investigates it from a new perspective: distortion of query-key embedding space. Based on the statistical analysis, we reveal that queries and keys are mapped in completely different spaces while only a few keys are blended into the query region. This leads to the collapse of the self-attention map as all queries are considered similar to those few keys. Therefore, we propose simple but effective solutions to activate self-attention. Concretely, we present an auxiliary loss that aligns queries and keys, preventing the distortion of query-key space and encouraging the model to find global relations by self-attention. In addition, the fixed sinusoidal positional encoding is adopted instead of undertrained learnable one to reflect appropriate positional clues into the inputs of self-attention. As a result, our approach resolves the aforementioned problem effectively, thus outperforming existing methods in both outdoor and indoor scenes.

Activating Self-Attention for Multi-Scene Absolute Pose Regression
Miso Lee
Jihwan Kim Sungkyunkwan University dami Camienoskku. edu
Jae-Pil Heo' Sungkyunkwan University jaepilheo@skku. edu
Sungkyunkwan University dlalth6570skku.
Abstract
Multi-scene absolute pose regression addresses the demand for fast and memory- efficient camera pose estimation across various real-world environments. Nowa- days, transformer-based model has been devised regress the camera pose directly in multi-scenes, Despite its potential, transformer encoders are underutilized due to the collapsed self-attention map, having low representation capacity. This work highlights the problem and investigates it from new perspective: distortion of query-key embedding space. Based on the statistical analysis, we reveal that queries and keys are mapped in completely different spaces while only few keys are blended into the query region. This leads to the collapse the self-attention map propose
all queries considered similar to those few keys. Therefore, simple but effective solutions to activate self-attention. Concretely, we present an auxiliary loss that aligns queries and keys, preventing the distortion of query-key space and encouraging the model to find global relations by self-attention. addition, the fixed sinusoidal positional encoding is adopted instead of undertrained learnable one to reflect appropriate positional clues into the inputs of self-attention. As result, our approach resolves the aforementioned problem effectively, thus outperforming existing methods in both outdoor and indoor scenes.
Introduction
Camera pose estimation is fundamental and essential computer vision task, adopted in numerous applications such as augmented reality and autonomous driving. Geometric pipelines [[] with 2D and 3D data have been mainstream with high accuracy. After extracting features and matching 2D-3D correspondences, camera pose is approximated via Perspective-n- Points (PnP) algorithm and RANSAC 77]. However, there still remains several challenges for real-world applications, including high computational cost and a huge amount of 3D point cloud.
Absolute Pose Regression (APR) tackles these issues by directly estimating the 6-DoF pose from single RGB image in an end-to-end manner. First introduced by Kendall al. [. subsequent APR methods [9118] have been devised based on convolutional neural networks (CNN). However, they still demand multiple models and individual optimization to be applied real-world multi-scene scenarios. In this regard, Multi-Scene Absolute Pose Regression (MS-APR) has emerged to satisfy the needs speed and memory efficiency across multiple scenes [99. MSTransformer (20) pioneers streamlined one-stage MS-APR approach with transformer architecture. It leverages the transformer decoder not only to improve memory efficiency but also to enhance accuracy significantly.
However, we point out that the learning capacity of transformer encoders is underutilized. As shown in Tab. [ MSTransformer's encoder self-attention modules do significantly improve or even degrade performance. Although we discovered low-rank, collapsed attention maps, which are known
*Coresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Ablation encoder self-attention modules in MSTransformer 20 of median position and orientation errors for each experiment.
We report the average
Outdoor [8]
Indoor [27]
MST
20
1.28m / /2773
0.18m 7.28
MST w/o encoder SA
1.21m/2.88 2..4
0.18m /7.49
Query-Key Space
Attention Map
Query-Ket Space
Attention Map
Query Key
Query
Key
Attendad Keys
Attendeed Keys
(a) Baseline
(b) Otts
The figure shows query-key spaces, self-attention maps, and attended keys from the orientation transformer encoder of the baseline and ours, respectively. (a) In the case of the baseline, queries and keys are mapped in separate regions, while only a few keys are blended into the query region. Consequently, the self-attention map collapses and whole image features are represented by meaningless few keys, indicating waste of learning capacity of transformer encoder. (b) However, our solution makes queries and keys interact with each other, activating self-attention. This allows the model to obtain crucial global relations within image features, capturing salient global features.
Figure 1:
to cause gradient vanishing 21 or training instability (22, general solutions (22-44 were not correct the problem in the case of MSTransformer This challenge motivates us to elucidate the phenomenon from another perspective in the context of MS-APR.
Therefore, this work proposes brand new analysis: distortion of query-key embedding space, where the attention score originates. We statistically substantiate that queries and keys are mapped in completely separated regions in the embedding space while only few keys are blended into the query region. In this situation, attention maps inevitably collapse, as all queries are represented similarly to those few keys. Intuitively, it means that all image patches are just represented by only few image patches, as illustrated in Fig. [ We conjecture that the task difficulty contributes to the issue; the model should extrapolate the camera pose from single RGB image across multiple scenes. To support this, we empirically reveal that the model tends to avoid exploring self-relationships at the beginning of the training. These findings align with our additional observations that learnable positional embeddings used in self-attention are also undertrained.
Nevertheless, we expect that image features incorporating global self-relations will be valuable to estimate the camera pose by capturing salient features such as long edges and corners. This work thus introduces simple but effective solutions that activate encoder self-attention modules for ASR Concretely, we design an auxiliary loss which aligns the query region and the key region. By forcing all queries and keys to be mapped into the close and dense space, they are highly encouraged to interact with each other by self-attention-iie and blowing the whistle! Furthermore, we explore various positional encoding methods based on empirical evidence since current undertrained one confuses the model to estimate the camera pose with incorrect positions. Finally, fixed 2D sinusoidal encoding is adopted instead of Icarnable parameter-based methods [6]. This enables the model to consider appropriate positional clues of each image feature during exploring self-relationships from the beginning of the training.
putting boxers the ring
As such, the model can obtain rich global relations from activated self-attention as shown in Fig. I These relations are incorporated into image features, acquiring more informative encoder output. Extensive experiments demonstrate that our solution recovers the self-attention successfully by preventing the distortion of query-key space and keeping high capacity of self-attention map [22 As result, our model outperforms existing MS-APR methods in both outdoor (8) and indoor 22
scenes without additional memory during inference, upholding the original purpose of MS-APR.

---

2 Related Work
Absolute Pose Regression (APR) is to estimate 6-DoF camera pose directly using only an RGB image as input. First proposed by Kendall et al. [ subsequent works p 10
13
17 have
introduced advanced architectures and training methods. However, Sattler et al. 22 demonstrated
that these models have a tendency to memorize training data, resulting in poor generalization when presented with novel views. In response, IRPNet [16] attempted fast APR by adopting pre-trained image retrieval model as feature extractor. On the other hand, E-PoseNet [T5] substituted vanilla CNN with Group-Eque Equivalent CNN to extract geometric-awane image features. Recently, NeRF- based models [1 12 29] have improved the APR performance significantly by combining existing APR models with NeRF-W [D] to capture geometric information or synthesize additional images. However, these models have the limitation of being scene-specific, making them impractical for real-world applications (20). NeRF-based models [1] 12 [2] are particularly vulnerable since mrommmooon,,
NeRF-W is not just scene-specifie, but also requires lots of time for taking
training, and generating additional data.
Multi-Scene Absolute Pose Regression (MS-APR) aims to estimate a camera pose from multiple scenes with a single model for scalability. Initially, MSPN [19] made feature weight database for each scene using CNN, and then fed the indexed features into scene-specific regressors for estimating camera pose in multi-scenes. However, still had drawbacks of training time and memory inefficiency. Against the backdrop. MSTransformer [20] enabled sound one-stage MS-APR with transformer-based architecture. They proposed to use decoder queries as scene queries to make both scene classification and camera pose regression possible at once. The whole decoder outputs are firstly used to predict the scene index, and then the decoder query corresponding to the scene is used to regress the camera pose. This allows it to become more general through various scenes. Recently, additional studies 32] have been proposed tine-tunigg methods, but reverting scene-specific again and using additional parameters. In contrast, propose simple but effective training methods for training MS MSTrnsfformrr from scratch, maintaining the original purpose of MS-APR.
Self-A Atteniinn is a attention mechanism which represents input sequences by inter-relationships among all elements. Following the remarkable performance of transformer [3 36) it became widely adopted in various fields. MSTransformer 120] also used self-attention both the transformer encoder and decoder. However, recent studies 221-44 37.40) have pointed out the failure of self-attention.
Firstly, several studies on natural language processing [37 38 have analyzed that self-attention map collapses to a low-rank matrix, by focusing attention scores to meaningless tokens such as
CLS and SEP. In the case of vision, several works [23] 40) have addressed the issue of collapsed self-attention in transformer encoder. There are also studies [21,22) [4]] 39) which fundamentally demonstrated that self-attention maps collapse into low-rank matrices in certain conditions. Similar to these findings, self-attention modules in MSTransformer's encoders do not significantly improve or even impair performance due to the collapse. However, found that pioneers not very effective in the case of MS-APR, which is shown in our experiments. Therefore, we analyse the problem from new viewpoint, distortion of query-key space and undertrained learnable positional embedding, and propose solutions that settle down the problem successfully in MS-APR.
3 Preliminary
Camera Pose. The location of a captured image I can be calculated from camera intrinsic and extrinsic parameters. The former is specific to the camera itself, and the latter corresponds to the pose of the camera with respect to a fixed world coordinate system. As such, the camera pose p can be represented as (t,r) r). Here, E R3 the camera position (translation) and r F the camera orientation (rotation). Note that r is an unit quaternion vector so p has 6-DoF.
Model Architecture. MSTransformer 20 suggests the model architecture inspired by DETR [35 is composed of CNN, transformer encoder-decoders, a scene classifier, and regressors. Firstly, given an image I, CNN features for the orientation f. is extracted from middle level while those for the position ft. is computed from middle-high level. fr and fr are projected by x convolution and reshaped as ft E N.D and f. E N.D to be cransforeer-oompatbbee inputs, where N₂ and N, the number of tokens for the position transformer and the orientation transformer, respectively.
3

---

Position Transformer Enc.
Orientation Transformer Enc.
Space
Postion Orientation
Occurrence Retio
Occummnca Rotto
6.0
Disiance b/w
0.0 0.1 0.2 24 06 07 0.8 09 1.0
0.0 0.1
0.2 0.5
Iterations 1500
2000 2500 1900
Query Region (a) Puriy Levels in Query Regions
Purlty Query
(b) Incresse Tendency of Distance
Figure 2: (a) shows the purity levels in query regions with the baseline on 7Scenes, referring the Eq. 2 Note that the purity is 1.0 when the query region is composed only with queries, but slightly lower than 1.0 when a small subset of keys resides in the query region. According to (a), statistical evidence supports the prevalent occurrence of the blending of few keys into the query region across the entire dataset, both the position and orientation transformer encoders. (b) illustrates the increasing tendency of distance between the query region and the key region the encoder. They lean away each other even at the beginning of the training. Here, the distance between query region and key region an average value across layers and heads.
The architecture transformer encoder-decoder follows the general transformer 33 35 utilizes decoder query scene query make it possible for both scene classification and camera pose regression at once, There are the position and the orientation transformers, each of which has scene queries 2t MR and 2, E RM where M is the number of scenes, respectively. In the encoder, there are several layers to reinforce the CNN features. Every layer of the encoder consists of multi-head self-attention module and multi-layer perceptron (MLP) with layer normalization and residual connection.
Mcanwhile, the decoder is designed to not only enhance the decoder queries but also learn the relationship between encoder features and decoder queries. It has multiple layers like the encoder but additionally includes multi-head cross-attention module in each layer. Afterwards, the scene classifier predicts the scene index m from concatenated decoder outputs = = [zz; zn] E Mx2D In the end, and f are predicted by MLP regressors from the m-th decoder outputs z th E RRP and 2, m D. respectively.
Self-Attention. There are three main components attention mechanism: query Q, key K, and value V. The final output is computed as weighted sum of V. with weights derived from the similarity score between Q and K. Formally, given the input E RVXD with N tokens of dimension D. the h-th attention head is defined follows:
Attn (X) An ho
(I)
where A. softmax
DD/H
Here, A is h-th attention map and // is the number of heads. When inputs to query, key, and value all the same, it is usually called self-attention.
4 Problem Analysis
4.1 Distortion of Query-Key Embedding Space
We introduce our viewpoint why encoder self-attention modules in MSTransformer are not utilized as shown Tab. [ Our main intuition that the model tends to avoid the self-attention mechanism [38 24 by distorting the query-key embedding space due to the learning difficulty. On the face of it, collapsed self-attention maps may seem to be the root cause. However, we further investigate the issue, focusing on the query-key embedding space. The attention map is determined by computed similarity scores between queries and keys, as expressed Eq. I In other words, the projection matrices and bias are what the model learns, and the attention map is merely calculated consequence. As such, the collapsed attention map which induces the model's bypassing comes from the distorted query-key key space, therefore we highlight it to address the problem.

---

Fixed PE Wisth
Learnalee PE Width
Fixed Width
Learnable Width
10
Height
Helghtt
Height
Position Transformer Encoder
Orientation Transformer Encoder
Figure 3: The figure shows L.2 distances between the top-left token and other tokens based on the fixed 2D sinusoidal positional encoding and learnable positional embedding, respectively. Here, the learnable positional embedding is the result of training with the baseline. The fixed positional encoding preserves the order input sequences, but in the case of the learnable positional embedding, tokens not aligned at the same height width were all treated randomly further away.
When collapsing, queries and keys are completely separated while a small subset of keys are blended into the query region across all layers and heads, both in the position and the orientation encoders of the baseline 120]. To gauge the statistical incidence of such cases, we first clarify the query region and the key region in the query-key embedding space. By clustering mean vector of queries q and mean vector of keys l initial centers, query query-dominant region Q is obtained from the cluster with center q. Then we define the purity of query region P as follows:
K through k-means using
1
QQq). Q
(2)
Intuitively, P is 1.0 when the query region composed only with queries, slightly lower than 1.0 when small subset of keys are blended into the query region, and about 0.5 when queries and keys are mixed together. Fig. 2 (a) illustrates the statistical results on 7Scenes [27] dataset. It shows that the phenomenon point orientation transformer encoders. Under this condition, the attention map inevitably collapses as all queries computed to be similar to those keys.
is predominant throughout the whole dataset both for the position and
We also verify that queries and keys lean away each other from the beginning of the training as shown in Fig. 2 (b). The model is trained to locate only a few keys close to the query region while keeping the long distance between the query region and the key region. Here, the distance is calculated using the 1.2 metric between their centers. These investigations imply that the model avoids the process of identifying self-relations due to the task difficulty only with the original task loss.
4.2 Undertrained Positional Embedding
Moreover, we discuss the importance of appropriate positional encoding for the task and the negative effects of the current positional embeddings in this section. As self-attention is a permutation innutation-invariantt invariant mechanism, positional encoding is widely adopted to the queries and keys to account for the order of the input sequence [34. It plays vital role utilize the encoder self-attention modules, particularly in MS-APR. To be specific, it is not only important to detect features like roofs and windows the image, but also crucial to know where they are located the image for accurate pose estimation. In other words, the model has to estimate the camera pose with shuffled image patches if uses inappropriate positional embedding.
Despite its importance, learnable positional embeddings in the baseline remain undertrained with deactivated self- self-attention modules. Fig. shows the visualization results of the distances between the top-left token and other tokens, based on the fixed 2D sinusoidal positional encoding and the learnable positional embedding trained the baseline. One can observe that distances between tokens are drastically larger if the token has different vertical or horizontal position with the top-left token, the case of learnable positional embeddings. This makes even harder for the model to learn geometric relationships between image features by self-attention mechanism.
5

---

2D Sinusuidal Positional Racading
Transformer Encoder
Trandfomeg
Nkm
1422
L
Position Peatumes
Image
Query: Space
LOKA,
MIP
Lucan
LQKA,
Odiettaion Festures f,
SA
VISHN
Trasfomerr
C,
Figure 4: Fig. 4 iillustrates the training pipeline with solutions. We apply additional objectives LOKA, and LOKA, to the model activate the self-attention modules. Specifically, queries Q and keys K interact with each other by forcing the centroid of query region q and the centroid of key region k to become closer. Here, we encode all input queries and keys with fixed 2D sinusoidal positional encoding to ensure active interaction between Q and K with reliable positional clues.
Activating Self-Attention for MS-APR
In this section, introduce our solutions with full objectives for training transformer-based MS-APR model successfully by mitigating self-attention map collapse. Firstly, we employ the L2 loss regress the camera position and orientation. Let denote the ground-trutt camera pose as (t, r) and the estimated camera pose (t, P). Then, the position loss and the orientation loss defined L. ||t i|12 and L, ||r T ||,, respectively. Here, is required balance two losses to make the model learn both camera position and orientation since the former a scalar value but the latter is unit quaternion 8 [4]. According to [14], the final pose loss is given by:
Lpace
Lg exp (-8t) + 8t + C, exp (-8,) 8r.
where 8, and s, are the learnable parameters to adjust the uncertainty.
Secondly, the model should be capable of classifying the scene from the input image to work with multi-scene dataset. Let denote y R as one-hot vector encoding the ground-truth scene index of the input image, and ý E R M as the predicted probability distribution of scene classes from concatenated decoder outputs z. Then, the scene classification loss proposed in [20 given by:
Legone
Σ Uj log ūy-
On top of it, we introduce an auxiliary loss to prevent the model from avoiding the self-attention depicted in Fig. 4 As discussed in Sec. [4.1 it is essential ensure that the queries and the keys do not become alienated from each other while encouraging their interaction by embedding in close proximity. To achieve this, propose the Kue Alignment (QKA) loss additional objective. It is defined as follows:
H
LOKA = H
1
where L, is the number of encoder layers, H is the number of heads, and g and ky are the mean vectors the queries and the keys per each encoder layer and head, respectively. By inducing queries and keys to be placed in shared compact area, ensure that they interact with each other in the

---

Table 2: Comparative analysis of MS-APRs the Cambridge Landmarks dataset (outdoor localiza- tion) 8
We report the median position/ooreentaionn errors in meters/degrees.
Method
King's College
Old Hospital
Shop Facade
St. Mary
Average
MSPN [19]
1.73/3.65
2.55/4.05
2.92/7.49
2.67/6.18
2.47/5.34
MST 20
0.83 1,47
1.81/2.39
0.86/3.07
1.62/3.99 1.57/3.50
1.28/2.73
+Ours
0.88 1.29
1.55/1.87
0.79/2.51
1.19/2.29
Table Comparative analysis of MS-APRs on the 7Scenes dataset (indooor localization) 27 We report the median position//orientaiion emrors in meters/degrees.
Method
Chess
Fire
Heads
Office
Pumpkin
Kitchen
Stairs
Average
MSPN 19
0.09 0.09/4.760.99/11 10.50 0.16/13.11 0.16/6.80 0.19/5.50 0.21/6.61 0.31/1L63
0.20/8.41
MST 20
0.11/4.66 0.24/9.60 0.14/12.19 0.17/5.66 0.18/4.44 0.17/5.94 0.26/8.45
0.18/7.28
+Ours
0..1 0.17/6666
self-attention module. Here, we encode all input queries and keys with fixed 2D sinusoidal positional encoding instead of undertrained learnable positional embedding. This guides the model to stably learn seff-relationships from the beginning the training. As a result, the accurate interaction can be accomplished by reflecting the verified correct positional information to input queries and keys.
We apply our QKA loss both the position and orientation transformer encoders, while adjusting the weight of the loss to integrate with the original task loss. Accordingly, the final auxiliary loss Laux Amxx (COKA, + LOKA.), where Auu is the weight of the auxiliary loss. LyKA, and LOKA, are QKA losses for the position and orientation transformer encoders, respectively. Putting all together, our full objective Cpoce + Lwenne Laux.
Experimental Results
6.1 Experimental Setup
Datasets. We train and evaluate the model on outdoor and indoor datasets [8] 27 which include RGB images labeled with 6-DoF camera poses. Firstly, we use the Cambridge Landmarks which consist of six outdoor scenes scaled from 875m² to 5600m Each scene contains 200 to 1500 training data. We conduct the experiment on four scenes in Cambridge Landmarks, which are typically adopted
for evaluating absolute pose regression [12 55 20 On the other hand, use the 7Scenes dataset which consists of seven indoor scenes scaled from 1m² to 18m2. Each scene includes from 1000 to 7000 images.
Training Details. We entirely follow the configuration of the baseline 200 We train the model with single RTX3090 GPU, Adam optimizer with 3 0.9, 83 0.999, 10010 10 and the batch size of 8. For the 7Scenes dataset, we train the model for 30 epochs with the initial learning rate of 1 x 10-1, reducing the learning rate by 1/10 every 10 epochs. In the case of Cambridge Landmarks dataset, we train the model for 500 epochs with the initial learning rate of 1 x 10 reducing the learning rate by 1/10 every 200 epochs. Afterwards, then fine-tune the position branch as same as the baseline 20). The model is trained for 60 epochs with the initial learning rate 10-4, reducing the learning rate by 1/ 10 every 20 epochs.
freeze the CNN and the orientation branch,
The learnable parameters 8; and 8, for the original task loss are initialized as in [14. Both for the position and orientation transformer encoder-decoder, the number of layers L is 6 and the number of heads H Lastly, we set the Aaux for our query-key alignment loss as 0.1.
6.2 Comparative Analysis
MS-APR Methods. We compare the model trained our solutions with other single-frame multi-scene APR models for fair comparison, i.e., excluding scene-specific methods. Tab, 2 Tab. 3 and Tab. 4 show the experimental results on four outdoor scenes in Cambridge Landmarks [8] and seven indoor scenes in 7Scenes [27), respectively. With our solutions, the model shows better performance through all scenes, compared to other MS-APR methods. This results verify that activating self-attention has
7

---

Table 4: Comparative analysis of the baseline and ours. We report the localization recall several thresholds on Cambridge Landmarks and 7Scenes datasets.
Cambridge Landmarks [
7Scenes [27]
Method
(1m, 5")
(1m, 10º) (2m, 5") (2m, 10")
(0.2m, 59) (0.2m, 10") (0.3m, 59)
(0.3m, 10%)
MST [20]
32.6
35.8
60.4
68.2
28.8
50.2
34.4
63.5
+Ours
35.8
38.5
65.5
72.7
32.6
52.6
39.5
67.1
Table 5: Comparison with alternative methods for collapsed SA and positional encoding methods.
We report the average of the median position/orientation errors on 7scenes dataset.
(a) Alternatve Methods for Collapsed SA
(b) Alternative Positional Encoding Methods
Method
7Scenes [27]
Method
7Scenes 127
Improved SN [23]
0.18m/ 7.044
T5 PE [25]
0.18m 6.97
1 I/caling 24
0.18m/ 0.18m/6.87 66.87
Rotary PE [26
0. 0.18m/ 6.94
Reparam [22
0. 0.19m/6.81 66.1
Fixed PE
0.17m 66.64
QK Alignment
0.17m/6. 6.664
the potential to become general module, generating task-relevant features across multiple scenes, regardless of the specific features of each scene.
Alternative Methods for Collapsed Self-Attention. We apply general-purpose techniques 22, proved to alleviate collapsed self-attention, to the baseline to compare the effectiveness of each method and ours in MS-APR. Here, we conduct experiments with fixed positional encoding, as incorrect inputs may suppress the potential of the solutions. Tab, sahwos that existing techniques have little or no effect terms of recovering the self-attention for the task. Meanwhile, ours significantly improves the baseline in both conditions. We conjecture that it is required to guide the model more direct manner to utilize self-attention for MS-APR.
Alternative Positional Encoding Methods. We conduct experiments with recent advanced learnable positional encoding methods [65 with our QKA loss. As shown in Tab. 5bb even advanced methods are not suitable to the task as they mainly depend on learnable parameters and relative position. We assume that absolute, purified positional clues important stabilize the training of the embedding space in the case of MS-APR.
6.3 Quantitative Analysis
Attention Entropy of Self-Attention Maps. To provide quantitative evidence that solutions enhance performance of model by activating the encoder self-attention, we measure attention en- tropy [2 defined as follows:
Ent(Attn(X)
Ent( (A))
(6)
I1
1
where Ent(An)
asj log (aiz)
N
Note that high attention entropy indicates training stability and high representation capacity of self-attention map (22).
Fig. 5 shows the attention entropy for each encoder layer in both the position and orientation transformers. As illustrated, the model employing solutions has significantly higher attention entropy than the original baseline model across all encoder layers. These results validate that our solutions successfully induce the transformer-basee model to activate the encoder self-attention, thus improving the representation quality self-attention map.
Purity Levels in Query Regions. To verify the effectiveness of our auxiliary loss rectifying the distortion of query-key space, we measure the purity levels in query regions defined Eq. E The statistical results for both the bascline and the model employing solutions presented in Fig.

---

Position Transformer Enc.
Orientation Transformer Enc.
Attention Entropy
Beseline
Actention Entropy
Baseline
5.0
Durs
8.0
Ours
4.5
5.5
4.0
5.0
25
Layer Index
Layer Index
Figure 5: The figure shows the attention entropy of the baseline and ours for each encoder layer. demonstrates that our solutions significantly improve the utilization of encoder's 's learning capacity.
Position Transformer Enc.
Orientation Transformer Enc.
Occumence Ratio
Occumennc
0.0% 8.6% 10% DOS
DOS 6.6% 2.05 D0% 0.%
0.% 10% 62% 0.6
0.4 1.55 pas 025
0.0
0.7 0.9 Purity of Query Region
0.2
0.8 0.9 Purity Query Region
044 06
Figure 6: The results show the change of purity of query region P, defined in Eq. across 7Scenes dataset. Note that the purity is 1.0 when the query region is composed only with queries, but slightly lower than 1.0 when few keys resides in the query region. Compared to the baseline, there cases where only a few keys are blended into the query region with our solution.
Here, the purity levels represent the proportion of the queries allocated to the guery-dominant cluster. Our observations reveal that the nearly half keys belong to the query region in almost all cases, indicating that the query and key regions are highly interleaved. This demonstrates that our solutions restrain all queries from being treated as similar only a few keys, thus preventing self-attention collapse and activating self-attention.
6.4 Qualitative Analysis
Fig. 7 reports visualization of query-key space by t-SNE and attention results of the baseline and the model employing solution. Note that the attention results depict the scores of the keys, averaged over the queries in the self-attention map. One can observe that the query regions and key regions of the baseline separate. In contrast, the regions reducing the occurrence of only a few keys blended into the query region. This condition necessitates that the model reflects global interactions between queries and keys, therefore, global relations incorporated into image features as depicted in Fig. 7 We can see that geometric relations such as long edges are usually obtained, which the critical points for the task. More examples in the supplementary material.
highly aligned by our solutions,
6.5
Ablation Study
We conduct an ablation study on our solutions to verify the effectiveness of each component. As shown in Tab. [6] our QKA loss enhances the performance in both the outdoor and indoor datasets. Additionally, overall performances are further improved with fixed positional encoding, rectifying the incorrect inputs in the self-attention. This ablation study indicates that several useful self-relation can be extracted from noisy content features by self-attention, but reliable positional clues are particularly important to learn various crucial relationship for estimating more accurate camera pose.
7 Discussion
Limitation and Future Work. The proposed auxiliary loss is designed based on the assumption that an image has many key features which are useful for estimating the camera pose. However, there

---

Query-Kor Key Space
Attended Keys
Query- Key Space
Attended Keys
Quety Key
Quary Key
Baseline
Query Key
Query Key
Ours
Figure 7: The figure shows the examples of t-SNE results of query-key space and attention results of the baseline and the model employing our solution. The query and key regions are separate in the baseline while our solutions align the regions, enabling the model to focus on salient global features and incorporate self-relations into image features.
Table Ablation on our solutions. We report the average of the median position/orieenation errors.
Method
Outdoor [
Indoor [27
MST [20]
1.28m/2.73'
11m//.22
Fixed PE
1.29m/ /2.46
0. 0.18m/66 55/6.800
MST+QKA
1.22m /2.68
0. 0.18m//2 8m/7.099
MST+Fixed PE+QKA
1.19m 2.29
0.17m/6.66
could be side effect if only few key features are in the image; ie., if dynamic moving object mainly occupies the image. Developing an algorithm decide whether global self-attention is useful for the image or not could be an interesting future work.
Broader Impacts. The camera pose estimation technology with this work can be expected have several positive social impacts, such as being utilized in programs to locate missing persons. On the other hand, could potentially be misused for purposes such as illegal location tracking programs.
8 Conclusion
This work has exposed that self-attention modules in transformer encoders not activated in MS-APR. Looking beyond the collapsed self-attention maps, we have discovered the distortion of query-key embedding space; queries and keys are entirely separate while small subset of keys resides in the query region. We have examined the predominance of the phenomenon, adducing statistical and empirical evidence. Additionally, the importance of accounting for the order of the input sequence has been addressed in the context of the task. Based on these analyses, this work has proposed simple but effective solutions to activate self-attention by rectifying the distorted query-key space with alignment loss and appropriate positional clues. As result, the model could learn salient global features through self-attention, thus achieving state-of-the-an performance both in outdoor and indoor scenes without additional memory in inference. Our extensive experiments have demonstrated the effectiveness of proposed solutions terms of recovering self-attention for MS-APR.
Acknowledgement
This work was supported in part by MSIT&KNPA/KIPP) (Police Lab 2.0, No. 206121 MSIT/ITP (No. 2022-0-00680, 2020-0-01821, 2019-0-00421, RS-2024-00499618 RS-2024. 00360227, RS-2024-00437100 RS-2024-004376333, and MSIT/NRE (No. RS- 2S-0024-00357729,
10

---

References
[1] Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Efficient & effective prioritized matching for large- scale image-based localization, IEEE transactions on pattern analysis and machine intelligence, 39(9): 1744-1756, 2016.
[2] Eric Brachmann and Carsten Rother. Learning less is more-6d camera localization via 3d surface regression. In Proceedings of the IEEF conference on computer vision and pattern recognition, pages 4654-4662, 2018.
[3] Kiru Park, Timothy Patten, and Markus Vineze. Pix2pose: Pixel-wise coordinate regression objects for 6d pose estimation. In Proceedings of the IEEEJCVE International Conference on Computer Vision, pages 7668-7677, 2019.
[4] Xiaotian Li, Shuzhe Wang. Yi Zhao, Jakob Verbeek, and Juho Kannala. Hierarchical scene coordinate classification and regression for visual localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1 11983-11922, 2020.
[5] Paul-Edous Eell-Edo Sarlin, Ajaykumar Unagar, Mans Larsson. Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, al. Back to the feature: Learning robust camera localization from pixels to pose. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3247-3257, 2021.
[6] Eric Brachmann and Carsten Rother. Visual camera re-localization from rgb rgb-d images using dsac. IEFE transactions on pattern analysis and machine intelligence, 44(9):58475585,, 2021.
Martin A Fischler and Robert C Bollcs. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6)3381-395, 1981.
[8] Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In Proceedings of the IEEE international conference on computer vision, pages 2938 2948-2996.220 2015.
[9] Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, and Jan Kautz. Geometry-aware learning of maps for camera localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2616 2625, 2018
[10] Ming Cai, Chunhua Shen, and Ian Reid. A hybrid probabilistic model for camera relocalization. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 238. BMVA Press, 2018. URL
org/ http://ottttttttecceeeee wttp//otttttttee//////e////////e pdf
[11] Shuai Chen, Zirui Wang, and Victor Prisacariu. Direct-posenet: Absolute pose regression with photometric consistency. In 202/ International Conference on 3D Vision (3DV), pages 1185-1185. IEEE, 2021.
[12] Shuai Chen, Xinghui Li, Zirui Wang, and Victor A Prisacariu. Dinet: with direct feature matching. In European Conference on Computer Vision pages 1-17. Springer, 2022.
Enhance absolute pose regression
[13] Alex Kendall and Roberto Cipolla. Modelling uncertainty in deep learning for camera relocalization. In 2016 IEEE international conference on Robotics and Automation (ICRA), pages 4762-4769, IEEE, 2016.
[14] Alex Kendall and Roberto Cipolla. Geometric loss functions for camera pose regression with deep learning. In Proceedings of the IEEF conference on computer vision and pattern recognition, pages 5974-5983, 2017.
[15] Mohamed Adel Musallam, Vincent Gaudilliere, Miguel Ortiz Del Castillo, Kassem Al Ismaeil, and Djamila Aouada. Leveraging equivariant features for absolute pose regression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6876-6886 2022.
[16] Yoli Shavit Ron Ferens. Do we really need scene-specific pose encoders? In 2020 25th International Conference on Pattern Recognition (ICPR), pages 3186-3192. IEEE, 2021.
[17] Florian Walch, Caner Hazirbas. Laura Leix-Taixe, Torsten Sattler, Sebastian Hilsenbeck, and Daniel Cremers. Image-based localization using Istms for structured feature correlation. In Proceedings of the IEEE international conference computer vision, pages 627-637, 2017.
[18] Xin Wu, Hao Zhao, Shunkai Li, Yingdian Cao, and Hongbin Zha. Sc-wls: Towards interpretable feed- forward camera re- re-localization. In European Conference on Computer Vision, pages 585-60 601. Springer. 2022.
11

---

[19] Hunter Blanton, Connor Greenwell, Scott Workman, and Nathan Jacobs. Extending absolute pose regression to multiple scenes. In Proceedings of the IEEE/CVE Conference Computer Vision and Puttern Recognition Workshops, pages 38-39, 2020.
[20] Yoli Shavit, Ron Ferens, and Yosi Keller. Learning multi-scene absolute pose regression with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2733-27444 2021.
[21] Yihe Dong, Jean- bean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793-2803. PMLR, 2021.
[22] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pages 40770 40803. PMLR, 2023.
[23] Kwonjoon Lec, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. In International Conference Learning Representations, 2021.
[24] Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198-27211, 2022.
[25] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Reseurch, 21()):5485-555) 2020.
[26] Jianlin Su, Murtadha Ahmed, Yu Lu,, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568:12706: 2024.
[27] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2930-2931 2013.
[28] Torsten Sattler, Qunjie Zhou, Mare Pollefeys, and Laura Leal- Taixe. Understanding the limitations of cnn-based absolute camera pose regression. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3302-3312, 2019.
[29] Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, and Arnaud de La Fortelle. Lens: Localization enhanced by nerf synthesis. In 5th Annual Conference on Robot Learning, 2021.
[30] Ricardo Martin Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7219, 2021.
[31] Yoli Shavit and Yosi Keller. Camera pose auto-cncoders for improving pose regression. In European Conference on Computer Vision, pages 140-157, Springer, 2022.
[32] Yoli Shavit, Ron Ferens, and Yosi Keller. Coarse- to tine mceti-scene pose regression with transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[34] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4188 2019.
[35] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[36] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL httpttt///oppeeeeiiiccc www.wwwwweeeerrrrr id=YicbFaiTly
12

---

[37] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. In Proceedingy of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4374-4374, 2019.
[38] Goro Kobayashi, Tatsuki Kuribayashi. Sho Yokoi, Kentaro Inui. Attention not only weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057-7075, 2020.
[39] Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In International Conference on Machine Learning, pages 5562 5571. PMLR, 2021.
[40] Jihwan Kim, Miso Lee, and Jae-Pil Heo. Self-feedback detr for temporal action detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10286-10296, 2023.
[41] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning. pages 6105-6114, PMLR, 2019.
13

---

A Appendix
A.1 Training Details
We entirely follow the setting of the baseline [20]. The settings are as below.
Data Augmentation. Data augmentation is performed in the same way [8] which is also adopted in the baseline. Afterwards, brightness, contrast, and saturation are randomly jittered for photometric robustness. Test images also resized to 256 x 256, but they are center-cropped to 224 x 224, without any photometric augmentation. Additionally, following the baseline, smaller scenes in the Cambridge Landmarks dataset g are oversampled during training deal with data imbalance problem.
Backhone. As did in the baseline (20). we utilize a CNN model, the EfficiennNNet-OO [1]. to extract local image features from input images. The image features ft for the position and fr for the orientation extracted from the fourth block and the third block, respectively. Accordingly, the size of ft is 14 x 14 x 112 and the size fr is x 28 x 40, i.e., the number of transformer encoder tokens N₂ and N, 196 and 784, respectively.
A.2 Visualization
Query-Key Spaces. addition to the statistical and quantitative results introduced the main paper, we visualize the query-key spaces across heads and layers. Fig. A1 reports the visualization results of query-key spaces from the baseline [20] across all even layers and heads. qualitatively shows that the distortion of query-key space occurs in most heads in transformer encoder of the baseline. In contrast, one can observe that queries and keys properly intertwine each other with our solutions as shown in Fig. A2
14

---

2 Layer
4th Layer
6h Layer
Query
Key
Query
Key
Query
Key
2 Head
4th Head
6th Head
8th Head
Figure A1: The figure shows the t-SNE results of query-key space from the baseline across even layers and heads. Overall, query and key regions are separate while small subset of keys are blended into the query region.
15

---

2nd Layer
4th Layer
6h Layer
Quary
Key
Quary
Key
Query
Key
2 Head
4th Head
6th Head
8th Head
Figure A2: The figure shows the t-SNE results of query-key space from the model with our solutions across even layers and heads. The problem point out resolved with solutions; similar query subsets and key subsets are grouped together.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We explain the paper's contributions and scope accurately in the abstract and
Sec. 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in Sec. g
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We explain all the formulas in Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We explain the exact method in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

reproducible, they Answer: [Yes]
Justification:
We include the code in the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides/ http://ooeeaaaioiiininnoeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips httpcc/cpplllgggiuueee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all details in Sec. 6 and supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since the train dataset and test dataset used for evaluation are completely separated, there are no issues arising from data splitting. For the fair comparison, our experiments follow the exact same configuration including random seeds as the baseline. The configuration and the code is included in the supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the compute resources in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conducted the research with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss broader impacts in Sec. g
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work uses public datasets and poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We checked the license of our bascline code and datasets.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not provide new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not contain crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not need study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "SdMnU3IUjF", "reviewer": "Reviewer_aJYL", "summary": "This paper is about a improving self attention in the transformer architecture for multi-scene APR. The show that the self attention module in the SOTA transformer model for APR is actually not helping much and offer an potential explanation. The paper claims that the keys and queries end up in different spaces, such that the inner product between keys and queries is very close to zero in most cases, leading the attention to collapse to zero. The paper proposes an additional loss term that encourages the mixing of queries and keys leading to much more overlap. The addition of this term results in a noticeable improvement in APR metrics across indoor and outdoor datasets.", "questions": "In Table 3, MST and +Ours is incorrectly bold for the position error for the Office scene.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The motivation for the paper is clear and concise and overall well laid out. Table 1 lists metrics with and without self attention showing the limited utility. It is shown empirically that the keys and queries are isolated form one another as well as theoretically explained why this is an issue. The proposed approach is well explained, straightforward, and has the desired effect of quantitatively improving attention and qualitatively improving APR metrics. This will clearly be used with mulitscene transformer based APR methods going forward because it's simple without requiring any extra data and is effective.\n\n+ The problem analysis Section 4 is thorough and useful. The side effects of the commonly used techniques are clearly explained and the arguments for why this is a problems is compelling. \n\n+ There are sufficient implementation details. I could easily reimplement this paper from the information provided. The hyper parameters are shared with [17] so it is unlikely that the performance gain is due to hyperparameter tweaking.\n\n+ Thorough ablation of different methods for solving the SA problem as described as well as different positional encoding methods. The methods proposed in the paper are validated as being the best on these datasets. \n\n+ This is one of the few papers I've seen using attention maps in pose regression where I feel like the visualizations and discussion around attention are actually meaningful.", "weaknesses": "- Table 5 is difficult to parse. I feel like it could easily be represented with histograms as in Figure 2a. Similarly I feel like the histograms a fairly course. The point comes across okay but I'm not sure why such a course histogram would be used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Histogram**\n\nA1. We thought that using coarse bins was suitable for visualizing and showing the general trend differences in purity between the baseline and our method.\nHowever, we agree with the reviewer's suggestion to provide a more specific analysis.\nAccordingly, we present the baseline's purity with a fine-level histogram and include the visualization for our method in Figure A2 on the global rebuttal page.\n\n**Q2. Incorrect bold**\n\nA2. Thank you for bringing this to our attention.\nWe will make sure to correct it in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DUZeqjzde9", "reviewer": "Reviewer_K2gT", "summary": "The paper analyzes the collapse of self-attention map in Multi Scene Pose Transformer model and proposes two simple but effective methods: auxiliary loss and fixed 2D sinusoidal encoding to solve this problem. The improved method delivers SOTA  performance on the Multi Scene Pose Regression task.", "questions": "The results in Table5 indicates that the loss function is a very hard constraint that limits the purity of query region to [0.4, 0.6), such constraint is not suitable for all situations. Do authors have comments?", "limitations": "The manuscript is limited in scholarship, missing references to more APR methods like [A].\n\n[A] Sc-wls: Towards interpretable feed-forward camera re-localization, ECCV 2022", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed auxiliary loss seems a good solution for query-key distortion in MS Transformer which is simple and effective.\n2. The figures and tables in this paper are exceptionally clear and well-organized, making the paper easy to understand and interpret.", "weaknesses": "1. Since query-key distortion is studied in the transformer literature and the fixed 2D sinusoidal positional encoding is an off-the-shelf module, the methods in this paper seems to be lack of novelty. It would therefore be nice to discuss more on why APR task will lead to query-key distortion rather than saying \"The model tends to avoid the self-attention mechanism due to the learning difficulty.\" in line 127-128. The paper reads as if it's discussing improvements to the transformer and not related to APR.\n\n2. Ablation study in Table6 indicates that most of the performance improvements come from the using of fixed sinusoidal positional encoding, a more complete ablation study may be helpful to show the ability of the proposed loss function.\n\n3. Section5: Activating Self-Attention for MS-APR is overly detailed, most of the information is a repetition of MS-Transformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Novelty**\n\nA1. We would like to highlight the novelty of our unique analysis and simple but effective solution.\nBy focusing on a different aspect from previous studies [18-21, 31-34], we identified the problem in self-attention modules of APR models and solved it, which previous works could not.\nSpecifically, they usually explored attention map theoretically to activate self-attention.\nHowever, as shown in our experimental results, these conventional solutions did not work for APR.\nWe think that we provide meaningful discussion by discovering the query-key distortion and **statistically and empirically validating the problem using the dataset specific to this task.**\n\nPositional encoding is no exception.\nPrevious studies primarily investigated various learnable positional embeddings to assist self-attention mechanism.\nHowever, our work found that these variations were not effective for this task and actually increased the difficulty of reasoning with self-attention.\nAlthough 2D sinusoidal positional encoding is off-the-shelf, it is sufficient to provide the essential position information for self-attention in APR.\n\nWe believe that establishing a practical and effective method that improves performance without compromising speed and memory efficiency is one of the most important discussions in MS-APR.\nFrom this perspective, our methods contribute to MS-APR by activating deactivated modules perfectly **without causing any slowdown or additional memory usage.**\n\n**Q2. Learning difficulty**\n\nA2. Rather than query-key distortion occurring because of estimating camera pose, we hypothesize that the problem stems from the **APR task being difficult due to limited data and the need to extrapolate 3D values from 2D inputs.**\nIn other words, self-attention, which has few biases and constraints, may fall back on shortcuts under difficult conditions for reasoning.\nHence, we suspect that similar issues may arise in other challenging fields with transformer-based models, which struggle with limited data and the need for sophisticated reasoning.\n\nHowever, it is too difficult to quantify the learning difficulty.\nTherefore, we have explored other demanding vision tasks to verify the effectiveness of our method.\nPlease refer to Table A2 and Figure A1 on the global rebuttal page for our experimental results on the DETR-based model in temporal action detection.\nSimilar issue of attention collapse in self-attention can be found, and our proposed auxiliary loss effectively solves the problem, leading to significant performance improvements.\n\n**Q3. Ablation study**\n\nA3. Please refer to Table A6 for the complete ablation study on the global rebuttal page.\n\n**Q4. Hard constraint**\n\nA4. Thank you for the great question.\nAs we discussed in the limitation, if only specific parts of an image contain information useful for camera pose estimation, such as when a dynamic moving object occupies a large portion of the image, our method might introduce side effects.\nHowever, these cases are not common.\nCamera pose estimation generally relies on the overall layout of the scene, like the long edges between the ceiling and walls, rather than specific parts of the image.\nTherefore, it is postulated that **applying a hard constraint to activate most features could be more beneficial overall for the task.**\nNamely, we presume that our method is effective for APR as it provides stronger regularization compared to other methods aimed at resolving attention collapse.\n\n**Q5. Missing references**\n\nA5. Thank you for bringing this to our attention.\nWe will make sure to include references to [A] and other relevant works in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9OknDP4sCb", "reviewer": "Reviewer_FKQN", "summary": "The paper investigates multi-scene absolute pose regression from a new perspective: query-key embedding space. Focusing on distortion of queries and keys, the paper solutions to activate self-attention, which includes an auxiliary loss to align queries and keys and fixed sinusoidal positional encoding. Experimental results demonstrate that the proposed method outperforms existing MS-APR methods in outdoor Cambridge Landmarks and indoor 7 Scenes.", "questions": "1.Currently, the 2D-3D correspondence based methods (also called coordinate regression based methods) still achieve the state-of-the-art localization performance in both static and dynamic scenes, such DSAC*, KFNet and so on. It is curious that whether the proposed method can apply to 2D-3D correspondence based methods? \n\n2.Does the proposed method affect the network training convergence time?", "limitations": "The authors have adequately addressed the limitation.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The paper proposes new analysis that focuses on distortion of query-key embedding space.\n\n2.The paper presents solutions to reduce the distortion that covers an auxiliary loss that aligns queries and keys and fixed sinusoidal positional encoding. \n\n3.Experimental results demonstrate that the proposed method can reduce the localization error.", "weaknesses": "1.The motivation of Multi-Scene Absolute Pose Regression is somewhat insufficient. The paper declares that Multi-Scene Absolute Pose Regression can satisfy the needs of speed and memory efficiency across multiple scenes. However, these are not validated in the experiments, especially compared with single-scene methods. \n\n2.The proposed method seems to only support transformer based APR, which are limited to its further applications. \n\n3.Some references are missing. Although the paper focuses on Multi-Scene Absolute Pose Regression, the references about single-scene camera relocalization methods still should be discussed, including single-scene ARP methods and 2D-3D correspondence based methods. \n\n4.The experimental results are limited, reflected by the following aspects. \n\n(1)The paper only lists the localization results in comparison with transformer based Multi-Scene APR methods, but the comparisons with other single-scene state-of-the-art methods are missing, especially the speed and memory efficiency which is declared as advantages of Multi-Scene methods.\n\n(2)From Tables 1,2, the improvements of the proposed method seem not obvious, which can not show the method superiority. More discussions are preferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Speed & memory efficiency**\n\nA1. We would like to bring to your attention that the advantages of MS-APR mentioned have already been claimed and proven by previous research, and are not new assertions from our side.\nFirstly, it is known that APR methods are much faster and more memory-efficient than 2D-3D correspondence-based methods because they do not require 3D point clouds or RANSAC algorithms [6, 13, 16, 17].\nAPR methods show almost the same speed; but scene-specific models have limitation of memory inefficiency for large-scale application.\nHence, MS-APR methods propose new architectures that ensure memory efficiency for multiple scenes.\nBesides, our methods do not require additional modules, thereby **maintaining the memory efficiency as the baseline.**\nIt is why we focus on performance differences in the experiment section.\nTo address the issue, we display the memory requirements of recent single-scene and multi-scene APR methods as the number of scenes increases in Table A3 on the global rebuttal page.\n\n**Q2. Comparison with single-scene methods**\n\nA2. Firstly, allow us to clarify by presenting the localization results in comparison with all MS-APR methods, including MSPN [16], which is a CNN and MLP-based model.\nSecondly, please consider that **to maintain a fair comparison**, we did not include 2D-3D correspondence-based methods or single-scene APR methods.\nFor 2D-3D correspondence-based methods, 3D point clouds are required, and thus many APR works [7-10, 13-14, 16-17] have excluded them from the comparison.\nIn this context, several recent single-scene APR methods, which adopt NeRF-W that also requires 3D point clouds, are also excluded from the fair comparison.\nAlthough other single-scene APR methods still pose fairness issues in terms of the number of model parameters, we show the comparison results in Table A4 on the global rebuttal page.\n\n**Q3. Performance improvement**\n\nA3. We understand that the improvements may not appear significant at first glance.\nHowever, please take into account two hidden aspects regarding the conventional evaluation of median errors in position and orientation in APR.\nFirstly, due to the small numerical values, the errors tend to appear minor. Despite this, **our method reduces the position/orientation error rates by 7%/17% in outdoor settings and by 6%/9% in indoor settings.**\n\nSecondly, median error, being a *median*, does not fully reflect the overall performance of the model.\nTo address this, we compared the performance with the baseline using the recall evaluation metric, which is widely adopted in 2D-3D correspondence-based camera pose estimation.\nAs shown in Table A1 on the global rebuttal page, **our method achieves 3-5%p higher recall across various thresholds and datasets compared to the baseline.**\nParticularly, there are substantial improvements of outliers, which are not evident in the median error metrics.\nBy forcing the model to find helpful cues for the task, it becomes especially effective in leveraging previously challenging edge cases.\nIn conclusion, we would emphasize our method's superiority and the remarkable performance improvements it provides over the baseline.\n\n**Q4. Applicability**\n\nA4. We would like to highlight the applicability of our method in two key aspects.\nFirstly, experiments in APR have only been conducted on small datasets, but it will not be long before APR methods need to be evaluated on large-scale datasets such as Aachen Day-Night or RobotCar, which are already tested in 3D-based methods.\nAccordingly, it is worth considering whether scene-specific models will remain competitive in terms of memory requirements as shown in Table A3.\n[16] proved that there is no need for a database in multi-scene by utilizing transformer-based model, drawing the potential of APR.\nTherefore, we believe that **transformer-based APR models will become more prevalent, and our method is applicable to any transformer with deactivated self-attention.**\n\nFurthermore, we would suggest that our analysis and method are **not limited to APR applications.**\nWhile our research began with addressing attention collapse in APR, our method can also be adopted as a solution for other vision tasks suffering from similar issues.\nTo validate this, we applied our method to the temporal action detection task, which uses transformer-based models and exhibits attention collapse.\nAs shown in Table A2 and Figure A1 on the global rebuttal page, the baseline DETR struggles with learning self-attention mechanism, and our method significantly improves performance by resolving this issue.\nTherefore, we ask for reconsideration of the strong applicability and scalability of our work in both APR and transformer research.\n\n**Q5. Application to 2D-3D correspondence-based**\n\nA5. That is an interesting question.\nOur method is designed to maximize the activation of image features that assist in camera pose estimation.\nIn this context, it could also be applied to 2D-3D correspondence-based methods that aim to reinforce image features through the self-attention mechanism.\nFor instance, it could be applied to the ViT Encoder used in [B].\nHowever, please note that they differ from APR methods in terms of task loss and learning complexity; the attention outcomes might differ from those observed in APR.\n\n[B] Revaud, Jerome, et al. Sacreg: Scene-agnostic coordinate regression for visual localization. CVPR, 2024.\n\n**Q6. Training convergence time**\n\nA6. Thank you for highlighting the new benefit of our research.\nWhile the baseline model is trained on Cambridge Landmarks for 600 epochs, our model achieved better performance with only 500 epochs of training, thus indicating a clear benefit in terms of training convergence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EBedmStnyW", "reviewer": "Reviewer_AzjT", "summary": "In this paper, the authors focus on improving the performance of multi-scene absolute pose regression models based on transformers. From statical analysis, the authors assume that the distortion between Q and K features in self-attention and the learnable position encoders are the reasons. Therefore, a Q-K alignment loss and the fixed-pose encoding method are adopted and experiments demonstrate the efficacy on indoor 7scenes and CambridgeLandmarks datasets.", "questions": "Please see the weakness.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The strengths of this paper are as follows.\n\n1.\tOriginality. The undistortion of Q-K features in self-attention is not a new direction as also mentioned in the related works section. While this paper may be the first one to apply it to the multi-scene absolute pose regression task and approve the efficacy. Besides, the authors also demonstrate that the fixed-position encoding works better than the learnable position encoding because learnable position encoding breaks the order the input sequences. These contributions are useful for improving the accuracy of the pose regression as shown in the experiments.\n\n2.\tQuality. The proposed algorithm is easy to follow, and the paper is well written.", "weaknesses": "The proposed method is easy to follow as I mentioned before, so I don’t have many concerns. Several minor concerns are as follows. \n\n1.\tContribution. I can understand this paper starts from analyzing the application of the absolute pose regression, but the key technique about the distortion of Q-K features in self-attention mechanism comes from previous works [19]. As this is the major contribution of this paper, I am not very sure if it is enough. Although an additional Q-K alignment strategy is proposed as an additional contribution, its improvement over prior strategies such as [19] is not significant (0.02m and 0.17 deg). These improves may come from a suitable learning rate or more proper hyper-parameters balancing different losses.\n\n2.\tResults. Overall, the improvements against the baseline method MSTransformer [17] on indoor (0.02m, 0.64deg) and outdoor (0.09m, 0.44deg) datasets are not significant, which further degrades the contribution of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Distinctive contribution**\n\nA1. We can understand why this might be a concern.\n[19] and our research both address the underlying problem of attention collapse.\nHowever, [19] and ours are **clearly different in terms of the analysis on the causes and the solution for the attention collapse.**\nFirstly, [19] points out that the large gradient norm and increasing spectral norm in the early layers of transformers induce the collapse with training instability and degraded performance.\nHence, they propose a method (σReparam) for regularizing the gradient and spectral norm with pre-layer normalization.\n\nOn the other hand, we identify the distortion of the query-key space and the undertrained positional encoding as the main issues.\nThat is, when the queries and keys are mapped into completely different spaces while only a few keys exist in the query space, the attention collapses and the components in self-attention modules are deactivated.\nNot just measuring attention entropy like [19], we also **define the new index, purity, to statistically demonstrate the predominance of this issue across the APR dataset.**\nFinally, we introduce an auxiliary loss which aligns the query and key regions as well as fixed positional encoding.\nTherefore, the major contribution stands apart from [19], and consideration of the novel contributions it offers would be appreciated.\n\n**Q2. Performance improvement**\n\nA2. We understand that the improvements may not appear significant at first glance.\nHowever, please take into account two hidden aspects regarding the conventional evaluation of median errors in position and orientation in APR.\nFirstly, due to the small numerical values, the errors tend to appear minor. Despite this, **our method reduces the position/orientation error rates by 7%/17% in outdoor settings and by 6%/9% in indoor settings.**\n\nSecondly, median error, being a *median*, does not fully reflect the overall performance of the model.\nTo address this, we compared the performance with the baseline using the recall evaluation metric, which is widely adopted in 2D-3D correspondence-based camera pose estimation.\nAs shown in Table A1 on the global rebuttal page, **our method achieves 3-5%p higher recall across various thresholds and datasets compared to the baseline.**\nParticularly, there are substantial improvements of outliers, which are not evident in the median error metrics.\nBy forcing the model to find helpful cues for the task, it becomes especially effective in leveraging previously challenging edge cases.\nIn conclusion, we would emphasize our method's superiority and the remarkable performance improvements it provides over the baseline.\n\n**Q3. Performance difference between [19] and ours**\n\nA3. First of all, please allow us to clarify that there is a trade-off between position and orientation [6, 12].\nThus, it is hard to see that [19] improves performance on the 7Scenes dataset since the median orientation error is decreased but the position error does not.\nIn addition, we report the comparative analysis between [19] and our QKA loss on Cambridge Landmark dataset in Table A5 of the global rebuttal page.\nAs the scale of the scene increases, it can be observed that our method shows a clear difference in median error compared to [19].\nAs mentioned earlier, median error may not be as visually striking as other evaluation metrics; however, please take into account that such a difference in median error is not marginal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for reviewing.

Responses to the questions can be found under each individual review.

The global rebuttal page includes the relevant figures and tables for your reference.

Meta-review: The reviewers generally agree that the manuscript is well written and is clearly motivated, that the proposed method is simple, has novelty, and is reproducible, and that the results and visualizations are compelling. An extensive rebuttal was submitted, which addressed a majority of the reviewers concerns, and also demonstrated that the proposed approach could benefit a different task (temporal action detection). After this phase, R-K2gT raised their rating to borderline-accept and the other three reviewers maintained their ratings. R-FKQN remained negative (borderline reject) but was not active during any of the discussion phases. Therefore, the AC reviewed the paper in detail, finding the review of R-aJYL to be the most compelling. The paper identifies a weakness in existing approaches, proposes a novel solution, and demonstrates a positive performance impact. Further, the AC found the weaknesses identified by R-FKQN to be insufficient reasons for rejection and adequately addressed by the authors. For example, multi-scene pose regression is an established research area and does not need to be re-motivated by the authors.  Additionally, penalizing the proposed approach for only supporting attention-based methods (i.e., transformers) is not viewed as a valid complaint, especially in the current era. As such, the AC reached an accept decision. Please take the reviewer feedback into account when preparing the camera ready version.
Decision: Accept (poster)