Cross-view Geo-localization with Layer-to-Layer Transformer
Authors: hongji yang
Xiufan Lu
Yingying Zhu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Cross-view Geo-localization

Abstract: In this work, we address the problem of cross-view geo-localization, which estimates the geospatial location of a street view image by matching it with a database of geo-tagged aerial images. The cross-view matching task is extremely challenging due to drastic appearance and geometry differences across views. Unlike existing methods that predominantly fall back on CNN, here we devise a novel layer-to-layer Transformer (L2LTR) that utilizes the properties of self-attention in Transformer to model global dependencies, thus significantly decreasing visual ambiguities in cross-view geo-localization. We also exploit the positional encoding of the Transformer to help the L2LTR understand and correspond geometric configurations between ground and aerial images. Compared to state-of-the-art methods that impose strong assumptions on geometry knowledge, the L2LTR flexibly learns the positional embeddings through the training objective. It hence becomes more practical in many real-world scenarios. Although Transformer is well suited to our task, its vanilla self-attention mechanism independently interacts within image patches in each layer, which overlooks correlations between layers. Instead, this paper proposes a simple yet effective self-cross attention mechanism to improve the quality of learned representations. Self-cross attention models global dependencies between adjacent layers and creates short paths for effective information flow. As a result, the proposed self-cross attention leads to more stable training, improves the generalization ability, and prevents the learned intermediate features from being overly similar. Extensive experiments demonstrate that our L2LTR performs favorably against state-of-the-art methods on standard, fine-grained, and cross-dataset cross-view geo-localization tasks. The code is available online.

Cross-view Geo-localization with Layer-to-Layer Transformer
Hongji Yang"
Xiufan Lu"
Yingying Zhu
College of Computer Science and Software Engineering
Shenzhen University
[yanghongji2224 1uxiufan2019 ema szu.. wad.edd..com cn thuyy@szu.edt...
Abstract
In this work, we address the problem of cross-view geo-localization, which esti- mates the geospatial location of street view image by matching with database of geo-tagged aerial images. The cross-view matching task is extremely chal- lenging due drastic appearance and geometry differences across views. Unlike existing methods that predominantly fall back on CNN, here we devise a novel layer-to-layer Transformer (I.2LTR) that utilizes the properties of self-attention in Transformer to model global dependencies, thus significantly decreasing visual ambiguities in cross-view gco-localization. We also exploit the positional encoding of the Transformer to help the L2LTR understand and correspond geometric config- urations between ground and aerial images. Compared to state-of-the-ar methods that impose strong assumptions on geometry knowledge, the L2LTR flexibly learns the positional embeddings through the training objective. It hence becomes more practical in many real- world scenarios. Although Transformer is well suited our task, its vanilla self-attention mechanism independently interacts within image patches in each layer, which overlooks correlations between layers. Instead, this paper proposes a simple yet effective self-cross attention mechanism to improve the quality of learned representations. Self-cross attention models global depen- dencies between adjacent layers and creates short paths for effective information flow. As result, the proposed self-cross attention leads to more stable training, improves the generalization ability, and prevents the learned intermediate features from being overly similar. Extensive experiments demonstrate that our L.2TRR performs favorably against state-of-the-art methods on standard, fine-grained, and cross-dataset cross-view geo-localization tasks. The code available online
1 Introduction
Estimating the geospatial location given image of paramount importance for robot naviga- tion [II] 3D reconstruction [12), and autonomous driving (5). Recently, cross-view geo-localization. which aims to match query ground images with geo-tagged database aerial/satellite images, has emerged promising proposal address this problem. Despite its appealing application prospect, the cross-view matching task is extremely challenging due to drastic viewpoint changes between ground and acrial images. Thus, it is critical to understand and correspond both image content (appearance and semantics) and spatial layout across views.
Towards the above goal, several recent works incorporate convolutional neural networks (CNNs) with Vea layers [8] capsule networks 20 or attention mechanisms 2 16
learn visually
"Equal contribution "ooresponning author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

discriminative representations. However, the locality assumption of their CNN architectures hinders their performance in complex scenarios, where visual interferences such as obstacles and transient objects (e.g., cars and pedestrians) may exist. Instead, the human visual system utilizes not only local information but also global context to make more accurate predictions when visual signals are ambiguous or incomplete. Another branch of works exploits geometry prior knowledge to reduce ambiguities caused by geometric misalignments. Though promising, these methods either rely heavily predefined orientation prior [ or make restrictive assumption that ground and aerial images are orientation-aligned [6]. Therefore, such a strong assumption limit the applicability of these approaches, which prompts us to seek more flexible approach for encoding position-awee representations.
Motivated by these observations, we introduce Transformer 211, which excels in global contextual reasoning and thus can naturally employed to reduce visual ambiguities in cross-view geo- localization. Besides, the positional encoding of the Transformer enables our network to learn position-dependent representations flexibly. Specifically, our proposed layer-to-layer Transformer (L2LTR) is built upon two independent Vision Transformer (ViT) [4] branches, which split a feature map into several sub-patches while modeling interactions between arbitrary patches. We show in the experiment that due to its context- and position-dependen natures, such a Transformer-based network a well-suited candidate for cross-view geo-localization and shows its superiority compared to the dominant CNN-based counterparts.
We also take deep look at self-attention map, which an integral part of the Transformer and is independently learned in each Transformer block. Nevertheless, such an independent learning strategy overlooks correlations between layers. Specifically, relating features from adjacent layers could improve the representation ability of the network [3]. To explore cross-layer correlations, we replace self-attention with a novel self-cross attention mechanism. Simple yet effective, the proposed self- cross attention learns pairwise similarities between features of adjacent blocks rather than that of the same blocks. Such a cross-block interaction strategy eases the information flow across Transformer blocks, thus leading to more stable network optimization. Moreover, we empirically show that self-cross attention can improve the network's generalization ability and prevent Transformer layers from producing overly similar intermediate features. As result, such mechanism significantly improves the quality of image representation without increasing the model complexity.
The key contributions of this work are as follows.
The L2LTR is the first model using Transformer for cross-view geo-localization to the best of our knowledge. The globally context-aware nature of the L,LTR effectively reduces visual ambiguities in cross-view geo-localization, while the positional encoding endows the L2LTR with the notion of geometry, thus decreasing ambiguities caused by geometry misalignments. Since the position embeddings are learned without imposing strong assumption on the position knowledge, the L.2LTR has wider practical applicability than ofte--of-theaate models.
We propose novel self-cross attention mechanism, which interacts within cross-layer patches to ensure effective information flow across Transformer blocks. This simple yet effective design consistently enhances the representation and the generalization ability of the L2LTR without adding additional computational cost.
Extensive experiments demonstrate that our L2LTR brings consistent and significant perfor- mance improvements for wide range of view matching tasks, including standard, fine-gras -ne-graided, and cross-dataset cross-view geo-localization. The L2LTR exhibits supe- riority in learning visually discriminative and position-aware representations on all these tasks and achieves a new sfate-of of-the-art art performance.
2
Related Work
The key to cross-view geo-localization is to understand and correspond both image content (appear- ance and semantics) and spatial layout across views. To this end, existing cross-view geo-localization methods can be roughly grouped into content-based and geometry-based
Content-based methods focus on learning image representations that are discriminative enough to distinguish between similar-looking images. Leveraging on the success of CNNs, Workman and
2

---

Jacobs [23 first introduce CNNs to the cross-view matching task. Later on, Hu et al. g incorporate two-branch VGG [19] backbone network with NetVlad layers [ to learn viewpoint-ivaanie representations. They also devise a weighted soft-margin triplet loss, which can speed up the network training. Sun et al. (20) apply the powerful ResNet ( as backbone networks. Coupled with capsule layers [5], their proposed GeoCapsNet can model high-level semantics. To steer where to focus in images, the attention mechanism is introduced to the field of cross-view geo-localization. Cai et al. 22 introduce a lightweight attention module that combines spatial and channel attention mechanisms to emphasize visually salient features. They also propose a novel reweighting loss that adaptively allocates weights to triplets according to their difficulties, thus improving the quality of network training. SAFA [6] employs a multi- spatial attention module to aggregate informative and diverse embedding maps. While promising, few of the above methods pay enough attention to the glohal dependencies of cross-view images, which hinders the discriminative ability of their embedded features. Different from existing methods, this work makes the first exploration introduce Transformer (21] to cross-view neo-localization. We demonstrate the importance of considering global dependencies for reducing visual ambiguities.
Geometry-basee methods aim to correspond geometric configurations between ground and aerial images, which helps to reduce ambiguities caused by geometry misalignments. To this end, Liu and Li [9] explicitly inject per pirel orientation information into the network. Nevertheless, this is based on the assumption of accessibility of the ground-trute orientation, which is not always satisfied in practice. Shi et al. [[6] employ a polar transform algorithm to warp satellite images so that acrial images are geometrically aligned with ground images. However, this method is only applicable to the ideal case where the ground images are orientation-alignel panoramas. Even though the dynamic similarity module proposed in [17] overcomes this limitation, the brute-force warping strategy of the polar transform overlooks the depth of the scene content. It results in obvious appearance distortions, hindering performance improvement. Regmi and Shah [14] attempt tackle this problem by synthesizing the corresponding satellite image from ground query using conditional GANs (cGANs), but the synthesized images are always granulated and lack details. In this paper, our L2LTR explicitly encodes learnable positional embeddings into the network without imposing a strong assumption. Unlike the previous works 16 17,, as shown in the experiments, our L2LTR does not impose strong assumption on the position knowledge but flexibly learns relative positional information.
3 Method: L2LTR
This paper proposes a novel layer-to- soyer-to-layer Transformer (L2LTR) architecture with self-cross attention mechanism for cross-view geo-localization. The following sections detail our problem setting, objective, the L2LTR architecture, and our proposed self-cross attention.
3.1 Problem Formulation and Objective
The goal of cross cross-view geo-localization is to localize query ground image by matching it with set of geo-tagged aerial images. We formulate this problem in the same way as prior works 18 16 (7).
Assume we have training set D {(), a1) a1),.., (gN: ax)) containing N cross-view image pairs of ground images g and aerial images a. To simplify the problem, let each ground image gs correspond to only one ground-truth aerial image cross-view image pair (g:, a;), we infer the corresponding image representations as (F), Ff). Then, for the jth exemplar, the weighted soft-margin triplet loss [8] L, which aims to bring matching pairs closer while pushing non-matching pairs far apart, can be defined as follows:
(i E (1, ..., N)) during the training phase. Given
2..,
L = log(1 1 + ((d(F)
(1)
where j E (1,2,..., N) and j + i. a is hyperparameter used to speed up training convergence, and d(---) denotes the L2 distance.
3.2 Transformer for Cross-view Geo-localization
We seek to develop an L.2TRR architecture that explores the global context and the positional information of cross-view images.
3

---

Backbone II
Encoder
Loss
Set featur
x12
Transforeer encodor
Forward
Ground image
Layer Norm
Add
Positional encoding
Descriptor
Triplet Loss
Backbone
Encoder
Catéelinon
Setee Sattre
Multi head Self cross Attention
x12
Transformer encoder
Acrial image
EE
Layer Norm
Layer Norm
Positional encading
Global Descriptor
Xy-1
Xj-2
(b)
Figure (a) Overview of our layer-to-layer Transformer (L2LTR). (b) Illustration of the encoder layer with self-cross attention in the L2LTR. x denotes the input of layer l.
Preliminaries: Vision Transformer. We first describe the Vision Transformer (ViT) 4 as background. Given an image, the ViT first splits into several patches. Then, the VIT receives as input sequence of linear projected patch embeddings x E R NxD, where N is the number of patches, and is the patch embedding size. After prepending a learnable global representation Xgtohol E D whose final hidden state serves an aggregate image descriptor for the cross-view geo-localization task and adding positional embeddings xpos to x, we gain Xg = [Xgtobat; x] + X Xpos and feed it into L-layer Transformer encoder. Each layer consists of a Multihead Self-Attention module (MSA), Feed Forward Networks (FFN), and LayerNorm blocks (LN). Note that the MSA consists of multiple self-attention heads and linear projection block. In order to make clear comparison with our proposed self-cross attention head, we denote the input of layer (1 E {1..... L)) X/-1 and formulate single self-attention head, the core of the vanilla MSA, follows:
Z (N(x-11
Q: = z W7. Kr = W Vi = z W
(3)
QKT V
A = softmar
(4)
where W7, w and WT are linear projection matrices.
Domain-specific Transformer. The drastic domain gap between the ground and aerial images makes it difficult to match cross-view images by embedding them into the same space through a domain-shared network. To suit the cross- view geo-localization task, employ fomainnsppciite Siamese-like architecture with two independent branches of the same structure. Such an architecture learns ground and aerial representations separately, and it can effectively project cross-view images into a shared space when optimized with Eq. [ The network overview is illustrated in Figure | (a). Each branch is a hybrid structure consisting of a ResNet backbone (denoted "Backbone" dashed boxes in Fig. [ (a)) extracting CNN feature map and a Transformer encoder (denoted as "Encoder" dashed boxes) modeling global context from the CNN feature map. The linear projection of patch embedding in the ViT is applied to the CNN feature map by regarding each x feature as patch.
Learnable positional embedding. Incorporating geometric cues 99] [6] helps avoid ambiguities caused by geometric misalignment across views, thus greatly simplifying the cross-view geo- localization. Instead of imposing predefined orientation knowledge on the network. this paper applies an efficient and flexible way to endow the network with the notion of geometry. Specifically, we use learnable ID positional embeddings in the ViT, i.e. Xpos E (N-11xx By adding positional embeddings to the patch embeddings, the transformed features become position-dependent. Furthermore, since we do not impose any assumption on position knowledge but learn it through our learning objective, our L2LTR has wider practical applicability. Experiments show that incorporating

---

the learnable positional embeddings helps capture relative positional information. This allows the network to geo-locate panoramas with unknown orientation and planar ground images with a limited field of view.
3.3 Self-cross Attention
In the vanilla VÝT, the attention map calculated independently in each layer. However, as mentioned before, such an independent learning strategy hinders the model's representation ability. To improve the quality of the learned representations, we propose novel self-cross attention mechanism to interact features between adjacent layers and name the novel network as a layer-to-layer Transformer (L2LTR). Namely, the L2LTR, the attention map of layer / is learned not only based on X but also X!-2 Considering the semantic gap between feature maps caused by different network depths, we do interact with all intermediate features but only the highly correlated adjacent features. Formally, in layer I, self-cross attention can be represented as:
Z: LN (x: 1), 21-1 LN (x:-x;
(5)
Q = 2, W K, = 21-1 W V1 W
(6)
Q:K7
A = softmait
V
DD
Note that, for = we 2 LN In Figure (b), illustrate the structure of self-cross attention-based encoder layer.
How self-cross attention affects feature learning. Compared to self-attention in Eq. 4 our proposed self-cross attention creates short path between ad- jacent layers (highlighted yellow in Figure (b)).
150
Self Aul
Variance of Accuracy
Self-cross Att.
thus allowing information to flow effectively across layers. This shares a similar spirit with ResNet 77]
100
To investigate how this affects our Transformer- based network, we report the variance of recall accu- racy at different training epochs on the CVUSA test set in Figure - Specifically, the variance curves are generated by calculating the accuracy variance over ten training epochs. Thus, a smaller variance indi- cates less fluctuation in recall accuracy and more stable model training. As shown, during the early training stage, the localization performance of self- attention-base based model fluctuates lot, while our L2LTR exhibits more stable performance as train- ing processes. Furthermore, we observe that by interacting cross-layer features, self-cross attention can decrease the representation similarity between layers and enhance the network's 's generalization ability, thus improving the network's representation ability. This is further discussed in the experiment.
50
20
40
60
80
100
Epoch
Figure 2: The variance curves of recall accu- racy in the first 100 training epochs on the CVUSA test set.
4 Experiment
We first introduce three benchmark datasets we used to evaluate our L2LTR, evaluation protocols, and implement details of our network. Then compare L2LTR with state-of-the-art models in Section 4.3 and present ablation studies to illustrate the advantages of the proposed L.2LTR in Section 4.4 Finally, we provide qualitative results in Section 4.5 to demonstrate the effectiveness of the positional embeddings in the L.2TRR.
4.1 Dataset and Evaluation Protocol
Dataset. To verify our model's effectiveness, we conduct extensive experiments on three widely used benchmarks: CVUSA (24) and CVACT [9] (including CVACT _al and (VACT_test), The CVUSA datasct provides 35,532 image pairs for training and 8,884 image pairs for testing. The CVACT
5

---

dataset contains 35,532 pairs for training and 8,884 pairs for validation (denoted as CVACT val). The CVACT also provides 92,802 image pairs with accurate geo-tags for testing (denoted as CVACT test) to support fine-gres fie-grained city-scale geo- gec-localization. For the CVUSA test set and CVACT_ A the correct match
ground image is corresponding aerial image. For the CVACT _eest, retrieved aerial image is considered correct as long as it is within the distance d=5m from the ground-truth location of the ground image. In the experiment, we denote the tasks performed on the CVUSA A test set and the CVACT_ val standard cross-view geo-localization and the tasks performed on the CVACT_test as fine-grained vress-view geo-localization.
Evaluation protocol. In line with 8 19 18 16 171, we evaluate our model by recall accuracy at top K (r@ K for short, K E {1, 5, 10, 1%3). which represents the probability of correct match(es) ranking within the first K results. For the CVUSA test set. r@ 1% indicates the recall accuracy at the top 1 % of the test set, and for the CVACT _al and the CVACT_test r@ 1% indicates the recall accuracy at the top 1 1% of the CVACT_val.
4.2 Implementation Detail
If not specified, the ground and aerial image sizes are set to 128 x 512 and 256 256, respectively. We empirically set model depth
and initialize our L.2LTR with pre-trained parameters on ImageNet [3]. The model is trained using Adam W [10] with a cosine learning rate schedule on 32GB NVIDIA V100 GPU. The learning rate is set le-4, the weight decay chosen 0.03, and the batch size is 32. For the weighted soft-margin triplet loss [8] a is set to 10.
4.3 Comparing L2LTR with State-of-the-art Models
Here we compare our method with
Table 1:
Comparisons with state-of-the-art methods on the CVUSA 24 dataset. For all the compared methods, we cite
several sfate-of-the-are methods on
the CVUSA (24), CVACT_ val [9]
the results from [17] and [[6] if not specified. "PT" indicates
and CVACT test [9 datasets. Un-
whether the model applies (w/ polar transform
16 to aerial
like state-of-the-art methods that predominantly fall back on CNN, our proposed L2LTR makes the first effort to introduce Trans- former to the field of cross-view geo-localization to learn globally context- and position-awar sentations. Below, we verify that our L2LTR exceeds in learning vi- sually discriminative and position- aware representations, thus achiev- ing outstanding performance in var- ious rrews-view cooloocalization tasks. Note that for a fair compar- ison with works [16. that use polar transform [6]. a kind of data pre-process pre-processing algorithm, we apply the same warping strategy aerial images before feeding them into the network (denoted as Polar-L22LTR) when comparing with these works. In this case, ground and warped aerial images are resized to 128 x 512.
images or not (w/o).
r@ 1
r@S (%)
r@ r@10
r@ 1%
PT
Model
(%)
(%)
(%)
Workman et al.
223
34.30
Vo and Hays
22
63.70
Zhai et al.
[4
43,20
repre-
CVM-Net 8
22.47
49.98
63.18
93.62
Liu and
9
40.79
66.82
76.36
96.12
w/o
Zheng et al.
(25
43.91
66.38
74.58
91.78
Regmi and Shah [14]
48.75
81.27
95.
Siam-FCAND
2
98.30
CVFT [8]
61.43
84.69
90.49
99.02
SAFA 16
81.15
94.23
96.85
99.49
L2LTR
91.99
97.68
98.65
99.75
SAFA
[16]
89.84
96.93
98.14
99.64
w/
Shi et al. [17
91.93
97.50
98.54
99.67
Polar-L2LTR
94.05
98.27
98.99
99.67
Standard cross-view geo-localization. We first evaluate our L2LTR on standard cross-view geo- localization. Tables [ and 22 show experimental results on the CVUSA and CVACT_ val datasets, respectively. From the results, we could conclude that our L2LTR significantly surpasses the com- peting approaches in learning visually discriminative representations and corresponding geometric configurations across views. particular, without applying the polar transform, our L2LTR achieves r@ of 83.14% on the CVACT val dataset compared to 78.28% obtained by the second- method, while on the CVUSA dataset, the L.LTR surpasses the second-best method by significant margin of 10.84 points at r@ 1. Moreover, when applying the polar transform, which geometrically aligns cross-view images, our L.2TRR outperforms the competing methods, gaining 84. 89% and 94. 94.05% on the CVACT val and CVUSA, respectively. The results indicate that the L2LTR is capable of

---

Table 2: Comparisons with state-of- models on the CVACT val (standard cross-view geo-
localization) and CVACT_test (fine-grained geo-loca (eo-localization) datasets.
CVACT_W val
CVACT
Jest
PT
Model
Code Length
r@ 1
r@5
10
r@ 1%
r@
r@5
r@
r@
(%)
(%)
(%)
(%)
(%)
(%)
(%)
(%)
CVM-Net g
4096
20.15
45.00
56,87
87,57
5,41
14.79
25,63
54,53
Liu and Li 9
1536
46.96
68.28
75,48
92.01
19.21
35,97
43,30
60.69
w/o
CVFT 08
4096
61.05
81.33
86.52
95.93
26.12
45.33
53,80
71.69
SAFA 16
4096
78.28
91.60
93.79
98.15
L2LTR
768
83.14
93.84
95,51
98,40
58.33
84.23
88.60
95.83
SAFA [16
4096
81.03
92.80
94.84
98.17
55.50
,9
85.08
94,49
w/
Shi al. [17
4096
82,49
92.44
93.99
97.32
35.63
60.07
69.10
84.75
Polar-I.2LTR
768
84.89
94.59
95.96
98.37
60.72
85.85
89.88
96.12
capturing visually discriminative features by modeling global context. Furthermore, could also find that removing the polar transform algorithm leads to significant performance degradation in SAFA (-4.21% the CVACT_ _aa and -8.69% on the CVUSA) while less degradation is noted in our L2LTR (- 1.77% on the CVACT_val and - -..06% on the CVUSA), Namely, the L2LTR does not have rely excessively
the polar transform establish cross-view geometric correspondence, which could save considerable image pre-processing time on large-scale datasets. This is because adding the positional embeddings the patch embeddings enables the L2LTR learn position- aware representations and correspond geometric configurations between ground and aerial images. Additional qualitative evidence for this provided in Section 4.5
Fine-grained cross-view geo-localization. To evaluate the representation ability of our model, we verify the L2LTR the fine-graing cross-view geo-localization task. Specifically, compare the L2LTR with state-of-ther methods on the challenging large-scale CVACT _est dataset, which fully GPS-tagged for accurate localization. Table[]shows the experimental results. Our L2LTR performs consistently better than all the competitors, achieving 58.33% and 60.76% at r@ I without and with the polar transform, respectively. These results further demonstrate that L2LTR has strong representation capability.
view geo-localization. In the context of cross-view geo- localization, the transferring performance determines whether a model could be practically usable for real-life scenarios, where a query image may be dramatically different from the training ground images. As the CVACT and CVUSA datasets are collected from two different countries, they have distinetly different scene styles. Based on this observation, to verify the transferring performance of our model, we train the L2LTR on the CVUSA dataset and test it on the CVACT_va val (denoted as CVUSA-CVVCCT could find that our L,LTR outperforms the second-best model r@ by a large margin of 13.89 points on the -CACCTT task while achieving 33.00% at r@ 1 compared to 21.45% gained by the second-bes model on the CVACT-CVUSA task. The transferring results demonstrate the outstanding generalization ability and practical applicability of our L2LTR.
Table 3: Cross-dataset cross-view geo-localization. The results are gained by retraining and evaluating the compared models using the released codes provided by their authors.
r@
r@:
r@ 10
r@ 1%
Model
Task
(%)
(%)
(%)
(%)
SAFA [6
30.40
52.93
62.29
85.82
Shi et al. 17
CVUSA-CVACT
33.66
52.17
59.74
79.67
Polar-L2LTR
47.55
70.58
77.39
91.39
SAFA [6]
21.45
36.55
43.79
69.83
Shi er al. 17
CVACT-CVUSA
18.47
34.46
42.28
69.01
Polar-L2LTR
33,00
51.87
60.63
84.79
and vice versa. Results are reported in Table 3 We
Localizing with unknown orientation and limited FoV. As verified in Section 4.5 our L2LTR can learn relative positional information without incorporating predefined position knowledge. This makes the network generalize well to panoramas with unknown orientation and planar ground images with a limited field of view (FoV). To evaluate this point, we test the performance of our L2LTR on orientation-unknoww and Fo ground images. Specifically, we follow the evaluation procedure in 17., randomly shifting and cropping ground panoramas (with FoV of 360") along the azimuthal direction on the CVUSA dataset. For a fair comparison, we set the cropped images with Fo of 180", 90" and 70 Then, train and test the performance ground images with the
7

---

Table 4: Comparisons with state-of-the-ar models for localizing ground images with unknown orientation and limited field of view on the CVUSA.
Trai Test FoV
180
90"
r69:0
Model
(%6
(%)
(36
(5)
(%)
(5)
(%)
(%)
(%)
(%)
(%)
(%)
CVM-Net 8
7.38
22.51
52.63
75.38
2.76
10.11
16.74
55.49
2.62
9.30
15.06
21.77
CVFT [8
8.10
24.25
34.47
75.15
4.80
14.84
23.18
61.23
3.79
12.44
19.33
55.56
Shi et al. 17
48.53
68.47
75.63
93.02
16.19
31.44
39.85
71.13
8.78
19.90
27.30
61.20
L,21TR
56.69
80.86
87.75
98.01
26.92
50.49
60.41
86.88
13.95
33.07
43.86
77.65
same FoV and report the results in Table 4 The results show that our L2LTR, benefiting from the learnable relative positional knowledge, consistently surpasses the competing methods over three experimental settings.
Comparison in terms of code length. To further illustrate the advantage of our method, we compare L2LTR with state-of-the-ar methods terms of image descriptor dimension (also called code length) in T We observe that our L2LTR has extremely short code length of 768, which is five times shorter than that of the SAFA [6]. CVFT [18]. and CVM-Net (8 A shorter code length not only implies the effective information encoding capability of the L2LTR but also means that the L2LTR provides an alternative with less storage space, lower computational complexity, and shorter running time to cross-view geo-localization.
4.4 Ablation Study
To investigate the effectiveness of the positional embeddings and self-cross attention mechanism, we conduct ablation studies by considering three scenarios: 1) where the positional embeddings are removed, 2) where the polar transform is removed and, 3) where self-attention replaces self-cross attention.
Positional encoding. We first analyze the importance of the positional embed- dings and report the ablation studies the
Table 5: Ablation studies L2LTR.
CVUSA datasct in Table [5] From the re- sults, tions. First, the positional encoding endows the network with the concept of position and yields consistent improvements. In particu- lar, adding positional embeddings improves the r@1 performance of the L,LTR from 89.0 99.04% 91.99% and brings 3.15 points improvement at r to the Polar-L2LTR, Second, we could also find that the positional embeddings complementary the polar transform (16) [7]. Specifically, combining the polar transform with our L2LTR improves the r@ I accuracy from 91.99% to 94.05% (+2.06%), while adding the positional embeddings Polar-L2LTR boosts the r@ performance from 90.90% 94.05%
Model
r@
r@5
r@ 10 (%)
r@ 1%
(%)
(%)
(%)
make the following observa-
Polar-L2LTR
94.05
98.27
98.99
99.67
w/o self-cross att.
93.26
97.91
98.75 78.78
99.68
w/o positional emb.
90.90
97.48
98.40
99.62
L2LTR
91.99
97.68
98.65
99.75
w/o positional emb.
89.04
96.88
98.44
99.61
(+3.15%).
Self-cross attention. In Table 5 we ablate self- cross attention mechanism by replacing it with self- attention on the CVUSA dataset. We can ob- serve from Table [ that, without imposing an in- crease model complex- ity. self-cross attention mechanism improves r@ performance from 93.266 23.26% to 94.05%, which mani- fests the effectiveness of
Table 6: Few-shot cross-view geo-localization on the CVUSA 24 We describe number training pairs of each subset its proportion (Prop.) to the original CVUSA dataset.
Training Pairs
r@] 1
r@5
r@ 10
r@ 1%
Prop.
Model
(%)
(%)
(%)
(%)
Polar-L2IITR
76.01
90.67
94.01
98.85
7,106
20%
self-cross att.
75.37
90.42
92.85
98.66
Polar-I.2I.T
86.06
95.80
97.16
99.38
14.212
40%
w/n self-cross att.
85.54
95.14
97.07
99,42
Polar-L2LTR
90.30
96.96
98.26
99.67
21,319
60%
w/o self att.
88.74
96.60
98.09
99.66
Polar-L2LTR
94.05
98.27
98.99
99.67
35,532
100%
w/o self-cross att.
93.26
97.91
98.78
99.68
8

---

(a) The ground PE of the (b) The acrial PE of the (c) The ground PE of the (d) The acrial PE of the L2LTR
L2LTR
Polar-L2LTR
Polar-L2LTR
Figure 3: Cosine similarity between the learnable positional embeddings (PE). Yellow indicates the two positional embeddings are closer. Better viewed in color and with zoom-in.
self-cross attention. Morcover, in Table ablate self-cross attention the few-shot cross-view geo-localization task. The few-shot task aims to learn a model that can achieve generalization from only a small number of training examples [6. To support this task, randomly select certain percentage (20%/40%/60%% of samples from the CVUSA dataset to generate three subsets. The size of each subset and its corresponding proportion to the CVUSA dataset is illustrated in Table 6 Results show that replacing self-cross attention with self-attention consistently harms the network performance on the few-shot task. This indicates that self-cross attention not only improves network performance but also enhances its generalization ability. Furthermore, we also investigate how self-cross attention affects feature learning.
In Figure 4 we compare the final representation with the output of each intermediate layer by mea- suring their cosine similarity. As observed, replac- ing self-attention with our proposed self-cross at- tention significantly and consistently decreases the representation similarity between layers. The re- sult implies that our self-cross attention can prevent the learned representations of Transformer layers from being overly similar, making the network more effective capturing rich representations.
0.8
Consine Similarity
0.6
0.4
0.2
Self Att.
Self-cross Att.
4.5 Qualitative Analysis
0
3
Block Index
We conduct a detailed qualitative analysis on the learnable positional embeddings Xpos + to investigate whether they encode and correspond geometric configurations across views and which positional information they can learn. To this end, we calculate the cosine similarity between two arbitrary positional embeddings of the L2LTR and acquire a distance matrix D = N (orm(xxps))MM) J.Nomm(xppp. (Xpo)) of shape (N + 1) x (N + +1). where Norm represents L2 normalization. Then, we visualize the distance matrix in Figure where yellow indicates the two positional embeddings are closer each other. From Figures 3(a) 3(b]
Figure 4: Cross-layer similarity between the last layer and previous layers.
we could find that each positional embedding is close to its neighbors with small location offsets. This implies that incorporating the learnable positional embeddings captures relative positional information. Additionally, we could observe that the visualization maps of the L2LTR are distinctly different across views in Figures 3(a)and (bb while the visualization maps the Polar-L.2LTR look similar to each other in Figures 3(c) and 3()) Such similar results of the Polar-L.2ITT are reasonable
since the polar transform geometrically aligns cross-view images. This result further confirms that the positional encoding could capture cross-view geometric configurations.

---

Conclusion and Future Work
This paper proposes a novel L2LTR architecture capable of learning globally context- and position- aware representations. We also propose novel self-cross attention to facilitate information flow across layers. Extensive experiments demonstrate that the L2LTR outperforms state-of-the-art methods in standard, fine-grained, and cross-dataset cross-view geo-localization tasks. In addition, we also conduct ablation studies and qualitative analyses to verify the effectiveness of the learnable positional embeddings and self-cross attention. One main limitation of the L.2TTR is its large demand for GPU memory. Moreover, the L2LTR is built on top of the pre-trained Transformer, which requires a large amount of data for training. We aim develop data-efficient Transformer-bias model with less memory consumption for cross-view geo-localization for future work.
Societal Impact
This paper addresses the problem of image-based geo-localization, which benefits a wide range of applications. On the one hand, the image-based geo-localization can serve as an alternative to GPS- based localization, especially when GPS signals are jammed, blocked by buildings, or not accurate enough for specific applications. Working in conjunction with GPS- positioning systems. our method can greatly improve the stability and safety several downstream applications, such as autonomous driving, robot navigation, and pedestrian navigation. On the other hand, image-han based geo-taggers can be used track the location intelligence of single image without GPS tags. This can be beneficial for applications such digital forensics, event detection, and scene annotation. Nevertheless, negligent or malicious use of our approach could also mislead positioning systems or expose people to privacy violations. Overall, our method has both positive and negative impacts. As long as the failure cases are handled properly, and method is not used for unethical tasks, our approach mostly leads to positive impacts.
Acknowledgments
This work was supported by: (i) National Natural Science Foundation China (Grant No. 62072318); (īi) Natural Science Foundation of Guangdong Province of China (Grant No. 0221A1155010111 (iii) Fundamental Research Project in the Science and Technology Plan of Shenzhen (Grant No.
JCYI200198081172007500
References
[1] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivie. Netvlad: Con architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5297-5307, 2016.
[2] Sudong Cai, Yulan Guo, Salman Khan, Jiwei Hu, and Gongjian Wen. Ground-to-acrial image geo-localization with hard exemplar reweighting triplet loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8391- 148000 2019.
[3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Icce, 2009.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[5] Christian Häne, Lionel Heng, Gim Hee Lee, Friedrich Fraundorfer, Paul Furgale, Torsten Sattler, and Mare Pollefeys. 3d visual perception for self-driving cars using multi-camera system: Calibration, mapping, localization, and obstacle detection. Image and Vision Computing, 68:14-27, 2017.
10

---

[6] Jun He, Richang Hong, Xueliang Liu, Mingliang Xu, Zheng-Jun Zha, and Meng Wang. Memory- augmented relation network for few-shot learning. In Proceedings of the 28th ACM International Conference Multimedia, pages 1236-1244, 2020.
[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[8] Sixing Hu, Mengdan Feng, Rang MH Nguyen, and Gim Hee Lee. Cvm-net: Cross-view matching network for image-based ground-to-ard -eariall geo-localization. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 7258-7267, 2018.
[9] Liu Liu and Hongdong Li. Lending orientation to neural networks for cross-view geo- localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5624-5633, 2019.
[10] Iya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
[11] Colin McManus, Winston Churchill, Will Maddern, Alexander D Stewart, and Paul Newman Shady dealings: Robust, long-ten termttrmm visual localisation using illumination invariance. In 2014 IEEE international conference on robotics and automation (ICRA), pages 901-906. IEEE, 2014.
[12] Sven Middelberg, Torsten Sattler, Ole Untzelmann, and Leif Kobbelt. Scalable 6-dof localization on mobile devices. European conference on computer vision, pages 268-283. Springer, 2014.
[13] Youwei Pang. Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu. Multi-scale interactive network for salient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9413-9422, 2020.
[14] Krishna Regmi and Mubarak Shah. Bridging the domain gap for ground-to-aerial image matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 470-479, 2019.
[15] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Proceedings of the st International Conference on Neural Information Processing Systems, pages 3859-3869, 2017.
[16] Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li. Spatial-aware feature aggregation for im- age based cross-view geo-localization. Advances in Neural Information Processing Systems, 32: 10:00000-0000, 2019.
[17] Yujiao Shi, Xin Yu, Dylan Campbell, and Hongdong Li. Where looking at? joint location and orientation estimation by cross-view matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4064 4072, 2020.
[18] Yujiao Shi, Xin Yu, Liu Liu, Tong Zhang, and Hongdong Li. Optimal feature transport for cross- view image geo-localization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11990-11997, 2020.
[19] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409:: 1556, 2014.
[20] Bin Sun, Chen Chen, Yingying Zhu, and Jianmin Jiang. Geocapsnet: Ground to aerial view image geo-localization using capsule network. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 742-747. IEFE, 2019.
[21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: T 170.003762. 2017.
[22] Nam and James Hays. Localizing and orienting street views using overhead imagery. In European conference on computer vision, pages 494-509, Springer, 2016.

---

[23] Scott Workman, Richard Souvenir, and Nathan Jacobs. Wide-are area image geolocalization with aerial reference imagery. In Proceedings of the IEEE International Conference on Computer Vision, pages 3961-3969, 2015.
[24] Menghua Zhai, Zachary Bessinger, Scott Workman, and Nathan Jacobs. Predicting ground-level scene layout from aerial imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 867-875, 2017.
[25] Zhedong Zheng, Yunchao Wei, and Yi Yang. University-1652: A multi-view multi-source benchmark for drone-based geo-localization. In Proceedings of the 28th ACM international conference on Multimedia, pages 1395-1403, 2020.
12


Reviews:
[{"review_id": "IlU142syFj", "reviewer": "Reviewer_kVJu", "summary": "The paper tackles the cross-view geo-localization problem, which seeks to match a given ground-level image to aerial images that cover the same area, selected from a fixed database.\n\nThe paper is the first to apply transformers to the problem, and reports state-of-the-art results. Notably, the transformer’s attention mechanism largely frees them from having to warp the aerial image to match the ground-level image’s panoramic projection, a procedure that can go awry in the face of significant occlusion in the ground-level panorama, and/or errors in the ground pano’s orientation data. The resulting accuracy improves the state of the art by a significant margin.\n\nThe paper also makes a few adjustments to the ViT transformer, performing ablation studies on some of them.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\nThe paper is the first to apply the transformer attention architecture to this problem. In addition, it modifies the ViT (Vision Transformer) [4] in a few ways: it replaces image patches with single-pixel columns through the feature maps, which strikes me as an improvement in elegance. It also computes the keys of layer i from the values of layer i-1. Finally, it introduces a ResNet-like pass-through connection through the transformer stack. The paper presents ablation studies of the effects of the first two changes above.\n\n# Quality\nThe submission is technically sound, and analyses presented show good evidence of its accuracy advantage over its predecessors. The ablation studies are useful and well presented. \n\nThere are some claims that are unsupported by evidence. Notably, the explanation in lines 180-198, of how self-cross attention affects feature learning, seems weak. \n\nLines 181-185 mention an architectural similarity to ResNet. In the paragraph below labeled “Clarity”, I talk about how the wiring of Fig 1b is messy. As it stands, the figure obscures any parallel with ResNet. Once fixed, this parallel should be more visually obvious. \n\nIn lines 189-191: “As shown, during the early training stage, the localization performance of the self-attention-based model fluctuates a lot.” It isn’t obvious that the early stage fluctuations of the red curve are significantly worse than those of the black curve. Could you quantify this, e.g. by computing a variance envelope over a moving time window? In any case, the final performance gap between the red and black curves could be attributable to many many things, and it’s not at all obvious why the early stability advantage of the black training curve (assuming there is one)  should be singled out as an explanation. Please remove the statement “As a result, the stable training of the EgoTR could improve the network performance.” Without direct evidence, this is purely speculative. In any case, even if the early stability did lead to better final performance, this only replaces one question (“what causes the better performance”) with a speculative statement (“stable training causes better performance”) tacked onto another question (“what causes the stability?”). That question is just as open as the original question.\n\n“Compared to the self-attention in Eq. 2, our proposed self-cross attention creates short path between adjacent layers,”\n\n# Clarity \nThere are a number of places where the clarity could be improved:\n\n## Abstract\n\n“Evolving”:\nI found the paper’s use of the term “Evolving” to be confusing, in the sense that it led me astray from what is actually being described. In a field where evolution mechanisms are closely studied as a part of the learning toolbox, I would stay away from using “evolve” to describe anything that doesn’t involve Darwinian (or even Lamarckian!) improvement. Even when taking the more relaxed view that “evolve” can refer to any sequential improvement process, it seems misapplied to the EvoTR. While the keys of layer i are a function of the features of layer i-1, It’s hard to say that the keys are “improved” from one layer to the next. Each set of keys are specific to that layer’s features, and comparing them across layers would be comparing apples to oranges. In that case, what is being “evolved”, i.e. sequentially improved over iterations? \n\nI recommend renaming EvoTR to replace “evolve” with something closer to what is being described, for example, L2LT (Layer-to-layer Transformer).\n\n## Section 3.2\nThe wiring in Fig. 1b is messy. At the bottom, x_{i-1} goes into Layer Norm, the output of which should be z_l according to Eq 3. This z_l is unnecessarily merged with the wire labeled x_{l-2} (did you mean for that to be z_{l-1}?). These wires should be kept separate: the z_l wire should go to the Q and V blocks, while the z_{l-1} wire should go to the K block. Most importantly, under the present figure it’s unclear where the x_{i-1} output at the top of the figure comes from. It can’t be the output of the Layer Norm; that’s z_l. It’s not the x_{l-2} input at the bottom; the index is different. Please fix the labels if necessary, and when two wires merge, make sure they only do so by going into a labeled block (e.g. “Add”).\n\nIn “Preliminaries: Vision Transformer”, please explain what “class” means, in the context of the cross-view geo-localization problem. Most readers will be familiar with “class” as used in image classification, so it is unusual to see “class” be part of the input data, as opposed to being a target output.\n\nAs you describe the ViT here, please be explicit about which parts correspond to the different dashed boxes in Fig 1a (even understanding that Fig 1a contains a modified ViT, rather than the vanilla ViT being described here).\n\nIn “Learnable positional embedding”, please describe why the positional embedding is added to the feature maps, rather than concatenated as additional channels. It’s a strikingly surprising design choice, at least to me.\n\nPlease separate Equations 2 and 3 into separate numbered equations. If you must keep the equations for Q_l, K_l, and V_l on a single line, please add more whitespace after the commas, to keep them visually separate.\n\n## Section 4.1\nPlease provide more details on the difference between CVACT vs CVACT_test. It sounds like the former provides just pairs of ground images with their corresponding aerial images. Is this correct? CVACT sounds the same, except that both images now have “geo-tags”, which I assume are like GPS coordinates? Is this correct? If so, How is this extra information used, during inference, training, and evaluation? Also, why is the test set (CVACT_test) ~3x the size of the CVACT training set?\n\n## Section 4.2\nConsider moving section 4.2 (“implementation detail”) to the beginning of section 4 (as 4.1).\n\nPlease provide more detail on the pretraining. In particular:\n* Did you train a single encoder to classify ImageNet images, then transfer those same trained weights to both the aerial and ground encoder networks? \n* Did you augment ImageNet images with positional information? \n* How does the network do without pre-training?\n\n## Page 8\nMost of the page makes reference to Table 4. Please try to move that figure to this page.\n\n## Section 4.5\nPlease go into more detail on how to interpret the distance matrices of figure 4. In distance image D, is D[i, j] the dot product of the positional embeddings of pixel V[i] and pixel V[j], where V is the flattened version of the ground/aerial image? Where does the checkerboard pattern come from? Without having this nailed down, it’s hard to evaluate the claims being made in this section, especially the one in lines 375-378. Lines 379-382 need an image illustrating the effect they describe, as it’s hard to visualize.\n\n## Nit\nThroughout the paper, there are misused articles, mostly in the form of unnecessary or missing “the'''s. Before submission, please run the paper by a colleague who speaks English natively, or if none are available, through a grammar checker (e.g. Grammarly).\n\n# Significance\nThe results are significant. The architecture presented is an improvement in accuracy over competing methods. They largely do away with the need to preprocess the aerial image through a panoramic projection, a potentially brittle step that we are well rid of. The claim is that the transformer’s attention mechanism handles the correspondence problem instead. If this is true, this work may also open the door to future work in geolocating planar ground-level images, which are far more common than ground panoramas. Unfortunately, this is not studied or mentioned in this paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In section 5, the authors highlight that the main limitation of EgoTR is its large demand on GPU memory. Please quantify this demand and compare it to that of the closest competitors.\n\nLine 391-392 says “Moreover, the EgoTR is built on top of the pretrained Transformer, which requires a large amount of data for training.” Is it the pre-training or the fine-tuning that takes a lot of data? Please quantify the data demands and compare it to that of the closest competitors, for example by comparing the number of epochs it takes to converge.\n\nCan you foresee any limitations that would impede success when using this method to match planar aerial images to planar ground images?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {"ethical_concerns": "In the checklist, item 1c (“did you discuss any potential negative societal impacts of your work?”), the answer provided is [N/A]. \n\nI can easily imagine one negative societal impact: people who have taken 360-degree photos (e.g. GoPro Max, Insta360 One X2) with geo-tagging deliberately turned off are now at risk of being geo-tagged. As the field matures, conventional planar photos may also be geotagged by the pixels alone.\n\nOn the positive end, such geotaggers could help Bellingcat-style digital forensics."}}, {"review_id": "HthqCO-P3mx", "reviewer": "Reviewer_Bnto", "summary": "a) This paper brings transformer to solve the cross-view geo-localization problem;\nb) The main contributions are two folds: one using transformer and the other using a modified attention scheme;\nc) Results are good and outperform previous method.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n\na) The paper is well-written;\n\nb) Using transformer for cross-view geo-localization is novel;\n\nc) Though modifying the attention equation is engineered, it is effective for this task;\n\nd) Experimental results are good.\n\n\nWeak points:\n\na) I would say the claim of the superiority of learned position embeddings is too strong and expect the following experiments.\n\nRandomly orientate the ground-view images and compare the proposed method with respect to [17].\n\nWithout this experiment, the proposed method can also benefit from the regularized configuration of aligned cross-view images.\n\n\nb)  Self-cross attention.\n\nx_{l-1} and x_{l-2} is used  in equation (3), can we use x_{l-3} to replace x_{l-2}?\n\nor, jointly using x_{l-1}, x_{l-2} and x_{l-3}?\n\nx_{l-3} can be used to build to Q_l.\n\n\nc) Time and model complexity comparisons.\n\nThough the code length [line 300] is short for the proposed method, as the transformer is time-consuming, I would expect detailed time and model complexity comparisons with respect to previous methods.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "DZBP99r3iB", "reviewer": "Reviewer_9M5m", "summary": "This paper proposes to use transformer for cross-view image geo-localization. It adopts a cross-layer attention block to further improve the performance. Experiments on CVUSA and CVACT show improvement over existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThe idea is simple and easy to follow.\n2.\tThe tables are clear.\n3.\tThe performance is higher than existing papers.\n\nWeaknesses:\n1.\tTransformer has been adopted for lots of NLP and vision tasks, and it is no longer novel in this field. Although the authors made a modification on the transformer, i.e. cross-layer, it does not bring much insight in aspect of machine learning. Besides, in ablation study (table4 and 5), the self-cross attention brings limited improvement (<1%). I don’t think this should be considered as significant improvement. It seems that the main improvements over other methods come from using a naïve transformer instead of adding the proposed modification. \n2.\tThis work only focuses on a niche task, which is more suitable for CV conference like CVPR rather than machine learning conference. The audience should be more interested in techniques that can work for general tasks, like general image retrieval.\n3.\tThe proposed method uses AdamW with cosine lr for training, while comparing methods only use adam with fixed lr. Directly comparing with their numbers in paper is unfair. It would be better to reproduce their results using the same setting, since most of the recent methods have their code released.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion about societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1ZfayI_gzi", "reviewer": "Reviewer_JyiQ", "summary": "This paper proposes EgoTR for cross-view geo-localization with evolving transformer. Specifically, it utilizes the properties of self-attention in transformers to model global dependencies, which reduces visual ambiguities in cross-view geo-localization. The positional encoding is also used in the proposed EgoTR to understand and correspond geometric configurations between ground and aerial images. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- This paper is well-written and easy to follow.\n\n- The paper claims that the use of positional encoder removes the need for strong assumptions on geometry knowledge required in some existing cross-view geo-localization methods. It is however not very obvious from the paper how this is achieved over the cross-view setting. It seems like the positional encoding mentioned in Line 160-171 is based on the image patches in the respective domains. Thus, shouldn't the positional knowledge be just within the image in the same domain, i.e. self-attention? How would the Siamese network architecture and the Triplet loss help in enhancing the cross-view geometry knowledge via the positional encoder? \n\n- The individual components of the proposed framework is not new. Transformer is from ViT [4], and the loss function is from [8]. The only new thing is the attention across different layers in the network proposed in Section 3.3. Therefore, this paper can be seen as another additional paper that introduces transformer into a well-established task / framework. \n\n- What is the explanation on self-attention across different layers can help cross-view localization? Similar to the positional encoding, it is also done within the same view. The only interaction across the two views is the final feature embeddings and the triplet loss. \n\n- Using transformers mean that the memory consumption might increase a lot compared to other existing methods. \n\n- The results on the two benchmark datasets show good performances on the proposed method. This seems to empirically suggest that the method that adopted transformer is effective on the task of cross-view geo-localization. \n\n- The ablation studies in Section 4.4 also seem to empirically support that claim that the positional encoder and self-cross attention across different layers are effective. However, as mentioned earlier, a more concrete explanation on why positional encoder and self-attention across layers can help cross-view geo-localization is missing. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed in conclusion, but not societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "none.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a siamese convolutional + vision transformer architecture for cross-view geolocalisation, from ground view equirectangular panorama images to satellite views (cartesian or polar coordinates). The transformer has a cross-attention mechanism between consecutive layers (I was confused by the word “evolving”, since no genetic algorithms were involved). The method is thoroughly evaluated on a US cross-modal localisation dataset where it achieves state-of-the-art results.

Reviewer kVJu praised the idea and significance of results but criticised some speculative or unsubstantiated claims in the paper, as well as the clarity, including the usage of term “evolution”; the authors responded with a long list of modifications and additional results. Reviewer JyiQ was uncertain why positional encoding and cross-layer self-attention specifically helped with the localisation task (the authors’ response was only empiriical but did not provide intuition about why it worked). Reviewer Bnto had questions about ground view image orientations, which were answered by the authors through additional experiments. Reviewer 9M5m made the usual comment about novelty (which I will dismiss) and narrowness of scope (again, that’s an easy critique to make), as well as a claim about learning rate (addressed by the authors).

All scores but for reviewer 9M5m (4) were positive (6, 6, 7). Reviewer 9M5m said they would update their score to 5 (which makes the paper average equal to 6). I have dismissed claims of “novelty” and “narrowness of scope” and decided to accept the paper.

Decision: Accept (Poster)