Cross-view Geo-localization with Layer-to-Layer Transformer

1 Introduction . Estimating the geospatial location of a given image is of paramount importance for robot navigation [ 11 ] , 3D reconstruction [ 12 ] , and autonomous driving [ 5 ] . Recently , cross-view geo-localization , which aims to match query ground images with geo-tagged database aerial/satellite images , has emerged as a promising proposal to address this problem . Despite its appealing application prospect , the cross-view matching task is extremely challenging due to drastic viewpoint changes between ground and aerial images . Thus , it is critical to understand and correspond both image content ( appearance and semantics ) and spatial layout across views . Towards the above goal , several recent works incorporate convolutional neural networks ( CNNs ) with NetVlad layers [ 8 ] , capsule networks [ 20 ] or attention mechanisms [ 2 , 16 ] to learn visually ∗Equal contribution †Corresponding author 3https : //github.com/yanghongji2007/cross_view_localization_L2LTR 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . discriminative representations . However , the locality assumption of their CNN architectures hinders their performance in complex scenarios , where visual interferences such as obstacles and transient objects ( e.g. , cars and pedestrians ) may exist . Instead , the human visual system utilizes not only local information but also global context to make more accurate predictions when visual signals are ambiguous or incomplete . Another branch of works exploits geometry prior knowledge to reduce ambiguities caused by geometric misalignments . Though promising , these methods either rely heavily on predefined orientation prior [ 9 ] or make a restrictive assumption that ground and aerial images are orientation-aligned [ 16 ] . Therefore , such a strong assumption limit the applicability of these approaches , which prompts us to seek a more flexible approach for encoding position-aware representations . Motivated by these observations , we introduce Transformer [ 21 ] , which excels in global contextual reasoning and thus can be naturally employed to reduce visual ambiguities in cross-view geolocalization . Besides , the positional encoding of the Transformer enables our network to learn position-dependent representations flexibly . Specifically , our proposed layer-to-layer Transformer ( L2LTR ) is built upon two independent Vision Transformer ( ViT ) [ 4 ] branches , which split a feature map into several sub-patches while modeling interactions between arbitrary patches . We show in the experiment that due to its context- and position-dependent natures , such a Transformer-based network is a well-suited candidate for cross-view geo-localization and shows its superiority compared to the dominant CNN-based counterparts . We also take a deep look at self-attention map , which is an integral part of the Transformer and is independently learned in each Transformer block . Nevertheless , such an independent learning strategy overlooks correlations between layers . Specifically , relating features from adjacent layers could improve the representation ability of the network [ 13 ] . To explore cross-layer correlations , we replace self-attention with a novel self-cross attention mechanism . Simple yet effective , the proposed selfcross attention learns pairwise similarities between features of adjacent blocks rather than that of the same blocks . Such a cross-block interaction strategy eases the information flow across Transformer blocks , thus leading to more stable network optimization . Moreover , we empirically show that self-cross attention can improve the network ’ s generalization ability and prevent Transformer layers from producing overly similar intermediate features . As a result , such a mechanism significantly improves the quality of image representation without increasing the model complexity . The key contributions of this work are as follows . • The L2LTR is the first model using a Transformer for cross-view geo-localization to the best of our knowledge . The globally context-aware nature of the L2LTR effectively reduces visual ambiguities in cross-view geo-localization , while the positional encoding endows the L2LTR with the notion of geometry , thus decreasing ambiguities caused by geometry misalignments . Since the position embeddings are learned without imposing a strong assumption on the position knowledge , the L2LTR has wider practical applicability than state-of-the-art models . • We propose a novel self-cross attention mechanism , which interacts within cross-layer patches to ensure effective information flow across Transformer blocks . This simple yet effective design consistently enhances the representation and the generalization ability of the L2LTR without adding additional computational cost . • Extensive experiments demonstrate that our L2LTR brings consistent and significant performance improvements for a wide range of cross-view matching tasks , including standard , fine-grained , and cross-dataset cross-view geo-localization . The L2LTR exhibits its superiority in learning visually discriminative and position-aware representations on all these tasks and achieves a new state-of-the-art performance . 2 Related Work . The key to cross-view geo-localization is to understand and correspond both image content ( appearance and semantics ) and spatial layout across views . To this end , existing cross-view geo-localization methods can be roughly grouped into content-based and geometry-based . Content-based methods focus on learning image representations that are discriminative enough to distinguish between similar-looking images . Leveraging on the success of CNNs , Workman and Jacobs [ 23 ] first introduce CNNs to the cross-view matching task . Later on , Hu et al . [ 8 ] incorporate a two-branch VGG [ 19 ] backbone network with NetVlad layers [ 1 ] to learn viewpoint-invariant representations . They also devise a weighted soft-margin triplet loss , which can speed up the network training . Sun et al . [ 20 ] apply the powerful ResNet [ 7 ] as backbone networks . Coupled with capsule layers [ 15 ] , their proposed GeoCapsNet can model high-level semantics . To steer where to focus in images , the attention mechanism is introduced to the field of cross-view geo-localization . Cai et al . [ 2 ] introduce a lightweight attention module that combines spatial and channel attention mechanisms to emphasize visually salient features . They also propose a novel reweighting loss that adaptively allocates weights to triplets according to their difficulties , thus improving the quality of network training . SAFA [ 16 ] employs a multi-head spatial attention module to aggregate informative and diverse embedding maps . While promising , few of the above methods pay enough attention to the global dependencies of cross-view images , which hinders the discriminative ability of their embedded features . Different from existing methods , this work makes the first exploration to introduce Transformer [ 21 ] to cross-view geo-localization . We demonstrate the importance of considering global dependencies for reducing visual ambiguities . Geometry-based methods aim to correspond geometric configurations between ground and aerial images , which helps to reduce ambiguities caused by geometry misalignments . To this end , Liu and Li [ 9 ] explicitly inject per-pixel orientation information into the network . Nevertheless , this is based on the assumption of accessibility of the ground-truth orientation , which is not always satisfied in practice . Shi et al . [ 16 ] employ a polar transform algorithm to warp satellite images so that aerial images are geometrically aligned with ground images . However , this method is only applicable to the ideal case where the ground images are orientation-aligned panoramas . Even though the dynamic similarity module proposed in [ 17 ] overcomes this limitation , the brute-force warping strategy of the polar transform overlooks the depth of the scene content . It results in obvious appearance distortions , hindering performance improvement . Regmi and Shah [ 14 ] attempt to tackle this problem by synthesizing the corresponding satellite image from a ground query using conditional GANs ( cGANs ) , but the synthesized images are always granulated and lack details . In this paper , our L2LTR explicitly encodes learnable positional embeddings into the network without imposing a strong assumption . Unlike the previous works [ 9 , 16 , 17 ] , as shown in the experiments , our L2LTR does not impose a strong assumption on the position knowledge but flexibly learns relative positional information . 3 Method : L2LTR . This paper proposes a novel layer-to-layer Transformer ( L2LTR ) architecture with self-cross attention mechanism for cross-view geo-localization . The following sections detail our problem setting , objective , the L2LTR architecture , and our proposed self-cross attention . 3.1 Problem Formulation and Objective . The goal of cross-view geo-localization is to localize a query ground image by matching it with a set of geo-tagged aerial images . We formulate this problem in the same way as prior works [ 9 , 18 , 16 , 17 ] . Assume we have a training set D = { ( g1 , a1 ) , ... , ( gN , aN ) } containing N cross-view image pairs of ground images g and aerial images a . To simplify the problem , let each ground image gi correspond to only one ground-truth aerial image ai ( i ∈ { 1 , 2 , ... , N } ) during the training phase . Given a cross-view image pair ( gi , ai ) , we infer the corresponding image representations as ( F g i , F a i ) . Then , for the ith exemplar , the weighted soft-margin triplet loss [ 8 ] L , which aims to bring matching pairs closer while pushing non-matching pairs far apart , can be defined as follows : L = log ( 1 + eα ( d ( F g i , F a i ) −d ( F g i , F a j ) ) ) ( 1 ) where j ∈ { 1 , 2 , ... , N } and j 6= i. α is a hyperparameter used to speed up training convergence , and d ( · , · ) denotes the L2 distance . 3.2 Transformer for Cross-view Geo-localization . We seek to develop an L2LTR architecture that explores the global context and the positional information of cross-view images . Preliminaries : Vision Transformer . We first describe the Vision Transformer ( ViT ) [ 4 ] as a background . Given an image , the ViT first splits it into several patches . Then , the ViT receives as input a sequence of linear projected patch embeddings x ∈ RN×D , where N is the number of patches , and D is the patch embedding size . After prepending a learnable global representation xglobal ∈ RD , whose final hidden state serves as an aggregate image descriptor for the cross-view geo-localization task and adding positional embeddings xpos to x , we gain x0 = [ xglobal ; x ] + xpos and feed it into an L-layer Transformer encoder . Each layer consists of a Multihead Self-Attention module ( MSA ) , Feed Forward Networks ( FFN ) , and LayerNorm blocks ( LN ) . Note that the MSA consists of multiple self-attention heads and a linear projection block . In order to make a clear comparison with our proposed self-cross attention head , we denote the input of layer l ( l ∈ { 1 , ... , L } ) as xl−1 and formulate a single self-attention head , the core of the vanilla MSA , as follows : zl = LN ( xl−1 ) ( 2 ) Ql = zlW q l , Kl = zlW k l , Vl = zlW v l ( 3 ) Al = softmax ( QlK T l√ D ) Vl ( 4 ) where Wql , W k l and W v l are linear projection matrices . Domain-specific Transformer . The drastic domain gap between the ground and aerial images makes it difficult to match cross-view images by embedding them into the same space through a domain-shared network . To suit the cross-view geo-localization task , we employ a domain-specific Siamese-like architecture with two independent branches of the same structure . Such an architecture learns ground and aerial representations separately , and it can effectively project cross-view images into a shared space when optimized with Eq . 1 . The network overview is illustrated in Figure 1 ( a ) . Each branch is a hybrid structure consisting of a ResNet backbone ( denoted as “ Backbone ” dashed boxes in Fig . 1 ( a ) ) extracting CNN feature map and a Transformer encoder ( denoted as “ Encoder ” dashed boxes ) modeling global context from the CNN feature map . The linear projection of patch embedding in the ViT is applied to the CNN feature map by regarding each 1× 1 feature as a patch . Learnable positional embedding . Incorporating geometric cues [ 9 , 16 ] helps avoid ambiguities caused by geometric misalignment across views , thus greatly simplifying the cross-view geolocalization . Instead of imposing a predefined orientation knowledge on the network , this paper applies an efficient and flexible way to endow the network with the notion of geometry . Specifically , we use learnable 1D positional embeddings in the ViT , i.e . xpos ∈ R ( N+1 ) ×D . By adding the positional embeddings to the patch embeddings , the transformed features become position-dependent . Furthermore , since we do not impose any assumption on position knowledge but learn it through our learning objective , our L2LTR has wider practical applicability . Experiments show that incorporating the learnable positional embeddings helps capture relative positional information . This allows the network to geo-locate panoramas with unknown orientation and planar ground images with a limited field of view .

Summary: The paper tackles the cross-view geo-localization problem, which seeks to match a given ground-level image to aerial images that cover the same area, selected from a fixed database.  The paper is the first to apply transformers to the problem, and reports state-of-the-art results. Notably, the transformer’s attention mechanism largely frees them from having to warp the aerial image to match the ground-level image’s panoramic projection, a procedure that can go awry in the face of significant occlusion in the ground-level panorama, and/or errors in the ground pano’s orientation data. The resulting accuracy improves the state of the art by a significant margin.  The paper also makes a few adjustments to the ViT transformer, performing ablation studies on some of them.