MFRGN: Multi-scale Feature Representation Generalization Network For Ground-to-Aerial Geo-localization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 05:58:36

Abstract: Cross-area evaluation poses a significant challenge for ground-to-aerial geo-localization (G2AGL), in which the training and testing data are captured from entirely distinct areas.
However, current methods struggle in cross-area evaluation due to their emphasis solely on learning global information from single-scale features. Some efforts alleviate this problem but rely on complex and specific technologies like pre-processing and hard sample mining. To this end, we propose a pure end-to-end solution, free from task-specific techniques, termed the Multi-scale Feature Representation Generalization Network (MFRGN) to improve generalization. Specifically, we introduce multi-scale features and explicitly utilize them for G2GAL. Furthermore, we devise an efficient global-local information module with two flows to bolster feature representations. In the global flow, we present a lightweight Self and Cross Attention Module (SCAM) to efficiently learn global embeddings. In the local flow, we develop a Global-Prompt Attention Block (GPAB) to capture discriminative features under the global embeddings as prompts. 
As a result, our approach generates robust descriptors representing multi-scale global and local information, thereby enhancing the model's invariance to scene variations.
Extensive experiments on benchmarks show our MFRGN achieves competitive performance in same-area evaluation and improves cross-area generalization by a significant margin compared to SOTA methods.

Main paper: 
 related work 2.1 cnn & transformer structures: 
 Early methods [2, 39] based on handcrafted features [3, 24] face challenges due to the large viewpoint and appearance differences between queries and references, resulting in extremely low retrieval accuracy. Subsequent efforts [7, [11] [12] [13] 17] shift towards deep learning-based methods to extract powerful deep features. Workman et al. [43] demonstrate the superiority of deep features over hand-crafted ones by utilizing a deep CNN network, achieving high accuracy. Building upon this pioneering work, Vo el et. [40] design Siamese-like CNN for learning image features, while Zhai et al. [47] employ a VGG16 architecture to predict a dense pixellevel segmentation of the ground image. Further advancements include CRN [14] , which learns contextual feature representations by reweighting CNN to produce a spatial weighting mask. CVM-Net [11] utilizes a full CNN to extract local features and a NetVLAD [1] layer to get the global descriptor. GeoCapNet [35] learns the spatial feature hierarchies to enhance representation based on ResNet [10] and capsule network [30] . Recently, Vision Transformer (ViT) [38] has achieved marvelous success in drawing global dependencies. Due to the significant computational burden of directly applying ViT, L2LTR [45] combines CNN and ViT (i.e. hybrid ViT) to reduce visual ambiguity by extracting global contextual descriptors, where CNN learns high-level semantic features from input images and ViT encodes them to produce global representations. Inspired by [31] , GeoDTR [49] introduces a geometric layout extractor based on ViT to generate attention maps from high-level semantics features of the CNN backbone. TransGeo [52] proposes a pure ViT-based model to learn global context directly from input images by using attention-guided non-uniform cropping, and achieves outstanding results in same-area evaluation, yet poor results in cross-area evaluation. To bridge the large domain gap between panoramic and satellite images, some methods specialize in designing specific techniques for G2AGL. For example, [29] introduces conditional GAN (cGAN) to synthesize overhead-view images from ground-view images, while [36] does the opposite. However, their achievements of high accuracy depend on the generated successful results of the cGAN. Alternatively, other methods [31, 33, 36 ] have adopted polar transformation to tackle the cross-view domain gap by leveraging the center alignment assumption: a reference image exactly centers at the location of any query image. For example, Shi et al. [31] apply this transformation to preprocess aerial images into ground-view images, which are similar to the domain of the street images, resulting in remarkably high retrieval accuracy. DSM [33] directly estimates the orientation alignment of the two-view images under this assumption. Later, numerous methods [25, 34, 45, 48] employ this transformation as a pre-processing technique. Moreover, some methods tend to mine samples. GeoDTR+ [48] , a recent outstanding method aimed at improving cross-area evaluation, introduces a hard sample mining strategy to generate additional challenging samples in training batches. Sample4geo [7] develops GPS-based sampling and dynamic similarity sampling strategies to sample hard negatives, outperforming previous SOTA results without polar transformation. Yet, it still relies on an assumption that both query and reference images have GPS tags in the raw data, which are unmet in the real world. In contrast, some works have proposed datasets that do not adhere to these limitations. For instance, the VIGOR [53] dataset allows ground images' GPS coordinates to be at any location of aerial images, while the University-1652 [51] dataset is primarily proposed for drone-to-aerial geo-localization but also provides ground images collected from ground cameras without GPS tags. 
methodology: 
 In this section, we present the proposed Multi-scale Feature Representation Generalization Network (MFRGN), aiming to enhance generalization for ground-to-aerial geo-localization. MFRGN is a Siamese neural network with two identical branches. Each branch consists of two main steps: multi-scale feature extraction (Sec. 3.1) and feature representation with a global representation flow (GRF) to learn global embeddings (Sec. 3.2), along with a local representation flow (LRF) to reinforce local feature representation with the global embeddings as prompts (Sec. 3.3). Consequently, we obtain a global descriptor from the GRF and a local descriptor from the LRF, then concatenate them together as the final output descriptor for similarity calculation. An overview of our MFRGN is presented in Fig. 3 . Following [7, 9, 19] , we utilize symmetric InfoNCE loss based on Noise-Contrastive Estimation (NCE) [26, 28] to train our model.EQUATIONwhere 𝑞 denotes an encoded query, and 𝑅 is a set of encoded references. The InfoNCE loss calculates cosine similarity and is low when 𝑞 and the positive match 𝑟 + are similar, and high when they are dissimilar. The temperature parameter 𝜏 is learnable.Due to the distance at which the images were taken and the different resolutions of the images captured, which are more pronounced across different areas, features favorable for localization are distributed across different scales. Therefore, we extract features at multiple scales as opposed to a single scale in most existing methods [31, 45, 49] . We employ a pre-trained CNN (e.g. RetNet [10] , Con-vNeXt [22] ) as our backbone with multiple stages for multi-scale feature extraction. Given that the output feature maps of the initial backbone stages (e.g. , stage0, stage1) tend to be large, leading to a substantial number of tokens if a ViT-like architecture is subsequently involved, we opt the output maps from the final three backbone stages, denoted as 𝑆1, 𝑆2, 𝑆3, as shown in Fig. 3 . 𝑆4 is acquired by further downsampling 𝑆3 by a ratio of 0.5. Ranging from 𝑆1 to 𝑆4, the feature scale progressively increases, capturing richer semantic information conducive to global matching. Conversely, lower-level features contain more detailed information beneficial for localized fine matching.Most approaches [21, [47] [48] [49] 52] for G2AGL achieve better results by generating global descriptors, which indicates the significance of global information. Leveraging the notable capability of ViT in capturing global dependencies, we adopt ViT to construct global representation. We designate 𝑆1 ∼ 𝑆4 as the 𝑄 (query), 𝐾 (key), and 𝑉 (value) for each ViT encoder. In this configuration, crossattention can be regarded as retrieving corresponding features among different scale features when the tokens of 𝑄, 𝐾, and 𝑉 come from different scale features. On the other hand, self-attention resembles retrieving corresponding features within the same scale features when tokens stem from the same scale features. Therefore, we propose a ViT-based Self and Cross Attention Module (SCAM) (shown in Fig. 3 ), which effectively exploits both self-attention and cross-attention mechanisms to capture global dependencies across different scale features, contributing to more effective feature representation.However, it is impractical to directly utilize features 𝑆1 ∼ 𝑆4 for ViT. This is because these multi-scale features would generate a large number of tokens, leading to excessively high model complexity, even rendering training infeasible. Inspired by Lite-DETR [18] , we present features 𝑆2 ∼ 𝑆4 as high-level features (𝐹 𝐻 ) and 𝑆1 as low-level features (𝐹 𝐿 ). The number of tokens in 𝐹 𝐻 accounts for 6% ∼ 33% of 𝐹 𝐿 , significantly reducing computational costs. Due to global descriptors primarily focusing on global semantics, we update 𝐹 𝐻 at a higher frequency. As shown in Fig. 3 and 4 , we firstly utilize 𝐹 𝐻 as the query 𝑄 and concatenate 𝐹 𝐿 with 𝐹 𝐻 as the key and value (𝐾 = 𝑉 ), which are then inputted into Lite-Transformer Encoder (LTE) for updating 𝐹 𝐻 . The update process of 𝐹 𝐻 is as follows:Flatten Cat Pool avg Pool avg Pool avg Pool avg 1 1 C   3 3 C   6 6 C   8 8 C   (1 1 3 3 6 6 8 8) 110 C C   +  +  +  =  C H W  𝑄 = 𝐹 𝐻 , 𝐾 = 𝑉 = 𝐶𝑜𝑛𝑐𝑎𝑡 (𝐹 𝐻 , 𝐹 𝐿 ) 𝑂𝑢𝑡𝑝𝑢𝑡1 = 𝐹 ′ 𝐻 = 𝑄 ′ = 𝐿𝑇 𝐸 (𝑄, 𝐾, 𝑉 ) 𝑂𝑢𝑡𝑝𝑢𝑡2 = 𝐹 𝐿 (2)where 𝐶𝑜𝑛𝑐𝑎𝑡 represents concatenating all scale features, 𝐹 𝐻 and 𝐹 𝐿 are the initial input features, and 𝐹 ′ 𝐻 are the updated features. 𝐹 𝐻 is updated 𝐴 times.𝐹 𝐿 contains only a small amount of global semantic information; therefore, it is updated only a few times. The updating process of 𝐹 𝐿 is as follows:𝑄 = 𝐹 𝐿 , 𝐾 = 𝑉 = 𝐶𝑜𝑛𝑐𝑎𝑡 (𝐹 𝐿 , 𝐹 ′ 𝐻 ) 𝑂𝑢𝑡𝑝𝑢𝑡1 = 𝐹 ′ 𝐿 = 𝐿𝑇 𝐸 (𝑄, 𝐾, 𝑉 ) 𝑂𝑢𝑡𝑝𝑢𝑡2 = 𝐹 ′ 𝐻 (3)where 𝐹 𝐿 represents the original low-level features, 𝐹 ′ 𝐻 represents the features updated from the previous iteration of 𝐹 𝐻 , and 𝐹 ′ 𝐿 represents the updated features. 𝐹 𝐿 is updated for 𝐵 iterations. The output of SCAM is represented as 𝑓 𝑔 (= Concat(𝐹 𝐿 ′ , 𝐹 𝐻 ′ )), followed by linear projection to generate the global descriptor (Desc 𝑔 ).However, despite the above steps taken to alleviate model costs, the computational demands of the ViT model remain high under multi-scale settings. To address this issue, we employ two strategies.Strategy 1: Considering that the dot-product attention with 𝑂 (𝑁 2 ) complexity in the vanilla ViT, we replace it with the linear attention [15] with only 𝑂 (𝑁 ) complexity, referred to as the Lite-Transformer encoder (Fig. 4 ). Formally, the linear attention substitutes the vanilla dot-product with an alternative attention function 𝜙 (𝑄) 𝜙 (𝐾) 𝑇 𝑉 , where 𝜙 (•) = 𝑒𝑙𝑢 (•)+1, and 𝑒𝑙𝑢 (•) denotes the exponential linear unit [6] . Fig. 4 illustrates the computation graph of this operation.Strategy 2: Given that the lower-level features occupy the majority of tokens and contain many redundant features, we employ a pyramid pooling sampling (PPS) strategy [50] to sample 𝑛 (𝑛 ≪ 𝐻 × 𝑊 ) representative feature points from 𝑆1. Fig. 5 shows the default configuration: four average pooling operations are initially performed to obtain four feature maps with sizes of 1 × 1, 3 × 3, 6 × 6, and 8 × 8, respectively. Subsequently, we flatten the four maps and then concatenate them together to obtain a sampled feature map with a length of 110. For G2AGL, we set 𝑛 to approximately twice the total number of tokens of 𝑆2 ∼ 𝑆4.Global Prompt Local Feature C H W   1 1 C   1 H W   C H W   Conv Conv Sig Sig Pool avg Pool max Pool avg 1 1 C  Based on the observation: a human typically confirms whether two images belong to the same scene by their overall appearance, and then verifies their correspondence-specific details. Generally, the "overall" refers to global semantics, while the "detail" refers to local textures. There is an interesting fact that local results are influenced by global confirmation, meaning that in this case where rough global matching is known, the local results would be more adept at capturing discriminative features conducive to localization. Therefore, we consider global information into the LRF. Specifically, we use the global representation (𝑓 𝑔 ) from the GRF as prompts to guide the learning of local representation (denoted as 𝑓 𝑙 ), shown in Fig. 3 . Due to the inconsistency between 𝑓 𝑔 and 𝑓 𝑙 , directly fusing them may hinder information expression. We project 𝑓 𝑔 into the local space, then reshape it to match the sizes of 𝑆1 ∼ 𝑆4, resulting in four global prompts (𝑓 𝑔 𝑖 ∼ 𝑓 𝑔 4 ). Subsequently, we propose a Global-Prompt Attention Block (GPAB) to achieve prompt learning, as shown in Fig. 6 . Each global prompt 𝑓 𝑔 𝑖 undergo operations according to Eq. 4, resulting in channel-wise and spatial-wise weights 𝐹 𝑐ℎ 𝑖 and 𝐹 𝑐ℎ 𝑖 , respectively.EQUATIONwhere pool 𝑎𝑣𝑔 represents average pooling along the channel direction, pool 𝑚𝑎𝑥 represents max pooling along the spatial direction, Conv denotes the 2D convolution operation, and Sig is the Sigmoid activation function.We utilize 𝐹 𝑐ℎ 𝑖 and 𝐹 𝑠𝑝 𝑖 to guide 𝑆 𝑖 to learn local discriminative information, ultimately leading to local representation 𝑓 𝑙 𝑖 :EQUATIONAll 𝑓 𝑙 𝑖 are concatenated to 𝑓 𝑙 that then generates local descriptor Desc 𝑙 by a linear projection. Similar to sampling strategies described in Sec. 3.2, we also use the PPS to sample 𝑆1 in LRF. 
experiments 4.1 experiments settings: 
 Datasets. Both CVUSA [43, 47] and CVACT [21] are cross-view datasets for ground-to-aerial geo-localization, each containing 35,532 pairs for training and 8,884 pairs for testing. CVUSA collects images from across the United States, while CVACT mainly covers urban scenery in Canberra, Australia. The two datasets are used for one-to-one matching, i.e. every satellite-view image has one corresponding street-view image, and share an assumption of center alignment: all pairs are aligned with similar spatial localization and orientation. For the two datasets, we use 256 × 256 for the aerial view and 128 × 256 for the ground view as input image size. VIGOR [53] is a challenging and large-scale dataset for one-to-more mapping (every reference covers 2∼4 queries), which is a more real-life setting. VIGOR collects 105,124 street panoramas and 90,618 satellite images from four cities, New York, Chicago, Seattle, and San Francisco. VIGOR provides two evaluation settings, i.e. same-area (training and testing on all cities)) and cross-area (training on New York and Seattle, testing on San Francisco and Chicago) evaluation for comprehensive performance in G2AGL. Exceptionally, the geographical location of a panorama can correspond to any location in the aerial image. Also, each satellite-view image has three semipositive images that cover regions of the street-view image. For VIGOR, we use 320 × 320 for the aerial view and 320 × 640 for the ground view as input image size.Metrics. Similar to existing methods [7, 8, 45, 49] , we choose to use the top-𝑘 recall accuracy (denoted as R@𝑘) with 𝑘 ∈ {1, 5, 10, 1%} for evaluation purposes. R@𝑘 assesses the probability of the ground truth reference image within the top 𝑘 ranked results given a query image. Besides, we also use hit rate to evaluate the retrieval performance of VIGOR following [7] . The hit rate is understood as the R@1 without semi-positive images.Implementation Details. Following [7, 19, 48] , we use weightshared ConvNeXt [22] as our CNN backbone for both views. We set 𝐴 = 2, 𝐵 = 1 in the SCAM. And, we set the latent dimension to 128 and the feedforward layer dimension to 2048 for each Lite-Transformer encoder with 4 heads. In PPS, we obtain four feature maps with sizes of 1 × 1, 6 × 6, 12 × 12, 21 × 21 for satellite view, and 1 × 1, 3 × 12, 6 × 24, 7 × 36 for street view, respectively. In the training phase, the model is trained on Nvidia Titan V GPUs for 50 epochs with AdamW [23] optimizer. The label smoothing of InfoNCE loss is set to 0.1, and the batch size to 64. We use a cosine annealing learning rate decay strategy with an initial learning rate of 10 -4 . The dimension of the final descriptor is 2048.Same-area evaluation results. Tab. 1 presents same-area results of various methods. In the "𝑤/ Pre-processing" column, all methods use a pre-cropping technology (i.e. pre-crop the edges of street images in CVACT) to accelerate model fitting, and some of them also use the polar transformation (marked with the †) to pre-process satellite images into ground-view images, which both rely on the center alignment assumption. We observe that MFRGN with only using pro-cropping achieves the best improvement across all four metrics. In the "𝑤/ Sampling" column, our method brings a slight performance fluctuation using hard negative sampling (HNS) (requiring the GPS-tagged query images) in [7] . When not using all of the above, our retrieval results surpass all others. Under this condition, an interesting observation is that the performance of our method far exceeds all methods with "Pre-processing" and reaches the same level as Sample4Geo with "Sampling" on CVUSA, even scores higher in terms of R@1, R@10 and R@1% on CVACT. Besides, MFGAN+ obtains the maximum gain with pro-cropping Table 1 : Same-area comparison results on CVUSA and CVACT benchmarks. † denotes using polar transformation. The best and second best results are bolded and underlined.CVUSA CVACT R@1 R@5 R@10 R@1% R@1 R@5 R@10 R@1% 𝑤/ Pre-processing SAFA [31] 81. 15 Method CVUSA→CVACT CVACT→CVUSA R@1 R@5 R@10 R@1% R@1 R@5 R@10 R@1% 𝑤/ Pre-processing SAFA † [31] 30. 40 and HNS, noting that without polar transformation. We further evaluate various methods without Pre-processing and Sampling on the highly challenging VIGOR (Tab. 3). Our MFRGN achieves the best scores compared with all methods. These results indicate that MFGRN can learn powerful feature representation that eliminates the need for these task-specific techniques to bridge the domain gap between the two distinct viewpoints, even in challenging scenarios.Table 3 : Same-area and cross-area comparison results on VIGOR benchmark. All methods do not use "Pre-processing" and "Sampling".Method R@1 R@5 R@10 R@1% Hit rate Cross-area evaluation results. We present the cross-area results of various methods to evaluate their generalization capabilities. The CVUSA→CVACT represents training on CVUSA and testing on CVACT, while CVACT→CVUSA represents the opposite. As shown in Tab. 2, our model performs the best results almost in all cases by a large margin. Specifically, our method achieves the SOTA performance by improving from the previous SOTA 37.81% to 51.61% on the CVUSA → CVACT, and 17.45% to 49.18% on the CVACT→CVUSA when no any special techniques are used. For the challenging VIGOR, MFRGN obtains significant cross-area improvements and improves 6.80% R@1 and 3.88% hit rate compared to Samole4Geo (Tab. 3). These considerable improvements can be attributed to the fact that our approach improves representation abilities by jointly learning global and local information from multiscale features, thereby generalizing to diverse cross-area scenarios.Impact of components. Tab. 4 performs a component analysis of MFRGN, such as the GRF (𝐺), the LRF with global embedings as prompts (𝐿 𝑝 ) or with the initial features (𝑆1 ∼ 𝑆4) as prompts (𝐿 𝑖𝑛𝑖 ), and so on. We use the highest level feature map from the last stage of the CNN backbone (i.e. 𝑆4) to directly output the descriptor, denoted as 𝐵𝑎𝑠𝑒𝑙𝑖𝑛𝑒. When adding multi-scale features (𝑀𝐵𝑎𝑠𝑒𝑙𝑖𝑛𝑒) into the 𝐵𝑎𝑠𝑒𝑙𝑖𝑛𝑒, the performance gains are slightly smaller than 𝐵𝑎𝑠𝑒𝑙𝑖𝑛𝑒 in the same area but are obviously larger in cross area. We conjecture that multi-scale features could introduce more local details but hamper information conveyance; whereas, in the context of the cross area, these features containing richer and more match-relevant information across different scales help adapt to diverse scenes. Furthermore, our model improves greatly cross-area performance after adding GRF or LRF on 𝑀𝐵𝑎𝑠𝑒𝑙𝑖𝑛𝑒, especially in the former. This proves that global information is more favorable for image-to-image matching than local information. However, jointly learning 𝐺 and 𝐿 𝑖𝑛𝑖 / 𝐿 𝑝 can further significantly Impact of lightweight strategies. Tab. 5 conducts comparative analysis for two lightweight strategies, including Linear Attention (LA) and Pyramid Pooling Sampling (PPS). We further report the FLOPs and memory (using a pair of street image and satellite image as input). We can see that using no LA and PPS suffers from poor results, especially in cross-area performance. This is a reasonable result since vanilla dot-product attention is more prone to overfitting than linear attention, as well as much redundancy from multi-scale features interfering with information representation. But this problem is alleviated by a combination of two strategies, and leading to an interesting observation is that the performance gain of using two strategies is more pronounced in the two area evaluations while using fewer FLOPs and memory.Impact of hyperparameter. Fig. 7 analyzes the impact of different dimension ratios of the global descriptor and local descriptor. As the proportion of dimensions of global descriptors relative to local descriptors increases, namely more global information, the same-area performance gradually improves while the cross-area performance notably enhances. Conversely, a decrease in same-area is dimension of Desc 𝑔 , and 𝐷𝑖𝑚 𝑙 is dimension of Desc 𝑙 . We take the average R@1 score as a baseline, and all cases are subtracted from it, respectively.results occurs slightly, but cross-area results rapidly decline. We also observe a severe deterioration when only local information is considered. This indicates that global information is crucial in image-to-image matching. However, relying on global information alone poses a limitation for performance. Therefore, it is indispensable to simultaneously learn both global and local information as our method for G2GAL'task. Based on the above results, we choose the ratio of dimensions between global and local descriptors to 7:1.T-SNE visualization. We use Sample4Geo and our MFRGN (both training on CVUSA) without Pre-processing and Sampling to obtain representation features of CVUSA and CVACT testing data. Fig. 8 visualizes the same-area and cross-area testing distribution by projecting the features onto a 2D space through t-SNE [37] . In Sample4Geo (a), the feature distributions of the same area and the cross area have a clear discrepancy. Our MFRGN (b) shows a coincidence between the cross-area distribution and the same-area distribution in both shape and scope. This proves that MFRGN can learn robust representation descriptors and effectively maps similar samples from same-area datasets and cross-area datasets to a similar feature representation space, which hence reduces the content distribution discrepancy between the two areas. Heatmap visualization. Fig. 9 visualizes response maps generated by the two methods following the same settings as above. Sampel4Geo only focuses on certain local regions, owing to its pure CNN architecture. Our method allows the response to simultaneously concentrate on both global and local regions, such as the highway with a large coverage area and the local area of the roof. Such a property is attributed to the fact that MFRGN explicitly models both global and local features, allowing the model to achieve better representation ability against variations in cross-area scenarios.  
conclusion: 
 In our work, we present a simple and effective method, i.e. MFRGN, to solve cross-area generalization for G2AGL by explicitly joint learning global and local representation from multi-scale features.We introduce a lightweight ViT-based Self and Cross Attention Module (SCAM) to efficiently learn global embedding from multi-scale features. Then, we utilize learned global embedding as prompts to boost local representation by the proposed Global-Prompt Attention Block (GPAB). Besides, we introduce a lightweight Transformer encoder and pyramid pooling sampling strategy to alleviate model burden and feature redundancy. Extensive experiments demonstrate competitive same-area performance and SOTA cross-area generalization of the proposed method without requiring strict assumptions that are unmet in real-life settings, or any task-specific techniques. Future work will design a more effective multi-scale feature exploitation model to achieve better performance in terms of samearea and cross-area evaluation. We also aim to generalize this paradigm to other cross-view geo-localization frameworks, such as UAV-to-satellite geo-localization, which enjoys widespread popularity within the community of UAVs in Multimedia1 . 
: 
 Figure 1 : Comparison of R@1 accuracy between recent methods without using task-specific techniques on VIGOR benchmark, encompassing both polar transformation [31, 45] and hard negative mining [7] . Our proposed MFRGN achieves the SOTA same-area and cross-area evaluation for G2AGL.in viewpoint and appearance make the two-view images distinct features, and even some contents of the one-view images are invisible to the other. Consequently, G2AGL is an extremely challenging problem.Existing approaches [5, 12, 13, 20, 43] typically learn a common representation across different views to convert image geolocalization into an image-to-image matching/retrieval task. However, these methods achieve high same-area performance, in which the training and testing data are captured from identical areas, but low cross-area evaluation, where data are collected from completely distinct areas. For example, TransGeo [52] demonstrates excellent same-area retrieval accuracy, e.g. ∼85% top-1 recall (R@1) accuracy on CVACT [21] benchmark, but notably low R@1 across different one, e.g. only 17.45% when training on CVACT (mostly included urban scenery in Canberra, Australia) and testing on CVUSA [43, 47] (collected from across the United States). Others attempt to improve cross-area generalization by complex pre-processing techniques [45, 49] , e.g., the polar transformation relying on a center alignment assumption between two views, which often unmeting in real-life settings. Additionally, elaborate hard sample mining techniques like dynamic similarity sampling strategy [7] and contrastive hard sample generation [48] have been explored. To make G2AGL more practical, some methods try to get rid of these taskspecific technologies, but there remains a notable performance gap between same-and cross-area evaluation for these algorithms without these techniques on the challenging VIGOR [53] benchmark, as shown in Fig. 1 .Motivated by these observations, this paper makes concerted efforts to improve cross-area performance by devising an efficiently pure structure devoid of specific techniques. As illustrated in Fig. 2 , the basic structures of the existing methods can be broadly divided into two categories: those based on CNN or Vision Transformer (ViT) [38] , and those based on a combination of CNN and ViT. However, single-scale features considered by most methods may be insufficient for new or changed scenes, which limits the final performance. It's noteworthy that features conducive to localization may be distributed across different scales due to variations in distance and resolution of image acquisition. On the other hand, since images from nearby yet distinct geographic locations often share common properties like street layout or building type, existing methods focusing only on learning global context fail to distinguish them, leading to matching with the same reference. This can be mitigated by incorporating local information, e.g. the diverse colors and textures found in two buildings. As a matter of fact, humans typically employ a "rough screening through global context, then fine matching through local details" mechanism when pairing two images. Hence, we introduce multi-scale features and jointly learn global and local information from them to bolster robustness in feature representations.In this paper, we introduce a Multi-scale Feature Representation Generalization Network (MFRGN) aimed at enhancing generalization for real-world applications. A simplified structure diagram of MFRGN is depicted in Fig. 2 (c), which differs from the two existing architectures (a) and (b). Firstly, we leverage a pre-trained CNN backbone to acquire multiple feature maps with richer information across different scales. Subsequently, we devise an efficient global-local information representation module to concurrently learn global and local representation. In this module, we introduce ViT-based Self and Cross Attention Module (SCAM) to efficiently learn multi-scale global context representations, and propose a CNN-based Global-Prompt Attention Block (GPAB) to encourage the model to learn distinctive local features under the global representations as prompts. Furthermore, to alleviate model burden and feature redundancy on multiple scales, we introduce a lightweight Transformer encoder and a concise Pyramid Pooling Sampling (PPS) strategy. As a result, MFRGN improves the abilities in terms of feature representation and model generalization by simultaneously learning global and local embeddings across multi-scale features. Our code will be released.The main contributions of this work are as follows: 1) We explicitly utilize multi-scale features and propose MFRGN to enhance the model's representational capability, thereby improving cross-area generalization.2) We simultaneously consider global and local information (possibly for the first time in G2AGL's task) by a well-designed dual-flow structure based on ViT and CNN.3) We present lightweight SCAM to efficiently model global dependencies and propose GPAB to boost local representation with learned global embedding as prompts, both of which are in the multi-scale case.4) Our framework is simple and efficient, and purely end-to-end requiring no neither complex pre-processing steps relying on strict assumptions, nor specific sample mining technologies. Our method outperforms previous works in generalization evaluation by a large margin. 
 
 Appendix: https://www.zdzheng.xyz/ACMMM2023Workshop/ 


Decision: Well written paper with a reasonable technical quality and convincing results.

Reviews: review: paper_summary: paper_summary: This paper presents a multi-scale feature representation method for Ground-to-Aerial Geo-localization, especially for enhancing the cross-area generalization. The authors propose a global-local information module with two flows (global and local flows) to improve feature representation. Global flow yields global embeddings using a Self and Cross Attention module. Local flow uses a Global-Prompt Attention block to capture discriminative features with the global representation as prompts.
strength_weakness: strength_weakness: 1. This paper is well-written and clearly presented. 
2. The proposed method is novel and the contributions are sufficient. The proposed method achieves good performance in cross-area evaluation without task-specific pre-processing or hard sample mining.
3. The experimental setup is reasonable and the experimental results also seem good.
limitations: limitations: 1. The proposed model architecture looks a bit complex compared to other methods, although the performance is good. Does this come at the cost of significantly increased inference runtime? It would be better to provide a comparison of inference runtime.
2. This proposed model seems to be directly applicable to UAV-to-satellite geo-localization, but it is considered as future work in this paper. I'm confused about this. How does the proposed method perform on the University-1652 dataset?
3. Although methods that use both global and local features are rare in cross-view geo-localization, they are indeed common in visual geo-localization (a.k.a. place recognition) [1][2][3]. It should be mentioned in Related Work.

[1] Transvpr: Transformer-based place recognition with multi-level attention aggregation. CVPR 2022.

[2] Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognition. ICLR 2024.

[3] Deep Homography Estimation for Visual Place Recognition. AAAI 2024.
paper_summary: paper_summary: This manuscript introduces a multi-scale feature representation generalization framework for the ground-to-aerial geo-localization problem. Self and cross-attention modules and global-prompt attention blocks are proposed.
strength_weakness: strength_weakness: + The performance of the proposed method, and its components seems to be decent, compared to its competitors.
+ The manuscript is well-organized.
limitations: limitations: + I suggest the authors to re-organize the contribution part. "Our framework is simple..." can't count as a technical contribution.
+ Can Sample4Geo adapt with pre-cropping? If so, please report its performance under the same setting as MFRGN+.  The proposed MFRGN seems to have a similar pipeline to Sample4Geo, so it should be easy to do so.
+ Why does multiscale representation work so well for this task? I think this part of the manuscript is not well-motivated.
+ Please report parameters/GFLOPs comparison with previous works.

I'm willing to discuss in the rebuttal phase, and I will raise the score if my concerns are alleviated.
paper_summary: paper_summary: This paper aims to address the challenge of cross-area generalisation in ground-to-aerial geo-localization using a multi-scale feature representation model. Despite the effort to improve upon current methods, several issues undermine the effectiveness and clarity of the proposed solution.
strength_weakness: strength_weakness: The model is evaluated on three datasets and compared with some recent work. These comparisons show the model's performance for the place recognition task.
limitations: limitations: 1- This paper primarily combines existing techniques without providing substantial innovation. The introduction of multi-scale features and the use of attention modules are well-explored areas, and the paper did not demonstrate how MFRGN significantly differs from or improves upon these established methods. Moreover, the paper does not provide adequate justification for the chosen architecture. For example, the reasons behind selecting a specific multi-scale (FPN) feature extraction method. 

2- Some technical details are missing. For instance, the specifics of the multi-scale feature extraction process, the exact architecture of the attention modules, and the training procedures are inadequately described.  

3- The authors should provide more comparisons, clearer justifications for their model design, and a better presentation of their methodology. Indeed, a clearer and more detailed presentation of the methodology, including algorithmic steps and parameter settings, would enhance the reproducibility of the work.