G3: An Effective and Adaptive Framework for Worldwide Geolocalization Using Large Multi-Modality Models
Authors: Pengyue Jia
Yiding Liu
Xiaopeng Li
Xiangyu Zhao
Yuhao Wang
Yantong Du
Xiao Han
Xuetao Wei
Shuaiqiang Wang
Dawei Yin
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Geolocalization
Image-to-GPS retrieval
Large Multi-Modal Models
CLIP
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Worldwide geolocalization aims to locate the precise location at the coordinate level of photos taken anywhere on the Earth. It is very challenging due to 1) the difficulty of capturing subtle location-aware visual semantics, and 2) the heterogeneous geographical distribution of image data. As a result, existing studies have clear limitations when scaled to a worldwide context. They may easily confuse distant images with similar visual contents, or cannot adapt to various locations worldwide with different amounts of relevant data. To resolve these limitations, we propose **G3**, a novel framework based on Retrieval-Augmented Generation (RAG). In particular, G3 consists of three steps, i.e., **G**eo-alignment, **G**eo-diversification, and **G**eo-verification to optimize both retrieval and generation phases of worldwide geolocalization. During Geo-alignment, our solution jointly learns expressive multi-modal representations for images, GPS and textual descriptions, which allows us to capture location-aware semantics for retrieving nearby images for a given query. During Geo-diversification, we leverage a prompt ensembling method that is robust to inconsistent retrieval performance for different image queries. Finally, we combine both retrieved and generated GPS candidates in Geo-verification for location prediction. Experiments on two well-established datasets IM2GPS3k and YFCC4k verify the superiority of G3 compared to other state-of-the-art methods. Our code is available online [https://github.com/Applied-Machine-Learning-Lab/G3](https://github.com/Applied-Machine-Learning-Lab/G3) for reproduction.

G3: An Effective and Adaptive Framework for Worldwide Geolocalization Using Large Multi-Modality Models
Pengyue Jia', Yiding Liu", Xiaopeng Li", Yuhao Wang', Yantong Du',
Xiao Han', Xuetao Wef Shuaiqiang Wang". Dawei Yin", Xiangyu Zhao
City University of Hong Kong, Baidd Inc., Southern University of Science and Technology ciityu. edu. fliuyiding tanh, hahahenha, shiigaaannnaassiiiiiinii..nn wang> com, duyantong94@hrbeu. edu en, www.ttssssesseeeeeeeeeeo edu. en, yindawei @acm org, www..raaeiihiiiiyy edu.hk
{jia. seii..pnguue,aaee
Abstract
Worldwide geolocalization aims to locate the precise location at the coordinate level of photos taken anywhere the Earth. It very challenging due 1) the difficulty of capturing subtle location-aware visual semantics, and 2) the heterogeneous geographical distribution of image data. As result, existing studies have clear limitations when scaled to worldwide context. They may easily confuse distant images with similar visual contents, or cannot adapt various locations worldwide with different amounts of relevant data. To resolve these limitations, we propose G3, a novel framework based on GarrivvalAAugennee Generation (RAG). In particular, G3 consists of three steps, i.e., Geco-aligmmen, Geo- crooddversification, and Geo-verification to optimize both retrieval and generation phases of worldwide geolocalization. During Geo-alignment, our solution jointly learns expressive multi- modal representations for images, GPS and textual descriptions, which allows us to capture location-aware semantics for retrieving nearby images for given query. During Geo-diversification, we leverage a prompt ensembling method that is robust to inconsistent retrieval performance for different image queries. Finally, we combine both retrieved and generated GPS candidates in Geo-verification for location prediction. Experiments two well-established datasets IM2GPS3K and YFCC4k verify the superiority of G3 compared to other state-of-the-an methods. Our code and data") are available online for reproduction.
Introduction
Worldwide image geolocalization 30 aims pinpoint the exact shooting location for any given photo taken anywhere on Earth, as illustrated in Figure I(a). Unlike geolocalization within specific regions (e.g., city level) [9 2 25 12 2 worldwide geolocalization (2 30 43) greatly unleashes the potential of geolocalization, which is useful for various real-world applications, such as crime tracking and navigation. However, worldwide image geolocalization is extremely challenging, as images collected from around the world are featured with a myriad of elements, including varying landscapes, weather conditions, architectural styles, etc.
"Corresponding author. http::/wwugginggge...oooaaassspp
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Similar Vision, not Geography
Heterogeneous Images
(28.77 .8147)
(35.86.-114.61)
Different Numbers of Effective References
Model
Query Image
Databaso
Image
Encoder
(a) Worldwide
(b) Challenges on extracting visual semantics
(c) Challenges on image heterogeneity
geolocalization
Figure Illustration of limitations of prevailing methods in worldwide geolocalization.
Extensive research efforts have been devoted to the task, which can be broadly categorized into 1) classification- based, 2) retrieval-based, and 3) retrieval-augmented generation (RAG) methods. 3 divide the entire geographical space into fixed grids, and classify each image into a particular grid. Retrieval- based methods convert image localization to either a image-to-image 2
In particular, classification-based methods 32 22
20
or an image-to-GPS (29) retrieval problem, where the final prediction a top-retrieved GPS exists in a given candidate database. Generation-base methods [3 recently achieved state-of-the-art performance on this task, via applying retrieval- augmented generation (RAG) pipeline that leverages the strong reasoning and generalization ability of large multi-modality models (LMMs). They usually integrate the retrieved GPS coordinates in the input prompts of LMMs as references, to generate more accurate predictions.
12 23
Despite their initial success, existing studies still have clear limitations when scaled to a worldwide context, mainly due to two challenges as shown in Figure T First, it very challenging to extract visual semantics that accurately indicate an image's geolocation, as two distant places could possibly have similar visual features. Conventional visual representations are ineffective in implying subtle location-aware semantics. Second, image data usually exhibits significant heterogeneity in its geographical distribution, which existing methods can hardly handle. For retrieval-based methods, it only performs well for image queries with many nearby images stored in the database, while many images at unpopular locations may have very few or even no similar data to be compared with, leading to large prediction errors. Worst still, such inconsistent retrieval performance significantly affects existing RAG-based methods that use a fixed number of references. As the retrieval performs inconsistently for different image queries, their generation process lacks the robustness to adapt to various image queries at different locations worldwide.
To address the aforementioned challenges, we propose G3, a novel RAG-based solution with expres- sive retrieval and robust generation for worldwide image geolocalization.
For the retrieval phase, we train multi-modality encoders to effectively capture location-aware visual similarity between images. Unlike existing RAG-based methods that only leverage conventional visual similarity for retrieval, we propose multi-modality alignment process, namely Glo-alignment, which learns the representations of GPS coordinates, images, and textual descriptions joint manner. By doing this, the visual representations are able to capture fine-grained location-aware semantics for retrieving other images close by. addition the numerical GPS coordinates, the textual descriptions (e.g., city/country names) can largely enrich the location information to be aligned with the visual representations. Moreover, to train the new dataset, namely MP16-Pro, by including textual geographical descriptions to the original MP16 dataset [11. We anticipate the dataset will benefit more future work for location-aware visual representation learning.
representations, we establish
For the generation phase, we leverage a prompt ensembling method, namely Geo-diversification. which improves the robustness of prediction generation for different types of images. More specifi- cally, it generates a diverse set predictions via multiple retrieval-augmeeted prompts, each of which might be more useful for a certain type of query images. As such, the generated GPS candidates are more likely to contain the ground truth coordinates. Subsequently, we conduct Geo-verification. which combines both the retrieved and the generated GPS candidates, and compares their similarities with the query image using the learned multi-modality representations. The most similar GPS returned as the final prediction. Extensive experiments are conducted on well-established datasets IM2GPS3k (30) and YFCC4K (26). and the results show the effectiveness of G3 compared to the other state-of-the are bascline methods. We summarize the key contributions of our work follows:

---

We present G3, a novel solution for the worldwide geolocalization task. Our proposed method leverages Geo-alignment to learn expressive location-aware representations of images, 2) Geo- diversification to improve the robustness of GPS candidate generation, and 3) Geo-verification to ensemble both retrieved and generated candidates for final prediction.
We release new dataset MP16-Pro, adding textual localization descriptions each sample based on the original dataset MP16 to facilitate future research in the field.
We extensively experiment with two well-established datasets IM2GPS3K and YFCC4K. G3 demonstrates superior performance compared to other state-of-the-art baseline methods.
2 Related Work
Image Geolocalization. Image Geolocalization is an important task in computer vision 48
spatial data mining 42 36 6), and GeoAl [40 39 35 37 Previous work in image geolocalization
can be divided into three main categories: classification-basee
methods, retrieval-basee basse methods,
divide the entire earth into multiple grid cells and assign the center coordinates as predicted values. Models are then trained to classify the input image into the correct cell. However, if the actual location of the image is far from the center of the predicted cell, there can still be significant errors, even if methods treat the image geolocalization task as
and generation- baaed methods. (1) Classification-
methods [22 30
18
32
20 3
the cell prediction is correct. (2) Retrieval-
retrieval problem. typically maintaining a database of images [3 17
46
34
44
27
24 45 or
a gallery of GPS coordinates [29). They take the most similar images and GPS coordinates to the query image as the predicted values. However, maintaining a global-level image database or GPS methods employ large multi-modality models to generate generation into the geolocalization task and took the retrieved similar images' coordinates as references to help generate predictions. However, they can not accurately extract visual semantics to indicate an image's location because they simply use visual similarity to retrieve references and suffer inaccurate prediction when facing heterogeneous query images. In this work, G3 introduces Geo-alignment to incorporate geographical information into image representations to help retrieve similar images in geography and proposes Geo-diversification and Seri-ieriiication enhance prediction performance and robustness.
gallery is infeasible. (3) Generation-basee
the predicted coordinates for images [14]. Zhou et al. 43 introduced eetrieval-augmente
in single domains like computer vision (31] and natural language processing [38] there has been increasing attention on large multi-modalig models. CLIP (1) aligns image and text representations through contrastive learning and achieves remarkable model generalization with simple optimization objectives. The Large sanguage-nnd-isiion Assistant (LLAVA) (16) effectively combines CLIP's visual encoder with the powerful language model Vicuna, enhancing the model's general understanding of both visual and textual information through two-stage instruction tuning. GPT4V M is large multimodal model released by OpenAl in 2023, allowing users to imput text and images to obtain answers.
Large Multi-modality Models. Inspired by the success of large models 9
10 15
retrieval-Augmenne Generation. To mitigate the hallucination issue in LLMs, retrieval augmeented generation (RAG) 13 has emerged as a popular and effective technique. It enhances the reliability of content generated by LLMs by incorporating facts fetched from external sources. Specifically, certain factual knowledge is retrieved by a retriever from external sources based on a query. LLMs can access these retrieval results during the generation process generate accurate outcomes. RAG preserves the generalization capabilities of LLMs while also introducing external information to enhance the reliability of generated content, efficiently alleviating the hallucination problem.
3 Methodology
Figure 2 iillustrates the comprehensive architecture of G3, which consists of Geo-alignment, diversifi- cation, and verification, with two phases: database construction and location prediction. During the database construction phase, introduced in Section 3.1 Geo-alignment aligns image representations with textual image descriptions and GPS information to incorporate geographical information into representations. the location prediction phase, introduced in Section similar images will be first retrieved based on the nearest neighbor search from the database; Geo-diversification will then
3

---

Database Construction
Controshe Leaming
Projection
Enooder
Vidan
Emoodor
prico taker from Veadland Narway
Tishnable
Frozon
(A) Geo-alignment
(B) Image Vectorization
Location Prediction Phase
Phamptati
RAG The
Cantitiaess
GPS
Final "Prdiction
Enooder
RAG Text Phoeell with
Predition Candistates
Preckston Candidales
Zero Pronct
Encodar
Simiarity
Query Image
Gerertted Condidates
Image
Geo-diversifiiation
(D) Geo-verification
Suppose you are an oxport in goolocal zotion, you have the ati give two number GPS ascrdiadionn gown an insepe Your amower musl be the caordimend vater lessd (beiluche, longluate) lommrt you reference, those are acordinates of some simlay images: (m). and these are images:
S ipppoen you aee an supert geolocalzation you have the atillty to give two number GPB ocordiaation given image. Your anower must the coordinatos in (astudo longllude
Prompt
Figure 2: Overview of the framework of G3.
combine their coordinates in RAG prompts generate diverse candidates, and Geo-verification finally selects the final predicted coordinates in a multi-modality space.
3.1 Database Construction
G3 requires an image database to preserve image representations. Existing work directly uses visual encoders (e.g., CLIP's ViT encoder or ResNet) encode images. However, visual similarity cannot completely represent geographical proximity. To overcome this issue, we propose Geo-alignment, which incorporates geographical information into image representations by multi-modality alignment.
Geo-alignment. Geographical features can be divided into continuous and discrete types, which are essential in geolocalization. On the one hand, according to the first law of geography (28). "everything related to everything else, but near things are more related to each other." " Climate, terrain, and vegetation are continuous features that gradually change along latitude or longitude. the other hand, discrete features (e.g., city/country names) are also conducive to determining geographical location. These features usually change abruptly at national borders. To encode images with representations tailored for geolocalization, we propose a multi-modality alignment method, Geo-alignment, shown in Figure A).
Image encoding. We use pretrained vision encoder and two trainable transformation layers to encode images: o fiex() oiggs Sgps (V,,,)), where o and image are the i-th image representations
the batch that need be aligned with textual geographical descriptions and GPS data. fiex and faps are the corresponding feed forward functions, V represents the fixed vision encoder, and I, is the 2-th image in batch.
GPS coordinate encoding. To encode GPS coordinates, an appropriate projection is needed to transform latitude and longitude into a Cartesian coordinate system. We choose not to adopt the equal earth projection (EEP) used in previous work 29 because EEP primarily focuses on area accuracy while overlooking angular distortions, which is significant in modeling the trends of geographical features along latitude and longitude. As result, we utilize Mercator projection for its conformal property. The formula of Mercator projection is shown below:
x R () 2o)
(I)
y R In [tan

---

where À and o are radians of longitude and latitude, Ao denotes the central meridian longitude. R is proportional constant of Earth radius. The output x and y denote the transformed plane coordinates.
After projection, we follow previous work [29 to capture high-frequency patterns and hierarchical rep- resentations using random fourier features (RFF) with various frequencies. RFF function ry will trans- form the projected coordinate G; = (x,, y:) first: ((;,) = [cos (2,,,, sin(2nSG,)) " M de- notes matrix sampled from a Gaussian distribution M (0, σ) to limit the frequencies. To capture hierarchical representations, we sum up the outputs with different σ: e = Σ fe(r(G,, σx)) where t is the encoded gps representations for i-th sample in a batch, K denotes the number of hierarchical patterns, f& is the feed forward function for k-th hierarchical layer. σ% controls the fre- quency for k-th layer and Jj 2 + +i ()(logg(0mmx) (oog))-oooggaa (o MiN-)) Ak E (1.....N).
mmin and Man controls the range of kk.
Text encoding. We initially employ geographical reverse encoding to obtain textual descriptions of GPS coordinates. For instance, as illustrated in Figure[[A) the GPS coordinates (60.37, 6.72) can be converted into the textual description "A photo taken from Vestland, Norway". 'These textual descriptions are inputs to a pre-trained text encoder, followed by feedforward networks for vector transformation. ett = f(T (T;)) where et denotes the encoded textual descriptions for i-th sample in batch, f is the feed forward transformation layer, T is the text encoder function, and T, the textual descriptions for i-th sample in batch.
Optimization. Geo- alognmiggment is optimized with the following objective to align image representa- tions with textual descriptions and GPS information:
7 Σ log( exp(logits,,)
ea
e |
Lab
logits
IT
exp'x.>
Σn j=11 exp(logits.y)
||ea
where L.,5 denotes the loss function of modality modality b, e is the encoded representations, and t is the temperature. G3 needs to align image representations with both textual descriptions
and GPS data, so the final optimization objective is shown below: L (Limage,text + Limage.gps Liesttimage image + Lygs,image) 1/2.
Image vectorization. As illustrated in Figure 2 B), after Geo-alignment, we will vectorize the images in the dataset and store them in a database. To maintain image representations tailored for geolocation tasks, we concatenate the original visual representations with representations aligned with geographical information: e' concat(eimage - -egps ). e' denotes the final representation, image and image the image representations aligned with textual geographical descriptions and GPS information.
image).
gimage represents the vector obtained directly through the pretrained vision encoder. ejxtt
3.2 Location Prediction
Figure [(C) and (D) illustrate the overview of the location prediction phase. Previous work directly incorporates the GPS coordinates of similar retrieved images as references into a single RAG prompt to generate predictions. However, due to the heterogeneity of query images, the number of reference GPS coordinates varies when each sample achieves optimal prediction performance. To address this issue, Geo-divensification expands the candidate pool with prompts containing different numbers of reference coordinates, including zero (i.e., in zero-shot manner), shown in Figure 2 C). Illustrated in Figure D). Geo-verification selects the best prediction coordinate for each sample to-S using the well-trained Image-to GPS encoders in Geo-aligno alignment. In the location prediction phase, reo-diversification and rre-ieeifiittion are introduced to enrich the diversity of generated predictions and select the predictions with the highest confidence.
Geo-diversification Due to the heterogeneity of query images, the number of reference coordinates introduced in the RAG process varies when each sample achieves optimal prediction performance. To solve this issue, we introduce Geo-diversification. Specifically, we first construct K RAG prompts with different numbers of reference coordinates (0 reference coordinates equals zero-shot generation), and each prompt will generate N results. This process can be represented as follows: (cf, ct = RAG(pp)) where ct denotes the candidate coordinate generated by the k-th RAG prompt, and p² is the k-th RAG prompt. The final candidate pool contains the top S coordinate candidates of retrieved similar images and the generated coordinate candidates as shown in Figure 2 The final candidate pool is denoted as {c1, C2, ... Jm where = K x N + S.
5

---

Geo-verification. Given the coordinate candidates set (c1, ... t, selecting the best guess is essential and challenging. We reinvent the well-trained Image-to-GPS model in aeo-alignment to achieve this target, as shown in Figure 2 D). The similarity between image representations e and image (T and the coordinate ej with the highest similarity selected as the final prediction by j argmax(sim,), j E (1,2, 2.... ,
GPS representations e are derived by sim egps
4 MP16-Pro Dataset
To facilitate subsequent research, we propose the MP16-Pro dataset by adding textual geographical descriptions to each sample from the MediaEval Placing Tasks 2016 (MP-16) dataset [1]] Specifi- cally, we utilize the open-source geocoding tool Nominatim to obtain multi-level geographical textual descriptions for each sample's s GPS location (total 4.72 million locations). There are eight geograph- ical unit levels: neighborhood, city, county, state, region, country, country code, and continent. Some examples are given in Appendix A. for reference. Geographical text descriptions provide additional information for geolocalization tasks and enable models to transcend the original paradigm solely supporting image and GPS alignment, facilitating more diverse modeling approaches.
Experiments
Datasets and evaluation metries: For database construction and model training, we use the MP Pro dataset we released. It contains 4.72 million geotagged images from Flickr(") However, given that the dataset released in 2016. currently, 4.12 million images within the dataset remain accessible. Following previous work [29] [43]. we evaluate G3 with public datasets (IM2GPS3K 77] and YFCC4K (261) and a threshold metric. Given the predicted coordinates and the ground truths, this metric quantifies the percentage of predictions where the distance to the ground truth falls within specified thresholds (1 k1,, 25km, 200km, 750km, and 2500km).
Implementation details: We use faiss to deploy the image database. The vision and text encoders are pretrained VIT-L/14 and masked self-attention transformer from CLIP [21]. The dimensions for two trainable layers of fiest. Saps. f are 768 and 768. The input dimension of GPS encoder is 512, and the dimensions for four hidden layers of f& in Equation 3. are 1024, the output dimension is 512. For the Earth radius, we set it as 6378137.0 For RFF, we use three hierarchies with o mmi as 20 and 07 maz as 28. GPT4V [ is selected as the LMMs in this paper. Its temperature is set to 1.2. The number of RAG prompts K is set to 4, and the number candidates for each RAG N is set to for IM2GPS3K and 1 for YFCC4K. The number of similar image coordinates taken into account in candidates O for IM2GPS3K and for YFCC4K. G3 is trained using Adam W optimizer with learning rate 3e-5 and weight decay 1e-6. A step linear scheduler is employed with gamma 0.87, and the training epoch is set to 10. Training batch size is set to 256 and temperature i in Equation 2 initialized 3.99. All experiments are conducted with Pytorch and one NVIDIA H800 GPU. Please refer to Appendix A.2 for more details on the training environment, training time, and API cost. We also mention the limitations of G3 Appendix A.4
Baselines: To evaluate G3 in geolocalization, we follow previous work (29 43 and select the
following baselines for comparison: [LIkNN,o== 30), PlaNet B2). CPlaNet [22 ISNs [18),
Translocator (20). GeoDecoder 33 GeoCLIP
29 Img2Loc (3), PIGEON 5 The detailed
deseriptions of baselines are in Appendix A.5 Due the lack of available implementations for
Img2Loc, we reproduce it based on its paper and release it in our repository for future research.
5.1 Comparison with State-of-the-art Methods
To verify the effectiveness of G3, we conduct comparative experiments on IM2GPS3K and YFCC4K
with other state-of-the-ale methods. The results are shown in Table [ (1) G3 superior to all the other baselines on almost all metrics. In addition, compared to the second best methods, G3 achieves 8.5%. 2.8 %, 3.3% improvements on IM2GPS3K in the 1km, 25km, 200km thresholds and 21.3% %. 16.9% 13.5%, 3.3%. 0.6% improvements on YFCC4K in the 1km, 25km, 200km, 750km, 2500km thresholds. (2) G3, Img2Loc, GeoCLIP, and PIGEON achieve leading results, which
hnttps//wwwwffickr.omm https///openai.com//

---

Table 1: Overall experimental results on IM2GPS3K and YFCC4K.
IM2GPS3K
YFCC4K
Methods
Street 1km
City 25km
Region 200km
Country 750km
Continent 2500km
Street Ikm
City 25km
Region 200km
Country 750km
Continent
2500km
[L]KNN, sigma=4
20
7.2
19.4
26.9
38.9
55.9
2.3
5.7
11
23.5
42
PlaNet
24.8
48
14.3
22.2
CPlaNet
22
10.2
26.5
34.6
48.6
64.6
7.9
14.8
21.9
36.4
55.5
ISN's [8
10.5
28
36.6
49.7
66
6.5
16.2
23.8
37.4
55
Translocator
20
11.8
31.1
46.7
58.9
80.1
8.4
18.6
27
GeoDecoder
3
12.8
33.5
45.9
61
76.1
24.4
33.9
68.7
GeoCLIP
29
14.11
34.47
50,65
69.67
83.82
9.59
19.31
32.63
55
74.69
Img2Loe
15.34
39.83
53.59
69.7
82.78
19.78
30.71
41.4
58.11
74.07
PIGEON
11.3
36.7
72.4
85.3
10.4
23.7
40.6
62.2
Ours
16.65
40.94
55.56
71.24
84.68
23.99
35,89
46.98
64.26
78.15
Table Ablation study on IM2GPS3K and YFCC4K.
IM2GPS3K
YFCC4K
Methods
Street 1km
City 25km
Region 200km
Country
Continent 2500km
Street 1km
City
Region
Country 750km
Continent
750km
25km
200km
2500km
w/o Geo-A
15.71
40.64
54.85
70.8
84.05
20.8
32.72
44.25
61.83
76.64
w/o Geo-D
16.35
40.51
53.89
69.2
83.11
20.28
31.87
43.67
60.84
76.25
w/o Geo-V
14.98
38.27
51.25
67.6
81.18
19.03
30.24
40.93
57.93
72.83
Ours 16.65
40.94
55.56
71.24
84.68
23.99
35.89
46.98
64.26
78.15
can be attributed to the other methods taking the worldwide geolocalization task as a classification problem, introducing inevitable systemic biases. (3) G3 demonstrates significant improvements over GeoCLIP because GeoCLIP is constrained by the settings of the GPS gallery, which can not cover the entire globe. (4) Compared to Img2Loc, G3, through alignmitmment that aligns images with discrete and continuous geographical features, achieves more precise retrieval of reference coordinates for subsequent RAG processes. Additionally, Geo-diversiffcation and reo-verification effectively expand the candidate pool and filter out the confident prediction results, further enhancing geolocalization performance. Overall, G3 achieves the best performance on all datasets across almost all metries, which verifies the effectiveness of G3.
5.2 Ablation Study
To understand the specific effects of each module in G3, design the following variants:
w/o Geo-A: G3 without Geo-alignment. Directly using ViT in CLIP for database construction.
w/o Geo-D: G3 without Seo-diversification. Generating prediction with one RAG prompt with 10 positive samples and 10 negative samples (the parameter has been tuned).
w/o Geo-V: V: G3 without Geo-verificatio Instead of using the well-trained Image-GPS model in Geo-alignment, this variant randomly selects the final prediction from candidates.
Table 2 shows the experimental results. We can draw the following conclusions: (1) All three modules significantly contribute to final performance. (2) G3 achieves better performance than w/o Geo-A for Geo-alignment incorporates geographical information into image representations. As a result, the retrieved images are geographically similar to the query image, enhancing effectiveness of references in the RAG process. (3) G3 is superior w/o Geo-D, for the number of reference coordinates varies when each sample achieves the optimal prediction performance facing heterogeneous query images. The absence of Geo-diversification leads to suboptimal candidates. (4) Comparing G3 with w/o Geo-V, V, we observe a significant performance drop in w/o Geo-V V, indicating the necessity of Geo- reo-verification.
5.3 Hyperparameter Analysis
In the generation process within G3, two hyperparameters directly impact the results: the number of RAG prompts and the number of candidate coordinates generated by each single prompt.
7

---

Street Level
City Level
Region Level
Country Level
Continent Level
18
43
Accuracy (%)
Accura cy (%)
72
85
16
41
55
71
Accuracy
0 39
70
Accuracy
83
14
69
12
37
35
51
67-
68
81
10
33
49
66
79
1
3
1
2
Num. of prompts
Num. prompts
Num. of prompts
Num. of prompts
Num. of prompts
Figure 3: Varying the number of RAG prompts on IM2GPS3K.
Street Level
City
Region Level
Country Level
Continent Level
Accuracy (%)
17.0
41.5
55.5
71.5
85.5
85.0
Accuracy
41.0
Accuracy
55.0
Accuracy
71.0
84.5
16.5
40.5
54.5
70.5
84.0
83.5
16.0
40.0
54.0
70.0
83.0
5
1
3 5
5
3
5
3 5
Num. of candidates
Num, of candidates
Num. of candidates
Num. of candidates
Num. of candidates
Figure 4: Varying the number of candidates for each RAG prompt on IM2GPS3K.
Number of RAG prompts. To investigate the impact varying numbers of RAG prompts, we design the following experiment: We employ four sets of RAG prompts with different reference coordinates: positive, negative; 5 positive, 5 negative; 10 positive, negative; 15 positive, Starting with the first prompt, subsequent prompts will be sequentially added to change the number of RAG prompts. The number of candidates generated by each prompt is fixed to As illustrated in Figure the influence of RAG prompt counts prediction performance is consistent across different metric thresholds. A significant enhancement is observed when the number increases from to 2. The reason is that the zero-shot prompt (RAG prompt with positive and 0 negative reference coordinates) fails to provide high-qual qual-tyality predictions for global images with insufficient information. The model's 's performance gradually improves as the number increases from 2 to 4 because having more candidates will increase the possibility of containing the ground truth coordinates.
negative.
Number of candidates. Figure 4 shows the results varying the number of candidates for each prompt. We fix the number of prompts to 4 in this experiment. We observe that the turning points where performance begins to decline exhibit increasing trend at different levels. Specifically, at the street level, performance declines after just one candidate, at the city level after three, at the region and country levels after five, and at the continent level after seven. Three key points merit attention: (1) The initial upward trend occurs because the generation of LMMs involves randomness. Introducing more candidates alleviate the randomness.
(2) As the number of candidates increases, performance ultimately drops, likely due to the introduction of more noise in the predictions from additional generations. (3) The turning points of decline differ by level because broader levels demonstrate greater tolerance to predictive bias when more noise candidates are included.
5.4 Effectiveness of Geo-alignment and Mercator Projection
To assess the effectiveness of Geo-alignment and Mercator projection, we conduct the following experiments focusing on the reference retrieval phase: We build image databases using different embedding techniques and then retrieve the Top-N images closest to the query image. The geodesic distances will be calculated between their coordinates and the query image. The embedding variants are illustrated as follows:
CLIP VIT: Directly using the visual encoder VIT in CLIP for image embedding.
G3+EEP: Geo-alignment with Equal Earth Projection (EEP).
Merrerrto
Geo-alignment with Mercator Projection.
Table 3
shows the statistics of the geodesic distances of retrieval reference images with different embedding methods. We can draw the following conclusions: (1) G3+EEP outperforms CLIP ViT as the latter only considers visual similarity, while image representations in G3+EEP encompass both

---

Table 3: Distance statistics of retrieval reference images with different embedding methods. Avg.. Md., Max., and Min. are the average, median, maximum, and minimum distances to the query image.
Top-5 Candidates
Top-10 Candidates
Top-15 Candidates
Methods
Avg.
Md.
Max.
Min.
Avg.
Md.
Max.
Min.
Avg.
Md.
Max.
Min.
CLIP ViT
2554.7
2244.6
5048.4
800.7
2645.9
2269.9
6376.9
513.3
2704.1
2307.7
7142.2
404.8
G3+EEP
2361.5
2089.0
4618.2
735.3
2434.5
2102.2
5808.9
479.3
2464.3
2104.3
6529.2
369.6
G3+Mercator
2299.2
5569.6
2046.9
373.0
Table 4: Impact of LMMs on G3.
Methods
Street Ikm
City 25km
Region 200km
Country 750km
Continent 2500km
Img2Loc (LLaVA)
10.21
29.06
$6.36
71.07
Img2Loc (GPT4V)
15.34
39.83
53.59
69.70
82.78
G3 (LLaVA)
14.31
35.87
49.42
66.93
81.78
G3 (GPT4V)
16.65
40.94
55.56
71.24
84.68
visual and geographical similarity, which is essential for geolocalization tasks. (2) G3+ per- forms better than G3+EEP, as the EEP projection method emphasizes area projection accuracy while overlooking angular distortions, which increases the training complexity and limits the performance.
5.5 Impact of LMMs G3
To explore the impact of LMMs on G3, we conduct the experiments of G3 and Img2Loc with LLaVA (LLaVA-Next-LLail)) --a-eext-LLMM----- on IM2GPS3K. From Table we find that: (1) After switching LMMs from GPT4V to LLaVA, the performance of G3 shows some decline across various metrics but remains competitive. (2) Additionally, compared to Img2Loc (LLaVA), G3 (LLaVA) significantly outperforms Img21 (LLaVA),, demonstrating the effectiveness the proposed modules. (3) Finally, by comparing the performance of G3 equipped with LLaVA and GPT4V Img21 equipped with LLaVA and GPT4V, we can observe that G3 shows more stable performance across different LMMs.
5.6 Necessity Analysis of Three Representations Alignment Geo-alignment
To verify the necessity of aligning the three representations in Geo-alignment, conduct experiments of the following variants on IM2GPS3K:
IMG: Directly using pre-trained CLIP vision encoder the encoder.
IMG+GPS: Aligning Image representations with GPS representations in Geo-alignment, the textual descriptions are not used.
IMG+GPS+TEXTIC:: Aligning three modalities simultaneously in Geo- -eiganigmment.
Table5]shooe that: (I) By comparing IMG+GPS+TEXT, IMG+GPS, and IMG, we find that adding GPS and text information can both enhance feature representation compared to using the orig- inal image information alone. (2) By comparing MG++GPSTEXX" with IMG+GPS, we find that IMG+GPS performs better smaller scales, while IMG+GPS+TEXT performs better larger scales. This might be because GPS is suitable for modeling variations at smaller scales, whereas text descriptions do not vary significantly at small scales and may even remain the same.
Table 5:
Results of necessity analysis.
Methods
Street km
City 25km
Region 200km
Country 750km
Continent 2500km
IMG
15.71
40.64
54.85
70.8
84.05
IMG+GPS
16.91
41.41
58.02
70.94
84.18
IMG+GPS+TEXTIC3)
55.56
71.24
84.68
5.7
Case Study
Case study on reference image retrieval. Figure 5 visually demonstrates the superiority of G3 in reference image retrieval. It is evident that if CLIP's VIT is used the image encoder, the model

---

Query Image
Top-5 Images retrieved by CLIP
Top-5 Images retrieved by G3
86.64 km 3267 km
3438 km 16.57 km 3214 km
0.026 km 3267 km
0.276 km
3438 km 0.241 km
Figure 5: Reference image retrieval with CLIP ViT and G3.
Query Image
Predictions
Query Image
Predictions
Query Image
Predictions
with reference 37.8700 32 4930
RAG with reference 40 40,2062, 16.3738
RAG with 0 reforence 37.7749, -122.4194
RAG with references 41.0168, 20.9721
RAG with references 37 8B33, -122.448
references
37.4377, 126.4102
RAG with 10 references
with 10 references
references 29.3768, 122.3009
37,8726, 32. 32.4814 41,0162, 28:9636
41.0162, 96966
29.9761, 3904
references:(41
(362).377 0ww.99955..220
1188,1115.. 1541174),338
1135,-15.17558
referencese 56 2560, 1001004442200 (24 4317.117.111.111111 118.1118.111 1377)
(51.5 (0....0),6.7777... 1..2243
(37. 4377, 4162)) (27.5444 106, 7081).02222 /2224476411
9721) (41 1..0044222 28, 9392) 28888886,88 52.5111,, (41.0
484),36
738),36, 1147 (177-15.17330.666
4.3624),02 (29. 9755, (22239115,220 52.29
207, 28 (41.0))441.012222))
01..0182.28 9638)6663664888 (41.8)))11.018,,2222 (11.11.1,,2.88888888 28.983)
0,699 (tt.339455..2222546 3),11 12.49977
7033, 122 (222444444.444444 6546, (28.449)).292... 278.9776,8 122.692223999)
Figure 6: Predictions given with different numbers of references facing heterogeneous images.
primarily focuses on the human figures in the image (i.e., 'two people posing together in the center of the photo') while neglecting background elements beneficial for geolocalization. In Geo-alignment, G3 incorporates geographical information into the image representations. As a result, retrieved images are more focused on geographical proximity (three reference images within km of the actual shooting location are retrieved in the top-5 candidate images). These valuable reference images further assist the RAG process to enhance final prediction performance.
Case study heterogeneous query image in RAG process. Figure provides three examples illustrating the best prediction occurs when using RAG prompts with different numbers of references facing heterogeneous query images. (1) RAG with 0 references achieves the best performance for the first query image. This is because, on the one hand, the references are filled with biased coordinates, and the other hand, the building in the figure is famous landmark named Selimiye Camii mosque. The pre- pre-trained LMMs effectively provide the longitude and latitude of this landmark based on its world knowledge. (2) For the second query image, RAG with 5 references performs best because the optimal reference appeared in the fifth position. More references do not add extra helpful information but instead introduce more noise, causing the performance of RAG with 10 references to decline; RAG with 0 references produces incorrect predictions due to the absence of clear landmark indicators in this image. (3) For the third query image, RAG with 10 references yields the best accuracy, as the references from 6 to 10 provide substantial helpful information, whereas the first five reference coordinates are far from the ground truth. Overall, from these examples, we can discern some common patterns: for images with prominent landmark features, RAG with 0 references often yields good results; for images with less informative content (such as oceans, skies, or indoor scenes), RAG with 10 references makes more comprehensive judgments based on a greater number of references; and for images with distinct regional features (images between the first two settings), RAG with references will achieve satisfactory prediction accuracy.
6 Conclusion
In this paper, we propose a novel worldwide geolocalization framework named G3. First, we introduce Geo-alignment to capture location-aware semanties in images by aligning images with textual geographical descriptions and GPS information. Second, Geo-diversiffcationn is proposed to improve the robustness of prediction generation via prompt ensemble technique. Finally, Geo- verification selects the final coordinate prediction using the learned multi-modality representations. G3 is evaluated on two well-established datasets, IM2GPS3K and YFCC4K, and achieves state-of- the-art performance. In addition, we release a new dataset MP16-Pro, adding textual localization descriptions each sample based on the original dataset MP16 to facilitate future research in the field. All the code and data used in this work have been released public.
10

---

Acknowledgements
This research was partially supported by Collaborative Research Fund (No.C1043-24GF), Research Impact Fund (oo.R10)-))), APRC CityU New Research Initiatives (No.9610565, Start-up Grant for New Faculty of CityU), CityU HKIDS Early Career Research Grant (No.9360163), Hong Kong ITC Innovation and Technology Fund Midstream Research Programme for Universities Project (No.IT//03//2MSS) Hong Kong Environmental and Conservation Fund (No. 88/2022), and SIRG CityU Strategie Interdisciplinary Research Grant (No.702004)) Huawei (Huawei Innovation Research Program), Tencent (CCF-Tencent Open Fund, Tencent Rhino-Birg Focused Research Program), Ant Group (CCF-Ant Research Fund, Ant Group Research Fund), Alibaba (CCF-Alimama Tech Kangaroo Fund (No. 2024002)), SeeF-BBahhuan--bbie Foundation Model Fund, and Kuaishou.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
[2] Bingyi Cao, Andre Araujo, and Jack Sim. Unifying deep local and global features for image search. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16. pages 726-743. Springer, 2020.
[3] Brandon Clark, Alec Kerrigan, Parth Parag Kulkarni, Vicente Vivanco Cepeda, and Mubarak Shah. Where we are and what we're looking at: Query based worldwide image geo-localization using hierarchies and scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23182 222222119,,2 2023.
[4] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
[5] Lukas Haas, Michal Skreta, Silas Alberti, and Chelsea Finn. Pigeon: Predicting image ge- olocations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12893-12902, 2024.
[6] Xiao Han, Xiangyu Zhao, Liang Zhang, and Wanyu Wang. Mitigating action hysteresis in traffic signal control with traffic predictive reinforcement learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 673-684, 2023.
[7] James Hays and Alexei A Efros. Im2gps: estimating geographic information from a single image. In 2008 ieee conference computer vision and pattern recognition, pages 1-8. IEEE, 2008.
[8] Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, (3(1):117-128, 2010.
[9] Pengyue Jia, Jingtong Gao, Xiaopeng Li, Zixuan Wang, Yiyao Jin, and Xiangyu Zhao. Second place overall solution for amazon kdd cup 2024. In Amazon KDD Cup 2024 Workshop, 2018.
[10] Pengyue Jia, Yiding Liu, Xiangyu Zhao, Xiaopeng Li, Changying Hao, Shuaiqiang Wang, and Dawei Yin. Mill: Mutual verification with large language models for zero-shot query expansion. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2498 2518, 2024.
[11] Martha Larson, Mohammad Soleymani, Guillaume Gravier, Bogdan Ionescu, and Gareth JF Jones. The benchmarking initiative for multimedia evaluation: Mediaeval 2016. IEEE MultiMe- dia, 24(1):4331)))9 96, 2017.
[12] Seongwon Lee, Hongje Seong, Suhyeon Lee, and Euntai Kim. Correlation verification for image retrieval. In Proceedings the recognition, pages 5374-5384, 2022.
the IEFF/CVF conference on computer vision and pattern
11

---

[13] Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih. Tim Rocktäschel, et al. Gearieval-augmennt generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 13:9459-94744 2020.
[14] Ling Li, Yu Ye, Bingchuan Jiang, and Wei Zeng. Georeasoner: Geo-localization with reasoning in street views using a large vision-language model. In Forty-first International Conference on Machine Learning.
[15) Xiaopeng Li, Lixin Su, Pengyue Jia, Xiangyu Zhao, Suqi Cheng, Junfeng Wang, and Dawei Yin. Agent4ranking: Semantic robust ranking via personalized query rewriting using multi-agent Ilm. arXiv preprint arXiv:2312. /5450, 2023.
[16] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[17] Liu Liu and Hongdong Li. Lending orientation to neural networks for cross-view geo- localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5624-5633, 2019.
[18] Eric Muller-Budack, Kader Pustu-Iren, and Ralph Ewerth. Geolocation estimation of photos using a hierarchical model and scene classification. In Proceedings of the European conference on computer vision (ECCV), pages 563-579, 2018
[19] Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In Proceedings of the IEEE international conference on computer vision, pages 3456-3465, 2017.
[20] Shraman Pramanick, Ewa M Nowara, Joshua Gleason, Carlos D Castillo, and Rama Chellappa. Where in the world is this image? transformer-based geo-localization in the wild. In European Conference on Computer Vision, pages 196-215. Springer, 2022.
[21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748- 1163.876...1 PMLR, 2021.
[22] Paul Hongsuck Seo, Tobias Weyand, Jack Sim, and Bohyung Han. Cplanet: Enhancing image geolocalization by combinatorial partitioning of maps. In Proceedings of the European Conference on Computer Vision (ECCV), pages 336-551, 2018.
[23] Shihao Shao, Kaifeng Chen, Arjun Karpur, Qinghua Cui, André Araujo, and Bingyi Cao. Global features are all you need for image retrieval and reranking. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11036-11046 2023.
[24] Yujiao Shi, Xin Yu, Dylan Camphell, and Hongdong Li. Where am looking at? joint location and orientation estimation by cross-view matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4064 4072, 2020.
[25] Fuwen Tan, Jiangbo Yuan, and Vicente Ordonez. Instance-level image retrieval using reranking transformers. In proceedings of the IEEE/CVF international conference on computer vision, pages 12105-12115, 2021.
[26] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 1fcm The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
[27] Yicong Tian, Chen Chen, and Mubarak Shah. Cross-view image matching for geo-localization in urban environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3608-3616. 2017.
(28) Waldo R Tobler. A computer movie simulating urban growth in the detroit region. Economic geography, (::supp)::3424400 1970.
12

---

[29] Vicente Vivanco Cepeda, Gaurav Kumar Nayak, and Mubarak Shah. Geoclip: slip-inspired alignment between locations and images for effective worldwide geo-localization. Advances in Neural Information Processing Systems, 36, 2024.
[30] Nam Vo. Nathan Jacobs, and James Hays. Revisiting im2gps in the deep learning cra. In Proceedings of the IEEE international conference on computer vision, pages 2621-26300 2017.
[31] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks, Advances in Neural Information Processing Systems, 36, 2024.
[32] Tobias Weyand, Ilya Kostrikov, and James Philbin. Planet-photo geolocation with convolutional neural networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 37-55. Springer, 2016.
[33] Scott Workman, Richard Souvenir, and Nathan Jacobs. Wide-area image geolocalization with aerial reference imagery. In Proceedings of the IEEE International Conference on Computer Vision, pages 3961-3969, 2015.
[34] Hongji Yang, Xiufan Lu, and Yingying Zhu. Cross-view geo-localization with layer-to-layer transformer. Advances Neural Information Processing Systems, 2-229999920200 2021.
[35] Zijian Zhang, Ze Huang, Zhiwei Hu, Xiangyu Zhao, Wanyu Wang, Zitao Liu, Junbo Zhang, S Joe Qin, and Hongwei Zhao. Mlpst: Mlp is all you need for spatio-temporal prediction. In Proceedings of the 32nd ACM International Conference Information and Knowledge 2023.
Management, pages
[36] Zijian Zhang, Xiangyu Zhao, Qidong Liu, Chunxu Zhang, Qian Ma. Wanyu Wang, Hongwei Zhao, Yiqi Wang, and Zitao Liu. Promptst: Prompt-enhanced spatio-temporal multi-attribute prediction. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 11195-3205 2023.
[37] Zijian Zhang, Xiangyu Zhao, Hao Miao, Chunxu Zhang, Hongwei Zhao, and Junbo Zhang. Autostl: Automated spatio-temporal multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages
2023.
[38] Wayne Xin Zhao, Kun Zhou, Junyi Li. Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.1822) 2023.
[39] Xiangyu Zhao, Wenqi Fan, Hui Liu, and Jiliang Tang. Multi-type urban crime prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1138-43990 2022.
[40] Xiangyu Zhao and Jiliang Tang. Modeling temporal-spatial correlations for crime prediction. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 497-506, 2017.
[41] Xiangyu Zhao, Tong Xu, Yanjie Fu, Enhong Chen, and Hao Guo. Incorporating spatio-temporal smoothness for air quality inference. In 2017 IEEE International Conference Data Mining (ICDM), pages 1177-1182, IEEE, 2017.
[42] Xiangyu Zhao, Tong Xu, Qi Liu, and Hao Guo. Exploring the choice under conflict for social event participation. In Database Systems for Advanced Applications: 21st International Conference, DASFAA 2016, Dallas, TX, USA, April 16-19, 2016, Proceedings, Part I 21, pages 396-411. Springer, 2016.
[43] Zhongliang Zhou, Jielu Zhang, Zihan Guan, Mengxuan Hu, Ni Lao, Lan Mu, Sheng Li, and Gengchen Mai. Img2loc: Revisiting image geolocalization using multi-modality foundation models and image-based retrieval-augmeened generation. arXiv preprint arXiv:2403. 19584, 2024.
13

---

[44] Sijie Zhu, Mubarak Shah, and Chen Chen. Transgeo: Transformer is all you need for cross-view image geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1162-1171, 2022.
[45] Sijie Zhu, Linjie Yang, Chen Chen, Mubarak Shah, Xiaohui Shen, and Heng Wang. R2former: Unified retrieval and reranking transformer for place recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19370-19380, 2023.
[46] Sijie Zhu, Taojiannan Yang. and Chen Chen. Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 3640-3649, 2021.
[47] Yuanshao Zhu, Yongchao Ye, Ying Wu, Xiangyu Zhao, and James Yu. Synmob: Creating high-fidelity synthetic gps trajectory dataset for urban mobility analysis, Advances in Neural Information Processing Systems, 36:22961-22991 2023.
[48] Yuanshao Zhu, Yongchao Ye, Shiyao Zhang, Xiangyu Zhao, and James Yu. Difftraj: Generating gps trajectory with diffusion probabilistic model. Advances in Neural Information Processing Systems, 36:65168-6511 2023.
[49] Yuanshao Zhu, James Jianqiao Yu, Xiangyu Zhao, Qidong Liu, Yongchao Ye, Wei Chen, Zijian Zhang, Xuetao Wei, and Yuxuan Liang. Controltraj: Controllable trajectory generation with sppoloyy-constrineed diffusion model. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4676-4687, 2024.
14

---

4faa0/39632168890.jpg
eb/a7/1993938478.jjg
4b/5c/8177901044.jjgg
Figure 7: Image data in MP16-Pro Dataset.
Table 6: More details training and inference parameters.
Parameter
G3
GPU
H800 80G
Training Time
7 hours / epoch 10 epoch 141,266,179
Total params
Trainable params
13,648,15 131 (3 (3.09%) 304.38
GFLOPS
Dataset Samples
4. 1M
Batch Size
256
GPU Memory
24G
Text Processor
Huggingface CLIP default text processor
Vision Processor
Huggingface CLIP default vision processor
Token per RAG prompt
Input: 200+30
Output: x each RAG prompt Input: 5. M,(55) Output: 2.16M($64.8)
Token for IM2GPS3K
Token for YFCC4K
Input: 7S22()72) Output: 3.06M($9118)
A Appendix
A.1 MP16-Pro Dataset Samples
To facilitate understanding of the MP16-Pro dataset, we provide some samples from the dataset in this section. The dataset contains two parts: the image data and the metadata for images.
Figure gives three examples from the MP16-Pro dataset. Take these three images as examples, MP16-Pro adds extra textual descriptions based on their coordinates:
For image 4f/a0/3963216888 16t1/999962189..j..... LAT: 47.217578, LON: 7.542092, neighbourhood: Wengis-
tein, city: Solothurn, county: Amtei Solothurn-Lebern, state: Solothurn, region: NA, country: Switzerland, country coud::yycode: ch, continent: NA.
For image cb/a771193934488.jppp LAT: 39.950477, LON: -75.157535, neighbourhood: Center City, city: Philadelphia, county: Philadelphia County, state: Pennsylvania, region: NA, country: United States, country_cooe code: us, continent: NA.
For image 4b/5c/8 1ttcc///7991147.jpgg LAT: -34.580365, LON: 58.425464, neighbourhood: Palermo, city: Buenos Aires, county: NA, state: Autonomous City of Buenos Aires, region: NA, country: Argentina, country_coode ar, continent: NA.
All the data has been released online
A.2 More Details on Training and Inference
In this section, we detail the training and inference process by providing information about the training environment, text and vision processors, and token costs. The Input token cost and output token cost for each RAG prompt are 200 + 30 and x n, where 200 the fixed token cost for prompts and images with resolution parameter "low", and the 30 is related to the reference
15

---

Error <
Images
IKM
25KM
200KM
750KM
2500KM
Figure 8: Example query images from IM2GPS3K that G3 localization error falls in 1km, 25km, 200km, 750km, 2500km thresholds.
coordinates, & denotes the number references in the prompt. For the output, 18 the token cost for one-time generation, and n is the times of generations.
A.3
G3 with Query Image
Figure R showcases some example query images from the IM2GPS3K dataset, with different rows representing the varying localization eтrors of G3 on these images. We can observe patterns in query images with errors ranging from kkm 2500km. Images with lower errors often feature prominent landmarks or regional characteristics, such as buildings, decorations, and symbols. On one hand, these images are more likely to have more similar images the database. On the other hand, LMMs are more sensitive to this type of information and can provide more accurate predictions. In contrast, query images with higher errors contain less effective information; large expanses of ocean, water bodies, and snow scenes offer limited assistance for geolocalization.
16

---

Table 7: Experimental results on textual description granularity on IM2GPS3K.
Street 1km
City
Region
Country 750km
Continent
Methods
25km
200km
2500km
G3-N
16.44
40.64
54.35
70.57
83,98
G3
16.65
40.94
55.56
71.24
84.68
A.4 Limitations
The limitation of G3 is mainly about its efficiency. G3 relies on large number of candidates generated through cro-diversification. However, the noise candidates generated by feo-diversification lead to low efficiency in the use of computational resources. A potential solution is to expand the database or enhance the geolocalization ability LMMs by fine-tuning help create prediction candidates more efficiently and effectively. In another aspect, the image vector length in Geo-alignment comprises three concatenated vectors, which increases storage demand and retrieval latency. potential solution is to use quantizers, such as ProductQuantizer storage pressure.
g to accelerate vector retrieval speed and reduce
A.5 Details on Baselines
[L [30]. KNN makes use of the top k. NN images and aggregates their coordinates to prediction point. As decreases, a decreases, this method transforms NN.
PlaNet [22]. PlaNet is the first work posing worldwide geolocalization as classification task by dividing the globe into thousands of multi-scale geographical cells. It can combine the complex clues in images to help pinpoint the shooting location of images.
CPlaNet [22). CPlaNet tries to solve the trade-off of cell granularity. It introduces combinatorial partitioning, which creates detailed output classes by intersecting broad earth partitions, with each classifier voting for overlapping classes.
ISNs 18 ISNs combines the hierarchical information that existed in the partitionings and the photo's s scene contextual information (e.g., indoor, natural, or urban, etc.) to give the prediction.
Translocator [20]. Translocator is dual-branch transformer network that focuses the detailed clues in images and generates robust feature representations. The semantic segmentation map and the entire image will be the input to translator.
GeoDecoder 33.
GeoDecoder argues that previous work fails to exploit the detailed cues in differ- ent hierarchical levels. It proposes a cross-attention network to capture the complex relationships between different hierarchical features.
GeoCLIP [29]. GeoCLIP is based on the CLIP backbone model and first introduces GPS encoder to transform coordinates into embeddings in worldwide geolocalization tasks.
Img2Loc [43. Imo2Loc combines the RAG paradigm into worldwide geolocalization and the latest work in this field. It first retrieves similar images via visual similarity and puts the coordinates of these images into RAG prompt help generate predictions.
PIGEON (5). PIGEON introduces an innovative approach combining semantic geocell creation, multi-task contrastive pretraining, and a novel loss function, and uniquely enhances guess accuracy through retrieval over location clusters.
A.6 More Experimental Results Geo-alignment.
In this section, we conduct experiments on incorporating more fine-grained textual descriptions in Geo-alignment. Specifically, in addition including the city, county, and country information in the textual descriptions of coordinates, we also introduce neighborhood information, which the most fine ine-grained data that can be obtained from Nominatim. We use G3-N to denote this variant and keep the other hyperparameters the same as G3. The experimental results on IM2GPS3K are presented in Table 7
From the results, we can see that G3 outperforms G3-N across all metrics. This may be because the text encoder's pre-training corpus contains very few instances of neighborhood-leww information,
17

---

resulting in weaker modeling capabilities for neighborhood names. Therefore, introducing neighbor- hood information into the textual descriptions of coordinates actually adds noise, which negatively impacts the effectiveness of Geo-alignment and subsequently reduces the model's prediction accuracy.
A.7 Hard Sample and Failure Analysis
Hard Sample. In this section, give a hard sample illustrate the effectiveness of G3. Figure shows a man holding an American flag in France, the text on the left says "United States of America" French. G3 can accurately give the prediction coordinate latitude: 48.8529 and longitude: 2.3632 located in Paris, France. This demonstrates that G3 can effectively avoid the influence of text in images, thereby focusing on the location where the imgge was taken, showing strong stability.
ÉTATS-UNIS D'AMÉRIQUE
Figure 9: Hard sample.
Failure Analysis. Figure 10 presents the results of G3 predicting the coordinates of the Eiffel Tower and its replicas. G3 is able to locate the Eiffel Tower in France and its replica in the USA, but fails to distinguish the replica in China. The prediction for the replica in the USA is correct, while the prediction for the replica in China is incorrect. This may be because there are more reference objects around the tower in the USA, which reduces the difficulty of the model's ' prediction. In contrast, the lack of reference objects around the tower in China confuses G3's judgment.
Ground Truth:
Paris, France
Ground Truth:
Las Vegas, USA
Ground Truth: Hangzhou, China
Prediction:
Paris, France
Prediction:
Las Vegas, USA
Prediction:
Paris, France
Figure 10: Experimental results of G3 on predicting coordinates of Eiffel Tower and its replica.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We summarize the contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have mentioned the limitations of our work in Appendix A.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification: We have contained the formula derivation process and the experimental results to prove the assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have introduced all the details of G3 in our paper. addition, our submission also contains the materials to reproduce the main experimental results, including code, data, hyperparameters settings, etc.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
20

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include the link to the GitHub repository in our paper to provide open access to the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppbllccguuudoueo Coaee//ddeeeSbbmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Section 5 Appendix A.2
A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conducted repeated experiments and calculated the average to mitigate the variability of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included the cost information in Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We mention the socictal impact on the introduction section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our released dataset is based on public dataset MP16 and will also follow the license of MP16.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets properly eredited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Yes, we have included the descriptions in this paper and in the supplementary document.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There are no crowdsourcing experiments or research involving human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No potential risks are found in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "vC0WEk2qo1", "reviewer": "Reviewer_TChJ", "summary": "This paper proposes a RAG-based framework for worldwide geo-localization. The first Geo-alignment stage projects input images to embedding spaces to align with GPS coordinates, and text description with contrastive learning. Given new input image, the system is able to retrieve similar GPS and text description. Then the retrieved candidates GPS and text prompts are fed to GPT4V with a pre-defined prompt template to generate GPS coordination. The final stage conducts a similarity-based verification based on multi-modal representations. The method is evaluated on two worldwide geo-localization datasets, i.e., IM2GPS3k and YFCC4K, with state-of-the-art performance.", "questions": "See the weaknesses.", "limitations": "The limitation is included in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "+ The RAG-design for geo-localization is interesting and promising.\n+ The writing is easy to follow.\n+ The performance is much better than previous methods. \n+ Ablation result is provided for the three stages. \n+ The case study and failure cases are informative and interesting.", "weaknesses": "- My major concern is that the current pipeline is highly dependent on the LMM which is the powerful closed-source model, GPT-4V. This model is expensive for large-scale applications and also hard to reproduce due to unannounced updates for API across time. It would be better to provide the results with open-source large multi-modal models, for example, LLaVA. I would expect a lower accuracy with open-source models. \n- The proposed MP16-Pro dataset is also claimed as a contribution, but there is no guarantee that the data will be released. Hope this will be provided in final version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: My major concern is that the current pipeline is highly dependent on the LMM which is the powerful closed-source model, GPT-4V. This model is expensive for large-scale applications and also hard to reproduce due to unannounced updates for API across time. It would be better to provide the results with open-source large multi-modal models, for example, LLaVA. I would expect a lower accuracy with open-source models.\n\n**Response:**\n\nThanks for your suggestions. Please refer to the Open-source LMM (LLaVA) experiments section in the global response to see our response.\n\n**W2**: The proposed MP16-Pro dataset is also claimed as a contribution, but there is no guarantee that the data will be released. Hope this will be provided in final version.\n\n**Response:**\n\nThank you very much for your attention to the MP16-Pro dataset presented in our paper. The MP16-Pro dataset includes image data (360GB) and metadata (700MB). Due to anonymous file-sharing platforms' file size limitations and supplementary file size limitations, we were unable to release this data during the review stage. To further enhance the credibility of MP16-Pro, we upload 100k rows of metadata in the anonymous repository's data folder. The URL of the anonymous repository is given in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55hZv8qokh", "reviewer": "Reviewer_uTH3", "summary": "This work focuses on the task of \"worldwide geolocalization\" with an effective and adaptive framework based on large multi-Modality models. A novel framework, i.e., G3, is proposed, including Geo-alignment, Geo-diversification, and Geo-verification. This work also releases a new dataset MP16-Pro. The experiment results show that G3 has superior performance on two well-established datasets IM2GPS3k and YFCC4K.", "questions": "1. Please address my above concerns on Weaknesses.\n2. The qualitative results show that most of the retrieved images are photos for tourists. What about using real world images from official company (e.g., Google Map)?\n3. Please briefly describe how does the MP16-Pro Dataset help (or improve) the G3 model.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The task of \"worldwide geolocalization\" is very important and quite interesting.\n2. This paper is easy to follow, it is well-written.\n3. The experiment results are solid. The G3 model achieve better results than GeoCLIP and Img2Loc on IM2GPS3k and YFCC4K.", "weaknesses": "1. In the Geo-diversification part, there is no ablation study on different LMMs and different RAG templates. I wonder whether it still works on open-source LLMs.\n2. Minor:\n\n    a. Figure 1 is often set as a teaser figure, which could show the basic design of the whole work. It would be better to indicate the solution, instead of only showing the limitations.\n\n    b. Table 1 needs citations for each previous work. And GeoCLIP should be noted as NeurIPS 2023 instead of arXiv.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: In the Geo-diversification part, there is no ablation study on different LMMs and different RAG templates. I wonder whether it still works on open-source LLMs.\n\n**Reponse:**\n\t\nThanks for your advice. \nFor the ablation study on different LMMs, please refer to the Open-source LMM (LLaVA) experiments section in the global response. \nRegarding the RAG templates, our template follows the previous work Img2Loc to ensure a fair comparison. We agree that exploring RAG templates is necessary. So we conduct experiments to explore the impact of incorporating different reference images' information into the RAG template on prediction performance. The experimental variants and results are as follows:\n\n* ZS: Zero-shot template, RAG template without any information from reference images.\n* GPS: RAG template incorporating the GPS coordinates of reference images.\n* GPS+Text: RAG template incorporating both the GPS information and textual descriptions of reference images.\n\n|   Variants   |   Street 1km   |   City 25km   |  Region 200km  | Country 750km | Continent 2500km |\n|:------------:|:--------------:|:-------------:|:--------------:|:-------------:|:----------------:|\n|      ZS      |     12.41      |     35.87     |      50.88     |     67.60     |       80.75      |\n|      GPS     |     16.65      |     40.94     |      55.56     |     71.24     |       84.68      |\n|  GPS+Text    |     16.75      |     41.44     |      55.68     |     71.60     |       85.01      |\n\nThe experimental results indicate that adding more modality information from the reference images to the RAG template can effectively improve prediction accuracy. We will include these experimental results in our final version. Thank you again for your assistance.\n\n**W2**: Figure 1 is often set as a teaser figure, which could show the basic design of the whole work. It would be better to indicate the solution, instead of only showing the limitations.\n\n**Reponse:**\n\nThank you for your comment. We understand that teaser figures are typically used to present the basic design of the entire work. We would like to clarify that our intention with the current figure was to highlight the issues with existing methods and help readers understand our motivation. In response to your suggestion, we will revise Figure 1 to include an overview of our solution, making it easier to comprehend our approach. Thank you for your valuable feedback.\n\n**W3**: Table 1 needs citations for each previous work. And GeoCLIP should be noted as NeurIPS 2023 instead of arXiv.\n\n**Reponse:**\n\nThanks for your comment. We will add citations for each previous work in Table 1. Additionally, we will update the reference for GeoCLIP to NeurIPS 2023 instead of arXiv and check the other references. We appreciate your attention to these details.\n\n**Q1**: The qualitative results show that most of the retrieved images are photos for tourists. What about using real world images from official company (e.g., Google Map)?\n\n**Reponse:**\n\nThank you for your insightful question. Adding more images from official companies could indeed enhance the robustness of the database, thereby improving the overall effectiveness of the pipeline. However, we used MP16 as the image source for this work for three main reasons:\n\n* Consistency: To maintain consistency with other works, such as GeoCLIP and Img2Loc, which also use MP16 as their data source.\n* Dataset Distribution Rights: We extended the text descriptions based on MP16 to create the new dataset MP16-Pro, which includes distribution attributes. In contrast, Google's street view images have clear distribution restrictions, so we did not include them in this work.\n\nYour insights are thought-provoking, and we greatly appreciate your constructive feedback.\n\n**Q2**: Please briefly describe how does the MP16-Pro Dataset help (or improve) the G3 model.\n\n**Reponse:**\n\nThanks for your comment. \nThe original MP16 dataset provides image data along with the GPS information of the image, such as: image 4f/a0/3963216890.jpg, LAT: 47.217578, LON: 7.542092. The MP16-Pro dataset enhances MP16 by adding textual descriptions about the location for each sample. \n\nThe enhanced sample becomes: image 4f/a0/3963216890.jpg, LAT: 47.217578, LON: 7.542092, **neighbourhood: Wengistein, city: Solothurn, county: Amtei Solothurn-Lebern, state: Solothurn, region: NA, country: Switzerland, country_code: ch, continent: NA**. \n\nMP16-Pro primarily improves G3 through the following aspects:\n* Adding Geographic Semantic Information: By incorporating additional geographic semantic information in geo-alignment, aligning it with GPS and image data, which makes the geographic information of samples in MP16 more accurate.\n* Enhancing Image Retrieval Accuracy: More precise geographic information enhances the accuracy of images retrieved based on the query image in geo-diversification, providing more effective references for the LMM to predict coordinates. For specific details, please refer to Section 5.5 Case Study on reference image retrieval.\n* Help Train the GPS Encoder: Introducing textual descriptions and aligning them with GPS information in geo-alignment helps train the GPS encoder more effectively, further increasing the reliability of the GPS encoder's judgments during geo-verification.\n\nThank you again for taking the time to review our work, and we hope our response addresses your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dTebJqfiUU", "reviewer": "Reviewer_YB9n", "summary": "This paper proposes three steps, i.e., geo-alignment, geo-diversification, and geo-verification to optimize both retrieval and generation phases of word-wide geo-localization.", "questions": "1. What is the purpose of image vectorization? Please provide a detailed explanation by the author.\n2.Figure 6 is difficult to understand, what does the author want to express? What does it mean that the number of references has a significant impact on the model?\n3. The model seems too large, the author should provide the number of parameters and gflops experiments.\n4. The author proposed a new dataset MP16 Pro, but it seems that the results have not been published on this dataset.\n5. The author obtain textual descriptions by geographical reverse encoding during the Database Construction. Are the descriptions generated for longitudes and latitudes with similar geographical locations consistent? Is the text description useful for images taken at similar locations? The author can add descriptive text experiments to the ablation experiment to prove that the text is helpful for feature representation.\n6. The experiment of paper lacks the results of baseline\n7. the author does not introduce enough details about the branch of  Geo-diversification Module.\n8. Fig.5/8 lacks a comparison with the visualization results of baseline.", "limitations": "The authors discuss the limitation in the paper, but not in enough depth.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clearly stated.\n2. The experimental results show the effectiveness of the proposed method.\n3. The proposed method achieves state-of-the-art performance.\n4. Code is publicly available.", "weaknesses": "The experiments are not insufficient. \nThe model seems too large, so the author should provide the number of parameters and gflops experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: The experiments are not insufficient. The model seems too large, so the author should provide the number of parameters and gflops experiments.\n\n**Response:**\n\nThanks for your question. We compile the data of model's parameters and computational load, as shown in the table below.\n\n| Total params   | Trainable params     | GFLOPs |\n|:--------------:|:--------------------:|:------:|\n| 441,266,179    | 13,648,131 (3.09%)   | 304.38 |\n\nSince the training parameters of G3 are concentrated in the geo-alignment stage, and most of the module parameters are frozen, only 3.09% of the parameters need to be optimized. \n\nThank you again for your suggestion, we will include this data in the final version of the paper.\n\n**Q1**: What is the purpose of image vectorization? Please provide a detailed explanation by the author.\n\n**Response**:\n\nThank you for your question. Image vectorization enables us to convert images into vector representations. These vectors allow us to calculate the similarity between a query image and images stored in our database. By doing so, we can efficiently retrieve reference images in RAG process that aid in accurately generating the geographic location of the query image.\n\n**Q2**: Figure 6 clarification.\n\n**Response**:\n\nThank you for your question. \nIn Figure 6, the left side of each example shows the query image, with the ground truth displayed in red text below it. On the right side are the predictions made by LMM using the same RAG template combined with different amounts of reference image information, with the most accurate prediction highlighted in red text. The blue reference text below indicates the coordinates of the top 10 most similar reference images retrieved from the database based on the query image.\n\nIn Figure 6, we aim to illustrate that introducing different numbers of coordinates as references during the RAG process significantly impacts the accuracy of the generated results. This is due to the heterogeneity of images in geographic space, which causes the significantly various number of effective reference images retrieved from the database for different query images, further affecting the RAG outcomes. This case study validates the necessity of geo-diversification. Geo-diversification mitigates the prediction sensitivity caused by the heterogeneous spatial distribution of images by generating candidate coordinates using multiple prompts that contain varying numbers of reference coordinates simultaneously. We will add more detailed descriptions of Figure 6 in its Caption.\n\n**Q3**: The author proposed a new dataset MP16 Pro, but it seems that the results have not been published on this dataset.\n\n**Response**:\n\nThank you for the opportunity to clarify our perspective. In the paper, the MP16 Pro dataset was used solely for training and constructing the database, while all testing was conducted on the Img2GPS3K and YFCC4K datasets. This approach is consistent with existing works such as GeoCLIP and Img2Loc.\n\n**Q4**: Questions about textual descriptions.\n\n**Response**:\n\nThanks for your questions. (1) Generating consistency. We perform geographic reverse geocoding using Nominatim. This process is error-free and is equivalent to directly inputting latitude and longitude coordinates into OpenStreetMap to obtain address information. (2) The effectiveness of text descriptions. \n\nplease refer to the Ablation study of Geo-alignment section in the global response.\n\n**Q5**: The experiment of paper lacks the results of baseline\n\n**Response**:\n\nThanks for your advice. We will further add the results of PIGEON to the overall results. The comparison between G3(GPT4V) and PIGEON on IM2GPS3K and YFCC4K are shown below:\n\n**IM2GPS3K**\n\n|     Model     |  Street 1km  |  City 25km  | Region 200km | Country 750km | Continent 2500km |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|PIGEON|11.3|36.7|53.8|72.4|85.3|\n|G3(GPT4V)|16.65|40.94|55.56|71.24|84.68|\n\n\n**YFCC4K**\n\n|     Model     |  Street 1km  |  City 25km  | Region 200km | Country 750km | Continent 2500km |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|PIGEON|10.4|23.7|40.6|62.2|77.7|\n|  G3(GPT4V)|23.99|35.89|     46.98    |     64.26     |       78.15      |\n\n\nFrom the results, we can find that G3 is superior to PIGEON on eight out of ten metrics on IM2GPS3K and YFCC4K. We will add the performance of PIGEON to the overall results in our final version.\n\n**Q6**: the author does not introduce enough details about the branch of Geo-diversification Module.\n\n**Response**:\n\nThanks for your advice. The purpose of Geo-diversification is to generate diverse predictions based on RAG prompts combined with different numbers of references. This approach aims to address the issue of inconsistent numbers of effective references retrieved due to spatial heterogeneity in images. In Geo-diversification, we combine the top S retrieved candidates with $K \\times N$ generated candidates. K represents the number of RAG prompts, and N represents the number of results generated per RAG prompt. Therefore, Geo-diversification produces a total of $K \\times N + S$ candidate predictions.\n\n**Q7**: Fig.5/8 lacks a comparison with the visualization results of baseline.\n\n**Response**:\n\nThanks for your advice. \n\nIn Figure 5, the CLIP ViT on the left represents the baseline, while the right side shows the retrieval results of G3. It can be observed that CLIP ViT focuses only on the visual features of the image (such as the presence of two people), whereas G3 pays more attention to the location where the image was taken. As a result, G3 can retrieve images that are geographically closer to the query image. \n\nIn Figure 8, we aim to visually demonstrate the performance of G3 under different error bounds. We can observe that the model's predictions are more accurate when the image contains clear geographic indicators such as buildings and decorations. However, when the image is filled with elements like the ocean or sky, which lack clear geographic indicators, the prediction error is larger.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgpCKlNxdQ", "reviewer": "Reviewer_tzbF", "summary": "This paper introduces G3, a RAG framework for geo-localization. By introducing a three-step process, the G3 framework achieves superior performance against other SoTA methods. To improve the expressiveness of the image embeddings, the paper proposed a new dataset, MP16-Pro, which adds textual descriptions to the existing MP-16 dataset. By comprehensive experiments, the paper demonstrates the necessity and merits of the G3 framework.", "questions": "1. Since the proposed dataset, MP16, is relatively large, I wonder if the authors have run decomtamination procedures to ensure there is no overlapping data between training and evaluation.\n2. The authors of Img2Loc provided experiment results with other LMMs (LLaVA), I wonder what is G3’s performance when switching the LMM to LLaVA compared to Img2Loc. I think by incorporating this result the authors can more robustly state their superiority over existing methods.\n3. As shown in Figure 2, the text descriptions of the location in MP16-Pro are not used during inference, I wonder if adding text descriptions to the prompt would work better.", "limitations": "The authors of this paper have addressed the limitation of the G3 framework by pointing out its high computation cost. The introduction of alignment and diversification brings on more computation cost and latency compare to existing methods, which limits the retrieval and inference speed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed method, G3, achieved competitive performance in geo-localization against existing classification-based, retrieval-based and RAG-based methods.\n2.\tCompared to the original MP16 dataset, the proposed MP16-Pro dataset additionally provides textual geolocation-related data, which could be beneficial to the community if fully open-sourced.\n3.\tComprehensive experiments and explanations are provided to prove the effectiveness and necessity of each G3 component.\n4.\tThe authors have open-sourced their project in a clear and instructive manner, which is very positive for reproduction.", "weaknesses": "1.\tThe necessity of the Geo-alignment module in the G3 framework has been indicated by experiments results in Table 2. However, the authors did not address the motivation for their particular design choice of the alignment module. Why do the image features have to align with both text features and gps features? Does aligning with simply one modality work just as well? The authors should clarify that by conducting the corresponding ablation study.\n2.\tOne of the main contributions claimed by the authors is the introduction of the MP16-Pro dataset. However, the description of the construction process for the MP16-Pro dataset is not sufficiently detailed.\n3.\tThe G3 framework was not the first work to incorporate RAG into geolocalization, nor was it the first work to use retrieval-based models. While the proposed method achieved superior performance, it lacks certain novelty to the field.\n4.\tIn the experiment setup in section 5(line 226), no retrieved coordinate is considered when evaluating G3 on IM2GPS3K, which is inconsistent to figure 2 of the paper. If no retrieved coordinate works better in certain cases, I wonder about the necessity and applicability of such design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: The necessity of the Geo-alignment module in the G3 framework has been indicated by experiments results in Table 2. However, the authors did not address the motivation for their particular design choice of the alignment module. Why do the image features have to align with both text features and gps features? Does aligning with simply one modality work just as well? The authors should clarify that by conducting the corresponding ablation study.\n\n**Response**:\n\nThank you for your suggestion. \n\nFirstly, from an intuitive perspective, GPS and text data can model continuous and discrete geographic information. Specifically, two cities located on a national border belong to different countries and may have inconsistent driving directions. This is suitable for alignment using text information and image data. On the other hand, their terrain, landscape, and climate are generally similar, which can be expressed by aligning GPS data with image data.\n\nSecond, We also conduct experiments to clarify the necessity of aligning image features with both text features and gps features. Please refer to the Ablation study of Geo-alignment section in the global response. We will also add these results in our final version paper.\n\n**W2**: One of the main contributions claimed by the authors is the introduction of the MP16-Pro dataset. However, the description of the construction process for the MP16-Pro dataset is not sufficiently detailed.\n\n**Response**:\n\nThanks for your comment. \n\nThe original MP16 dataset provides image data along with the GPS information of the image, such as: image 4f/a0/3963216890.jpg, LAT: 47.217578, LON: 7.542092. The MP16-Pro dataset enhances MP16 by adding textual descriptions about the location for each sample. \n\nThe enhanced sample becomes: image 4f/a0/3963216890.jpg, LAT: 47.217578, LON: 7.542092, **neighbourhood: Wengistein, city: Solothurn, county: Amtei Solothurn-Lebern, state: Solothurn, region: NA, country: Switzerland, country_code: ch, continent: NA**. \n\nIt is also worth noting that we perform geographic reverse geocoding using Nominatim. This tool is error-free and is equivalent to directly inputting latitude and longitude coordinates into OpenStreetMap to obtain address information. We will add these details of MP16-Pro construction in our final version.\n\n**W3**: The G3 framework was not the first work to incorporate RAG into geolocalization, nor was it the first work to use retrieval-based models. While the proposed method achieved superior performance, it lacks certain novelty to the field.\n\n**Response**:\n\nThanks for the opportunity for us to clarify the motivation and contribution of G3. Our work builds upon img2loc, but img2loc and other existing approaches suffer from two serious issues: they may easily confuse distant images with similar visual contents, or cannot adapt to various locations worldwide with different amounts of relevant data. To address these issues, we propose G3 and achieve significant performance improvements. Additionally, to further advance the field, we introduce the MP16-Pro dataset, which supplements the original MP16 dataset with geographic text descriptions for each sample.\n\n**W4**: Necessity of retrieved coordinate.\n\n**Response**:\n\nThank you for your comment. The setting of retrieved coordinates is to ensure the lower bound of the pipeline's effectiveness, especially when the LMM model's capabilities are not strong enough. We conduct experiments on the LLaVA model on IM2GPS3K with the following variants:\n* LLaVA S=0: LLaVA variant without retrieved coordinates.\n* LLaVA S=1: LLaVA variant with one retrieve coordinate.\n* LLaVA S=2: LLaVA variant with two retrieved coordinates.\n\n|Model| Street 1km | City 25km | Region 200km | Country 750km | Continent 2500km |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| LLaVA S=0|14.15|35.57|48.78|66.53|81.68|\n| LLaVA S=1|14.41|35.64|48.88|66.40|81.55|\n| LLaVA S=2|14.31|35.87|49.42|66.93|81.78|\n\nFrom the experimental results, we can see the necessity of retrieved coordinates in the LLaVA experiments. As the number of retrieved coordinates increases from 0 to 2, all metrics generally show an improvement.\n\n**Q1**: Since the proposed dataset, MP16, is relatively large, I wonder if the authors have run decomtamination procedures to ensure there is no overlapping data between training and evaluation.\n\n**Response**:\n\nThank you for your insightful question. The dataset we use, MP16, is consistent with those used in prior works in this domain. This consistency ensures comparability and reproducibility of our results. \n\n**Q2**: LMM Ablation Study.\n\n**Response**:\n\nThanks for your advice. We conduct the experiments with LLaVA on IM2GPS3K, please refer to the Open-source LMM (LLaVA) experiments Section in the global response.\n\n**Q3**: As shown in Figure 2, the text descriptions of the location in MP16-Pro are not used during inference, I wonder if adding text descriptions to the prompt would work better.\n\n**Response**:\n\nThank you for your insightful suggestion. Based on your recommendation, we design the following experiments. The experimental variants and results are as follows:\n* ZS: Zero-shot template, RAG template without any information from reference images.\n* GPS: RAG template incorporating the GPS coordinates of reference images.\n* GPS+Text: RAG template incorporating both the GPS information and textual descriptions of reference images.\n\n|Variants|Street 1km|City 25km|Region 200km|Country 750km|Continent 2500km|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|ZS|12.41|35.87|50.88|67.60|80.75|\n|GPS|16.65|40.94|55.56|71.24|84.68|\n|GPS+Text|16.75|41.44|55.68|71.60|85.01|\n\nThe experimental results indicate that adding more various modality information from the reference images to the RAG template can effectively improve prediction accuracy. We will include these experimental results in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cx54tAZ1n2", "reviewer": "Reviewer_CzZb", "summary": "In the paper, authors propose a novel framework, G3, for worldwide geolocalization of a given photograph anywhere on Earth. The authors address the challenges of capturing location-specific visual cues and handling variations in image data distribution across the globe. G3 utilizes a three-step process: Geo-alignment, which learns location-aware image representations, Geo-diversification, which employs multiple retrieval-augmented prompts for robust location prediction, and Geo-verification, which combines retrieved and generated location data for final prediction. The authors also introduce the MP16-Pro dataset to support location-aware visual representation learning. Experiments on the IM2GPS3k and YFCC4K datasets demonstrate the superiority of G3 over existing methods.", "questions": "* There is only a marginal improvement in performance when including the geo-diversification step considering it is potentially the most expensive step during the inference.\n* Currently, the text associated with each coordinate only includes the country and city labels. Will the performance of the framework improve by including fine-grained details such as region and/or street name?", "limitations": "Limitations are included but failure cases are missing.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* All the modules in the G3 framework: Geo Alignment, Geo Diversification and Geo Verification seem logical and rational. Three kinds of embedding coming from the vision encoder are used for retrieval. LLM is used to generate a set of plausible coordinates by providing positive and negative examples.\n* The method achieves superior performance over several baselines at various levels of granularity on IM2GPS3k and YFCC4K.\n* Overall, the method is interesting and novel, the writing and flow of the paper is meaningful.", "weaknesses": "* The only limitation discussed is regarding the efficiency of inference. However, there is no mention of how much compute time and memory (in number) is required to geo-localize a given input image. \n* There are no concrete qualitative example of failures reported in the paper. Can the system be fooled easily? For example, if an image from Italy contains a human with a flag of The Netherlands, is the system capable of correctly geolocalizing the image? How does the RAG system along with the LLM perform in such a case?\n* Limited evaluation considering the state-of-the-art. No mention of recent works such as Pigeon [1] or a GeoReasoner [2].\n* Why did the authors choose to use CLIP vision encoder for extracting image features? Recent works have shown that purely image-based pretrained models such as DINO-v2 are better feature extractors than CLIP. No ablation study done for the choice of the vision encoder.\n* Overall, from discussion in **L316-L333** and Figure 4, it looks like the number of references provided to LLM highly depends and varies based on the image content. The performance is highly sensitive to this hyperparameter and a single value cannot guarantee optimal performance. This can make the framework highly unreliable for practical use cases.\n\n[1] Haas, Lukas, Michal Skreta, Silas Alberti, and Chelsea Finn. \"Pigeon: Predicting image geolocations.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12893-12902. 2024\n\n[2] Li, Ling, Yu Ye, Bingchuan Jiang, and Wei Zeng. \"GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Model.\" In Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: No mention of how much compute time and memory (in number) is required to geo-localize a given input image.\n\n**Response**:\n\nThanks for your comment. We gather the compute time and memory cost in Geo-diversification and Geo-verification, as Geo-alignment is not directly used in inference. We will add this information in our final version. The statistics are listed as follows:\n\n|Phase|Time cost|Memory cost|\n|:-:|:-:|:-:|\n|Geo-diversification|LLaVA generating time: 4s/10 prompts; Loading Model: 10s|GPU Memory: 20.81GB; Memory: 6GB|\n|Geo-verification|Evaluating time on IM2GPS3K: 56s|GPU Memory: 8.85GB; Memory: 10.37GB|\n\n**W2**: Hard and failure cases analysis.\n\n**Response**:\n\nThanks for your interesting question. \n\nWe identify an image similar to the situation you mentioned, depicting a man in Paris holding an American flag. **Detailed images can be found in the PDF file in global response**. Upon testing, G3 accurately predicts the location as Paris without being fooled. \n\nAdditionally, we use images from the failure analysis section of GeoReasoner concerning the Eiffel Tower for testing, which are also included in the PDF. We find that G3 can accurately identify the replica of the Eiffel Tower in the USA but fails to recognize the replica in China, which is better than GeoReasoner. This discrepancy may be due to the presence of more iconic buildings in the images of the USA replica, which aid in location determination, whereas the images of the Chinese replica lack clear geographic indicators, leading to incorrect predictions by the model. All these results and analyses will be included in our formal version.\n\n**W3**: Limited evaluation considering the state-of-the-art. No mention of recent works such as Pigeon [1] or a GeoReasoner [2].\n\n**Response**:\n\nThank you for your comment. We present a comparison of the performance of G3 and PIGEON on the commonly used datasets. There are two reasons we do not take GeoReasoner as a baseline: \n1. GeoReasoner is specifically fine-tuned for country classification, which differs  from our focus on worldwide geolocalization. \n2. GeoReasoner conduct experiments on filtered IM2GPS3K dataset, which includes only highly locatable data. Therefore, the experimental results in the GeoReasoner paper are not comparable.\n\nThe results of PIGEON and G3 are shown below:\n\n**IM2GPS3K**\n\n|     Model     |  Street 1km  |  City 25km  | Region 200km | Country 750km | Continent 2500km |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|PIGEON|11.3|36.7|53.8|72.4|85.3|\n|G3(GPT4V)|16.65|40.94|55.56|71.24|84.68|\n\n\n**YFCC4K**\n\n|     Model     |  Street 1km  |  City 25km  | Region 200km | Country 750km | Continent 2500km |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|PIGEON|10.4|23.7|40.6|62.2|77.7|\n|  G3(GPT4V)|23.99|35.89|46.98|64.26|78.15|\n\nFrom the experimental results, we can find that G3 demonstrates superior performance across most metrics. Additionally, it is worth noting that PIGEON, during its training process, incorporates not only the MP16 dataset but also an additional 340k images from the Google Landmark v2 dataset, utilizing a larger training dataset compared to our work. \nThanks again for your advice, we will include these discussions in our final version.\n\n**W4**: Why using CLIP Vision encoder.\n\n**Response**:\n\nThanks for your questions. In this paper, we use CLIP's vision encoder to align our work with previous studies such as GeoCLIP and Img2Loc, ensuring the results are comparable. \n\n\n**W5**: Overall, from discussion in L316-L333 and Figure 4, it looks like the number of references provided to LLM highly depends and varies based on the image content. The performance is highly sensitive to this hyperparameter and a single value cannot guarantee optimal performance. This can make the framework highly unreliable for practical use cases. \n\n**Response**:\n\nThank you for your question. \n\nAs mentioned in lines L316-L333, the number of references provided to the LLM highly depends on and varies based on the image content. You are correct. This indicates that the heterogeneity in the spatial distribution of images can make LMM's predictions highly sensitive. To address this issue, we propose Geo-diversification. By combining RAG templates with different numbers of reference coordinates, we comprehensively consider all candidates to enhance the robustness of the model's predictions.\n\nFigure 4 illustrates the trade-off between different levels of prediction accuracy. If the application scenario requires high accuracy for small-scale predictions, a smaller number of candidates for each RAG prompt should be chosen. Conversely, if the scenario requires high accuracy for large-scale predictions, a larger number of candidates for each RAG prompt is preferable. Overall, selecting 5 as the hyperparameter provides a balanced performance, achieving optimal results at the region, country, and continent levels, and near-optimal results at the street and city levels.\n\n**Q1**: There is only a marginal improvement in performance when including the geo-diversification step considering it is potentially the most expensive step during the inference.\n\n**Response**:\n\nThanks for your comment. Through the ablation study in Table 2, we observe that removing Geo-diversification results in a significant drop in performance. On the Im2GPS3K dataset, the average accuracy across five scales decreases by 2.14%, and on the YFCC4K dataset, the average accuracy across five metrics drops by 8.28%. These results demonstrate the necessity and effectiveness of Geo-diversification.\n\n\n**Q2**: Including fine-grained details in Geo-alignment.\n\n**Response**:\n\nThank you for your question. Exploring the impact of finer-grained text descriptions on performance requires rerunning the Geo-alignment, which takes approximately 70 hours of training time. We have not yet completed this experiment, but we will provide the results during the discussion phase next week.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

We would like to extend our sincere gratitude to all the reviewers for your valuable comments and constructive feedback on our manuscript. Your insights have been instrumental in improving the quality of our work.

We find two common concerns raised by multiple reviewers, which we response through this global response.

## Open-source LMM (LLaVA) experiments

We conduct the experiments of G3 with LLaVA (LLaVA-Next-LLaMA3-8b) on Im2GPS3K. Since Img2Loc did not specify the version of LLaVA they used, we also re-ran the experiments on Img2Loc. The results are as follows:

| Model       | Street 1km | City 25km  | Region 200km | Country 750km | Continent 2500km |
|:-:|:-:|:-:|:-:|:-:|:-:|
| GeoCLIP     | 14.11  | 34.47  | 50.65  | 69.67   | 83.82     |
| Img2Loc (LLaVA)     | 10.21  | 29.06  | 39.51  | 56.36   | 71.07     |
| Img2Loc (GPT4V)     | 15.34  | 39.83  | 53.59  | 69.70   | 82.78     |
| G3(LLaVA)   | 14.31  | 35.87  | 49.42  | 66.93   | 81.78     |
| G3(GPT4V)   | 16.65  | 40.94  | 55.56  | 71.24   | 84.68     |

We can find that:

* After switching LMMs from GPT4V to LLaVA, the performance of G3 shows some decline across various metrics but remains competitive: compared to GeoCLIP, it performs better at the street level and city level. 
* Additionally, compared to Img2Loc(LLaVA), G3(LLaVA) significantly outperforms Img2Loc(LLaVA), demonstrating the effectiveness of the proposed modules. 
* Finally, by comparing the performance of G3 equipped with LLaVA and GPT4V to Img2Loc equipped with LLaVA and GPT4V, we can observe that G3 shows more stable performance across different LMMs.

## Ablation study of Geo-alignment

To verify the necessity of aligning the three modalities in geo-alignment, we conduct experiments on the following variants.
* IMG: Directly using pretrained CLIP vision encoder as encoder.
* IMG+GPS: Aligning Image representations with GPS representations in Geo-alignment, the textual descriptions are not used.
* IMG+GPS+Text (G3): Aligning three modalities simultaneously in Geo-alignment.

| Model       | Street 1km | City 25km  | Region 200km | Country 750km | Continent 2500km |
|:-:|:-:|:-:|:-:|:-:|:-:|
| IMG                  | 15.71  | 40.64  | 54.85  | 70.8    | 84.05     |
| IMG+GPS              | 16.91  | 41.41  | 55.02  | 70.94   | 84.18     |
| IMG+GPS+Text (G3)    | 16.65  | 40.94  | 55.56  | 71.24   | 84.68     |

From the experimental results, we can draw the following conclusions:

* By comparing IMG+GPS+Text, IMG+GPS, and IMG, we find that adding GPS and text information can both enhance the feature representation compared to using the original image information alone.
* By comparing IMG+GPS+Text with IMG+GPS, we find that IMG+GPS performs better at smaller scales, while IMG+GPS+Text performs better at larger scales. This might be because GPS is suitable for modeling variations at smaller scales, whereas text descriptions do not vary significantly at small scales and may even remain the same.

We hope these responses address your concerns. Thank you once again for your time and effort in reviewing our manuscript.

Meta-review: This paper tackles the challenging problem of worldwide geo-localization by introducing a new dataset, MP16 Pro, and a novel retrieval-augmented generation framework, G3, which leverages recent LLMs to explore diverse clues such as retrieved images, GPS data, and textual information. The reviewers engaged actively in the discussion and reached a consensus with positive evaluations. The AC concurs with the reviewers' assessment that this paper addresses an important problem and offers a novel solution. The authors are encouraged to incorporate the additional feedback provided during the rebuttal and discussion period. Additionally, it is recommended that they include details on the number of trainable parameters and GFLOPs of the comparison algorithms, and ensure that the proposed dataset and code are made publicly available as promised.
Decision: Accept (poster)