Swarm Intelligence in Geo-Localization: A Multi-Agent Large Vision-Language Model Collaborative Framework

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:19:35

Abstract: Visual geo-localization demands in-depth knowledge and advanced reasoning skills to associate images with real-world geographic locations precisely.
In general, traditional methods based on data-matching are hindered by the impracticality of storing adequate visual records of global landmarks.
Recently, Large Vision-Language Models (LVLMs) have demonstrated the capability of geo-localization through Visual Question Answering (VQA), enabling a solution that does not require external geo-tagged image records. However, the performance of a single LVLM is still limited by its intrinsic knowledge and reasoning capabilities.
Along this line, in this paper, we introduce a novel visual geo-localization framework called smileGeo that integrates the inherent knowledge of multiple LVLM agents via inter-agent communication to achieve effective geo-localization of images. 
Furthermore, our framework employs a dynamic learning strategy to optimize the communication patterns among agents, reducing unnecessary discussions among agents and improving the efficiency of the framework.
To validate the effectiveness of the proposed framework, we construct GeoGlobe, a novel dataset for visual geo-localization tasks. Extensive testing on the dataset demonstrates that our approach significantly outperforms state-of-the-art methods.
The source code is available at https://anonymous.4open.science/r/ViusalGeoLocalization-F8F5/ and the dataset will also be released after the paper is accepted.

Main paper: 
 introduction: 
 Visual geo-localization, referred to the task of estimating geographical identification for a given image, is vital in various fields such as human mobility analysis [1, 2, 3, 4, 5] and robotic navigation [6, 7, 8, 9, 10, 11] . In general, accurate visual geo-localization without the help of any localization equipment (e.g., GPS sensors) is a complex task that requires abundant geospatial knowledge and strong reasoning capabilities. Traditional methods [12, 13, 14, 15] typically formulate it as an image retrieval problem where to geo-localize the given image by retrieving similar images with known geographical locations. Thus, their effectiveness is limited by the scope and quality of the geo-tagged image records.Recently, the success of Large Vision-Language Models (LVLMs) has enabled Visual Question Answering (VQA) to become a unified paradigm for multi-modal problems [16, 17] , providing a novel solution for visual geo-localization without the need for external geo-tagged image records.However, the performance of a single LVLM on the geo-localization task is still limited by its inherent geospatial knowledge and reasoning capabilities. Along this line, in this paper, we introduce a novel multi-agent framework, named swarm intelligence Geo-localization (smileGeo), which aims to adaptively integrate the inherent knowledge and reasoning capabilities of multiple LVLMs Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.to effectively and efficiently geo-localize images. Specifically, for a given image, the framework initially elects K suitable LVLM agents as answer agents for initial location analysis. Then, each answer agent chooses several review agents via an adaptive social network, which imitates the collaborative relationships between agents with a target on the visual geo-localization task, to discuss and share their knowledge for refining its location analysis. Finally, our framework conducts free discussion among all of the answer agents to reach a consensus. Besides, we also design a novel dynamic learning strategy to optimize the election mechanism along with the adaptive collaboration social network of agents. We hope that by the effectiveness of the election mechanism and the review mechanism, our framework can discover the mode of communication among agents, thereby enhancing geo-localization performance through multi-agent collaboration while minimizing unnecessary discussions. In summary, our contributions are demonstrated as follows:• A novel swarm intelligence geo-localization framework, smileGeo, is proposed to adaptively integrate the inherent knowledge and reasoning capability of multiple LVLMs through discussion for visual geo-localization tasks.• A dynamic learning strategy is introduced to discover the most appropriate discussion mode among LVLM agents for enhancing the effectiveness and efficiency of the framework.• A new visual geo-localization dataset named GeoGlobe1 is collected, containing a wide variety of images globally. The diversity and richness of GeoGlobe allow us to evaluate the performance of different models more accurately. Moreover, extensive experiments demonstrate our competitive performance compared to state-of-the-art methods.The remainder of this paper is organized as follows: Section 2 discusses the related literature. In Section 3, the proposed framework is introduced. Section 4 provides the performance evaluation, and Section 5 concludes the paper. 
related work: 
 Visual Geo-localization. Recent research in visual geo-localization, commonly referred to as geo-tagging, primarily focuses on developing image retrieval systems to address this challenge [3, 18, 19, 20, 21, 22] . These systems utilize learned embeddings generated by a feature extraction backbone, which includes an aggregation or pooling mechanism [23, 24, 25, 26] . However, the applicability of these retrieval systems to globally geo-localize landmarks or natural attractions is often limited by the constraints of the available database knowledge and the restrictions imposed by national or regional geo-data protection laws. Alternatively, some studies treat visual geo-localization as a classification problem [27, 28, 29, 30] . These approaches posit that two images from the same geographical region, despite depicting different scenes, typically share common semantic features.Practically, these methods organize the geographical area into discrete cells and categorize the image database accordingly. This cell-based categorization facilitates scaling the problem globally, provided the number of categories remains manageable. However, while the number of countries globally remains relatively constant, accurately enumerating cities in real-time at a global scale is challenging due to frequent administrative changes, such as city reorganizations or mergers, which reflect shifts in national policies. Additionally, in the context of globalization, this strategy has inherent limitations. The recent advent of LVLMs offers promising compensatory mechanisms for the deficiencies observed in traditional geo-localization methodologies, making the exploration of LVLM-based approaches significantly relevant in current research.Multi-agent Framework for LLM/LVLMs. LLM/LVLM agents have demonstrated the potential to act like human [31, 32, 33] , and a large number of studies have focused on developing robust architectures for collaborative LLM/LVLM agents [34, 35, 36, 37, 38] . These architectures enable each LLM/LVLM agent that endows with unique capabilities to engage in debates or discussions.For instance, [34] proposes an approach to aggregate multiple LLM/LVLM responses by generating candidate responses from various LLM/LVLM in a single round and employing pairwise ranking to synthesize the most effective response. While some studies [34] utilize a static architecture potentially limiting the performance and generalization of LLM/LVLM, others like [38] have implemented dynamic interaction architectures that adjust according to the query and incorporate user feedback.Recent advancements also demonstrate the augmentation of LLM/LVLM as autonomous agents capable of utilizing external tools to address challenges in interactive settings. These techniques include retrieval augmentation [39, 40, 41] , mathematical tools [40, 42, 43] , and code interpreters [44, 45] . With these capabilities, LLM/LVLMs are well-suited for various tasks, especially for geo-localization. However, most LLM/LVLM agent frameworks mandate participation from all agents in at least one interaction round, leading to significant computational overhead. To address this issue, our framework introduces a dynamic learning strategy electing only a small number of agents to geo-localize different images, which significantly enhances the efficiency of LLM/LVLM agents by reducing unnecessary interactions.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: At present, the LVLM agent framework we proposed can only search the Internet autonomously. Our agent still has shortcomings in the use of other multiple tools.We stated in our future outlook that our follow-up work will solve this problem. 
methodology: 
 In this section, we first present the overall framework and then introduce each part of smileGeo in detail for geo-localization tasks.In this paper, we denote the social network of LVLM agents by G, where G = {V, E}.V stands for the agent set and E presents the edge set. Each agent v i ∈ V, i ∈ [N ] is an LVLM, which is pre-trained by massive vision-language data and can infer the possible location Y of a given image X. Besides, each edge e ij ∈ E, i, j ∈ [N ] is the connection weighted by the improvement effect of agent v i to agent v j via discussion regarding the geo-localization performance.As illustrated in Figure 1 , smileGeo contains the process of the review mechanism in agent discussions along with a dynamic learning strategy of agent social networks:The review mechanism in agent discussions is a 3-stage anonymous collaboration approach to allow LVLM agents to reach a consensus via discussion. In the first stage, for a given image X, our framework elects the most suitable K agents as answer agents by agent election probability Lst. In the second stage, these answer agents respectively select R review agents by the adaptive collaboration social network A to refine their answer via discussion. Finally, our framework facilitates consensus among all agents through open discussion to reach a final answer. Both Lst and A are analyzed from the given image X, allowing our framework to minimize unnecessary discussions, thereby significantly enhancing its efficiency while maintaining its accuracy. Moreover, the multi-stage discussion facilitates communication among agents, maximizing the integration of their knowledge and reasoning abilities to generate an accurate response Y .To get Lst and A, we specifically design a dynamic learning module, which initially deploys the encoder component of a pre-trained image variational autoencoder (VAE) to extract features from the given image X. The extracted features, combined with agent embeddings Emb, are employed to determine the suitability of agents w.r.t. Lst for agent discussions and predict agent collaboration connections A in the geo-localization task.LLM/LVLM have demonstrated remarkable capabilities in complicated tasks and some pioneering works have further proven that the performances can be further enhanced by ensembling multiple LLM/LVLM agents. Thus, to improve the geo-localization capability of LVLMs, we propose a cooperation framework to effectively integrate the diverse knowledge and reasoning abilities of multiple LVLMs. Inspired by the fact that community review mechanisms can improve the quality of manuscripts, an iterative 3-stage anonymous reviewing mechanism is proposed for helping agents share knowledge and reasoning capability with each other through their collaboration social network: i) answer agent election & answering, ii) review agent selection & reviewing, and iii) final answer conclusion.Initially, we select K agents with the highest agent election probabilities Lst as answer agents and let them geo-localize independently as the preliminary step for further discussion. By initiating the discussion with a limited number of agents, we aim to reduce potential chaos and maintain the efficiency of our framework as the number of participating agents increases.It's located in ... A๖ Some one says it's located ... Initizalize: , After the answer agents are elected, we send the image X to all answer agents and let them give the primary analysis. Each answer must contain three parts: one location (city, country, and so on), one confidence (a percentage number), and a detailed explanation.In this stage, for each answer agent, we choose R review agents by performing a transfer-probabilitybased random walk on the agent collaboration social network G for answer reviewing. The transfer probability p(v i , v j ) from node v i to node v j can be calculated as follows:EQUATIONwhere N (v i ) is the 1-hop neighbor node set of node v i .For each selected review agent, it reviews the results as well as the explanations generated by the corresponding answer agent and gives its own comments. After that, each answer agent would summarize their preliminary analysis and the feedback from all of its review agents to get the final answer, which must include three parts as well: one location, one confidence, and an explain.In the previous stage, each answer agent produces a refined result based on feedback. When K > 1 in Stage 1, the proposed framework generates multiple independent results, which may not be consistent.However, we aim to provide a definitive answer rather than multiple options for people to choose from. To address this, we allow up to Z rounds of free discussion among those answer agents to reach a unified answer:First, we maintain a global dialog history list, diag, recording all replies agents respond. In addition, discussions are executed asynchronously, which means that any answer agent can always reply based on the latest diag, and replies would be added to the end of diag as soon as they are posted. Each answer agent is allowed to speak only once in each discussion round, and after Z rounds of free discussion, we determine the final result using a minority-majority approach, i.e., we choose the reply with the most agreement as the final conclusion. If all agents reach a consensus, we early stop this stage and adopt the consensus answer as the final answer. If none of any consensus is reached, we only select the reply of the first answer agent elected from Stage 1 as the final result.In our framework, choosing the appropriate answer agents and review agents for knowledge sharing and discussion is vital to its effectiveness and efficiency. Therefore, we propose a dynamic learning strategy to optimize them. Specifically, for each training sample, i.e., a geo-tagged image, we would first estimate the optimal answer agent election probability Lst and the optimal collaboration social network of agent Ĝ by its actual location. Then we train an attention-based graph neural network, which aims to predict Lst and G, by such estimated ground truth.To estimate the optimal Lst and Â for agents to geo-localize image X, we first initialize the agent social network G (0) by a fully connected graph with the agent set V. Besides, we initialize the agent election probabilityLst (0) = [0.5, 0.5, • • • ],with all agents having 50% probability of being chose as answer agents.Then, we iteratively conduct our 3-stage discussion framework to get the prediction answer. Lst (l) and G (l) is updated at the end of each round l ∈ L by comparing the answers Y (l) vi from each answer agent with the ground truth Ŷ .After L rounds of agent discussions, the updated agent election probability for an image X, Lst :=Lst (L) (X) = [P (L) v1 , P (L) v2 , • • • , P (L)v N ], determines whether an agent v i gives the correct/wrong answers Y (L)vi by comparing it with the ground truth Ŷ . Here, the definition ofP (l)vi of agent v i at round l is as follows:P (l) vi :=      0, if D( Ŷ , Y (l) vi ) > th 1, if D( Ŷ , Y (l) vi ) ≤ th 1 2 , if v i did not participate in the discussion (2)where th is a pre-defined threshold for determining whether the predicted location is close enough to the actual location. In the distance function D(•), we first deploy geocoding to convert natural language into location intervals in a Web Mercator coordinate system (WGS84) by utilizing OSM APIs, and then compute the shortest distance between two two location intervals.Please note that, rather than electing the top-K answer agents in each round, we choose each agent with probability P vi during the training period to ensure that every agent has the opportunity to participate in the discussion for more accurate estimation, as shown at the left part of the dynamic learning strategy module of agent collaboration social networks in Figure 1 .In addition, the agent collaboration social network would also be updated by comparing the actual location with the generated answer of each answer agent at the same time. For l-th round, we strengthen the link between the correctly answered agent and the corresponding review agents while weakening the link between the incorrectly answered agent and the corresponding review agents:EQUATIONwhereA (l-1) ij (X)is the weight of the connection between answer agent v i and review agent v j at round l -1 when geo-locating image X, AEQUATIONis the number of consecutive times an agent has answered correctly, which is used to attenuate the connection weights when updating them, preventing the performance of an agent on a certain portion of the continuous dataset from interfering with the model's evaluation of the current agent's performance on the entire dataset.Then, we try to learn an attention-based graph neural network to predict the corresponding optimal agent election probability Lst = h(X, G|Θ) and the optimal agent collaboration connectionsEQUATIONwhere W , Emb ∈ Θ are two learnable parameters,Emb := [Emb v1 , Emb v2 , • • • ] ⊤ is the agent embedding and W is the weight matrix, σ(•) is the LeakyReLU function, σ ′ (•) is the Sigmoid function, VAE Enc (•)is the encoder of the image VAE that compresses and maps the image data into the latent space. It is used to align the image features with the agent embedding, and d k is the dimension of the F ea. Our learning target can be formalized as:EQUATIONwhere D(•) denotes the distance between the places an LVLM agent answered and the ground truth,1(•) is the indicator function, Y vi := Y (L) vi = g vi (X, Y (L-1) vj ), g vi (•) represent the LVLM agent v iwith fixed parameters and Y (0) vi = g vi (X) is the answer that LVLM agent v i generates at the initial stage of discussion. 
experiments: 
 To evaluate the performance of our framework, we conducted experiments on the real-world dataset that was gathered from the Internet to answer the following research questions:• RQ1: Can smileGeo outperform state-of-the-art methods in open-ended geo-localization tasks?• RQ2: Are LVLM agents with diverse knowledge and reasoning abilities more suitable for building a collaboration social network of agents?• RQ3: How does the setting of hyperparameters affect the performance of smileGeo? we utilize several open-source fine-tuned LVLMs: Infi-MM2 , Qwen-VL3 , vip-llava-7b&13b 4 , llava-1.5-7b-base&mistral&vicuna 5 , llava-1.6-7b&13b&34b-mistral&vicuna 6 , CogVLM 7 . As for the closed-source LVLMs, we chose the models provided by three of the most famous companies in the world: Claude-3-opus 8 , GPT-4V 9 , and Gemini-1.5-pro 10 . Besides, 99% of images (about 290,000 samples) from the original dataset are randomly chosen as training samples. For the open-world geolocation problem, we construct the test dataset using approximately 4,000 samples, of which nearly 66.67% samples reflected different locations not present in the training dataset. More details about the deployment of smileGeo and the related parameter settings can be found in Appendix C.Baselines. In this work, we compare the proposed framework with three kinds of baselines: single LVLMs, LLM/LVLM-based multi-agent frameworks, and image retrieval approaches. Firstly, we use each LVLM alone as an agent directly for the geo-localization task and compute the performance of these single LVLMs under the same dataset. In addition, we experiment with multi-agent collaborative frameworks, including LLM-Blender [34] , PHP [35] , Reflexion [36] , LLM Debate [37] , and DyLAN [38] . Finally, several state-of-the-art image retrieval approaches, including NetVLAD [3] , GeM [26] , and CosPlace [46] , are also used to be part of the baselines. We set the training dataset as the geo-tagged image database of each image retrieval system and use images in the test dataset for the retrieval system to generate answers.Evaluation Metrics. We use Accuracy (Acc) to evaluate the performance:Accuracy = Ncorrect Ntotal ,where N correct is the number of samples that the proposed framework correctly geo-localizes, and N total refers to the total number of testing samples.In this paper, we first geo-encode the answers with the ground truth, i.e., we transform the addresses described through natural language into latitude-longitude coordinates. Then, we calculate the distance between the two coordinates. When the distance between the two coordinates is less than th = 50km (city-level), we consider the answer of the framework to be correct.We divide the baseline comparison experiment into three parts: i) comparison with single LVLMs, ii) comparison with LLM/LVLM-based agent frameworks, and iii) comparison with image retrieval systems. Bold indicates the statistically significant improvements (i.e., two-sided t-test with p < 0.05) over the best baseline. 5 https://huggingface.co/llava-hf/llava-1.5-xxx 6 https://huggingface.co/liuhaotian/llava-v1.6-xxx 7 https://github.com/THUDM/CogVLM 8 https://anthropic.com/ 9 https://openai.com/ 10 https://gemini.google.com/ Firstly, the performance of all single LVLM baselines is shown in Table 1 , in terms of the metric Acc. The data in Table 1 indicate that open-source LVLMs with diverse knowledge and reasoning capabilities exhibit significant variations, particularly in geo-localization tasks. This may be due to the difference in the overlap between the pre-training datasets used by different LVLMs and the dataset we constructed. Therefore, in addition to querying the LVLM locations about images, we also incorporated real-time image search results from Google to provide the model with more comprehensive information. These results from Internet retrievals are incorporated into the chain-ofthoughts (CoT) [47] of LVLMs as external knowledge. At this time, models with larger parameters, such as llava-1.6-34b, demonstrate superior reasoning abilities compared to smaller models (7b or 13b). In addition, closed-source large models also show more consistent performance than their opensource counterparts and are more adept at analyzing and utilizing external knowledge for accurate inferences. Compared to all single LVLMs, our proposed LVLM agent framework surpasses all single LVLM baselines in accuracy. This improvement confirms the effectiveness of different LVLMs collaborating by engaging in discussions and analyzing various types of images, thus producing more precise results. Secondly, the comparative results across various LLM/LVLM agent frameworks are presented in Table 2 . It is evident that the majority of LLM/LVLM agent frameworks surpass individual LVLMs in terms of geo-localization accuracy. This improvement can primarily be attributed to the ability to integrate knowledge from multiple LVLM agents, thereby enhancing the overall precision of these frameworks. However, LLM-Blender and LLM Debate exhibit lower accuracy due to statements of some agents misleading others during discussions, which impedes the generation of correct outcomes.Our framework, smileGeo, guarantees the highest accuracy while being able to accomplish the geo-localization task with the lowest token costs. The average number of tokens our framework spent per query is 18,876, and it is less than the computational overhead of LLM-Blender (23, 662) , which has the simplest agent framework structure but the lowest accuracy among all baselines. This is mainly due to a 'small' GNN-based dynamic learning model being deployed for agent selection stages and significantly reducing unnecessary discussions among agents. As illustrated in Figure 2 (a), the framework achieves optimal accuracy with 4 or 5 agents. Beyond this number, the framework's performance begins to deteriorate. This shows that using models with the same knowledge and reasoning capabilities as different agents has limited improvement in the accuracy of the framework. Despite this decline, the performance of frameworks other than LLM-Blender and LLM Debate remains superior to that of a single agent. LLM-Blender and LLM Debate, however, have a significant decrease in model accuracy when the number of agents exceeds 11. This is mainly because both of them involve all LVLMs in every discussion, which suffers from excessive repetitive and redundant discussions. sults indicate that optimal performance can be achieved with relatively small values of K or R. However, the computational cost, measured in tokens, increases exponentially with higher values of K and R. To balance both the efficiency and the accuracy of smileGeo, for the experiments presented in this paper, we set both K and R equal to 2. 
conclusion: 
 This work introduces a novel LVLM agent framework, smileGeo, specifically designed for geolocalization tasks. Inspired by the review mechanism, it integrates various LVLMs to discuss anonymously and geo-localize images worldwide. Additionally, we have developed a dynamic learning strategy for agent collaboration social networks, electing appropriate agents to geo-localize each image with different characteristics. This enhancement reduces the computational burden associated with collaborative discussions among LVLM agents. Moreover, we have constructed a geo-localization dataset called GeoGlobe and will open-source it. Overall, smileGeo demonstrates significant improvements in geo-localization tasks, achieving superior performance with lower computational demands compared to contemporary state-of-the-art LLM/LVLM agent frameworks.Looking ahead, we aim to expand the capabilities of smileGeo to incorporate more powerful external tools beyond just web searching. Additionally, we plan to explore extending its application to complex scenarios, such as high-precision global positioning and navigation for robots, laying the cornerstone for exploring LVLM agent collaboration to handle different complex open-world tasks efficiently. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We announce the compute resources in the appendix (Implementation Details). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] widely used in robot positioning and navigation in the future. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: The data sets we collect have been manually reviewed twice, and all data containing various types of sensitive information or copyright risks have been filtered out. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: In this paper, we provide the algorithm of the code and introduce the dataset in detail (in the appendix).Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper aims to address visual geo-localization tasks and does not contain any experiments with human subjects. 
 
 Appendix: We summarize all notations in this paper and list them in Table 4 . The predicted (ground truth of) geospatial location in the natural language form. G ( Ĝ)The predicted (ground truth of) LVLM-based agent collaboration social network. A ( Â)The predicted (ground truth of) adjacency matrix of the agent social network. Lst ( Lst)The predicted (ground truth of) scalar of agent election probability.The set of LLM agents.The set of connections between LLM agents.The number of agents.The number of agents to be elected as answer agent(s).The number of agents to be selected as review agent(s).The number of agent discussion rounds.The maximum number of rounds in which answer agents harmonize opinions.The learnable parameters of the agent social network learning model.The images in this dataset are copyright-free images obtained from the Internet via a crawler. We divide the images into two main categories: man-made landmarks as well as natural attractions. Then, we filter out the data samples that could clearly identify the locations of the landmarks or attractions in the images. As a result, we filter out nearly three hundred thousand data samples, and please refer to Table 5 and Figure 4 for details. Due to the fact that a large number of natural attractions in different geographical regions with high similarity are cleaned, the magnitude of the data related to natural attractions in this dataset is smaller than that of man-made attractions. For an open-world geo-localization task, the relationship between the training and test samples in the experiment could greatly affect the results. We label the training samples as Z train , and the test sample set as Z test , and use two metrics, coverage as well as consistency, to portray this relationship:EQUATIONAs for the samples in this paper, coverage ≈ 4.6564%, and consistency ≈ 33.2957%.In all experiments, we employ a variety of LVLMs, encompassing both open-source and closed-source models, to be agents in the proposed framework. Unless specified otherwise, zero-shot prompting isapplied. Each open-source LVLM is deployed on a dedicated A800 (80G) GPU server with 200GB memory. As for each closed-source LVLM, we cost amounting to billions of tokens by calling APIs as specified by the official website. To avoid the context length issue that occurs in some LVLMs, we truncate the context before submitting it to the agent for questions based on the maximum number of V 1 = {v 1 a , v 1 b , • • • } ← Lst, where |V 1 | = K 6: for each answer agent v 1 do 7:Obtain the location:Y 1 v 1 ← Ask v 1 (X) 8:Get the confidence percentage:C 1 v 1 ← Ask v 1 (X, Y 1 v 1 ) 9:Store the further explanation: Select the review agents:T 1 v 1 ← Ask v 1 (X, Y 1 v 1 )V 2 = {v 2 a , v 2 b , • • • } ← RandomWalk v 1 (G), where |V 2 | = R 13:for each review agent v 2 do 14:Obtain the commentT 2 v 2 ← Review v 2 (X, Y 1 v 1 , C 1 v 1 )15:Get the confidence percentage:C 2 v 2 ← Ask v 2 (X, T 2 v 2 ) 16:end for 17: end for Stage 3: 18: for each selected answer agent v 1 do 19:Summary the final answer:Y 3 v 1 ← Summary v 1 (Y 1 v 1 , C 1 v 1 , T 2 v 2 1 , C 2 v 2 1 , T 2 v 2 2 , , C 2 v 2 2 , • • • ) 20:Get the final confidence percentage:C 3 v 1 ← Ask v 1 (Y 1 v 1 , C 1 v 1 , T 2 v 2 1 , C 2 v 2 1 , T 2 v 2 2 , , C 2 v 2 2 , • • • ) 21: end for 22: Generate the final answer: Y ← Discussion Z (Y 3 v 1 1 , C 3 v 1 2 , Y 3 v 1 2 , C 3 v 1 2 , • • • )The dynamic learning strategy module: 23: Initialize Lst (0) , G (0) 24: for round l in total L rounds do 25:for each selected answer agent v 1 do 26:Obtain coordinates:Coors ← GeoEmb(Y 3 v 1 ), Coors Truth ← GeoEmb(Y Truth ) 27:if Dis(Coors, Coors Truth ) ≤ th then 28:A (l) ← Enhance(e|e contains v 1 , e ∈ E) 29: Update Lst (l) [v 1 ] = 1 30:else 31:A (l) ← Weaken(e|e contains v 1 , e ∈ E) 32: Update Lst (l) [v 1 ] = 0 33: end if 34:end for 35: end for36: Â ≈ A (L) , Lst ≈ Lst (L) 37: Update: Θ ← Loss( Ŷ , Y , Â, A, Lst, Lst)tokens that each agent supports. Besides, noting that images are token consuming, we only keep the freshest response for agent discussions.The detailed algorithm of smileGeo is illustrated in Algorithm 1. In the initialization stage, we initialize or load the parameters of the agent social network learning model, as delineated in line 1.Next, we treat each LVLM agent as a node, establishing the LVLM agent collaboration social network and computing the adjacency relationships among LVLM agents as well as the probability that each agent is suited for responding to image X, as shown in line 2. Then, line 3 initializes the agent 100%. I agree with the previous identification. The building in the image is indeed the Uptown Tower located in Dubai, United Arab Emirates. This is confirmed by the logo "DMCC" at the top of the building and the unique architectural design of the tower, which matches the description of the Uptown Tower in Dubai. The information provided from the internet search results that mention "DUBAI | Uptown Tower" further supports this identification. The tower is a notable feature of Dubai's skyline and serves as an office building for the Dubai Multi Commodities Centre.You say this picture is located in Dubai, United Arab Emirates, with confidence 100%. Here are others comments with the confidence: 1. I agree with the previous identification. The ...., 100%; 2. This image features the Uptown Tower in Dubai, ...., 100%. Please summarize all informations by this format: <city>, <country>, <confidence percentage> : Dubai, United Arab Emirates, 100%. Pv๗=0.65 0 0.9 0.5 0.8 0.5 0.9 0 0.5 0.5 0.5 0.5 0.5 0 0.5 0.6 0.8 0.5 0.5 0 0.5 0.5 0.5 0.6 0.5 0 ratio 1:8) and extract its representations. We define the embedding dimension of the nodes to be 1024 and the hidden layer dimension of the network layer to be 1024. we use Adam as an optimizer for gradient descent with a learning rate of 1e -5 . For each stage of the LVLM agent discussion, we use a uniform template to ask questions to different LVLM agents and ask them to make a response in the specified format. In addition, the performance of our proposed framework is the average of the last 100 epochs in a total training of 2500 epochs.ª º « » « » « » « » « » « » ¬ ¼ A v๖ v๖Case 1: In Figure 5 , we illustrate the application of smileGeo in a visual geo-localization task.For this demonstration, we randomly select an image from the test dataset and employ five distinct LVLMs: LLaVA, GPT-4V, Claude-3, Gemini, and Qwen. The agent selection model selects two answer agents, as depicted in the top part of the figure. Subsequently, stages 1 through 3 detail the process of generating the accurate geo-location. Initially, only one answer agent provided the correct response. However, after several rounds of discussion, the agent that initially responded incorrectly revised the confidence level of its answer. During the final internal discussion, this agent aligned its response with the correct answer. This outcome validates the efficacy of our proposed framework, demonstrating its ability to integrate the knowledge and reasoning capabilities of different agents to enhance the overall performance of the proposed LVLM agent framework. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. Because GeoGlobe is relatively large (about 32GB), we are unable to provide it as an attachment during the double-blind review stage. We will publish it once the paper is accepted.https://huggingface.co/Infi-MM/infimm-zephyrhttps://huggingface.co/Qwen/Qwen-VLhttps://huggingface.co/llava-hf/vip-llava-xxx 


Decision: The paper received final ratings of borderline reject, weak accept, and borderline reject. The reviewers indicated general enthusiasm for the high-level approach but expressed concerns about the appropriateness of the model across various task definitions (robot navigation vs general image geolocalization vs landmark image geolocalization), missing comparison to important previous work, insufficient qualitative examples, and limited discussion/analysis of failure cases. 

The AC carefully read the paper, the reviews, the rebuttals, and subsequent discussions. The following are the major factors in recommending rejection:
- while the new dataset is potentially interesting, there needs to be more clarity about the construction process. For example, the supplemental states, "we filter out the data samples that could clearly identify the locations of the landmarks or attractions". This seems to indicate that samples were removed that had clear indications of the location, which seems to disagree with statements in the rebuttal that samples were filtered that weren't location discriminative. Regardless, the exact procedure for this filtering should be clarified.

- there are existing datasets for global scale geolocalization and this work seems to dismiss them because they include some challenging images. This work would have been much stronger if it had directly compared to results in recent geolocalization papers that use the IM2GPS3K and YFCC4K datasets. Some attempt was made in the rebuttal phase, but it would have been much better if the original numbers from the other papers had been used rather than a quick attempt to retrain their models.

- for this method to be useful, I would expect it would need to be significantly more accurate than recent work given the high computational complexity and slow run-time relative to methods that are based on image-only processing

- there is also significant potential for data leakage since testing images were drawn from Internet repositories which are also likely to have been used as training data for the LVLMs.

Given these factors, this paper needs an additional round of reviews before it can be accepted.

Reviews: review: paper_summary: paper_summary: The paper introduces smileGeo, a novel framework for visual geo-localization, which involves identifying the geographic location of an image. The authors argue that while Large Vision-Language Models (LVLMs) show promise in this area, their individual performance is limited. SmileGeo leverages the concept of "swarm intelligence" by enabling multiple LVLMs to collaborate and refine their location predictions through a multi-stage review process. To enhance efficiency, the framework incorporates a dynamic learning strategy that optimizes the selection of LVLMs for each image. Furthermore, the paper introduces "GeoGlobe," a new dataset designed to evaluate visual geo-localization models in open-world scenarios where many images depict locations not seen during training. Experimental results demonstrate that smileGeo outperforms existing single LVLMs and image retrieval methods, highlighting the effectiveness of collaborative learning for visual geo-localization.
strength_weakness: strength_weakness: * The paper only seems to tackle the problem of geolocalizing **landmark images**. While this is a challenging problem, the current literature [1, 2, 3] has already tried to address the problem of geolocalizing arbitrary ground-level images. The latter problem requires learning sophisticated geographic and visual features. I think even searching the internet cannot effectively solve the geolocalization problem for non-landmark images.
* Limited applicability: The framework is built entirely upon the capabilities of different LVLMs (e.g. GPT4, LLaVA, etc). It seems the framework cannot generalize beyond the training data used for training LLMs.
* The work fails to address the practical applications and real-life use cases of the framework. Why do we require such a framework?
* The limitation and failure cases are not adequately mentioned in the paper.

[1] Vivanco Cepeda, Vicente, Gaurav Kumar Nayak, and Mubarak Shah. "Geoclip: Clip-inspired alignment between locations and images for effective worldwide geo-localization." Advances in Neural Information Processing Systems 36 (2023).

[2] Haas, Lukas, Michal Skreta, Silas Alberti, and Chelsea Finn. "Pigeon: Predicting image geolocations." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12893-12902. 2024.

[3] Berton, Gabriele, Carlo Masone, and Barbara Caputo. "Rethinking visual geo-localization for large-scale applications." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4878-4888. 2022.
questions: questions: * At present, each agent sees the same information. It might be interesting to incorporate different kinds of information that is revealed differently to the agents, such as multi-view images or panorama images.
* How much compute time is used for a single inference run?
* Why does the performance of some LLMs decrease with web searching?
limitations: limitations: Limitations are insufficiently addressed in the paper. The future works mentioned in the conclusion are vague and fail to specify specific future directions for the work.
paper_summary: paper_summary: This works proposes a new visual geo-localization framework with multiple LVLM (Large Vision Language Model) agents. The agents communicate with each other to estimate the geo-location of the input image. A dynamic learning strategy is proposed to optimize the communication patterns among agents to improve efficiency. The method is evaluated on the proposed GeoGlobe dataset.
strength_weakness: strength_weakness: - The authors could make the geo-localization setting more clear in the introduction, for example, the paper focuses on worldwide city-level geo-localization. There are lots of different settings for geo-localization problem and this could be confusing for some researchers.
- This paper provides a comparison with three traditional geo-localization methods, i.e., NetVLAD, GeM, and CosPlace. However, these three methods are either retrieval-based landmark matching methods or fine-grained classification-based place recognition methods. It would be better to provide a direct comparison with worldwide geo-localization method on city-level setting, e.g., [A]. Although I believe LVLM-based method is better at this setting, a comparison can make it more convincing.

[A] Pramanick, Shraman, et al. "Where in the world is this image? transformer-based geo-localization in the wild." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
- There are only two qualitative results in the appendix. Given that the accuracy is over 60%, it should be easy to find successful and failed cases to demonstrate the actual output cases of the proposed methods. It can also better illustrate how multiple agents help the geo-localization process.
- There are also some existing worldwide geo-localization datasets that could be used for more comprehensive evaluation, e.g., IM2GPS3K, YFCC4K.
questions: questions: See the weaknesses.
limitations: limitations: The authors mentioned the limitations in the checklist.
paper_summary: paper_summary: Proposed a graph based learnable multi-agent framework. The framework consists of multiple stages : Forwarding: Election (K: Answer agents; R: Reviewer) -> Review -> K Discuss till a final conclusion is reached. Proposed a mechanism to learn the graph connections dynamically. 

The major Contributions Introduced in the paper: (A)  A new swarm intelligence geo-local framework smileGeo; (B) Dynamic learning strategy; (C) A new Geo-dataset (test mainly).
strength_weakness: strength_weakness: The major weaknesses are as follows:

(a) Comparisons with baselines seems unfair. 

(b) Missing details of the evaluation setup, metrics, etc.
questions: questions: Here are my questions to the authors:

(a) Table 1 involves comparison between open/closed source single LVLMs with smileGeo-single. However, smileGeo appears to primarily focus on a multi-agent framework, without introducing any new single LVLM architectures.

(b) The comparative results of different agent frameworks without web searching are reported in table 2. How are 'acc' and 'tks' determined for each framework? Which types of LVLMs are employed? Did they aggregate all LVLMs and calculate averages per framework, or utilize the best-performing LVLM specific to each framework? It's important not to unfairly advantage one framework over others by using superior LVLMs.

(c) Question about the comparison with LLM/LVLM-based agent frameworks:

For the integration frameworks you compared in Table 2, what specific LVLMs were integrated within the LLM-Blender, LLM Debate, and smileGeo frameworks? Did you use the same LVLM combinations for the different frameworks in the comparison? Different LVLM combinations may have different underlying behavior on metrics.
limitations: limitations: yes, the authors adequately addressed the limitations.

Comments: comment: I appreciate the authors' response. However, I remain unconvinced by their explanation regarding the implementation of the majority rule after a certain number of discussion rounds when consensus cannot be reached. The method for determining the number of discussion rounds requires further analysis, as it is a crucial aspect of their proposed framework. This situation is likely to occur frequently in practice. Additionally, after reviewing the author's discussion with the reviewer mcak, I agree with most of the concerns raised by the reviewer mcak. I find the authors' response to be unconvincing, especially the motivation and cost issues. Therefore, I also recommend a borderline reject.
As we approach the end of the author-reviewer discussion period, we respectfully wish to check in and ensure that our rebuttal has effectively addressed your concerns regarding our paper. Should you have any remaining questions or need further clarification or additional experimental results, please do not hesitate to let us know. We appreciate the thoughtful reviews and the time you’ve invested in providing us with valuable feedback to improve our work. If you believe that our responses have sufficiently addressed the issues raised, we kindly ask you to consider the possibility of raising the score.
I thank the authors for providing additional clarifications. However, after reading the responses, I am still unsure about the motivation for using such a general framework for the task of geolocalization. The response time of 25 seconds seems significant and the dependence of the framework on closed-sourced LLMs raises questions about whether the framework is cost-effective. Hence, I shall keep my rating unchanged.
Thank you for your positive feedback. We appreciate your time and effort throughout this reviewer-author discussion stage.
Thanks for the rebuttal. It addresses the concerns and I will keep the rating.
**Response to Q1:**

Thank you for your questions. Geo-localization is a complex task that requires extensive geospatial knowledge and strong reasoning abilities. LVLMs offer a novel approach to visual geo-localization by leveraging their powerful visual question-answering (VQA) capabilities, eliminating the need for external geo-tagged image records. This motivation led us to design a discussion framework around LVLMs to fully utilize their strengths and achieve better results. Since different LVLMs have different memory and reasoning capabilities for geo-localization tasks, we designed an LLM agent selection module in the proposed framework, which can select the most suitable agents for geo-localization of the target image for discussion, thus improving the efficiency of the framework. When the selected LLM agents cannot reach a high-confidence conclusion, they can autonomously call an internet-based geographic image search tool to supplement it with additional positioning information.

We also appreciate your acknowledgment of our framework’s potential as a general approach that could be extended to other fields, reinforcing that the underlying concept of the LLM-based discussion framework is versatile. In the revised version, we will mention our intention to explore its application to other areas in future work.

**Response to Q2:**

Thank you for your questions. This was the motivation behind our first comparative experiment, where we compared different single LVLMs in Table 1. Even without retrieval assistance, most closed-source large model agents (such as GPT-4V and Gemini-1.5-pro) and some open-source large models (like Qwen-VL) achieved higher experimental accuracy than some image retrieval-based methods (as shown in Table 3). This demonstrates that LVLMs inherently possess the ability to analyze and process geo-location data, as well as the capacity to retain geo-location knowledge.

Furthermore, our framework allows LVLM agents to search the internet and obtain sufficient world knowledge directly. As mentioned in Section 4.2, "models with larger parameters, such as llava–1.6–34b, demonstrate superior reasoning abilities compared to smaller models," leading to significant improvements in accuracy and outperforming traditional retrieval-based geo-localization methods.

These experiments confirm that LLM agents, particularly closed-source large models and LVLMs with larger parameters, exhibit strong memory and reasoning capabilities for geo-localization tasks, both independently and with additional geo-location information. 

Additionally, there are also many reports verifying that using LVLMs for geo-tagging has gained widespread acceptance; please check the following links [1-3].

[1] https://x.com/itsandrewgao/status/1785827031131001243

[2] https://lingoport.com/i18n-term/llm/

[3] https://www.assemblyai.com/blog/llm-use-cases/ 

**Response to Q3:**

Thank you for your concerns. In this paper, we propose a framework that effectively addresses the geo-localization task, which could be a critical component of robot navigation. Robot navigation typically involves many stages, such as localization, trajectory planning, and execution of the planned route. Tasks like localization and path planning often prioritize accuracy over real-time processing, especially in city-level navigation, where incorrect trajectory planning can lead to significant resource wastage.

Several studies [1][2] utilize LMM/LVLM agents for UAV dispatching, a specific aspect of robot navigation. While inference using LMM/LVLM is known to be very time-consuming, the successful application of methods in these studies indicates promising prospects for LMM/LVLM-based geo-localization. Additionally, we believe that as LLM technology advances—through methods like quantization, distillation of open-source LLM agents, and calling of more lightweight and faster closed-source LLM agents (e.g., GPT-4o-mini)—our proposed framework will soon be capable of real-time responses. We will include this prospect in the revised paper.

[1] Liu, S., Zhang, H., Qi, Y., Wang, P., Zhang, Y., & Wu, Q. (2023). Aerialvln: Vision-and-language navigation for uavs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 15384-15394).

[2] Zhao, H., Pan, F., Ping, H., & Zhou, Y. (2023). Agent as Cerebrum, Controller as Cerebellum: Implementing an Embodied LMM-based Agent on Drones. arXiv preprint arXiv:2311.15033.
I thank the authors for their responses. I have a few clarifying questions:

[1] What is so __unique__ about the framework that it could only be employed for geolocalization? I think the proposed framework is very general and not unique to geolocalization. 

[2] "A single LVLM indeed faces these challenges, as many large models attempt to consume vast amounts of data for pre-training. Our motivation is to address the biases in pre-trained LVLMs by combining their strengths." 

How do you ensure that the LLM agents have sufficient world knowledge that is relevant for the task of geolocalization? Are there any empirical studies to prove the statement?

[3] Regarding practical application: How can the framework be used in robot navigation? Robot navigation requires fast response times for decision-making at each stage. The authors have mentioned that "99% Response Time for smileGeo is less than 25 seconds." This does not make the framework scalable for real-time applications.