GeoLLM: Extracting Geospatial Knowledge from Large Language Models
Authors: Rohin Manvi
Samar Khanna
Gengchen Mai
Marshall Burke
David B. Lobell
Stefano Ermon
Venue: ICLR 2024 poster (2024)
Keywords: geospatial
knowledge extraction
large language models
pre-trained language models
natural language processing
Primary area: general machine learning (i.e., none of the above)

Abstract: The application of machine learning (ML) in a range of geospatial tasks is increasingly common but often relies on globally available covariates such as satellite imagery that can either be expensive or lack predictive power.
Here we explore the question of whether the vast amounts of knowledge found in Internet language corpora, now compressed within large language models (LLMs), can be leveraged for geospatial prediction tasks. 
We first demonstrate that LLMs embed remarkable spatial information about locations, but 
naively querying LLMs using geographic coordinates alone is ineffective in predicting key indicators like population density. 
We then present GeoLLM, a novel method that can effectively extract geospatial knowledge from LLMs with auxiliary map data from OpenStreetMap.
We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods.
Across these tasks, our method demonstrates a 70\% improvement in performance (measured using Pearson's $r^2$) relative to baselines that use nearest neighbors or use information directly from the prompt, and performance equal to or exceeding satellite-based benchmarks in the literature.
With GeoLLM, we observe that GPT-3.5 outperforms Llama 2 and RoBERTa by 19\% and 51\% respectively, suggesting that the performance of our method scales well with the size of the model and its pretraining dataset.
Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe.
Crucially, GeoLLM shows promise in mitigating the limitations of existing geospatial covariates and complementing them well.
Code is available on the project website: https://rohinmanvi.github.io/GeoLLM

Published as a conference paper at ICLR 2024
GEOLLM: EXTRACTING GEOSPATIAL KNOWLEDGE FROM LARGE LANGUAGE MODELS
Rohin Manvi
Samar Khanna
Gengchen Mai!
Marshall Burke*
David Lobell'
Stefano Ermont
ABSTRACT
The application of machine learning (ML) in a range of geospatial tasks is increas- ingly common but often relies on globally available covariates such satellite imagery that can either be expensive or lack predictive power. Here we explore the question of whether the vast amounts of knowledge found in Internet language corpora, now compressed within large language models (LLMs), can be leveraged for geospatial prediction tasks. We first demonstrate that LLMs embed remark- able spatial information about locations, but naively querying LLMs using geo- graphic coordinates alone ineffective in predicting key indicators like popula- tion density. We then present MM, a novel method that can effectively extract geospatial knowledge from LLMS with auxiliary map data from OpenStreetMap. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Across these tasks, our method demonstrates 70% improvement in performance (measured using Pearson's r²) relative to baselines that use nearest neighbors formance equal to or exceeding satellite-based benchmarks in the literature. With GeoL.LM, observe that GPT-3.5 outperforms Illaa and RoBERTa by 19% and 51 % respectively, suggesting that the performance of our method scales well with the size of the model and its pretraining dataset. Our experiments reveal that LLMs are remarkably sample-efficient rich in geospatial information, and robust across the globe. Crucially, GeoLLM shows promise in mitigating the limitations of existing geospatial covariates and complementing them well. Code is available on the project website: https://rohinmanv httss//ohiimaanvi.iihbubbboooo githhb.iioGeooLLMM.M...
information directly from the prompt, and per-
INTRODUCTION
Geospatial predictions with ML are widely used across various domains, including poverty estima- tion (Jean et al. 2016 Chi et 202T), public health (Nilsen et al., 2021 Areed al. 2022 Chang et al. (022). food security (ak 2018). biodiversity preservation (Mai et al. 2023c b). and environmental conservation (Sofaer et al. 2019). The covariates used these predictions include
geographical coordinates, remote sensing data, satellite imagery. human mobility data (Chang et al.) 2022], and phone metadata (Blumenstock et al. 2015) Burke et al. 2019). While having access to quality covariates is essential, it can be challenging due limited spatiotemporal coverage, high costs, and accessibility barriers (Ball et al. 2017). For example, researchers often use publicly avail- able satellite images estimate socio-economic indicators like asset wealth, population density, and infrastructure access, because they are free and globally available (Yeh et al. 2020) Robinson et al.) 2017 Head et al.) 2017). However, model predictive power can be limited due to the fact that important features may not be visible from space.
Large language models (LLMs) have proven be highly effective foundation models that can be fine-tuned or prompted to perform tasks in various domains including healthcare, education, law,
"Corresponding author, rchinnecs................ stanford.
1 Stanford University
University of Georgia

---

Published as a conference paper at ICLR 2024
Describe the acdress Avenue, Manhattan, New York" one detailed paragraph.
Masic Prompt: conndinetes: (40, 76238, -73, 96042)
Popolation Density (On, Scale 0.0 9,99:
GPT-3.5:
6th Aecac,, Karhattan, Nox York, is a nustling choroughtare that cutss through the heart of this sconts norough,
Finetuned GPT-3.5; 5.2
the soutbern tip Nanhattar, where kieses the edge of the verdent Mashington Squere Park, and kinds its way northward, pessing througa diverse reighborhoods like Creanwich Villags, Chelsea, and Ristom. the svcaus is a symphony ot activity, with constant tlw of people, vehicles, and erarcy. Skysarasers toxer overheed, casting long chacovs that provide nonestary respite tron the arilliant cunlicht that often betaes the sidewalks rustie commuerra, atreet vandors padale thair wares, and the distant hen of traffic biends with the occasional blaze of taxi
stretches
Our Prompt: Coordinates: (40,96200, -73.980421
Adiress:
"Calyon Bu:lding, Avense, Monhattan Community Board 5, Marhattan, Now York County, City of New York, Nov York, United States"
Nearby Flaces:
0.6 km Scuta-West: Theater District
0.7 kn Nerth: columbes Circle
0.7
Kidtown East
hurried
0.5 kn South West: Kid:own
1.0 km Wost: Hell's Kitchen
1.2 km North: Liacoir. Bquaze
1.3 South-West:
District
storefronta 6ch Avenue are a vibrant tapestry surinesees, raaçing tron coty zottes shees and quisky boutiques to corsraate offices
line
1.4 kn Bouth-East: Furele Bay
1.4 km South: Jan Kazski Corner
1.4 km South: Midrown Bouth
cateries. It's
glace where easence of New York city, with its unceasing vitality and diverse cultuze, is palpable wish every stop.
Population Denaity (On Scale fron 3.0 to 9.9);
Finetuned -3T-3.5:
9.0
(a) GPT-3.5 describing an address.
(b) Prompting with coordinates strategy.
Figure 1: Example prompts and corresponding GPT responses. In fig. la we show GPT-3.5 demonstrate its geospatial knowledge by asking it to describe an address. However, in ng. Tb (top) prompting GPT-3.5 with just coordinates and finetuning it population density is insufficient. We demonstrate our prompting strategy fig. (bottom) with which finetuned GPT-3.5 is able solve the task correctly (the expected value is 9.0).
finance, and scientific research (Bommasani al. 2021 Zhao et al. 2023). This is because LLMs have compressed the knowledge contained in their training corpus, which includes billions or tril- lions of tokens of data from the internet (Del etang et al.) 2023). Here, we seek to understand whether LLMs possess geospatial knowledge and explore methods to extract this knowledge to of- fer novel set of geospatial covariates that can enhance various geospatial MI. tasks.
As shown in fig. Ta substantial amount of geospatial knowledge in LLMs can be revealed sim- ply by querying them to describe an address. However, extracting this knowledge from LLMs is not trivial. While the most natural interface involves using geographic coordinates like latitude and longitude to retrieve specific location information, this often yields poor results as shown in fig. [b] The inherent challenge lies in the LLMs' capability to understand and interpret these numeric coor- dinates in relation to real-world locations, especially when the location is remote or lesser-known.
In this paper, we introduce Geo novel method that can efficiently extract the vast geospatial knowledge contained in L.LMs by fine-tuning them on prompts constructed with auxiliary map data from OpenStreetMap OpenStreetMap contributors (0117). With our prompting strategy shown in fig. Ib we can pinpoint a location and provide the LLM with enough spatial context information, thereby enabling to access and leverage its vast geospatial knowledge. We find that including infor- mation from nearby locations in the prompt can improve GPT-3.5's (OpenAl, 2023b) performance by 3.3x, measured in Pearson's 's 22, over simply providing the target location' coordinates.
We find that popular LLM models such as GPT-3.5 and Llama 2 (Touvron et al., 2023) can be fine-tuned to achieve state-of-the-an performance on a variety of large-scale geospatial datasets for tasks including assessing population density, asset wealth, mean income, women's 's education and more. With GeoLLM, GPT-3.5, Llama 2, and RoBERTa (Liu et al., 2019), show a 70%, 43%, and 13% improvement in Pearson's 's y2 respectively over baselines that use nearest neighbors and use information directly from the prompt, suggesting that the models' geospatial knowledge scales with their size and the size of their pretraining dataset. Our experiments demonstrate LLMs' remarkable sample efficiency and consistency worldwide. Importantly, GeoLLM could mitigate the limitations of existing geospatial covariates and complement them well.
2

---

Published as a conference paper at ICLR 2024
We present the following main research contributions:
1. Language models possess substantial geospatial knowledge and our method unlocks this knowl- edge across a range of models and tasks.
2. Constructing the right prompt is key to extracting geospatial knowledge. Our ablations find that prompts constructed from map data allow the models to more efficiently access their knowledge.
2 RELATED WORK
Identifying Knowledge Pre-training instills large amounts of knowledge into language models.
Since the introduction of pre-trained language models such as BERT (Devlin et al. 2019). many
works try to quantify the amount of knowledge they can encode (Roberts et al. 2020 Jiang et al,
2019) and see if they can be used knowledge bases (Petroni et al. 2019). Other works have
looked for more specific types of knowledge such as commonsense (Feldman et al. 2019). temporal
(Dhingra et al.] 2021] Liu al., 202T), biomedical (Sung et al.) 2021), numerical (Lin al. 2020).
scale (Zhang et al. 2020), and many others. With the recent introduction of instruction-finttune
LLMs, directly querying knowledge with prompts is a potentially simpler method. In this paper, we focus on showing the quantity and quality of geospatial knowledge in pre-trained language models.
Knowledge Extraction Many works aim improve knowledge extraction from language models by fine-tuning or prompt tuning. Works on prompt tuning explore mining prompts from the web (Jiang at. 2019]. optimizing prompts in the discrete space of words and tokens (Shin et al., 2020), optimizing prompts in the continuous embedding space (Zhong et al.) (02T). and using adapters (Newman et al. 202T). However, prompts are often difficult to craft (Qin & Eisner [202]] Diang al. 2019). and are sensitive to small changes. Fine-tuning language models for downstream tasks has shown to an effective and simple method to tune and elicit specific knowledge for evaluation (Radford & Narasimhan 2018: Dong al., 2019). It is also shown that fine-tuning may result in higher performance gains compared to prompt tuning (Fichtel et al. 2021). Additionally, LoRA and quantization are shown to greatly reduce the computational cost of fine-tuning (Hu al. 2021 Dettmers et al., 2023). For these reasons, we choose to extract geospatial knowledge with fine- tuning.
NLP for Geospatial Tasks Natural language processing (NLP) has been used for many geospatial tasks with various data sources. Hu al. (2017) extracted semantics relations among cities by lever- aging news articles. Sheehan et al. (2019) utilized Wikipedia to predict poverty. Xu et al. (2016) accessed traffic by using OpenStreetMap data. Signorini et al. (201T): Paul & Dredze (011) utilized
tweets to predict track-levels disease activities. Recently, researchers started to explore LLMs' ca- pability on various geospatial tasks. Mai al. (2023a) demonstrated the usability of large language models on various geospatial applications including fine-grained address recognition, time-series dementia record forecasting, urban function prediction, and so on. GeoGPT (Zhang et al., 2023) has been proposed as GPT-3.5-based autonomous AI tool that can conduct geospatial data collection, processing, and analysis in an autonomous manner with the instruction of only natural language. However, both works mainly relied on the pre-trained LLMs and did not explore the possibility of fine-tuning LLMs for a geo-specific foundation model. Deng et al. (2023) developed an LLM in geoseience, so-called K2, by fine-tuning on geoscience text corpus which has shown remark- able performances on various NLP tasks in the geoscience domain. However, the K2 geoscience language foundation model is still limited the common NLP tasks such as question answering. summarization, text classification, etc. In contrast, in this work, we conduct a deep investigation into the possibility of extracting geospatial knowledge from LLMs and whether we can use it for various real-world geospatial tasks like predicting population density. By fine-tuning various LLMs, we demonstrate the quantity, quality, and scalable nature of geospatial knowledge contained in LLMs.
METHOD
Abstractly, we want to map geographic coordinates (latitude, longitude) and additional features to response variable, such as population density or asset wealth. A naive approach would be to spatially interpolate using the coordinates. However, this is far from ideal, especially for small sample sizes. Features or covariates can help, but such data can be difficult to obtain. Given this challenge, we aim to determine how much L.MMs know about these coordinates and how
use that knowledge to
3

---

Published as a conference paper at ICLR 2024
better predict response variables of interest. In this section, we describe approach of extracting this knowledge with the fine-tuning of language models with prompts generated using map data.
3.1 MINIMUM VIABLE GEOSPATIAL PROMPT
minimum viable prompt needs to specify the location and the prediction task. Our basic prompt consists of geographic coordinates and description of the task. The use of geographic coordi- nates is essential to extracting high-resolution data as they are a universal and precise interface for knowledge extraction. The description comprises the task's ' name and the scale indicating the range of possible labels. The ground truth label, is also added when training an LLM with unsupervised learning. We present example of this simple structure in the top part of fig. Tb
While we find that adding the name of the task helps, more specific information, such as the name of the dataset, generally does not help. This suggests that basic context sufficiently assists LLMs in efficiently accessing the knowledge in its weights.
Since the label has to be specified through text for L.Mss, we use classification instead of regression. However, we find that a classification task closer to regression (c.g. 0.0 to 9.9 as opposed to 1 to 5) is beneficial, despite the fact that LLMs use softmax. If the distribution in the original dataset already has an approximate uniform distribution, we simply scale the values to the range from 0 to 9.9 and round to one decimal place. If the dataset not inherently uniform, we distribute the values evenly among 100 bins to maintain uniform distribution. Each bin is then associated with the range of values from 0.0 to 9.9.
3.2 PROMPT WITH MAP DATA
The prompt described above, however, has a major limitation. Through zero-shot prompting, one can easily find that LLMs struggle to identify the locations of coordinates. Not knowing where the coordinates are located would detrimental for geospatial tasks. This is why we focus on making sure the prompt contains additional context that helps the model understand where the coordinates are located. We also design our prompt to be descriptive and useful for a range of geographical scopes such as ZIP code areas single square kilometer.
Our prompt, shown in the lower part of fig. Ib consists of two additional components constructed using map data:
1. Address: A detailed eeverse-geocodder description, encompassing place names from the neighbor- hood level up to the country itself.
2.
Nearby Places: A list of the closest 10 locations in a 100 kilometer radius with their respective names, distances, and directions.
We use OpenStreetMap (OpenStreetMap contributors 2017 for our map data as can be queried for free using various APls, although any map data can be used. In our testing, map data from Google is of higher quality, but was prohibitively expensive. We generate addresses using reverse geocoding from Nominatim (Hoffmann) 2023) and the names and locations of nearby places from Overpass API (Olbrichi] (023). comprehensive prompt template and additional prompt example is provided in appendix A.2
3.3 FINE-TUNING AND INFERENCE WITH LANGUAGE MODELS
RoBERTa ROBERTaBASE is 125 million parameter encoder-only transformer model trained on 160 GB worth of text (Liu al. 2019). We choose this as it and other encoder-only models like BERT (Devlin et al. 2019) are often used for classification and regression tasks (Bommasani al., 2021 J. We fine-tune the model with all of its parameters being trainable. It takes the entire prompt and outputs continuous value which round to the nearest bin.
Llama 2 Llama 2 7B is a billion parameter decoder-only transformer model trained on 2 trillion tokens Touvron et al. 2023). Despite being unsuited for regression tasks, it has the potential to per- form well as it has been trained on significantly more data. Similar to how the model was pretrained, fine-tune Ilama 2 using unsupervised learning the prompts and labels concatenated. We find
4

---

Published as a conference paper at ICLR 2024
Task
Source
Sumples
GPT-3.5
Llama
RoBERTa
GPT-2
MLP-BERT
XGBoust FT
XGBBoott
K-NN
Nightlight
10,000
0.78
0.73
0.60
0.26
0.55
0.53
0.46
0.36
0.68
Population
WorktPop
1,000
0.73
0.63
0.41
0.18
0.42
0.34
0.32
0.11
0.62
100
0.14
0.39
0.54
Asset Wealth
DHS
1,000
10,000
0.75
0.71
0.74
0.69
0.23
0.40
0.62
0.64
0.48
0.63
0.59
0.49
0.55
Women
DHS
10,000
0.64
0.61
0.58
0.29
0.52
0.55
0.55
0.54
0.33
1,000
0.60
0.07
0.3G
0.40
0.40
0:25
Sanitation
DES
10,000
0.65
0.63
0.32
0.61
0.57
0.57
0.41
1,000
0.64
0.16
0.43
0.42
0.43
0.35
10,000
0.68
0.37
0.59
0.09
0.58
0.31
1.000
0.58
0.56
0.50
0.09
0.45
0.43
0.46
0.50
0.21
1,000
0.27
0.21
0.45
Population
100
0.62
0.38
0.00
0.00
0.20
0.13
0.15
0.08
0.34
Income
100
1,000
0.55
0.46
0.29
0.01
0.04
0.00
0.08
0.03
0.04
0.18
0.10
0.22
0.09
0.07
Hisponic Rotio
1,000
0.74
0.07
0.06
0.57
0:24
100
0.64
0.32
0.11
0.00
0.36
0.19
0:23
0.40
0.14
Home Value
ZiBlow
100
1.000
0.87
0.74
0.43
0.00
0.08
0.00
0.35
0.13
0.50
0.27
0.59
0.40
0.06
0.14
Table 1: Pearson's 's 2 for all models across all tasks and training sample sizes.
that this unsupervised learning works better than using supervised finetuning. This means that the model is learning to generate the prompt as well the label. One could imagine that it is beneficial to learn to generate the address given the coordinates or the nearby places using the coordinates and address. During inference time, we provide with the prompt and let it generate the three tokens required for the prediction (e.g.. "9"; "." and "9"). QLoRA (Dettmers et al. 2023) allows us to greatly reduce computational cost and effectively fine-tune with just 33 million trainable param- eters. However, since it also quantizes the frozen model to 4-bit, this comes with the potential cost of diluting the knowledge contained in the weights.
GPT-2 GPT-2 is 124 million parameter decoder-only transformer model trained on 40 GB worth of text (Radford et al. 2019). We use the same finetuning and inference process as Llama 2. We use this model as an LLM baseline and way gain insight into the effect of the size of the model and its dataset on performance.
GPT-3.5 GPT-3.5 (gpt-3. 5-t.....tttuooos1- OOpeAA 2023b) is chat based LLM. It is the successor of GPT-3 which is decoder-only transformer model with 175 billion parameters (Brown et al. 2020). GPT-3.5 shows promise in containing substantial amount of knowledge (Bian al.;
2023 We fine-tune GPT-3.5 through OpenAl's fine-tuning API (OpenAI 2023a). Similar to Llama provide it the prompt and let generate the three tokens required for the completion.
The details of the finetuning of these models and their hyperparameters are specified in ap- pendix A..3 We also demonstrate the importance of fine-tuning appendix A.4 by testing the few-shot performance of GPT-3.5 and GPT-4.
EXPERIMENTS
We construct a comprehensive benchmark that encompasses wide range of geospatial prediction tasks. This benchmark includes various domains and geographies, ranging from women's education levels in developing countries to housing prices in the US. We develop robust set of baselines to demonstrate the performance attainable using the information directly the prompt. Finally, we present our results and an ablation study.
4.1 TASKS AND SOURCES
WorldPop WorldPop Tatem 2017 is a research group focused on providing spatial demographic data for the use of health and development applications, including addressing the Sustainable Devel- opment Goals. We use their population counts dataset that covers the whole world for the population density task. More specifically,
their unconstrained global mosaic for 2020 that has res-
5

---

Published as a conference paper at ICLR 2024
XGBoost
GPT-3.5
Population (USCB)
Home Value (Z3low)
XGBoost
GPT-3.5
Assett Wealth (DHS)
XGBoost-FT
GPT-3.5
Populitiio (WorttPPpp)
New York
Mumbal
Liberia
South West of Yemen
Coast of France
XGBoostF1
GPT-3.5
Figure 2: Plots absolute error comparing the best baselines utilizing no pretraining and GPT-3.5 on tasks from each source with 1,000 samples. We also provide high-resolution plots from various locations around world for the population density task from WorldPop. We show that GeoLLM not only outperforms baselines variety of tasks (table but also demonstrates remarkable geographic consistency across the world.

---

Published as a conference paper at ICLR 2024
olution of 30 are (approximately 1km the equator) (WorldPop & CIESIN 2018 To ensure
comprehensive representation of populations, we employed importance sampling, weighted by pop- ulation. This allows us capture wide range of populations; without it, our samples would largely consist of sparsely populated areas. To construct a learning curve for this task, we use training sample sizes of 100, 1,000, and 10,000.
DHS SustainBench Yeh et al. 2021) includes tasks derived from survey data from the Demo- graphic and Health Surveys (DHS) program (DHS) 2023). These surveys constitute nationally rep- resentative household-level data on assets, housing conditions, and education levels, among other attributes across 48 countries. While DHS surveys provide household-level data, SustainBench sum- marizes the survey data into "cluster-level" labels, where a "cluster" roughly corresponds to village or local community. The geocoordinates are "jittered" randomly up to 2km for urban clusters and 10km for rural clusters to protect survey participant privacy. From their datasets, we use asset wealth index, women educational attainment, sanitation index, and women BMI. Since their datasets cover wide scope, we use training sample sizes of 1,000 and 10,000.
USCB The United States Census Bureau (USCB) USS] 2023) is responsible for producing data about the American people and economy. In addition to the decennial census, the Census Bureau continually conducts annual demographics surveys through the American Community Survey (ACS) program. From these two sources, Hispanic ratio by what they call ZIP Code Tabulation Areas. Since their data only covers the US, training sample sizes of 100 and 1,000.
get tasks for population, mean income, and Hispanic to Non-
Zillow Zillow provides the Zillow Home Value Index (ZHVI) ZRR 2023): : A measure of the typical home value and market changes across a given region and housing type. It reflects the typical value for homes in the 35th to 65th percentile range. We use the smoothed, seasonally adjusted measure that they provide. Like with the USCB data, we also use this data on ZIP code level. This serves as our home value task. Since this dataset also only covers the US, we use training sample sizes of 100 and 1,000.
When explicit coordinates are not given in the datasets, we approximate them by determining the center of relevant areas. For instance, we approximate the center of each of the ZIP code area so that we can generate prompts for the tasks from USCB or Zillow. We evaluate performance using the squared Pearson correlation coefficient 72 between the predictions and labels so that comparisons can be made with previous literature (Perez et al.) 2017 Jean et al.] 2016: Yeh et al. 2020 2 We use 2,000 samples for our test and validation sets across all tasks. We split the data into training, test, and validation partitions early in the process before sampling different sizes. We also explicitly check again for any overlap between the train and test samples before fine-tuning or training.
4.2 BASELINES
The purpose of the first three baselines is to demonstrate the performance attainable using the in- formation in the prompt. Pre-trained language models can surpass these baselines only if they ef- fectively use the knowledge embedded in their weights. This helps understand the extent of knowledge encapsulated within pre- pre-trained language models. The next baseline has pretrained component but provides further insight into performance attainable using sentence embeddings. The last baseline allows for comparison with a well-established satellite-hased method.
k-NN Our simplest baseline is k-NN. takes the mean of the closest neighbors. This is a reasonable baseline as physical distance is very relevant for geospatial tasks.
XGBoost Our second baseline uses XGBoost (Chen & Guestrin 2016). This baseline incorporates all numerical and categorical data from the prompt, including coordinates, distances, and directions. could outperform k-NN since it leverages not just the coordinates, but also the distance and direc- tion data. We established this baseline believing that the direction and distance information could be covariates for some of the tasks we use.
XGBoost-FT
Our third baseline, XGBoost-FT, leverages both XGBoost and fastText (Joulin al.) 2016). This baseline uses all information contained in the prompt. In addition to using the coordi- nates, distances, and directions, it also uses embeddings generated from the address and the names of the nearby places. We use fastText to learn sentence embeddings on any given dataset using strings created by concatenating the address and the names of the nearby places.
7

---

Published as a conference paper at ICLR 2024
MLP-BERT Our fourth baseline, MLP-BERT, also uses coordinates, distances, directions, and an embedding of the address and the names of the nearby places. Unlike XGBoost- it uses sentence embeddings from the 110 million parameter pre-trained BERT (Devlin et al. 2019). To accommodate for the larger embedding size, we use a 2-layer MI.P
Nightlight Our final baseline uses publicly available satellite images of luminosity at night ("night- lights") from VIIRS which comes at a resolution of 500 meters (Elvidge et al. (017). Nightlight
images are commonly used as they are a proxy for economic development (Yeh et al. 2020 Shee- han et al., 2019 Jean et al. 2016). We find that images of sizes 16 km by 16 km work best for our diverse set of tasks. We use GBTs as they have been shown to be effective for single-band nightlight imagery (Perez et al.) 2017). We find that ResNet (He et al. 2015) models, as used in (Yeh et al.)
2020) require large numbers of samples to outperform GBTs.
All baselines except GPT-2, treat the tasks as a regression problem. Their output is rounded to the nearest bin (0.0 to 9.9) for classification. We get the average of the values in a 25 square kilometer area. We then use those values to get the ground truth ranking that is used in the experiments.
4.3 PERFORMANCE ON TASKS
From the results presented in table ! and table 4 it is evident that LLMs are the pest-perfonming models. Among them, GPT-3.5 stands out as the best-performing LLM. Its Pearson's r² ranges from 0.46 to 0.74, 0.55 to 0.87, and 0.63 to 0.78 for sample sizes of 100, 1,000, and 10,000, respec- tively. While these variations indicate that the tasks differ in difficulty, LLMs generally outperform all prompt-based baselines and even the nightlight baseline. This suggests that LLMs possess a substantial amount of geospatial knowledge.
Not only does GPT-3.5 outperform all other models on every test, but its performance is also rel- atively consistent across tasks and sample sizes. Llama 2 does better than all baselines for 17 out of the 19 total tests and consistently does better than RoBERTa. RoBERTa consistently does bet- ter than all prompt-based baselines at 10,000 training samples, but struggles at lower sample sizes. Furthermore, GPT--3.5 and Llama 2 retain much more acceptable level of performance at a sample size of 100, emphasizing their sample efficiency.
GPT-3.5 and Llama 2 do especially well with the population tasks from WorldPop and the USCB compared to the baselines. GPT-3.5 also especially impressive with the home value task from Zillow with a Pearon's 22 of up to 0.87. However, the difference in performance between the models is less pronounced for the tasks from the DHS. This might be due to the noise that added when the coordinates for these tasks are "jittered" up to 5 kilometers. With the added noise, it is potentially more difficult to achieve a high Pearson's 's ?
As shown in fig. GPT-3.5 outperforms Llama 2 and RoBERTa by 19% and 51% respectively, indicating that the method scales well. Fig. again shows that the sample efficiencies of Llama 2 and GPT-3.5 are exceptional, especially when making predictions a global scale. Note that with larger sample sizes the gaps in performance will decrease as the physical distances between the training coordinates and test coordinates become smaller.
It is visually clear in fig. 2 that GPT-3.5 does substantially better than the best prompt-based base- lines trained from scratch for each task. It also appears that GPT-3.5 generally only struggles where the baselines struggle, suggesting that those areas might just be difficult to estimate rather than it be- ing a specific failing of the model. Notably, GPT-3.5 shows has impressive geographic consistency.
4.4 ABLATIONS ON THE PROMPT
The ablation study shown in table 2 demonstrates the importance of using a highly descriptive prompt that helps the model understand where the coordinates are located and efficiently access the knowledge contained in its weights. It also suggests that all components of the prompt con- tribute to the overall performance of the models. In addition to performing better than the other language models, GPT-3.5 is more resilient to the removal of the map data. This is evident from the fact that prompt without map data is completely ineffective for Llama and RoBERTa. This suggests that GPT-3.5 can more effectively access its knowledge.
8

---

Published as a conference paper at ICLR 2024
Ablation
GPT-3.5
Llama 2
RoBERTa
Whole prompt
0.73
0.63
0.41
Removed coordinates
0.72
0.63
0.36
Removed address
0.68
0.51
0.38
Removed nearby places
0.58
0.47
0.28
Only nearby places
0.65
0.51
0.38
Only address
0.57
0.48
0.28
Only coordinates
0.22
0.00
0.00
Table 2: Pearson's 's x² for ablations with the population density task from WorldPop and 1,000 training samples.
5 DISCUSSION
LLMs not only match but also exceed the performance of methods that use satellite imagery. satellite-based methods achieve Pearson's y2 of 0.56 (Jean et al. 2016), 0.71 (Perez et al. 2017). and 0.67 (Yeh et al.) (2000) in predicting asset wealth in Africa. With 10,000 training samples (ap- proximately 3, 5,700 of which are from Africa), fraction of the over 87,000 data points available from the DHS, GPT-3.5 achieves an y² of 0.72 when evaluated in Africa.
The performance of the LLMs is particularly notable considering an inherent disadvantage they have relative to the baselines and RoBERTa. Generating multiple tokens with softmax is far less intuitive than regression which the other models use. Despite this, there are substantial improvements in performance and sample efficiency with the LLMs.
It also appears that the significant increase in performance is not simply due to the number of param- eters being trained during fine-tuming iinettunig. With QLoRA, Llama has 33 million trainable parameters as opposed to RoBERTa's 's 125 million trainable parameters. This suggests that the performance increase is due the knowledge contained Llama 2's frozen weights.
This method's performance appears to scale with both the size of the model's pretraining dataset and its capacity for data compression, capability influenced by the model's size. This suggests our method likely scales with the general capability of the model as well (Kaplan et al. 2020). For example, GPT-4 (OpenAl, 2023c) would likely outperform GPT-3.5.
One could potentially use our method to better understand the biases of LLMs that they inherit from their training corpora. Surprisingly, the performance across countries and continents appears very consistent, as evident from the maps of absolute errors shown in fig. 2 However, preliminary evidence of biases towards sparsely populated or underdeveloped areas is shown in appendix A.6
This work lays the foundation for the future of LLMs for geospatial tasks. Additional dimensions could extend the applicability of our method. For instance, one could augment prompts with names of amenities or roads to enrich the geospatial context and allow for even higher-resolution knowledge extraction. Temporal data such as dates could be integrated into the prompts for making forecasts. Multimodal LLMSs could potentially also leverage satellite imagery street view images. Overall, there are likely many ways to extend GeoLLM.
CONCLUSION
In this work, we introduced GeoLLM, a novel method that can efficiently extract the vast geospatial knowledge contained in LLMs by fine-tuning them on prompts constructed with auxiliary map data from OpenStreetMap. We found that including information from nearby locations in the prompt greatly improves L.MMs performance. We demonstrated the utility of our approach across various tasks from multiple large-scale real-world datasets. Our method demonstrated a 70% improvement in Pearson's 's x2 over traditional baselines including k-NN and XGBoost, exceeding the performance of satellite-based methods. With GeoLLM, we observed that GPT-3.5 outperforms Llama 2 by 19% and RoBERTa by 51%, suggesting that the performance the model and its pretraining dataset. Our simple method revealed that LLMs are sample-efficiee rich in geospatial information, and robust across the globe. Crucially, GeoLLM shows promise in substantially mitigating the limitations of traditional geospatial covariates.
our method scales well with the size of

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
This research is based upon work supported in part by the Office of the Director of National Intelli- gence (ODNI, Intelligence Advanced Research Projects Activity (IARPA), via 2021-201 10000044 NSF(#165 151655), ARO views and conclusions contained herein are those of the authors and should be interpreted as necessarily representing the official policies, either expressed implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for gov- ermmental purposes not wot-ttihstanding any copyright annotation therein.
ONR (N00014-23-1-21555
CZ Biohub, HAL The
REFERENCES
The demographic and health surveys (dhs) program, 2023. URL http::/ww..dspprgg//
Con
Census bureau tables, 2023. http:tss///aaa..cassusoo///////e
Housing data, 2023. URL https: pestps://ww..r. www./www..o. ow. com wwwp://llww..com//rreerrccc/ccea
Wala Draidi Areed, Aiden Price, Kathryn D Arnet,, and Kerrie Lee Mengersen. Spatial statisti- machine learning models to assess the relationship between development vulnerabilities and educational factors in children in queensland, australia. BMC Public Health, 22, 2022. URL
httpss///ppi www.seeeett... semantiicscheeee cecccocchoooooooooo www.:larrrrr/oooooottooo 2540011 901
John E. Ball, Derek T. Anderson, and Chee Seng Chan. Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community. Journal of Applied Remote Sensing. 11, 2017. URL URI.https:///ap
semant cscholar: org/ Co Crrg/Copuss. DD: 000::009990 7069900
Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. Chatgpt is knowledge- able but inexperienced solver: An investigation of commonsense problem in large language
models. ArXiv, 115/2303.1421., 2023. URL https: api.
org/
ID: C
Joshua Evan Blumenstock, Gabriel Cadamuro, and Robert On Predicting poverty and wealth from mobile phone metadata. Science, 350:1073 - 1076, 2015. URL https://appi
semanticscholas org/CorpusID 22063::996669999
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.07258: 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Vonl Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, 0ttp/20055.1166.. 2020. URL https: /api. semanticscoolar. orrg CorpusID:21897111
M. Burke, Anne Driscoll, D. Lobell, and Stefano Ermon. Using satellite imagery under- stand and promote sustainable development. Science, 371, 2019. URL https://api.
semanticscholeat htt.://aa...aeee/oooooosaa22.22.
Ting Chang, Yingjie Hu, Dane Taylor, and Brian M Quigley. The role of alcohol outlet visits derived from mobile phone location data in enhancing domestic violence prediction the neighborhood level. Health & Place, 73:102736, 2022.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016. URL https://aap
ar.
10

---

Published as a conference paper at ICLR 2024
Guanghua Chi, Han Fang, Sourav Chatterjee, and Joshua Evan Blumenstock. Microestimates of wealth for low- and middle-income countries. Proceedings of the National Academy of Sci- ences of the United States of America, 119, 2021. URL/ttps::/ http:///aapaaeetatttcccccoocoeee
org/CorpustD ID: 233289683
Gr' egoire Del etang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Wenliang Kevin Li, Matthew Aitchison, Laurent Orseau, Mar- cus Hutter, and Joel Veness. Language modeling is compression. //api semant www.rniiscchaa....commmm hat.aar.orrrgooorurrt.2220..0-.0
2023. URL https:
Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Le Zhou, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al. Learning foundation language model for geoscience knowledge understanding and utilization. arXiv preprint arXiv:2306. 050644 2023.
Tim Dettmers, Artidoro Pagmoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. ArXiv, abs/2305.1433 2023.
org/CorpusID: 258841328
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, Abs//1810.0485,, 2019. URL
https: www.seinnnncchaaate..eeeemm wwt.:aaarrrrr//oouuuoeee :t2
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273, 2021. URL https://api.
semanticscholar: wwwp://aaarrr///ooooooo Serg/CorpusID :2::::2355566
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, M. Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understand- ing and generation. In Neural Information Processing Systems, 2019. URL https://apll semanticschole
Christopher D. Elvidge, Kimberly E. Baugh, Mikhail N. Zhizhin, Feng-Chi Hsu, and Tilottama Ghosh. Viiss night-time lights. International Journal of Remote Sensing, 38:5860 5879, 2017. URL https: /satps//aap... www.sttesacsscaaoe ar Sttg:/CoouSID::24555505
Joshua Feldman, Joe Davison, and Alexander M. Rush. Commonsense knowledge mining from pretrained models. ArXiv, aws//a09.000510 2019. URLhttps:://agge http://ppp..eeantcccchchoooll/
org/Corpus 202541043
Leandra Fichtel, Jan-Chrise Chrr-hhistophh Kalo, and Wolf-Tilo Balke. Prompt tuning fine-tuning in- vestigating relational knowledge in pre-trained language models. In 3rd Conference on Auto- mated Knowledge Base Construction, 2021. URL httt:ss//openrevieew.nee//e
id=o7sM1 pr
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015.
URL https: /tss://app... www.nnicschooioooee.o...ccccccce
org/ Corpus ID: 20:55
Andrew Head, Mélanie Manguin, Nhat Tran, and Joshua Evan Blumenstock. Can human devel- opment be measured with satellite imagery? Proceedings of the Ninth International Confer- ence Information and Communication Technologies and Development, 2017. URL https:
Sarah Hoffimann. Nominatim, 2023. URL htthhtss:/// ptttpp:://nominaiiin im..iiritiii.ggg...
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021. uttpttts::
semant ccscholas www.larr.ororrpooo ID: 23:4::3350000
Yingjie Hu, Xinyue Ye, and Shih-Lung Shaw. Extracting and analyzing semantic relatedness be- tween cities using news articles. International Journal of Geographical Information Science, 31 (12):2427-2451, 2017.
11

---

Published as a conference paper at ICLR 2024
Neal Jean, M. Burke, Sang Michael Xie, W. Matthew Davis, D. Lobell, and Stefano Ermon. Com- bining satellite imagery and machine learning to predict poverty, Science, 353:790 - 794, 2016.
semant sem.nncccholarrr.......c.. org/ htt.:/aaa......ooooutui:::::::::
Zhengbao Jiang, Frank F. Xu, J. Araki, and Graham Neubig, How can know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2019. URL https::/aapppp htt.://ap..ssmmaacccccccccoooooa
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01755 2016.
Jared Kaplan, Sam McCandlish, T. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, 0as/2001.08366 2020. URL htttss:/app...emaatiiccccs..s semanticsccolll org/ CorpusID:2108600
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! nu- mersense: Probing numerical commonsense knowledge of pre-trained language models. In Conference Empirical Methods in Natural Language Processing, 2020. URL https:
//api. semant wwwpaaaaiiccccooaorrooooooo
Leo Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing across time: What does roberta know and when? In Conference on Empirical Methods in Natural Language Processing, 2021. URL httpthtts:////t semanticscholar..... org/ www.arorrr/////oDD00/
233289478
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692 2019. http:::t::/pppp www.ntttsccoooooaa......ccccccee org/ Corpus ID: 1D:199953378
Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, et al. On the opportunities and challenges of foundation models for geospatial artificial intelligence. arXiv preprint arXiv:2304.0673 2023a.
Gengchen Mai, Ni Lao. Yutong He, Jiaming Song, and Stefano Ermon. Csp: Self-supervised con- trastive spatial pre-training for geospatial-visual representations. In the Fortieth International Conference on Machine Learning (ICML 2023), 2023b.
Gengchen Mai, Yao Xuan, Wenyun Zuo, Yutong He, Jiaming Song, Stefano Ermon, Krzysztof Janowicz, and Ni Lao. Sphere2vec: A general-purpose location representation learning over spherical surface for large-scale geospatial predictions. ISPRS Journal of Photogranmetry and Remote Sensing, 202:439 2023c.
Catherine Nakalembe. Characterizing agricultural drought in the karamoja subregion of uganda with meteorological and satellite-based indices. Natural Hazards, 91:837-862 2018. URL https:
Benjamin Newman, Prafulla Kumar Choubey, and Nazneen Rajani. P-adapters: Robustly extracting
factual information from language models with diverse prompts. ArXiv, abs/2110.07256 2021.
URL https: www.sttesanccchooooooooom
wttp:oooooooo 238856973
Kristine Nilsen, Natalia Garavio-,, Douglas R. Leasure, Chigozie Edson Utazi, Corrine War- ren Ruktanonchai, Adelle Wigley, Claire A. Dooley, Zoe Matthews, and Andrew J. Tatem. A review of geospatial methods for population estimation and their use in constructing repro- ductive, maternal, newborn, child and adolescent health service indicators. BMC Health Ser- vices Research, 21, 2021. URL
237487421
Roland Olbricht. Overpass api, 2023. Utt.:ppps//Mtt....eesrreeee.pogp
Overpass API
OpenAl. Gpt-3.5 turbo: Fine-tuning and api updates, 2023a. URL https:////ppea..coomm b blog/gpt-23 5 3- wwwerroo-fineetttiiin -nn-unnnnn--------ddda-------ee-
12

---

Published as a conference paper at ICLR 2024
OpenAl. Introducing chatgpt, 2023b. URL ott.tttess//ppena...oo/////////p
OpenAL. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023c.
URL https://api.
semanticscholar: wtt//orpps1D::2577333222
ppeStreetMapponttrbbn Planet dump retrieved from https://planet.omm.org http:::ttt////tto openstzeetmap. org 2017.
Michael J. Paul and Mark Dredze. You are what you tweet: Analyzing twitter for public health. Proceedings of the International AAAI Conference on Web and Social Media, 2011. URL
wttpo/aaarrrgooooooooeooooo wwtg/orppsssi.ccoooooooooo 92: 9270435
Anthony Perez, Christopher Yeh, George Azzari, M. Burke, D. Lobell, and Stefano Ermon. Poverty prediction with public landsat 7 satellite imagery and machine learning. ArXiv, abs/1711 1.06544, 2017. URL https: /ttps::/api..stee http:ppppp.mmmattcccccccccaaaaaa org/ Corpuorrpussaaa 237481777
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases? ArXiv, Abss/190,,0166 2019.
URL URL htkthhtps://agg htw.aappisemannncschhhooeee org. wtt.cco......ooooooooopp22222222
Guanghui Qin and Jas' Eisner. Learning how to ask: Querying Ims with mixtures of soft
prompts. ArXiv, 0/s/2104406599, 2021. URL https::/
NorpusID:2332314:
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018. URL httttttss///app wewantsemmntti.mm iesmmnntiscco" wwwchollrrr..........c. law.oorrroooouuase 4 3245
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models unsupervised multitask learners. 2019. URL https://api.
: 660025533
Adam Roberts, Colin Raffel, and Noam M. Shazeer. How much knowledge can you pack into the parameters of language model? In Conference Empirical Methods in Natural Language Pro- cessing, 2020. hettrpsss/ppp..mmmmcccccccoooo
//attps::/pp.... wwwanticscoolos.too.....cc
Ott://orrpusI:::122250 21120001121111111
Caleb Robinson, Fred Hohman, and Bistra N. Dilkina. A deep learning approach for popu- lation estimation from satellite imagery. Proceedings of the Ist ACM SIGSPATILL Work- shop on Geospatial Humanities, 2017. URL
CorpusID:12713973
Evan Shechan, Chenlin Meng, Matthew Sze Piao. Tan, Burak Uzkent, Neal Jean, D. Lobell, M. Burke, and Stefano Ermon. Predicting economic development using geolocated wikipedia articles. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discov- ery & Data Mining, 2019. URL https: semanticscholaat wtw.:/ccchhoaarooooooooo.ss...e
146120697
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Elic- iting knowledge from language models using automatically generated prompts. In Confer- ence on Empirical Methods Natural Language Processing, 2020. URL httts:://ppi.
seemanticscoolar.oore httpaa/o..............ooooaaaaaa
Alessio Signorini, Alberto Maria Segre, and Philip M. Polgreen. The use of twitter to track levels of disease activity and public concern in the u.s. during the influenza a hlnl pandemic. PLOS ONE, 6. 2011. URL /api. semant csccholar
/trr//oruussaaa ID: 18:8801
Helen R. Sofaer, Catherine S. Jarnevich, Ian S. Pearse, Regan L. Smyth, Stephanie Auer, Ger- icke Cook, Thomas C. Edwards, Gerald F. Guala, Timothy G. Howard, Jeffrey Morisette, and Healy Hamilton. Development and delivery of species distribution models to inform decision- making. BioScience, 2019. URL https::/ //api. wwwanticscholeeeee.eeeeeeeccceee
1953847755
Mujeen Sung, Jinhyuk Lee, Sean S. Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. Can language models be biomedical knowledge bases? ArXiv, 0ttps//19.0071... 2021. URL https: //ap1.
semant i cschol ar otg.t//o....rrr/ooooooot///3333
13

---

Published as a conference paper at ICLR 2024
Andrew J. Tatem. Worldpop, open data for spatial demography. Scientific Data, 4, 2017. URL http://///ap...saantiics... www.eaatttecccoll...oooooorrrrre Ct:
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Bsiobu, Jude Fernan- des, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Ma- dian Khabsa, Isabel M. Kloumann, A. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mi- haylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan. Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang. Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tumed chat models. ArXiv, 2ww/2307.09288. 2023. https:tss: /api semant eeemanticschollle...... org/
TD:
WorldPop and Columbia University CIESIN. Global high resolution population denominators project funded by the bill and melinda gates foundation, 2018. URLhttps: /xxx doi. org/
SCTON /WP 64 www. www.worldpop.org School of Geography and Environmental Science, University of Southampton; Department of Geography and Geosciences, University of Louisville; Departement de Geographie, Universite de Namur.
Frank F. Xu, Bill Yuchen Lin, Qi Lu, Yifei Huang, and Kenny Q. Zhu. Cross-region traffic prediction for china on openstreetmap. Proceedings of the 9th ACM SIGSPATIAL International Workshop on Computational Transportation Science, 2016. URL WRR.hhts:://app.. semant cacholar.
org/CorpusID: 1 15751676
Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, D. Lobell, Ste- fano Ermon, and M. Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature Communications, 11, 2020. URL https:
//api. hawpaaatccccccacaaoooo//roooooaa
Christopher Yeh, Chenlin Meng, Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Jihyeon Lee, Marshall Burke, D. Lobell, and Stefano Ermon. Sustainbench: Benchmarks for monitoring the sustainable development goals with machine learning. ArXiv, abs/2111.04424, 2021. URL ar httphhaaaa.rrrrorrrrrtaa// 24:8:44388888
ht.ps::/app. htt.:/pp...mmattttsssssss
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. Do language embed- dings capture scales? In BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NI.,, 2020. URL htttps://api semanti
Yifan Zhang, Cheng Wei, Shangyou Wu, Zhengting He, and Wenhao Yu. Geogpt: Understanding and processing geospatial tasks through autonomous gpt. arXiv preprint arXiv:2307.07930. 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong. Yifan Du, Chen Yang. Yushuo Chen, Z. Chen, Jinhao Jiang. Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. A survey of large language models. ArXiv, abs/2303. 182223....12,,.. 2023. URL https://app. semant. i cscho l ar. org/ Co TD: : 2572257777 900969
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]: Learning vs. learning to recall. In North American Chapter of the Association for Computational Linguistics, 2021. URL
https: / //ppi..../// semanticschol. ort//orrpusD::332200
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
ADDITIONAL VISUALIZATIONS OF PERFORMANCE
0.8
0.68
0.6
0.57
2 performance
0.45
0.4
0.37
0.44
0.30
0.3
K-NN
XGBoosi
0.2
XGRoost-FT
0.1
GPE35
Lians
ROBERTS
XGBbout-PP F7
XiBoost
E:NN
1,000
Training Sumple
Figure 3: Mean Pearson's 's for models across all tasks at 1,000 training samples
Figure 4: Learning curves for population density task from WorldPop.
A.2 PROMPT TEMPLATE
To aid the reproducibility of results, we provide the prompt template, shown in fig. 5a that is used to construct all prompts across all tasks. In addition to the example shown in the lower part of fig. Lb we provide an example used for the asset wealth task shown in fig. 5b that further clarifies how the prompt template is used.
natedinates::
(clasitudeo, clangitude>)
Coordi netex: 63706, 8 8.48625)
Address: *caddress>"
Address: "Kanchok,,
Nigeria"
Nearby Placea:
Nearby Places:
ediscance> adit <placs>
4.7
Kanohok
sdistance> sdleection::
<place>
9.1
Horchs Jankasa
odiatances kn <di rect ends
splaces
10.0 km Rest Matagani
adistance> kn Soit
splaces
12.0 kn Besl: Raguzo
<distance> edisectlears odiatanncd
Souths Tukun Kasa
rect ends
<place>
South- Peat: Manyi
edistance? kn cdic splace>
15.3
ka South Weals Pačan
<distance> ka edisectienr: ediatance> <dicwetionss <place>
splaces
17.1
km South: Wazo Taho
<placeo
North:
Souths
RanaCi
<label>
Wealch
Scale 0.0 9.91s 2.1
(a) Template of prompt and its label.
(b) Example of a prompt and its label.
Figure 5: The <latitude>
Kaddress> <langitude> are always rounded 5 decimal places.
is reverse geocoded address. The <di stance> is always in kilometers and is rounded to one decimal place. The <di rect on> is one of eight cardinal and intercardinal directions which are "North", "Fast", "South". "West", "North-Eat East", "South-East" "South- West", or "North-West". The <place> the actual name of the place. <task>
name of the task. In our experiments, the <task> is one of "Population Density", "Asset Wealth", "Women's Education". "Sanitation". "Women's BMP", "Mean Income", "Hispanic/Latino to Non- Hispanic/LLatin Ratio", or "Home Value". The <label> is the ground truth (or complction, which is prediction from LLM during inference) which can be any number between 0.0 and 9.9 rounded to one decimal place.
15

---

Published as a conference paper at ICLR 2024
A.3 LANGUAGE MODEL FINETUNING DETAILS
RoBERTa We fine-tune the model for regression with all of its parameters being trainable. With a single V100 GPU, finetuning never takes more than 2 hours. We use learning rate of 1 x 10-5 with AdamW optimizer, training for 8 epochs with a batch size of 16, warmup ratio of 0.1, weight decay of 0.1, and a cosine learning rate scheduler.
Llama Similar to how the model was pretrained, we fine-tune Llama 2 using unsupervised learning with the prompts and labels concatenated. With QLoRA, finetuning never takes more than hours for 10,000 samples with a single A A100 GPU. For LoRA, we use rank of 64, alpha of 16, and dropout probability of 0.1. We also use 4-bit quantization with 4-bit NormalFloat. We train for 4 epochs with bíloat16 mixed precision training, a batch size 8, gradient accumulation steps of 2, gradient checkpoints enabled, maximum gradient norm of 0.3, initial learning rate of 1.5 x 10-3 for the AdamW optimizer, a weight decay of 0.001, a cosine learning rate scheduler, and warmup ratio of 0.03. Interestingly, the model's performance still improves even when the loss on the validation set increases by the fourth epoch. This suggests that the model is less prone to overfitting on the prediction tasks compared to the prompts.
GPT-2 We use the same finetuning process for GPT-2 as Llama 2. The only difference is that we train the model for 8 epochs instead of Other changes in hyperparameters do seem to increase performance significantly.
GPT-3.5 We find that 4, and 2 epochs work well for 100, 1,000, and 10,000 training samples respectively. This costs roughly $0.67, S5, and $33 for those sample sizes. It is worth noting that the specific fine-tuning procedures and internal architecture of GPT-3.5 are not publicly disclosed by OpenAl.
A.4 FEW-SHOT
We demonstrate GPT-3.5 and GPT-4 few-shot performance by providing them with a series of the 10 closest (by physical distance) training samples for each test example. We use system messages "You are a detailed and knowledgeable geographer" and "You complete sequences of data with predictions/essimates" before providing the 10 examples.
The test set consists of 200 samples. The size reduction of the test set is required 10-shot is quite expensive, especially with GPT-4. We also reevaluate the performance of the finetuned GPT-3.5 model with this smaller test set be able to make direct comparisons.
The results in table 3 demonstrate that while sho-shot does work, it performs significantly worse than the respective finetuned model as can be seen with the performance difference between the GPT-3.5 models. Interestingly, GPT-4 does far better than GPT-3.5 the few-shot setting, suggesting that fine-tuned version of this model could potentially perform significantly better.
Task
Source
Total Samples
10-Shot GPT-4
10-Shot GPT-3.5
Finetuned GPT-3.5
Population
WorldPop
10,000
0.63
0.53
0.80
Asset Wealth
DHS
10,000
0.65
0.62
0.78
Population
USCB
1.000
0.49
0.39
0.73
Home Value
Zillow
1,000
0.79
0.55
0.86
Table 3: Pearson's 2 for few-shot performance.
A.5 MEAN ABSOLUTE ERROR
As potentially more interpretable metric, present the mean absolute error (MAE) for all models across all tasks and training sample sizes in table 4 One can observe that the same conclusions that made with the Pearson's ? can be made when comparing the MAE of the various models. In particular, the relative ranking in performance of the models within tasks is consistent with Pearson's 2
16

---

Published as a conference paper at ICLR 2024
Task
Source
Sumples
GPT-3.5
Llama
RoBERTa
GPT-2
MEP-BERT
XGBoust FT
XGBooe
K-NN
Nightiihtt
10,000
1.02
1.12
1.44
2.32
1.49
1.56
1.65
1.81
1.26
Population
WorktPop
1,000
1.13
1.40
1.75
2.69
1.81
1.88
1.92
2.26
1.38
100
2.26
1.55
Asset Wealth
DHS
1,000
10,000
0.94
0.97
1.06
1.64
1.22
1.19
1.45
1.20
1.21
1.42
1.34
Women
DHS
10,000
0.89
0.92
0.98
1.38
1.09
1.00
1.01
1.00
1.27
1,000
0.95
3.62
1.19
1.30
Sanitation
10,000
1.44
1.49
1.46
2.44
1.80
1.37
1.64
1.57
1.96
1,000
2.98
1.96
1.93
2.08
0.37
0.50
0.39
0.39
0.39
0.51
1.000
0.40
0.60
0.46
0.45
0.43
0.54
Population
1,000
100
1.35
1.98
2.50
3.07
2.59
2.15
2:27
2.24
2.11
2.34
1.63
1.82
Encomee
1,000
100
1.61
2.41
2.44
3.24
4.28
2.36
2.19
2.48
2.53
2:14
2.38
2.45
2.32
Hisponic Rotio
1,000
1.11
3.05
1.48
2,02
100
1.30
2.43
2.43
3.60
1.94
2.13
2:31
1.78
2:27
Home Value
ZiBlow
1,000
0.74
2.82
1.74
1.63
2.23
100
1.11
1.81
2.51
3.26
1.94
2.29
2.03
1.81
2.44
Table Mean absolute error (MAE) all models across all tasks training sample sizes.
A.6 PRELIMINARY INVESTIGATION ON PERFORMANCE BIAS
While there does not appear to be any obvious signs of performance bias across countries or conti- nents seen in fig. 2 the existence of biases in performance is inevitable it is very likely that the internet training corpora LLMS are inherently biased towards developed and densely populated areas. We find preliminary signs of this as the increase in MAE shown in table indicates ate performance bias towards densely populated and developed areas. However, further research is needed for comprehensive analysis the geospatial biases LLMs and their training corpora. As we have demonstrated here, GeoLLM has the potential to be used as a tool to reveal LLM biases on a geographical scale.
indicates moder-
Task
Source
Above Median Ground Truth Below Median Ground Truth
Population
WorldPop
0.94
1.09
Asset Wealth
DHS
0.88
0.98
Table 5: Mean absolute errors of GPT-3.5 trained on 10,000 samples for population density and asset wealth tasks. Performance is shown for the subsets above and below the median ground truth values (ground truth population density and asset wealth respectively).
17


Reviews:
[{"review_id": "2m1HJE2dOK", "reviewer": "Reviewer_xypS", "summary": "This paper presents a method to extract prior geospatial knowledge from pretrained LLM. \nThe queried knowledge includes spatial demographics data, census data, and survey data. \nThe findings in the paper are interesting, however seem very preliminary. And the experiments are limited.", "questions": "•\tIs there any specific tokenization process introduced for GPS coordinates? Normally, the direct tokenizers with LLMs could split the GPS point into several different tokens, which might undermine the ability to understand GPS coordinates correctly.\n\n•\tWhy using classification setting? The current loss functions that LLMs are using is a kind of binary, where only predictions that exactly match the masked word are considered correct and rewarded, while all other predictions are considered incorrect and penalized. So, there is no has no awareness and no sense of being close to or far away the correct answer. Normally, for the task described in this paper, this kind of sense should be valuable in getting better performance.\n\n•\tWhat about the deployment cost of the proposed GeoLLM? For example, compared to the baselines, what are the fine-tuning costs and inference costs of using GPT-3.5 or other LLMs. From my experience, fine-tuning GPT-3.5 can be very expensive.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper is well presented, and the experiments cover several different geospatial datasets and tasks in relation to census and demographic data.", "weaknesses": "•\tIt seems that the proposed GeoLLM can only perform one specific task. After fine-tuning, is the fine-tuned LLMs (e.g., GPT-3.5) able to retain the ability to answer general questions that is not related to the specific task? Including some discussions about the generalization part could be useful.\n\n•\tThe proposed model can only handle static/tabular geo information. It does not handle other types of spatial data, or spatiotemporal data and tasks.\n\n•\tBaselines are too simple. Considering some more recent deep-learning-based or Transformer-based baselines could be more convincing.\n\n•\tThe experiments are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hqj5SHk7RV", "reviewer": "Reviewer_5MUc", "summary": "This study explores leveraging large language models (LLMs) for geospatial prediction tasks, addressing limitations of traditional covariates like satellite imagery. The authors introduce GeoLLM, an approach that effectively extracts geospatial knowledge from LLMs with auxiliary map data. The proposed method demonstrates a 70% improvement in performance compared to baselines, rivaling satellite-based benchmarks. GPT-3.5 outperforms other models, highlighting the scalability of the proposed approach. This research underscores LLMs' efficiency, global robustness, and potential to enhance geospatial analysis.", "questions": "Have you tried the zero-shot or few-shot performance of various LLMs on the presented tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It proposed a novel method for efficiently extracting geospatial knowledge from large language models.\n2. The paper outlined experiments to evaluate extracting geospatial knowledge from large language models, which included constructing a comprehensive benchmark, developing a robust set of baselines, and presenting results and an ablation study.\n3. The paper revealed that GeoLLMs are sample-efficient, rich in geospatial information, and robust across the globe.", "weaknesses": "1. The paper does not provide a detailed analysis of the potential biases of LLMs and their training corpora. \n2. It would be better to compare the GeoLLM’s performance with the results from satellite images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sbi9HCKfrX", "reviewer": "Reviewer_tFFk", "summary": "This paper proposes GeoLLM, a prompt method that enriches geographic coordinates with auxiliary map data for geospatial prediction task (e.g., population density, economic livelihoods). This prompt approach outperforms some baseline methods in several tasks.", "questions": "1.  While Peason's r^2 is used for comparisons with previous literature, can I see more comparisons with baselines on other evaluation metrics like MAE or MSE?\n    \n2.  Could the paper present the label distribution to evaluate performance variations between urban and rural areas, or densely versus sparsely populated regions?\n    \n3.  It would be beneficial to include a wider range of baselines, particularly \"weaker\" text embedding models like Doc2Vec, ELMo, and BERT, paired with basic predictors, for a holistic performance comparison.\n    \n4.  I tried to use GPT 4.0 for the example in Figure 1, and it looks like it can solve this question pretty well with only coordinates:  \n    \"\"\"User: Coordinates:  \n    (40.76208, -73.98042)  \n    Population Density (On a Scale from 0.0 to 9.9):\n    \n    ChatGPT:  \n    Given the coordinates (40.76208, -73.98042), this location is in New York City, specifically in Manhattan. Manhattan has one of the highest population densities of any area in the United States.  \n    On a scale from 0.0 to 9.9, where 9.9 would represent the most densely populated, Manhattan would be close to the top of that scale. While I can't provide an exact number without more specific criteria for each point on the scale, it would be reasonable to rate Manhattan, particularly Midtown which is where these coordinates roughly point to, above 9.0 on a density scale capped at 9.9, considering its population density surpasses 10,000 people per square kilometer.\n    \n    User:  \n    Just return a number\n    \n    ChatGPT:  \n    9.5  \n    \"\"\"  \n    The paper could enhance its soundness by examining the performance with more powerful LLMs. Performance insights using LLama 13b and 70b, as well as the latest GPT-4 model, would be valuable.\n\n\n\n\n5.  Can I see the performance on LLMs without finetuning? It would be necessary to see the impact of fine-tuning in geospatial predictions.\n    \n6.  Few-shot learning capabilities are an intriguing aspect of LLMs. An analysis of the model’s few-shot performance, both pre and post fine-tuning, could significantly add to our understanding of the method's efficiency and flexibility.\n    \n7.  The current ablation study does not fully clarify the independent contribution of coordinates to the model's performance. Results from experiments that exclude coordinates from prompts would help demonstrate that the enhanced performance is not solely attributable to the auxiliary map data.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  This paper is well structured and good at the clarity of presentation.\n\n2.  The significance of the problem is high, potentially impacting a wide range of geospatial applications and offering a new way to view large language model capabilities.\n\n3. The paper provides a detailed discussion of how large language models can be linked to geospatial applications. This can potentially bring new research opportunities to the field of spatial data mining.", "weaknesses": "1.  The method presented seems to build incrementally on existing methodologies to generate auxiliary texts for prompt engineering, which may not represent a significance in technical novelty.\n\n2.  The scope of experiments could be further expanded to support the conclusion. Please consider the Questions below for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WXyNXhlg5X", "reviewer": "Reviewer_Gcfb", "summary": "This paper introduces a new method called GEOLLM, which can effectively extract spatial knowledge from large language models and fine tune it by using auxiliary map data from OpenStreetMap. Experiments on various tasks of several large-scale real-world sets show that the method is practical and effective. In the experiment, the author found that the pre-training language model has rich geospatial knowledge, and their method can unlock this knowledge. In addition, the author also discusses how to construct appropriate hints to extract geospatial knowledge, and how to find a balance between knowledge extraction and sample efficiency.\nThe contributions of the paper include demonstrating the quantity, quality, and scalable nature of geospatial knowledge contained in LLMs and presenting a simple and efficient method for extracting this knowledge.", "questions": "Besides what is mentioned in weaknesses, what does “the prompts’ mean in “This suggests that the model is less prone to overfitting on the prediction tasks compared to the prompts” on page 4?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1: This paper delves into the internal structure of large-scale language models in extracting geographical knowledge. Utilizing auxiliary map data from OpenStreetMap to extract geospatial knowledge from large language models represents a new approach that has been sparingly explored in previous research. By fine-tuning basic models of different structures and scales and employing prompt templates containing addresses and nearby places, geographical knowledge within pre-trained models is effectively extracted. \nS2: In terms of significance, this paper addresses a challenge in natural language processing by exploring how to extract geospatial knowledge from large-scale language models. The paper proposes a new approach to knowledge extraction based on fine-tuning and prompt strategies, establishing a new concept of \"geospatial covariates.\"", "weaknesses": "W1: Lack of clear prompt templates and answer pairs design. The paper mentions the use of multiple data sources and various tasks; however, there are significant deficiencies in the theoretical approach. Specifically, the paper does not clearly specify whether the prompt formats are the same for each task and lacks detailed descriptions of how prompt templates and answer pairs were designed for different tasks. This lack of clarity makes it challenging for readers to understand the experimental design and poses obstacles for replication and further research.\nW2: Lack of clear experimental design details. Furthermore, in terms of experimental design, there are shortcomings in this paper. It does not provide a clear overview of the experimental training process, including the selection of data, preprocessing steps, and hyperparameter tuning. This omission makes it difficult to evaluate the detailed experimental procedure, hindering the reproducibility of the experiments.\nW3: Absence of comparison with ground-truth. Additionally, the paper does not thoroughly explore the comparison between experimental results and real-world geospatial data (Ground Truth). Accuracy of geospatial knowledge is crucial in practical applications; however, the paper lacks comparative analysis with actual geographical data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLr8uJtKCN", "reviewer": "Reviewer_92yi", "summary": "This paper explores possible ways to extract and tease out the geospatial information embedded inside the knowledge bases like LLMs. Specifically, they fine-tune LLMs such as chatGPT and Llama-2 with training data curated using OpenMaps with labels derived from various sources for geospatial tasks. On a variety of tasks, they show that fine-tuning LLMs on such data can effectively outperform several baselines. They further establish the geographical consistency of their results as well as ablation on suitable prompts.", "questions": "Overall, I have several requests to further delineate the performance improvements due to various aspects, while also using stronger baselines as detailed above. I am ready to raise my rating upon satisfactory rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper reveals an innovative way to use LLMs towards a expert tasks such as computing geospatial metrics without access to various kinds of data. \n\n- The access to various covariates, as mentioned, might not always be available - so the motivation of this work is sound and valid.\n\n- The experiments cover a wide variety of tasks and settings, indicating the power of the proposed approach.\n\n- The experiments involve comparison between multiple LMs like RoBERTa, GPT3.5 and LLama-2, with several useful observations pertaining to performance as well as the data efficiency of each of those.", "weaknesses": "- There are no experiments which really delineate where the performance improvements of the final model are coming from. The chosen baselines are not that strong. To delineate the knowledge offered by the LLMs compared to that of the training data, can the authors also conduct an experiment where they fine-tune a normal neural network (not LLM) on the created training data? For example, we can first pass the prompt to a sentence encoder, and then train an MLP/Neural Network which uses this embedding to predict the output as a regression or classification task  (using the same training data used during fine-tuning). This would separate the contributions offered by the LLM compared to the training data which was used in fine-tuning. \n\n-  Adding to the above, can the authors also provide results with using zeroshot capabilities of the LLMs? This would help to separate the importance of fine-tuning on custom prompts. \n\n- Assuming the covariates are indeed available, does the current method offer complimentary benefits to further improve the performance?\n\n- The authors, if possible, should also make comparisons with fine-tuning several other geo-spatial LLM type models, like K2 (Deng et. al. 2023) through fine-tuning on the same set of data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)