GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding
Authors: Zekun Li
Wenxuan Zhou
Yao-Yi Chiang
Muhao Chen
Venue: EMNLP 2023 Main (2023)
Keywords: geospatial grounding
language model
TL;DR: A language model that bridges geospatial databases and NL corpora

Abstract: Humans subconsciously engage in geospatial reasoning when reading articles. We recognize place names and their spatial relations in text and mentally associate them with their physical locations on Earth. Although pretrained language models can mimic this cognitive process using linguistic context, they do not utilize valuable geospatial information in large, widely available geographical databases, e.g., OpenStreetMap. This paper introduces GeoLM, a geospatially grounded language model that enhances the understanding of geo-entities in natural language. GeoLM leverages geo-entity mentions as anchors to connect linguistic information in text corpora with geospatial information extracted from geographical databases. GeoLM connects the two types of context through contrastive learning and masked language modeling. It also incorporates a spatial coordinate embedding mechanism to encode distance and direction relations to capture geospatial context. In the experiment, we demonstrate that GeoLM exhibits promising capabilities in supporting toponym recognition, toponym linking, relation extraction, and geo-entity typing, which bridge the gap between natural language processing and geospatial sciences. The code is publicly available at https://github.com/knowledge-computing/geolm.

Spatial reasoning and semantic understanding of natural language text arise from human communi- cations. For example, "I visited Paris in Arkansas see the smaller Eiffel Tower", human can eas- ily recognize the toponyms "Eiffel Tower", "Paris", "Arkansas", and the spatial relation "in". Implic- itly, a human can also infer that this "Eiffel Tower" might be replical ! of the original one in Paris, GeoLM: Empowering Language Models for Geospatially
Grounded Language Understanding
Zekun Li Wenxuan Zhou Yao Chiang M Muhao Chen
E
Department of Computer Science and Engineering, University of Minnesota, Twin Cities
f Department of Computer Science, University of Southern California Department of Computer Science, University of California, Davis (1i002666, yaoyi j@umn. edu; zhouwenxêusc. edu; muhchen@ucdavis..cm edu
Abstract
France. Concretely, the process of geospatially grounded language understanding involves core tasks such as recognizing geospatial concepts be- ing described, inferring the identities of those con- cepts, and reasoning about their spatially qualified relations. These tasks are essential for applications that involve the use of place names and their spa- tial relations, such as social media message analy- sis (Hu al., 2022), emergency response (Gritta et al., 2018b), natural disaster analysis (Wang al., 2020), and geographic information retrieval (Wall- grün et al., 2018).
Humans subconsciously engage in geospatial reasoning when reading articles. We recog- nize place names and their spatial relations in text and mentally associate them with their physical locations on Earth. Although pre- trained language models can mimic this cog- nitive process using linguistic context, they do not utilize valuable geospatial information in large, widely available geographical databases, e.g., OpenStreetMap. This paper introduces GEoLM (L)). D. a geospatially grounded lan- guage model that enhances the understanding of geo-entities in natural language. GEOLM leverages geo-entity mentions as anchors to connect linguistic information in text corpora with geospatial information extracted from ge- ographical databases. GEOLM connects two types of context through contrastive learn- ing and masked language modeling. It also incorporates spatial coordinate embedding mechanism to encode distance and direction relations to capture geospatial context. In the experiment, demonstrate that GEoLM exhibits promising capabilities in supporting toponym recognition, toponym linking, rela- tion extraction, and geo-entity typing, which bridge the gap between natural language pro- cessing and geospatial sciences. The code is publicly available at https: //gi thub. com/ knowledge- knwwledgg--computingg
Pretrained language models (PLMs; Devlin et al. 2019; Liu et al. 2019; Raffel et al. 2020) have seen broad adaptation across various domains such as biology (Lee et al., 2020), healthcare (Alsentzer et al., 2019), law (Chalkidis al., 2020; Douka et al., 2021), software engineering (Tabassum al., 2020), and social media (Röttger and Pierrchum- bert, 2021; Guo al., 2021). These models ben- efit from in-domain corpora (e.g., PubMed for the biomedical domain) to learn domain-specific terms and concepts. Similarly, a geospatially grounded language model requires training with geo-corpora. The geo-corpora should cover world- wide geo-entity names and their variations, geo- graphical locations, and spatial relations to other geo-entities. Although some geo-corpora are avail- able, e.g., LGL (Lieberman et al., 2010) and Spa- tialML (Mani et al., 2010), the sizes of the datasets are relatively small.
1 Introduction
In contrast, Wikipedia stores many articles de- scribing places worldwide and can serve as com- prehensive geo-corpora for geospatially grounded language understanding. However, training with Wikipedia only solves partial challenges in geospa- tial grounding, it only provides the linguistic context of a geo-entity with sentences describing history, demographies, climate, etc. The infor- mation about the geospatial neighbors of geo- entity is still missing. On the other hand, large-
The small Eiffel Tower in Paris, Arkansas, United States: https: http://wwww.kaaaaa.a.. arkansas. wwr.aaaas...//paaaaeeeeeea.eeeae com/pariss httplpaaaaacomaapaaoppo accommodations ei ffel towl-tower-ppar parower-prkk.....

---

Los Angeles, often referred to by initials larpest cnty Califoriia With population of..
New York, often called New York NYC, populous city United
MLM
Los Angeles is the capital Los Angeles County
Angeles [MASK] of los [MASK], Montebello, Pizo [MASK], West
Los Angeles the United States America
Los Angeles is on the continent North America
GeoLM
Angeles
Contrastive
Los Angeles, Montebello, Picc Rivera, West Hallywood...
Loss
Rati 34.05, 33.98,
Now York
(aar
118.24, 118.18, 118.12, 111..99
Angelee gro-entity neightors
Figure 1: Outline of GEOLM. Wikipedia and Wikidata form the NL corpora, and OpenStreetMap (OSM) form the pseudo-sentence corpora (Details in $2.2). GEOLM takes the NL and pseudo-sentence corpora as input, then pretrain with MLM and contrastive loss using geo-entities anchors. (See $2.3)
scale geographical databases (e.g., OpenStreetMap) and knowledge bases (e.g., Wikidata) can pro- vide extensive amounts of geo-entity locations and geospatial relations, enriching the information sourced from Wikipedia. Additionally, the geo- locations from OpenStreetMap can help connecting the learned geo-entity representations to physical locations on the Earth.
2 GEOLM
This section introduces GEOLM's mechanism for representing both the linguistic and geospatial con- text ($2.1), followed by the detailed development process of pretraining tasks ($2.3) and corpora ($2.2), and how GEOLM further adapted to vari- ous downstream geospatial NLU tasks ($2.4).
To address these challenges, we propose GE- OLM, a language model specifically designed to support geospatially grounded natural language un- derstanding. Our model aims to enhance the com- prehension of places, types, and spatial relations in natural language. We use Wikipedia, Wikidata, and OSM together create geospatially grounded training samples using geo-entity names as anchors. GBoLM verbalizes the Wikidata relations into nat- ural language and aligns the linguistic context in Wikipedia and Wikidata with the geospatial con- text in OSM. Since existing language models do not take geocoordinates as input, GEOLM further incorporates spatial coordinate embedding mod- ule to learn the distance and directional relations between geo-entities. During inference, model can take either natural language or geographical subregion (i.e., a of nearby geo-entities) as in- put and make inferences relying on the aligned Linguistic-geospaae information. We employ two training strategies: contrastive learning (Oord et al., 2018) between natural language corpus and lin- earized geographic data, and masked language modeling (Devlin et al., 2019) with concatena- tion of these two modalities. We treat GEOLM as a versatile model capable of addressing various ge- ographically related language understanding tasks, such as toponym recognition, toponym linking, and geospatial relation extraction.
2.1 Representing Linguistic and Geospatial Context
The training process of GEOLM aims to simul- taneously learn the linguistic and geospatial con- text, aligning them in the same embedding space obtain geospatially grounded language representa- tions. The linguistic context refers to the sentential context of the geographical entity (i.e., geo-entity). The linguistic context contains essential informa- tion to specify geo-entity, including sentences describing geography, environment, culture, and history. Additionally, geo-entities exhibit strong correlations with neighboring geo-entities (Li al., 2022b). We refer to these neighboring geo-entities as the geospatial context. The geospatial context encompasses locations and implicit geo-rclations of the neighbors to the center geo-entity.
To capture the linguistic context, GEOLM takes natural sentences input and generates entity-level representations by averaging the token representa- tions within the geo-entity name span. For the geospatial context, GEOLM follows the geospatial context linearization method and the spatial em- bedding module in our previous work SPABERT (Li et al., 2022b), a PLM that generates geospa- tially contextualized entity representations using
Here, we assume that all geo-entities in the geographical datasct are represented as points.

---

NL Input
Tokens
[CLS]
Los
Angeles
is
the
commercial
financial
and
cultural
[SEP]
Position ID
0
1
2
5
8
Segment
0
0
0
0
0
0
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
V-Coord
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
DSEP
concat
Geospatial Input
Tokens
Los
Angeles
[SEP]
Glen
##dale
[SEP]
Pasadena
[SEP]
Al
##ham
Position ID
0
1
3
4
5
7
8
9
Segment ID
1
1
1
1
X-Coord
34.05
34.05
DSEP
34.17
34.17
DSEP
34.16
DSEP
34.08
34.08
Y-Coord
-118.24
-118.24
DSEP
-118.25
-118.25
DSEP
-118.13
DSEP
-118.13
-118.13
Figure 2: Sample inputs to GEOLM. Note that segment IDs for the NL tokens are zeros, and for the pseudo-sentence tokens are ones.
point geographic data. Given center geo-entity and its spatial neighbors, GEOLM linearizes the geospatial context by sorting the neighbors in as- cending order based on their geospatial distances from center geo-entity. GEOLM then concate- nates the name of the center geo-entity and the sorted neighbors to form a aseudo-sentence To preserve directional relations and relative distances, GEoLM employs the geocoordinates embedding module, which takes the geocoordinates as input and encodes them with sinusoidal position em- bedding layer.
(lat, Ing) into 2-dimensional World Equidistant Cylindrical coordinate system EPSG:4087, This because (lat, Ing) represent angle-based values that model the 3D sphere, whereas coordinates EPSG:4087 are expressed in Cartesian form. Ad- ditionally, when generating the pseudo-sentence, we sort neighbors of the center geo-entity based on the Haversine distance which reflects the geodesic distance on Earth instead of the Euclidean distance.
2.2 Pretraining Corpora
We divide the training corpora into two parts: 1) pseudo-sentence corpora from geographical dataset, OpenStreetMup (OSM), provide the geospatial context; natural language corpora from Wikipedia and verbalized Wikidata to pro- vide the linguistic context.
To enable GEOLM process both natural lan- the fol-
guage text and geographical data, lowing types of position embedding mechanisms for each token and the token embedding (See Fig. 2). By incorporating these position embed- ding mechanisms, GEOLM can effectively process both natural language text and geographical data, allowing the model to capture and leverage spatial information.
Geographical Dataset. OpenStreetMap (OSM) is a crowd-sourced geographical database containing massive amount of point geo-entities worldwide. In addition, OSM stores the correspondence from OSM geo-entity Wikipedia and Wikidata links. We preprocess worldwide OSM data and gather the geo-entities with Wikidata and Wikipedia links to prepare puired training data used in contrastive pretraining. To linearize geospatial context and prepare the geospatial input in Fig. 2, For each geo-entity, we retrieve its geospatial neighbors and construct pseudo-sentence (See Fig. 1) by con- catenating the neighboring geo-entity names after sorting the neighbors by distance. In the end, we generate 693,308 geo-entities with the same num- ber of pseudo-sentences total.
Position ID describes the index position of the token the sentence. Note that the position ID for both the NL and geospatial input starts from zero. Segment ID indicates the source of the input tokens. the tokens belong to the natural language input, then the segment ID is zero; otherwise one. X-coord and Y-coord are inputs for the spatial coordinate embedding. Tokens within the same geo-entity name span share the same X-coord and Y-coord values. Since NI. tokens do not have as- sociated geocoordinate information, we set their X-coord and Y-coord to be DSEP, which is con- stant value distance filler.
In addition, GEOLM projects the geocoordinates
EPSG:4087: htt.:////..tttts///pp...ooo/

---

Natural Language Text Corpora. We prepare the text corpora from Wikipedia and Wikidata. Wikipedia provides large corpus of encyclope- die articles, with a subset describing geo-entities. We first find the articles describing geo-entities by scraping all the Wikipedia links pointed from the OSM annotations, then break the articles into sen- tences and adopt Trie-based phrase matching (Hsu and Ottaviano, 2013) to find the sentences contain- ing the name of the corresponding OSM geo-entity, The training samples are paragraphs containing at least one corresponding OSM geo-entity name. For Wikidata, the procedure is similar to Wikipedia. We collect the Wikidata geo-entities using the QID identifier pointed from the OSM geo-entities. Since Wikidata stores the relations as triples, we convert the relation triples to natural sentences with a set of pre-defined templates (See example in Fig. 1). After merging the Wikipedia and Wikidata samples, we gather 1,458, 150 sentencessparagraahss describ- ing 472,067 geo-entities.
Таponym Recognition
Wilton town in Saratoga County. New York. The population was 17,361 at 2020 census.
Tоponyth Linking
Map marker thoue planes ramed "Wihoor" & ind cate the cornect Inting resul:
Geospatial Relation Extraction
Wilton
Soratoga County
Cale
Costo
Tips:
Geo-antity Typing
Cart
Ropel
Figure 3: Downstream tasks to evaluate geospatially grounded language understanding.
where 7 temperature, and sim(-) denotes the cosine similarity.
To improve GEOLM's s ability to disambiguate geo-entities, we include in-batch hard negatives comprising geo-entities with identical names. As result, each batch is composed of 50% random negatives and 50% hard negatives.
2.3 Pretraining Tasks
We employ two pretraining tasks to establish con- nections between text and geospatial data, enabling GEOLM to learn geospatially grounded represen- tations of natural language text.
Additionally, we employ a masked language modeling task (Devlin al., 2019) on concate- nation of the paired natural language sentence and geographical pseudo-sentence (Fig. 2). This task encourages GEOLM to recover masked tokens by leveraging both linguistic and geographical data.
The first a contrastive learning task using an InfoNCE loss (Oord et al., 2018), which contrasts between the geo-entity features extracted from the two modalities. This loss encourages GEOLM to generate similar representations for the same geo- entity, regardless of whether the representation is contextualized based the linguistic or geospatial context. Simultaneously, GEOLM learns dis- tinguish between geo-entities that share the same name by maximizing the distances of their repre- sentations the embedding space.
2.4 Downstream Tasks
Our study further adapts GEOLM to several down- stream tasks to demonstrate its ability for geospa- tially grounded language understanding, including toponym recognition, toponym linking, geo-entity typing, and geospatial relation extraction (Fig. 3). Toponym recognition or geo-tagging (Gritta al., 2020) is to extract toponyms (i.e., place names) from unstructured text. This is crucial step in recognizing geo-entities mentioned in the text be- fore inferring their identities and relationships in subsequent tasks. We frame this task as multi- class sequence tagging problem, where tokens are classified into one of three classes: B-topo (begin- ning of toponym), I-topo (inside toponym). or (non-toponym). To accomplish this, we ap- pend a fully connected layer to GEOLM and train the model end-to-end on a downstream dataset to classify each token from the text input.
Formally, let the training data D consist of pairs of samples (spl, s,50), where sid is linguistic sample (a natural sentence or verbalized rela- tion from a knowledge base), and s000 is a pseudo- sentence created from the geographic data. Both samples mention the same geo-entity. Let f()) be GEOLM that takes both sب and 0g as input and produces entity-level representation had - ) and hTTe = )) Then the loss function is:
chim(hai 1/200/7
_ coontrast_ = -log
2 1 1iziee (ha! e ///
Toponym linking, also referred to as toponym res-

---

olution and geoparsing (Gritta al., 2020, 2018a; Hu et al., 2022), aims to infer the identity or geoco- ordinates of a mentioned geo-entity in the text by grounding geo-entity mention to the correct record in geographical database, which might contain many candidate entities with the same name as the extracted toponym from the text. During inference, we process the candidate geo-entities from the ge- ographical databases the same way during pre- training, where nearby neighbors are concatenated together to form pseudo sentences. For this task, we perform zero-shot linking by directly applying GEOLM without fine-tuning. GEOLM extracts representations of geo-entities from linguistic data and calculates representations for all candidate en- tities. After obtaining the representations for both query and candidate geo-entities, the linking results are formed as a ranked list of candidates sorted by cosine similarity in the feature space.
3.1 Toponym Recognition
Task Setup. We adopt GEOLM for toponym recog- nition on the GeoWebNews (Gritta al., 2020) dataset. This dataset contains 200 news articles with 2,601 toponyms. 5 The annotation includes the start and end character positions (i.c., the spans) of the toponyms in paragraphs. We use 80% for training and 20% for testing.
Evaluation Metrics. We report precision, recall, and F1, and include both token-level scores and entity-level scores. For entity-level scores, a pre- diction is considered correct only when it matches exactly with the ground-truth mention (i.e., no miss- ing or partial overlap).
Models in Comparison. We compare GEOLM with fine-tuned PLMs including BERT, SimCSE- BERT (Gao et al., 2021), SpanBERT (Joshi et al., 2020) and SapBERT (Liu et al., 2021). SpanBERT is BERT-based model with span prediction pre- training. Instead of masking out random tokens, SpanBERT masks out continuous span and tries to recover the missing parts using the tokens right next to the masking boundary. SapBERT learns to align biomedical entities through self-supervised contrastive learning." 7 We compare all models in base versions.
Geo-entity typing is the task of categorizing the types of locations in geographical database (e.g.,
OpenStreetMap) (Li et al., 2022b). Geo-entity typ- ing helps understand the characteristics of region and can be useful for location-basee recom- mendations. We treat this task as a classification problem and append a lne-layer classification head after GEOLM. We train GEoLM to predict the type of the central geo-entity given subregion of a geo- graphical area. To accomplish this, we construct a pseudo-sentence following Fig. 1, then compute the representation of the geo-entity using GEOLM and feed the representation to the classification head for training.
Results and Discussion. Tab. shows that our GEOLM yields the highest entity-level F1 score, with BERT being the close second. Since GE- OLM's weights are initialized from BERT, the improvement over BERT shows the effectiveness of in-domain training with geospatial grounding. For token-level results, SpanBERT has the best F1 score for I-topo, showing that span prediction during pretraining is beneficial for predicting the continuation of toponyms. However, GEOLM is better at predicting the start token of the toponym, which SpanBERT does not perform as well.
Geospatial relation extraction is the task of clas- sifying topological relations between pair of lo- cations (Mani et al., 2010). We treat this task as an entity pair classification problem. We compute the average embedding of tokens to form the entity embedding and then concatenate the embeddings of the subject and object entities to use as input for the classifier, then predict the relationship type with softmax classifier.
3.2 Toponym Linking
Task Setup. We run unsupervised toponym linking on two benchmark datasets: Local Global Corpus (LGL; Lieberman et al. 2010) and Wikipedia To-
3 Experiments
We hereby evaluate GEOLM on the aforemen- tioned four downstream tasks. All compared mod- els (except GPT3.5) are finetuned on task-specific datasets for toponym recognition, geo-entity typ- ing, and geospatial relation extraction.
'We only consider the place names associated with valid geocoordinates toponyms and do not count the literal pression (e.g.. the word "street", "blocks" and "intersection") toponyms.
"We use the unsupervised pretrained weights. ? "Although SapBERT's is trained with biomedical corpus, it generalizes well on geo-entity recognition and linking since the study of discuses often relates to places and regions.
'Toponym linking is an unsupervised task.

---

Token(B-topp)
Token (I-topo)
Entity
Geo NewNews
Prec
Recall
Prec
Recall
F1
F1
Prec
Recall
F1
BERT
90.00
89.28
78.55
79.44
78.99
84.46
77.03
83.42
80.10
SimCSE-Ble BERT
83,86
90.26
86.95
74.61
82.07
78.16
82.67
72.76
83.68
77.84
SpanBERT
85.98
88.37
87.16
86.13
89.19
87.63
87.38
75.32
81.16
78.13
SapBERT 83.12
88.32
85.64
76.26
81.11
78.61
82.22
72.48
80.16
76.12
GEOLM 91.15
90.43
90.79
79.16
84.27
81.63
86.33
82.18
85.67
83.89
Table 1: Toponym recognition results Geo WebWees dataset. Bolded and underlined numbers are for best and second best scores respectively.
BERT
GeoLM
03
60
(Paris, France) Input: Paris (French: (pasi)) the capital and most populous city of France. Situated on the Seine River, in the north of the country, it is the centre of the lie-de-France region, also known the région parisienne, "Paris Region"...
BERT
GeoLM
6.0
6.6
(Paris, AK, US) Input: Paris is a city Logan County, Arkamsas, United States, and serves as the county seat for the northern district of Logan County; its southern district counterpart is Booneville...
Figure 4: Visualization of BERT and GEOLM predictions on two "Paris" samples from the WikToR dataset. The two left figures are BERT results, and the two right figures are GEOLM results. The colored cireles in the figure denote all the geo-entities named "Paris" in the GeoNames database. The circle's color represents the cosine similarity (or ranking score) between the candidate geo-entitity from GeoNames and the query toponym Paris. It is worth noting that the BERT predictions almost the given the two sentences because BERT does not align the linguistic context with the geospatial context.
ponym Retrieval (WikToR; Gritta et al. 2018b). LGL contains 588 news articles with 4,462 to- ponyms that have been linked to the GeoNames database. This dataset has ground-truth geoco- ordinates and GeoNames ID annotations. This dataset has only sample with unique name (among 4,462 samples in total). On average, each toponym corresponds to 27.52 geo-entities with the same in GeoNames. WikToR is derived from 5,000 Wikipedia pages, where each page describes a unique place. This dataset also contains many toponyms that share the same lexical form but refer to distinct geo-entities. For example, Santa Maria, Lima, and Paris are the top ambiguous place names. For WikToR, there is no sample with unique name. The least ambiguous one has four candidate geo-entities with the same name in Gamess
average, each toponym corresponds to 70.45 geo- entities with the same name in GeoNames. This dataset is not linked with the GeoNames database, but the ground-truth geocoordinates of the toponym provided instead.
Evaluation Metrics. For LGL., we evaluate model performance using two metrics. 1) R @ k is stan- dard retrieval metric that computes the recall of the top-k prediction based on the ground-truth GeoNames ID. 2) @@D, following previous studies (Gritta al., 2018b) 8b), computes the distance-based approximation rate of the top-ranked prediction. If the prediction falls within the distance threshold D from the ground truth geocoordinates, we con- sider the prediction as a correct approximation!
Here, P P@D the same as Acc (or 1000100ee reported by Gritta et al. (2018b).

---

LGL
R@1
R@55
R@
P@D 161
scores on P@D161- Since baselines are able to har- ness only the linguistic context, while GEOLM can use the linguistic and geospatial context when tak- ing sentences as input. The improvement of R@! and P@ 1161 from BERT shows the effectiveness of the geospatial context. On WikToR, GEOLM performs the best on all metries. Since WikToR has many geo-entities with the same names, the scores of GROLM indicate strong disambiguation capability obtained from aligning the linguistic and geospatial context.
BERT
34.6
67.5
78.1
41.2
RoBERTa
24.2
48.7
60.6
27.9
SpanBERT
25.2
48.8
61.0
28.8
SapBERT
30.8
58.8
72.2
35.1
GEOLM
38.2
65.3
72.6
44.1
WikToR
P@D2o
P@ Dou
P@
BERT
16.1
16.3
16.9
17.6
RoBERTa
11.7
11.9
12.4
13.0
SpanBERT
5.5
5.7
5.9
6.3
SapBERT 25.9
26.3
27.0
28.3
GEOLM
32.5
33.4
34.3
35.8
Table 2: Toponym linking results LGL and Wik- TOR datasets. Bolded numbers are the best scores and underlined numbers are the second-best scores. @@ measures whether ground-truth GeoNames ID presents among the top retrieval results. P@D mea- sures whether top retrieval within the distance from the ground-truth. D20. D50 Dioo and Diei indi- cate the distance thresholds of (20, 50, 100, 161) km.
Fig. 4 shows the visualization of the toponym linking results given "Paris" mentioned in two sen- tences. The input sentences are provided in the figures. Apparently, the first Paris should be linked to Paris, France, and the second Paris goes Paris, AK, US. However, the BERT model fails to ground these two mentions into the correct loca- tions. BERT ranks the candidate geo-entities only slightly differently for the two input sentences, in- dicating that BERT relies more on the geo-entity name rather than the linguistic context when per- forming prediction. On the other hand, our model could predict the correct location of geo-entities. This because even though the lexical forms of the geo-entity names (i.e., Paris) are the same, the linguistic context describing the geo-entity are dis- tinct. Fig. 4 demonstrate that the contrastive learn- ing helps GEOLM to map the linguistic context to the geospatial context in the embedding space.
For WikToR, the ground-truth geocoordinates are given, and we follow Gritta et al. (2020, 2018b) to report P@ D metrics with various D values.
Models in Comparison. We compare GEOLM with multiple PLMs, including BERT, RoBETRa, SpanBERT (Joshi al., 2020) and SapBERT (Liu et al., 2021). In the experiments, we use all base versions of the above models. For a fair compar- ison, to calculate the representation of the candi- date entity, we concatenate the center entity's s name and its neighbors' names then input the concate- all base-
nated sequence (i.e., pscudd-senteneey line PLMs to provide the linguistic context, follow- ing the same unsupervised procedure in $2.4.
3.3 Geo-entity Typing
Task Setup. We apply GEOLM on the super- vised geo-entity typing dataset released by Li et al. (2022b). The goal is to classify the type of the cen- ter geo-entity providing the geospatial neighbors as context. We linearize the set of geo-entities a pseudo-sentence that represent the geospatial con- text for the center geo-entity then feed the pseudo- sentence into GEOL.M. There are 33,598 pseudo- sentences for nine amenity classes, with 80% for training and 20% for testing. We train multiple language models to perform amenity-type classifi- cation for the center geo-entity.
Results and Discussion. One challenge of this task is that the input sentences may not have any information about their neighbor entities; thus, in- stead of only considering the linguistic context or relying neighboring entities appear in the sentence, GEOLM's 's novel approach aligns the lin- guistic and geospatial context so that GEOLM can effectively map the geo-entity embeddings learned from the linguistic context in articles to the embed- dings learned from the geospatial context in geo- graphic data and perform linking. The contrastive learning process during pretraining is designed to help the context alignment. From Tab. 2, GEOLM offers more reliable performance than baselines. On WikTor and LGL datasets, GEOLM obtains the best or second-best scores in metries. On LGI,, GEOLM demonstrates more precise top-1 retrieval (R @ 1) than other models, and also the highest
Evaluation Metric. Following (Li et al., 2022b), report the Fl score for each class and the miero FI for all the samples.
Models in Comparison. In addition to BERT, SpanBERT and SimCSE-BERR BERT, this task also takes LUKE (Yamada et al., 2020) and SpaBERT (Li

---

Classes
Edu.
Ent.
Fac.
Fin.
Hea.
Pub.
Sus.
Tra.
Miero F1
67.4
63.4
76.3
92.9
85.6
83.5
SpanBERT
63.3
58.9
60.8
$2.4
81.9
SiMCSE-BERT
62.3
59.0
50.4
92.5
85.7
81.0
47.0
LUKE
64.8
60.8
59.8
94.5
85.7
86.7
85.4
85.1
51.7
82.5
SpaBERT
67.4
65.3
68.0
95.9
90.0
88.3
70.3
85.2
GEOLM
72.5
70.9
73.0
97.8
91.5
83.6
90.5
90.8
62.2
87.8
Table 3: Comparing GEOLM with the state-of-the-ar LMs geo-entity typing. Column names the OSM classes (education, entertainment, facility, financial, healthcare, public service, sustenance, transportation and waste management). Bolded and underlined numbers are for best and second best scores respectively.
Settings
Prec
Recall F1
70
GEOLM
82.18
85,67
83,89
88
GEOLM (No Contrastive)
70.67
77.98
74.14
66
GROL.M M (No Spatial Fimhed))
75.05
87.00
80.59
Table 4: Ablation study on toponym recognition to show the GEOLM performance after removing various components.
60
58
BERT
SpanBERT GeeLM
et al., 2022b) into comparison. LUKE is designed to solve entity-related tasks with specially de- signed entity tokenizer. SpaBERT generates geo- entity representations given small geographical re- gions as input. addition, BERT, SpanBERT, SimCSE-BERT, and LUKE rely only on linguistic information, and SpaBERT relies only on geospa- tial context to make predictions.
Figure 5: Relation extraction results on SpatialML com- paring with other baseline models.
ing an NA class indicating that there no relation). The dataset consists of 1,528 sentences, with 80% for training and 20% for testing. Within the dataset, there total of 10,592 entity pairs, of which 10,232 pairs do not exhibit any relation.
Results and Discussion. Tab. shows that the performance of GEOLM surpasses the baseline models using only linguistic or geospatial informa- tion. The experiment demonstrates that combining the information from both modalities and aligning the context is useful for geo-entity type inference. Compared to the second best model, SpaBERT, GEOLM has robust improvement on seven types. The result indicates that the contrastive learning during pretraining helps GEOLM align the lin- guistic context and geospatial context, and although only geospatial context is provided as input during inference, GEOLM can still employ the aligned linguistic context to facilitate the type prediction.
Evaluation Metric. We report the micro Fl score on the test dataset. We exclude the NA instances when calculating the Fl score.
Models Comparison. We compare GEOLM with other pretrained language models. including BERT, RoBERTa, and SpanBERT, as well as a large language model (LLM), GPT-3.5.
Results and Discussion. Fig. 5 shows that GE- OLM achieves the best F1. The results suggest that GFoLM effectively captures the topological rela- tionships between entities. Furthermore, GEOLM demonstrates better ability lear geospatial grounding between the mentioned entities.
3.4 Geospatial Relation Extraction
3.5 Ablation Study
We conduct two ablation experiments to validate the model design using toponym recognition and toponym linking: 1) removing the spatial coordi- nate embedding layer that takes the geo coordinates as input during the training; 2) removing the con- trastive loss that encourages the model learn similar geo-entity embeddings from two types of context (i.e., linguistic context and geospatial con- text) and only applying MLM on the NL corpora.
Task Setup. We apply GEOLM to the SpatialML topological relation extraction dataset released by Mani et al. (2010). Given a sentence containing multiple entities, the model classifies the relations between each pair of entities into six types? (includ-
"The types come from the RCC8 relations, which contain eight types of geospatial relations. The Sp dataset merges four of the relations (TPP, TPPi, NTTP, NTTPi) into the "IN" relation thus resulting in five geo-relations in the datasct.
For the toponym recognition task, we compare

---

the entity-level precision, recall, and Fl scores on the Geo GeoWebNews dataset. Tab. shows that re- moving either component could cause performance degradation. For the toponym linking task, in the first ablation experiment, the linking accuracy (P@D161, i.c., Acc@ 161km or Acc@ 100miles) drops from 0.358 to 0.321 after removing the spa- tial coordinate embedding, indicating that the geo- coordinate information is beneficial and our em- bedding layer design is effective. In the second ab- lation experiment, the linking accuracy drops from 0.358 to 046,, showing that contrastive learning is essential.
and Lu (2015) use keyword extraction approach and Yang al. (2022) use language model based (e.g., BERT) classification to solve the geospatial relation extraction problem. However, these mod- els often struggle to incorporate the geospatial con- text of the text during inference. Our previous work SpaBERT (Li et al., 2022b) is related to GEOLM in terms of representing the geospatial context. It is language model trained on geographical datasets. Although SpaBERT can learn geospatial informa- tion, it does not fully employ linguistic information during inference.
GEOLM is specifically designed to align the lin- guistic and geospatial context within a joint embed- ding space through MLM and contrastive learning.
4 Related Work
Geospatial NLU. Understanding geospatial con- cepts in natural language has long been topic of interest. Previous studies (Liu et al., 2022; Hu, 2018; Wang and Hu, 2019) have used general- purpose NER tools such as Stanford NER (Finkel et al., 2005) and NeuroNER (Dernoncourt et al., 2017) to identify toponyms in text. In the geo- graphic information system (GIS) domain, tools such as the Edinburgh geoparser (Grover et al., 2010; Tobin et al., 2010a), Yahoo! Placemaker and Mordecai (Halterman, 2017, 2023) have been developed to detect toponyms and link them to geographical databases. Also, several heuristics- based approaches have been proposed (Woodruff and Plaunt, 1994; Amitay et al., 2004; Tobin et al., 2010b) to limit the spatial range of gazetteers and associate the toponym with the most likely geo-entity (c.g., most populous ones). More re- cently, deep learning models have been adopted to establish the connection between extracted to- ponyms and geographical databases (Gritta et al., 2017, 2018a; DeLozier et al., 2015). For instance, TopoCluster (DeLozier al., 2015) learns the as- sociation between words and geographic locations, deriving a geographic likelihood for each word in the vocabulary. Similarly, CamCoder (Gritta et al.. 2018a) introduces an algorithm that encodes to- ponym mentions in a geodesic vector space, predict- ing the final location based geodesic and lexical features. These models utilize supervised training. which assumes that the testing and training data cover the same region. However, this assumption may limit their applicability in scenarios where the testing and training regions differ. Furthermore, Yu
I Language Grounding. Language grounding in- volves mapping the NL component to other modal- ities, and it encompasses several areas of research. In particular, vision-language grounding has gained significant attention, with popular approaches in- cluding contrastive learning (Radford et al., 2021; Jia et al., 2021; You et al., 2022; Li et al., 2022a) and (masked) vision- -laguuge model on distantly parallel multi-modal corpora (Chen et al., 2020, 2021; Su et al., 2020; Li et al., 2020). Addition- ally, knowledge graph grounding has been explored with similar strategies (He et al., 2021; Wang et al.. 2021). GEOLM leverages both contrastive learn- ing and MLM on distantly parallel geospatial and linguistic data, and it represents a pilot study on the grounded understanding of these two modalities.
5 Conclusion
In this paper, we propose GEOLM, a PLM for geospatially grounded language understanding. This model can handle both natural language in- puts and geospatial data inputs, and provide con- nected representation for both modalities. Tech- nically, GEOLM conducts contrastive and MLM pretraining on massive collection of distantly par- allel language and geospatial corpora, and incorpo- rates a geocoordinate embedding mechanism for improved spatial characterization. Through evalua- tions on four important downstream tasks, toponym recognition, toponym linking, geo-entity typing and geospatial relation extraction, GEOLM has demonstrated competent and comprehensive abili- ties for geospatially grounded NLUU In the future, we plan to extend the work to question answering and autoregressive tasks.
Y'yahoo! placemaker: https://ssimnn wwl.ssss.nnneet net/

---

Limitations
Matthew McDermott. 2019. Publicly available clin- ical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA, Associ- ation for Computational Linguistics.
The current version of our model only uses point ge- ometry to represent the geospatial context, ignoring polygons and polylines. Future work can expand the model's 's capabilities to handle those complex geometries. Also, it is important to note that the OpenStreetMap data were collected through crowd- soureing, which introduces possible labeling noise and bias. Lastly, model pre-training was conducted on a GPU with at least 24GB of memory. Attempl- ing to train the model on GPUs with smaller mem- ory may lead to memory constraints and degraded performance.
Finat Amitay, Nadav Har Ron Sivan, and Aya Soffer. 2004. Wib-a-wheree geotagging web content. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 273-280.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ton Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 2898- 2904, Online. Association for Computational Lin- guistics.
Ethics Statement
The model weights in our research are initialized from a pretrained BERT model for English. In ad- dition, the training data used in our research are primarily extracted from crowd-sourced databases, including OpenStreetMap (OSM), Wikipedia, and Wikidata. Although these sources provide exten- sive geographical coverage, the geographical distri- bution of training data exhibits significant dispar- ities, with Europe having the most abundant data. At the same time, Central America and Antarctica are severely underrepresented, with less than 1% of the number of samples compared to Europe. This uneven training data distribution may introduce bi- ases in the model's performance, particularly in regions with limited annotated samples.
Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. 2021. Learning the best pooling strategy for visual semantic embedding. In Proceed- ings of the IEEE/CVE conference on computer vision and pattern recognition, pages 15789- 15798..
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, Au- gust 23-28, 2020, Proceedings, Part XXX, pages 104- 120. Springer.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive entity retrieval. arXiv preprint 27X1v:2010.00004.
Grant DeLozier, Jason Baldridge, and Loretta London. 2015. Gazettser-independent toponym resolution using geographic word profiles. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 29.
Acknowledgements
We appreciate the reviewers for their insightful comments and suggestions. We thank the Min- nesota Supercomputing Institute (MST) for provid- ing resources that contributed to the research re- sults reported in this article. Zekun Li and Yao-Yi Chiang were supported by the University of Min- nesota Computer Science & Engineering Faculty startup funds. Wenxuan Zhou and Muhao Chen were supported by the NSF Grant IIS 2105329, the NSF Grant ITE 2333736, and the DARPA MCS program under Contract No. N66001 1924033 with the United States Office of Naval Research, a Cisco Research Award, two Amazon Research Awards, and Keston Research Award.
Franck Dernoncourt Ji Young Lee, and Peter Szolovits. 2017. Neuroner: an easy-to-use program for named- entity recognition based on neural networks. arXiv preprint 1005:07705.05487.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Lunguage Tech- nologies, Volume (Long and Short Papers), pages 417 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Stella Douka, Hadi Abdine, Michalis Vazirgiannis, Ra- jaa El Hamdani, and David Restrepo Amariles. 1021. JuriBERT: masked masked-language model adaptation for French legal text. In Proceedings of the Natural Le- gal Language Processing Workshop 2021, pages 95- 101, Punta Cana, Dominican Republic. Association for Computational Linguistics.
References
Emily Alsentzer, John Murphy, William Boag. Wei- Hung Weng, Di Jindi, Tristan Naumann, and

---

Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local informa- tion into information extraction systems by gibbs sampling. In Proceedings of the 43rd annual meet- ing of the asxociation for computational linguistics (ACI705), pages 363-370.
Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 4536-4542, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Bo-June Hsu and Gusseppe Ottaviano, 2013. Space- efficient data structures for top-k completion. In Proceedings of the 22nd international conference on World Wide Web, pages 583-594.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.
Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint arXiv:2104. 104.82..
Xuke Hu, Yeran Sun, Jens Kersten, Zhiyong Zhou, Friederike Klan, and Hongchao Fan. 2023. How can voting mechanisms improve the robustness and generalizability of toponym disambiguation? Inter- national Journal of Applied Earth Observation and Geoinformation, 117:103191.
Milan Gritta, Mohammad Taher Pilehvar, and Nigel Col- lier. 8a. Which Melbourne? augmenting geocod- ing with maps. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 1285-1296, Melbourne, Australia. Association for Computational Linguistics.
Xuke Hu, Zhiyong Zhou, Hao Li, Yingjie Hu, Fuqiang Gu, Jens Kersten, Hongehao Fan, and Friederike Klan. 2022. Location reference recognition from texts: A survey and comparison. arXiv preprint urXiv:2207.01683
Milan Gritta, Mohammad Taher Pilehvar, and Nigel Col- lier. 2020. A pragmatic guide to geoparsing evalua- tion: Toponyms, named entity recognition and prag- matics. Language resources and evaluation, 54:683- 712.
Yingjie Hu. 2018. Eupeg: Towards extensible and unified platform for evaluating geoparsers. Pro- ceedings of the 12th Workshop on Geographic Infor- mation Retrieval, pages 1-2.
Milan Gritta, Mohammad Taher Pilehvar, Nut Lim- sopatham, and Nigel Collier. 2017. Vancouver wel- comes you! minimalist location metonymy resolu- tion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1 1248-1259, Vancouver, Canada. Association for Computational Linguisties.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham. Quoc Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pages 4916. PMLR.
Milan Gritta, Mohammad Taher Pilchvar, Nut Lim- sopatham, and Nigel Collier. 2018b. What's missing in geographical parsing? Language Resources and Evaluation, 52:603-622 623.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span- BERT: Improving pre- pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Claire Grover, Richard Tobin, Kate Byrne, Matthew Woollard, James Reid, Stuart Dunn, and Julian Ball. 2010. Use of the edinburgh geoparser for georefer- encing digitized historical collections. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 368( (892))38777- 3889.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 1234-:2230
Yanzhu Guo, Virgile Rennard, Christos Xypolopoulos, and Michalis Vazirgiannis, 2021. BERTweetER Do- main adaptation of pre-trained language models for French tweets. In Proceedings of the Seventh Work- shop on Noisy User- seer-geneeated Texi (W-NUT 2021), pages 445-450, Online. Association for Computa- tional Linguistics.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. 2020. What does BERT with vision look at? In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 5265-5275, Online, Association for Computational Linguistics.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei 7hang, Jenq-Neng Hwang, et al. 2022a. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Patern Recognition, pages 10965- 10975.
Andrew Halterman. 2017. Mordecai: Full text geop- arsing and event geocoding. The Journal of Open Source Software, 2(9).
Andrew Halterman. 2023.
Mordecai 3: A neu- ral geoparser and event geocoder. arXiv preprint arXiv:2303.16655
7ekun Li, Jina Kim, Yao Yi Chiang, and Muhao Chen. 2022b. SpaBERT: A pretrained language model from geographic data for geo-entity representation. In Findings of the Association for Computational
Lei He, Suncong Zheng, Tao Yang, and Feng Zhang. 2021. KLMo: Knowledge graph enhanced pretrained language model with fine-grained relationships. In

---

Linguistics: EMNLP 2022, pages 2757-2769, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
Jeniya Tabassum, Mounica Maddela, Wei Xu, and Alan Ritter. 2020. Code and named entity recognition in StackOverflow. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 4913-4926, Online. Association for Computational Linguistics.
Michael D Lieberman, Hanan Samet, and Jagan Sankaranarayanan 2010. Geotagging with local lexi- cons to build indexes for textually-specifeed spatial data. In 2010 IEEE 26th international conference on data engineering (ICDE 2010), pages 201-212. LEEE.
Richard Tobin, Claire Grover, Kate Bymee, James Reid, and Jo Walsh. 2010a. Evaluation of georeferencing, In Proceedings of the 6th Workshop on Geographic Information Retrieval, GIR '10, New York, NY, USA. Association for Computing Machinery.
Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021. Self-alignment pretraining for biomedical entity representations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4228 022844238, Online. Association for Computa- tional Linguistics.
Richard Tobin, Claire Grover, Kate Byme, James Reid, and Jo Walsh. 2010b. Evaluation of georeferencing, In proceedings of the 6th workshop geographic information retrieval, pages 1-8.
Jan Oliver Wallgrün, Morteza Karimzadeh, Alan M MacFachren, and Scott Pezanowski. 2018. Geocor- pora: building a corpus to test and train microblog geoparsers. International Journal of Geographical Information Science, 32(1):1-29,
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeulemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1909.
Jimin Wang and Yingjie Hu. 2019. Enhancing spatial and textual analysis with eupeg: An extensible and unified platform for evaluating geoparsers. Transac- tions in CIS, 23(6):1393313) 1419.
Zilong Liu, Krzysztof Janowicz, Ling Cai, Rui Zhu, Gengehen Mai, and Meilin Shi. 2022. Geoparsing: Solved or biased? an evaluation of geographic biases in geoparsing. AGILE: GIScience Series, 3:9.
Jimin Wang, Yingjie Hu, and Kenneth Joseph. 2020. Neurotpr: A neuro-net toponym recognition model for extracting locations from social media messages. Transactions in GIS, 24(3):
Inderjeet Mani. Christy Doran, Dave Harris, Janet Hitze- man, Rob Quimby, Justin Richer, Ben Wellner, Scott Mardis, and Seamus Clancy. 2010. Spatialml: anno- tation scheme, resources, and evaluation. Language Resources and Evaluation, 44:263-280.
Xiaozhi Wang. Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transac- tions of the Association for Computational Linguis- tics, 9:17
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprini arXiv: 1807,03745
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
Allison Gyle Woodruff and Christian Plaunt. 1994. Gipsy: Automated geographic indexing of text docu- ments. Journal of the American Society for Informa- tion Science, 45(9):645 665.655..
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6442-666 6454, On- line. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Leurning Rexearch, 21(1):54555551
Jiannan Yang, Hong Jia, and Hanbing Liu. 2022. Spatial relationship extraction of geographic entities based on bert model. In Journal of Physics: Conference Series, volume 2363, page 012031. IOP Publishing.
Paul Röttger and Janet Pierrehumbert. 2021. Temporal adaptation of BERT and performance on downstream document classification: Insights from social media. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2400-2412, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Haoxuan You. Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. 2022. Learning visual representation from modality contrastive language-image pre-training. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23- 27, 2022, Proceedings. Part XXVII, pages 69.87. Springer.
Weijie Su, Xizhou 7hu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. VI-bert: Pre-training of generic visual- linguisti representations. In Inter- national Conference on Learning Representations.

---

Li Yu and Feng Lu. 2015. bootstrapping algorithm for geo-entity relation extraction from online ency- clopedia. In 2015 23rd International Conference on Geoinformatics, pages 1-5. IEEE
Place names from OSM
Los
Las
"Los Angeles"
Angeles
Caloss
Vegas
*Los Angeles County"
*Los Angeles Elementary School" Cabos"
County
Elementary
A Distribution of Geo-entities
*Las Vegas'
School
We analyze the distribution of geo-entities in the pretraining corpora and the downstream datasets. When considering only the name (without geoco- ordinates), the overlapping percentages of the pre- training data and downstream datasets are shown in Tab. 5. Ablation experiments show that for the geo- entities already included during the pretraining, the average P@ D161 score is 0.362. For the ones that are not included during the pretraining, the average P@ D161 score is 0.341, which is not significantly different from the prior one. This indicates that the improved performance of GEOLM comparing with other models benefits from enhancing the geospa- tial representations.
Figure 6: Example of the constructed Trie if Open- StreetMap only contains "Los Angeles", "Los Angeles County", "Los Angeles Elementary School", "Los Ca- bos" and "Las Vegas". Each node is word in a place name. The gray color indicates that this node the last word of place name.
the scores for linking only, we use the scores re- ported in Voting, which assumes gold toponyms inputs for toponym linking (same as ours).
The scores in P@ D1 (or Accuracy @ 16] km) are shown Tab. 6. The models that perform better than GEOLM are all supervised learning models (i.e. CamCoder, GENRE and Voting) while GE- OLM is unsupervised. Within the unsupervised group, GFOLM performs the best. The benefit of the unsupervised nature of GEOLM is that GE- OLM can handle new samples without the need for extra training data, which is often difficult to gather, Also, new geo-entity names can appear in docu- ments and the way people call the same geo-entity can change over time, the unsupervised approach has the advantage of handling ever-changing doc- uments, e.g., online text, while the supervised ap- proach focuses existing names of geo-entities. We acknowledge that there is gap between GF- OLM and the domain-specift, geoparsers, and we will aim narrow this gap in the future.
B Comparison with Other Methods
For the toponym recognition task, compare the results of GEOLM with other existing mod- els designed specifically for this problem. Ac- cording (Gritta et al., 2020), the token-level F1 achieved by Yahoo! Placemaker, Edinburgh Geop- arser, Spacy NLP, Google Cloud Natural Language, and NCRF++ are 63. 2%, 63.6%, 74.9%, 83.2% and 88.6% respectively. GEOLM has token-level Fl of 86.3%. which is better than all existing ones except NCRF++. The reason is that NCRF++ fine-grained toponym taxonomy boost the to- ponym recognition performance. However, the fine- grained labels can sometimes be difficult to collect for large-scale datasets. In addition, NCRF++ is a specific toponym recognition model that does not support other geospatial ccospatial-nffernnce inference tasks. With GE- oLM, we can generate representations useful for various tasks.
C Trie
Trie supports tree structure for efficient search for geo-entity names, and it helps distinguish be- tween two geo-entities with shared substrings in their names. For example. Trie helps extract the geo-entity "Los Angeles High School" from the sentence "I work at the Los Angeles High School," instead of extracting the geo-entity "Los Angeles". Fig. 6 shows an example Trie constructed from a set of geo-entity names.
For the toponym linking task, we compare our model with the other existing geoparser models mentioned in EUPEG (Hu, 2018) and Voting (Hu et al., 2023), including CLAVIN!! TopoClus- ter (DeLozier et al., 2015), CamCoder (Gritta al., 201 8a), Modecai (Halterman, 2017), GENRE (De Cao et al., 2020), and Voting (Hu et al., 2023). Since EUPEG evaluates the toponym resolution and toponym linking together and does not provide
We use all geo-entity names in the worldwide OpenStreetMap database construct worldwide Trie, where each node is a single word in the name. When using Trie to preprocess the Wikipedia doc- uments, we apply the Trie searching to find all
"CLAVINsttps: httpg//u....oo////tttt/////////

---

Tasks
Dataset
Total Records Intersection Percentage
Tooonnm Recognition
GeoWebNews
35.6%
Toponym Linking
WikToR
1280
67.9%
Geo-entity Typing
OpenStreetMap Subset
23195
2.3%
Relation Extraction
SpatialMI.
309
91
29.4%
Table 5: The number of geo-entities in the downstream datasets, and the overlapping portion of geo-entities with the pretraining corpora.
Unsupervised
Supervised
listed below: disconnected (DC): Entity A and have no spatial intersection, both terms of interiors and boundaries; externally connected (EC): Entity and B touch each other only at their boundaries; equal (EQ): Entity and B are identical; partially overlap- ping (PO): Interiors of entity A and B overlap but neither is completely contained within the other; within (IN); One entity is part of the other entity; Others: No relation between tities]
Modecai
0.15
CamCoder 0.63
CLAVIN
0.22
GENRE 0.81
TopoCTuster 0.24
Voting 0.85
GEOLM (ours) 0.35
Table 6: Comparison with other toponym linking mod- both unsupervised and supervised category.
Types
Keywords
Others
Other; No relation
Equal (RQ)
EQ
Disconnected (DC)
DC
Externally connected (EC)
EC
Within (LN)
In: Within
Partially overlapping (PO)
PO
User: Sentence: [input-sentence] Entityl: [entityl-amme] Entity2: (entity2-name] Re- lationship:
Table 7: Relation types and the corresponding accept- able keywords in GPT outputs
To address the randomness the GPT outputs and robustly evaluate the performance, we only look for some particular keywords from the GPT outputs. If the output contains the desired keyword. we consider the prediction correct. Tab. 7 lists the keywords for each relation type.
the mentioned geo-entity names. To mitigate the disambiguation error, use the Wikipedia page title filter out the mentions that do not describe the "entity-of-interest., After this step, the dis- ambiguation error only occurs when two distinct geo-entities with the same name occur on the same Wikipedia page, which is pretty rare. This does not fully resolve the disambiguation error issue, and there may still be noises in the training data. However, as long most of the data is clean, the model can still learn meaningful information from the data. Thus have compiled quite large pretraining NL corpora with 1,458, 150 sen- tences/paragraphs describing 472,067 geo-entities.
GPT-3.5 Prompt
We use GPT-3. 5-tubo to help predict the rela- tions between two geo-entities. With the system role, prompt the model with the general task description and ask the model to choose from one of the possible relationships. With the user role, we provide the input sentence and the two geo-entity names. The example prompt is shown below.
Syseem:Given a sentence, and two entities within the sentence, classify the relationship between the two entities based on the pro- vided sentence. All possible Relationships are


Reviews:
[{"review_id": "5xWAchHh44", "reviewer": "Reviewer_zGPk", "summary": "", "questions": "A. What is the distribution of geo-entities in both the pretraining corpora and downstream task datasets and what are the overlapping sets of geo-entities for both data types? Does the improved performance result from introducing more geo-entity knowledge or from enhancing geospatial representations?\n\nB. For the downstream task of toponym linking, could you elaborate on the methods used to calculate representations for all candidate entities across different PLMs? Are all models purely name-based?\n\nC. For spatial relation extraction, it seems that a significant portion of the dataset belongs to the NA class. Is the F1 score shown in Figure 5 calculated excluding the NA class? If the NA class is included, could you provide the score for each individual spatial relation?\n\nD. If during the pre-training process, people can already access the coordinates to the location entities, then for some tasks like relation extraction, why cannot directly get the results from coordinates/maps?\n\nE. For the relation extraction task, in line 482, the author stated that there are 7 types for the relations, but the dataset they mentioned (Mani et al., 2010) only has 5 link types (excluding the NA class). Can the authors list all the link types they have in their experiments?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: No ablation studies discuss each module's effectiveness in the paper.**\n\nWe conducted two additional ablation studies to show the validity of the model design using the **toponym recognition** and **toponym linking** as the testing ground:  \n\n* 1) Removing the spatial coordinate embedding layer that takes the geo coordinates as input during the training \n* 2) Removing the contrastive loss that encourages the model to learn similar geo-entity embeddings from two different types of context (i.e., linguistic context and spatial context) and only applying MLM on the NL corpora\n\nFor the toponym recognition task,  we compare the entity-level precision, recall, and F1 scores on the GeoWebNews dataset and observe that removing either of the components could cause performance degradation. \n\n\n| Setting     | Precision |  Recall | F1 | \n| -------- | ------- | ------- | ------- |\n| GeoLM | 82.18 | 85.67 | 83.89 | \n|GeoLM (No Contrastive) | 70.67 | 77.98| 74.14 |\n| GeoLM (No Spatial Embedding)| 75.05 | 87.00 | 80.59 |\n\n\nFor the toponym linking task, in the first ablation experiment, the linking accuracy (P@D161, i.e, Acc@161km or Acc@100miles) dropped from 0.358 to 0.321 after removing the spatial coordinate embedding, indicating that the geocoordinate information is beneficial and our embedding layer design is effective.  In the second ablation experiment, the linking accuracy (P@D161) dropped from 0.358 to 0.146, showing that contrastive learning is essential. \n\n\n&nbsp; \n\n**Q2. The distribution of geo-entities in the pretraining corpora and downstream task datasets and the overlapping sets of geo-entities for both data types.**\n\nWe analyze the distribution of geo-entities in the pretraining corpora and the downstream tasks. When considering only the name (without geo-coordinates), the overlapping percentages of the pretraining data and downstream datasets are shown in the table below\n\n| Task     | Dataset | # Total Records | # Intersection | Percentage |\n| -------- | ------- | ------- |------- | ------- |\n| Toponym Recognition  | GeoWebNews | 912  | 325  | 35.6%  |\n| Toponym Linking | WikToR     | 1886 | 1280  |67.9%  |\n| Geo-entity Typing    | OpenStreetMap Subset   | 23195 | 544  | 2.3%  |\n| Relation Extraction    | SpatialML   | 309 | 91  | 29.4%  |\n\nWe further analyze the toponym linking experiment results. The results show that for the geo-entities already included during the pretraining, the average P@D161 score is 0.362. For the ones that are *not* included during the pretraining, the average P@D161 score is 0.341, which is not significantly different from the prior one. This indicates that the improved performance of GeoLM benefits more from enhancing the geospatial representations. \n\nRegarding the overlapping sets of geo-entities for both data types (i.e., NL corpora and geographical dataset), all the geo-entities exist in both data types to train the model with the contrastive objective. \n\n\nWe will add the distribution and analysis in the revised version. \n\n&nbsp; \n\n\n**Q3. For the downstream task of toponym linking, could you elaborate on the methods used to calculate representations for all candidate entities across different PLMs? Are all models purely name-based?**\n\nAll of the compared PLMs are purely name-based. For a fair comparison, to calculate the representation of the candidate entity, we concatenate the pivot entity’s name and its neighbors’ names and input the concatenated sequence (i.e., pseudo sentence in the paper) to the model to provide linguistic context, following the same procedure as GeoLM. The positional encoding used here reflects the spatial order of the neighbors to the pivot entity. \n\n&nbsp; \n\n**Q4. Is the F1 score shown in Figure 5 calculated excluding the NA class?**\n\nThe F1 score excludes the NA class instances that are treated as negative instances. This is consistent with the common practice of evaluating relation extraction tasks where the majority of entity pairs are not related [1].\n\n[1] Zhang, Yuhao, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. \"Position-aware attention and supervised data improve slot filling.\" In Conference on Empirical Methods in Natural Language Processing. 2017.\n\n&nbsp; \n\n**Q5. If during the pre-training process, people can already access the coordinates to the location entities, then for some tasks like relation extraction, why cannot directly get the results from coordinates/maps?**\n\nDirectly obtaining the results from map data (location entities with coordinates) only works when individual toponyms in the documents are all correctly resolved (i.e., linked to their correct geocoordinates). However, toponym linking (resolution)  is another challenging problem, as we show in Section 3.2. More importantly, documents can contain much richer semantic information about entity relations than map data. For example, “I went to the Disneyland in Los Angeles” is commonly said, reflecting the “IN” relation between these two geo-entities. This represents people’s conceived space (i.e., what is thought) that is useful but can be inconsistent with the perceived space (i.e., what is seen) in map data. In this case, Disneyland is actually located in Orange County, but not in Los Angeles. \n\n&nbsp; \n\n**Q6. For the relation extraction task, in line 482, the author stated that there are 7 types for the relations, but the dataset they mentioned (Mani et al., 2010) only has 5 link types (excluding the NA class). Can the authors list all the link types they have in their experiments?**\n\nWe apologize for the typo. There should be six types in total: five geo-relations plus “no-relation”. The types come from RCC8 relations, which contain eight types of geo-spatial relations, and the SpatialML dataset merged four of the relations (TPP, TPPi, NTTP, NTTPi) into “IN” relation thus resulting in five geo-relations in the dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and the motivation is clear. Geospatial language understanding is an essential task in the geospatial community.\n2. The method is shown to be effective and outperforms some previous SOTAs like SpaBERT.\n3. The idea of incorporating coordinates in the pre-training is well-formulated.", "reasons_to_reject": "1. The authors only conducted experiments on the downstream tasks. There are no ablation studies discussing the effectiveness of each module in the paper.\n2. See the questions below.", "questions_to_authors": "A. What is the distribution of geo-entities in both the pretraining corpora and downstream task datasets and what are the overlapping sets of geo-entities for both data types? Does the improved performance result from introducing more geo-entity knowledge or from enhancing geospatial representations?\n\nB. For the downstream task of toponym linking, could you elaborate on the methods used to calculate representations for all candidate entities across different PLMs? Are all models purely name-based?\n\nC. For spatial relation extraction, it seems that a significant portion of the dataset belongs to the NA class. Is the F1 score shown in Figure 5 calculated excluding the NA class? If the NA class is included, could you provide the score for each individual spatial relation?\n\nD. If during the pre-training process, people can already access the coordinates to the location entities, then for some tasks like relation extraction, why cannot directly get the results from coordinates/maps?\n\nE. For the relation extraction task, in line 482, the author stated that there are 7 types for the relations, but the dataset they mentioned (Mani et al., 2010) only has 5 link types (excluding the NA class). Can the authors list all the link types they have in their experiments?", "paper_topic_and_main_contributions": "The paper proposes a geospatially grounded language model for geospatial language understanding named GeoLM. And the authors showed that this pre-trained model can address various geographical-related language understanding tasks by utilizing both geographical and natural language knowledge. GeoLM is tested on various traditional downstream tasks and is shown to be more effective than the compared baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "wuV3ELCeuu", "reviewer": "Reviewer_HVHc", "summary": "", "questions": "A - In Section 2.2, the authors mention the use of \"trie-based phrase matching\" to find the sentences that contain the name of the target geo-entity. This procedure should ideally be explained in more detail, and the authors should comment on the fact that this procedure can also produce disambiguation errors.\n\nB - Section 2.3 should explain the combination of the two pre-training tasks in more detail. The contrastive learning objective seems to use in batch negatives, and the linguistic/verbalized sentences need thus to be encoded separately by the model. On the other hand, the MLM objective uses the concatenation of the linguistic/verbalized sentences and they are encoder jointly by the model. How are then the two tasks combined together when processing a batch of instances? Also, the use of negative samples should be explained in more detail (e.g., how are the hard negative samples selected).\n\nC - In Section 2.4, the procedure associated to toponym linking should also be explained in more detail. How are the candidate entities from geographical data encoded by GEOLM (e.g., does the model encode simply the name for the entity? Or instead the name together with a textual description? Or perhaps a sentence verbalizing the entity name together with names for neighbouring entities, as done during pre-training?)\n\nD - Also in Section 2.4, the relation extraction task should be better explained. What does it mean to \"concatenate the average embedding of entities in the final layer of GEOLM\"?\n\nE - The toponym recognition and toponym linking tasks, in particular, should ideally be evaluated with more datasets. Results should also be compared against those from previous methods specifically focusing on these tasks, instead of just making the comparison against BERT/SpanBERT and the previous work named SapBERT. Results against SapBERT are encouraging, but ideally the paper should also explain if the baselines were fine-tuned to the in-domain data in the same way as GEOLM.\n\nF - The experimental evaluation should ideally consider an ablation in which GEOLM is just pre-trained with masked language modeling on the same Wikipedia data, instead of considering the contrastive objective and the verbalized sentence. A separate ablation should ideally also consider an alternative which does not use the geographic/coordinate embeddings.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: the toponym recognition and disambiguation tasks did not compare with results against previous studies specifically focusing on these problems (e.g., see previous benchmarks such as EUPEG).**\n\nFor the **toponym recognition task**, we compare the results of GeoLM with other existing models designed specifically for this problem. According to Gritta et al. [1], the token-level F1 achieved by Yahoo! Placemaker, Edinburgh Geoparser, Spacy NLP, Google Cloud Natural Language, and NCRF++ are 63.2%, 63.6%, 74.9%, 83.2% and 88.6% respectively. GeoLM has a token-level F1 of 86.3%, which is better than all existing ones except NCRF++. The reason is that NCRF++ uses fine-grained toponym taxonomy to boost the toponym recognition performance. However, the fine-grained labels can sometimes be difficult to collect for large-scale datasets. In addition, NCRF++ is a toponym recognition specific model which does not support other geospatial-inference tasks. With GeoLM, we can generate representations useful for various tasks. \n\n\nFor the **disambiguation (toponym linking)** task, we compare our model with the other existing geoparser models mentioned in EUPEG [2] and Voting [3], including CLAVIN, TopoCluster, CamCoder, Modecai,  GENRE, and Voting. Since EUPEG evaluates the toponym resolution and toponym linking together and does not provide the scores for linking only, we use the scores reported in Voting, which assumes gold toponyms as inputs for toponym linking (same as ours). \n\nThe scores in P@D161 (or Accuracy@161km) are shown in the table below:\n\n| Unsupervised | | | |\n| -------- | ------- | ------- |------- | \n| GeoLM (ours)     | CLAVIN | TopoCluster | Modecai | \n| **0.35** | 0.22 | 0.24 | 0.15 | \n\n| Supervised | | |\n|------- |------- | ------- |\n| CamCoder | GENRE | Voting | \n| 0.63 | 0.81  | 0.85 | \n \n\nNote that the models perform better than GeoLM are all supervised learning models (i.e. CamCoder, GENRE and Voting) while GeoLM is unsupervised. Within the unsupervised group, GeoLM performs the best. The benefit of the unsupervised nature of GeoLM is that it can handle new samples without the need of extra training data, which is often difficult to gather. Also, *new* geo-entity names can appear in documents and the way people call the same geo-entity can *change* over time, the unsupervised approach has the advantage of handling ever changing documents, e.g., online text, while the supervised approach focuses on existing names of geo-entities. We acknowledge that there is a gap between GeoLM and the domain-specific geoparsers, and we will aim to narrow this gap in the future. \n\n[1] Gritta, Milan, Mohammad Taher Pilehvar, and Nigel Collier. \"A pragmatic guide to geoparsing evaluation: Toponyms, Named Entity Recognition and pragmatics.\" Language resources and evaluation 54 (2020): 683-712.\n\n[2] Wang, Jimin, and Yingjie Hu. \"Enhancing spatial and textual analysis with EUPEG: An extensible and unified platform for evaluating geoparsers.\" Transactions in GIS 23, no. 6 (2019): 1393-1419.\n\n[3] Hu, Xuke, Yeran Sun, Jens Kersten, Zhiyong Zhou, Friederike Klan, and Hongchao Fan. \"How can voting mechanisms improve the robustness and generalizability of toponym disambiguation?.\" International Journal of Applied Earth Observation and Geoinformation 117 (2023): 103191.\n\n\n&nbsp;\n\n**Q2: Justify why the use of the verbalized sentences and/or the coordinate embeddings provides any benefits. The experimental evaluation should ideally consider an ablation in which GEOLM is just pre-trained with masked language modeling on the same Wikipedia data, instead of considering the contrastive objective and the verbalized sentence. A separate ablation should ideally also consider an alternative which does not use the geographic/coordinate embeddings.**\n\n\nA verbalized sentence (pseudo-sentence) is a simple form to linearize a small geospatial region. The order of the place names in the pseudo sentence reflects the distance relation with the pivot entity. The spatial coordinate embedding helps learn the distance and orientation relations between geo-entities. \n\nWe add the two ablation experiments as suggested by the reviewer and evaluate the model on the toponym recognition and toponym linking task. In the first experiment, we pretrain the model using only the MLM objective on the NL corpora consisting of Wikipedia and Wikidata. In the second experiment, we remove the spatial coordinate embedding during pretraining and evaluate the model on the downstream tasks.\n\n\nFor the **toponym recognition** task,  we compare the entity-level precision, recall, and F1 scores on the GeoWebNews dataset and observe that removing either of the components could cause performance degradation. \n\n| Setting     | Precision |  Recall | F1 | \n| -------- | ------- | ------- | ------- |\n| GeoLM | 82.18 | 85.67 | 83.89 | \n|GeoLM (No Contrastive, only MLM) | 70.67 | 77.98| 74.14 |\n| GeoLM (No Spatial Embedding)| 75.05 | 87.00 | 80.59 |\n\n\nFor the **toponym linking** task, in the first ablation experiment, P@D161 dropped from 0.358 to 0.321 after removing the spatial coordinate embedding, indicating that the geocoordinate information is important and our embedding layer design is effective. In the second ablation experiment, P@D161 dropped from 0.358 to 0.146, showing that contrastive learning is essential. \n\n&nbsp;\n\n**Q3: Explain Trie in more detail, comment on the fact that this procedure can also produce disambiguation errors.**\n\nUsing Tries has two advantages. First, Tries support a tree structure for efficient search for geo-entity names. Second, Tries helps differentiate between two geo-entities with shared substrings in their names. For example, Tries helps extract the geoentity “Los Angeles High School” from the sentence “I work at the Los Angeles High School,” instead of extracting the geoentity “Los Angeles”. \n\nWe use all geo-entity names in the worldwide OpenStreetMap database to construct a worldwide Trie, where each node is a single word in the name. When using Trie to preprocess the Wikipedia documents, we apply the Trie searching to find all the mentioned geo-entity names. To mitigate the disambiguation error, we use the Wikipedia page title to filter out the mentions that do not describe the “entity-of-interest”. After this step, the disambiguation error only occurs when two different geo-entities with the same name occur in the same Wikipedia page, which is pretty rare. We admit that this does not fully resolve the disambiguation error issue, and it may still cause noise in the training data. However, as long as most of the data is clean, the model can still learn meaningful information from the data. Thus we have compiled a quite large pretraining NL corpora with 1,458,150 sentences/paragraphs describing 472,067 geo-entities. \n\n&nbsp;\n\n**Q4: Section 2.3 should explain the combination of the two pre-training tasks in more detail, also explain how in-batch hard negatives are selected** \n\nThe two losses are calculated on the same batch of data. With a batch size of 32, we will have 16 samples of NL (linguistic) from Wikipedia & Wikidata and 16 samples of pseudo (verbalized) sentences constructed from OpenStreetMaps (OSM). These samples in two modalities are paired and each pair describes the same geo-entity; thus we have 16 positive pairs in total. Moreover, among the 16 pairs, 8 of them have different geo-entity names (easy negatives), and 8 of them have the same geo-entity name but different OSM ID (hard negatives). To calculate the MLM loss, we merge the NL sentence and pseudo sentence to feed into the model. To calculate the contrastive loss, we keep the NL sentence and pseudo sentence separate and sample from positive and negative pairs to calculate the InfoNCE (contrastive) loss. \n\n&nbsp;\n\n**Q5: How are the candidate entities from geographical data encoded by GEOLM**\n\nFollowing the convention in SpaBERT [1], to capture the spatial context of candidate entities from a geographical database (e.g., GeoNames) (which does not contain textual descriptions), the model encodes the name for the entity, using the neighbor entity names as the context,  as done during pre-training. The pivot entities are concatenated with the neighbor geo-entities to provide the token input, and their geo-coordinates are embedded with the spatial coordinate embedding layer. \n\nDifferent from SpaBERT, GeoLM is a novel model that combines both linguistic and spatial context in a contrastive way so that even when the candidate entities are only encoded with spatial context, GeoLM can still match the corresponding entity encoded with linguistic context. \n\n[1] Li, Zekun, Jina Kim, Yao-Yi Chiang, and Muhao Chen. \"SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation.\" In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2757–2769.\n\n&nbsp;\n\n**Q6: Also in Section 2.4, the relation extraction task should be better explained. What does it mean to \"concatenate the average embedding of entities in the final layer of GEOLM\"?**\n\nWe compute the average embedding of tokens to form the entity embedding and then concatenate the embeddings of the subject and object entities to use as input for the classifier. We'll make it clear in the revision.\n\n&nbsp;\n\n**Q7: Missing References**\nWe will add the references in the revised version.\n\n&nbsp;\n\n**Q8: Suggestion on how motivation should be framed**\n\nWe appreciate this suggestion and agree with the reviewer’s comment. To tackle the problem of small geo-corpora, GeoLM incorporates Wikipedia to constitute the pretraining corpora. As the reviewer pointed out, Wikipedia constitutes a very specific text genre when describing the geo-entity (e.g., history, environment, etc.). GeoLM overcomes this problem by capturing broader relations between geo-entities learned from Wikidata relations and the OpenStreetMap database. Granted that GeoLM does not entirely solve these two challenges. Still, the experiments show that GeoLM provides a pilot solution to these problems, demonstrated in the downstream tasks, including geo-entity linking and relation extraction. We will modify the motivation section of the paper in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written, mostly clear in its descriptions, and it addresses an interesting problem.\n\n* The experimental results are encouraging. The paper assesses the model on many different tasks, with positive results in comparison to a previous related work named SPABERT (Li et al., 2022b).", "reasons_to_reject": "* The experimental evaluation in the toponym recognition and disambiguation tasks is not particularly convincing, since the authors do not compare results against previous studies specifically focusing on these problems (e.g., see previous benchmarks such as EUPEG).\n\n* The descriptions provided in the paper are not entirely convincing, in terms of justifying why the use of the verbalized sentences and/or the coordinate embeddings provides any benefits. Ablation experiments focusing on these components would help in assessing this claim.", "questions_to_authors": "A - In Section 2.2, the authors mention the use of \"trie-based phrase matching\" to find the sentences that contain the name of the target geo-entity. This procedure should ideally be explained in more detail, and the authors should comment on the fact that this procedure can also produce disambiguation errors.\n\nB - Section 2.3 should explain the combination of the two pre-training tasks in more detail. The contrastive learning objective seems to use in batch negatives, and the linguistic/verbalized sentences need thus to be encoded separately by the model. On the other hand, the MLM objective uses the concatenation of the linguistic/verbalized sentences and they are encoder jointly by the model. How are then the two tasks combined together when processing a batch of instances? Also, the use of negative samples should be explained in more detail (e.g., how are the hard negative samples selected).\n\nC - In Section 2.4, the procedure associated to toponym linking should also be explained in more detail. How are the candidate entities from geographical data encoded by GEOLM (e.g., does the model encode simply the name for the entity? Or instead the name together with a textual description? Or perhaps a sentence verbalizing the entity name together with names for neighbouring entities, as done during pre-training?)\n\nD - Also in Section 2.4, the relation extraction task should be better explained. What does it mean to \"concatenate the average embedding of entities in the final layer of GEOLM\"?\n\nE - The toponym recognition and toponym linking tasks, in particular, should ideally be evaluated with more datasets. Results should also be compared against those from previous methods specifically focusing on these tasks, instead of just making the comparison against BERT/SpanBERT and the previous work named SapBERT. Results against SapBERT are encouraging, but ideally the paper should also explain if the baselines were fine-tuned to the in-domain data in the same way as GEOLM.\n\nF - The experimental evaluation should ideally consider an ablation in which GEOLM is just pre-trained with masked language modeling on the same Wikipedia data, instead of considering the contrastive objective and the verbalized sentence. A separate ablation should ideally also consider an alternative which does not use the geographic/coordinate embeddings.", "paper_topic_and_main_contributions": "The paper presents GEOLM, a Transformer encoder model that is pre-trained with language descriptions of geospatial locations, paired with geospatial information extracted from geographical databases such as OSM or Wikidata, in an attempt to ground the language model and enhance the understanding of geospatial entities in natural language. The pre-training procedure leverages masked language modeling together with a contrastive objective, where ??. The evaluation of the model considered different tasks (i.e., toponym recognition, toponym disambiguation/linking, geospatial relation extraction, and geo-entity typing), and results show that the proposed model outperforms standard BERT and other extensions focused on better modeling entities and/or spans of text.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There's a recent paper that discusses current challenges associated to text geoparsing, which should be analyzed in the context of this work:\n\nGeoparsing: Solved or Biased? An Evaluation of Geographic Biases in Geoparsing\nhttps://agile-giss.copernicus.org/articles/3/9/2022/\n\nThere's also paper describing a benchmark platform named EUPEG for evaluating text geoparsing over multiple datasets, going beyond the datasets used in the present paper. Ideally, the authors should compare their results on toponym recognition/disambiguation against results reported for other systems/approaches, and EUPEG can perhaps help on this:\n\nEnhancing spatial and textual analysis with EUPEG: an extensible and unified platform for evaluating geoparsers\nhttps://geoai.geog.buffalo.edu/EUPEG/\nhttps://github.com/geoai-lab/EUPEG\n\nAnother recent system for toponym recognition and disambiguation is Mordecai, which is conveniently available as open-source software. Passing texts through Mordecai and comparing results against the proposed approach would also be relatively simple:\n\nMordecai 3: A Neural Geoparser and Event Geocoder\nhttps://arxiv.org/pdf/2303.13675.pdf\nhttps://github.com/ahalterman/mordecai3", "Typos_Grammar_Style_and_Presentation_Improvements": "The following comment is not exactly a typo, but instead a suggestion in terms of how the motivation is framed in the introduction of the paper.\n\nSpecifically, the authors state that some geo-corpora are available (e.g., LGL and SpatialML), but they only cover limited geographical regions. I would argue that this is not the main limitation, given that some of the existing corpora are of global coverage. The main issue is, nonetheless, the fact that the existing geo-corpora are relatively small. Also in the introduction the authors state that Wikipedia does not always contain geo-coordinates and spatial relations between geo-entities, but again this is not the main limitation (i.e., many entities in Wikipedia, not just places, are indeed associated to geospatial coordinates of latitude and longitude). The issue with Wikipedia is that its descriptions constitute a very specific genre of text, different from other documents where one may which to use geographic text analysis methods."}}, {"review_id": "vHiHsSEFBa", "reviewer": "Reviewer_TG2x", "summary": "", "questions": "Are the compared models in section 3.1, 3.3 and 3.4 also fine-tuned on respective down-stream tasks?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Are the compared models in section 3.1, 3.3 and 3.4 also fine-tuned on respective down-stream tasks?**\n\nYes, all the compared models in Section 3.1, 3.3, and 3.4 are fine-tuned on the same task training data as GeoLM and GeoLM outperforms these compared models. We’ll make that clear in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written with clear definitions and details for the proposed approach as well as the experimental evaluation on four tasks. It is also shown that the model achieves somewhat improved performance with compared counterparts.", "reasons_to_reject": "The main criticism would about the compared models: BERT, SpanBERT, RoBERTa, SapBERT, LUKE. The Section 3 provides details about each down-stream task and the acquired results presented in tables. It is not really clear whether the compared models were also fine-tuned on the respective task or not. For example, the Section 3.1 clearly states that GeoLM is trained on 80% and tested on 20%. But it is not clear whether the compared models were also trained on the down-stream task of toponym recognition. The same detail is missing for 3.3 and 3.4. Section 3.2 clearly mentions that it is performed in an unsupervised fashion. ", "questions_to_authors": "Are the compared models in section 3.1, 3.3 and 3.4 also fine-tuned on respective down-stream tasks?", "paper_topic_and_main_contributions": "The paper proposes a new language model that is trained on text data that includes geo-entities. Specifically, the approach merges data from Wikipedia, Wikidata, and Open Street Map where every data point is represented as text data. The proposed GeoLM model is pre-trained on this collected data using BERT model as a base. The training procedure includes two tasks: contrastive learning and masked language modelling. The approach is evaluated on 4 down-stream tasks that are related to parsing geo-entities in text. \n\nThe main contribution is the proposed GeoLM language model that is pre-trained on the collected data points.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new language model (GeoLM) that is pre-trained using masked language modelling and a contrastive objective on descriptions of geospatial locations and information with the goal of enhancing geospatial abilities of the model. This is evaluated using a suite of downstream tasks including toponym recognition, toponym disambiguation, geospatial relation extraction and geo-entity tagging and shows improvements over baselines and existing methods.

All reviewers agree on multiple aspects, including that the paper is well written and motivated, that it addresses a relevant problem and highlight that the results are good in comparison with relevant baselines. Evaluating on four downstream tasks gives robustness to the results.

There were some questions about some details of the work (e.g. trie based matching, evaluation setup) that were addressed well in the author response. Ablation experiments were also mentioned by two of the reviewers, and the response provided results on this aspect as well.  This leads to all reviewers agreeing that the paper methodology and results are sound.

Overall, the authors should consider including the details provided in the discussion in their camera ready version of the work. Beyond this, there are few pending comments from the reviewers.
Decision: Accept-Main