Venue: NAACL 2024

# Improving Toponym Resolution by Predicting Attributes to Constrain Geographical Ontology Entries

Zeyu Zhang

Egoitz Laparra and Steven Bethard

School of Information

University of Arizona

Tucson, AZ {zeyuzhang,laparra,bethard}@arizona.edu

###### Abstract

Geocoding is the task of converting location mentions in text into structured geospatial data. We propose a new prompt-based paradigm for geocoding, where the machine learning algorithm encodes only the location mention and its context. We design a transformer network for predicting the country, state, and feature class of a location mention, and a deterministic algorithm that leverages the country, state, and feature class predictions as constraints in a search for compatible entries in the ontology. Our architecture, GeoPLACE, achieves new state-of-the-art performance on multiple datasets. Code and models are available at [https://github.com/clulab/geonorm](https://github.com/clulab/geonorm).

## 1 Introduction

Geocoding is the task of matching locations in text to geospatial coordinates or entries in a geographical database. Geocoding systems support document categorization and retrieval Bhargava et al. (2017), historical event analysis Tateosian et al. (2017), monitoring the spread of infectious diseases Hay et al. (2013), and disaster response mechanisms Ashktorab et al. (2014); de Bruijn et al. (2018). Geocoding is challenging because identical place names may refer to different geographical locations (e.g., _San Jose_ in Costa Rica vs. _San Jose_ in California, USA), while distinct names can represent the same geographical location (e.g., _Leeuwarden_ and _Ljouwert_ in the Netherlands).

The traditional paradigm for geocoding systems is to train machine learning algorithms that encode the location mention, its context, and the geographical ontology entries together when predicting a label for the mention. CamCoder Gritta et al. (2018), ReFinED Ayoola et al. (2022), and GeoNorm Zhang and Bethard (2023) all take this approach, with the latter showing that explicit countries and states in the context are especially helpful in this paradigm. However, these approaches are not able to take advantage of implicit context, such as countries and states that are not mentioned in the text but are inferrable from it.

We propose an novel prompt-based approach to geocoding that automatically identifies the implicit geographic information necessary to resolve location mentions. In this new paradigm for geocoding, we first apply a text classification approach that takes a prompt containing the location mention and some document context as input and predicts ontology attributes such as the location's enclosing country and state. For example, our approach would predict that _Paris_ in a document about Texas would have the attributes _"a Populated Place located in Texas in the United States."_ The constraints implied by these predictions are used to deterministically filter the ontology entries. Our novel architecture, GEOgraphical normalization by Predicting Location Attributes to Constrain ontology Entries (GeoPLACE) is illustrated in Figure 1.

Our work makes the following contributions:

* We introduce a new paradigm for geocoding, predicting implicit geographic information to enable deterministic filtering of the ontology.
* We design a transformer network for predicting the country, state, and feature class of a location mention, combining a novel prompt for geographic text classification with a masked language modeling objective.
* We introduce a novel deterministic algorithm that leverages the country, state, and feature class predictions as constraints in a search for compatible entries in the ontology.
* Our proposed approach achieves new state-of-the-art performance on multiple datasets.

## 2 Related Work

Prior work in geocoding included models based on hand-crafted rules and heuristics Grover et al. (2010); Tobin et al. (2010); Lieberman et al. (2010);

Lieberman and Samet, 2011; Berico Technologies, 2012; Karimzadeh et al., 2013), and traditional machine learning models such as support vector machines Martins et al. (2010); Freire et al. (2011); Lieberman and Samet (2012); Speriosu and Baldridge (2013); Zhang and Gelernter (2014); DeLozier et al. (2015); Kamalloo and Rafiei (2018); Wang et al. (2019). However, most recent approaches to geocoding use neural networks.

Neural network based models have approached geocoding both as a ranking problem, trying to sort ontology entries by their appropriateness as a label for a location mention Hosseini et al. (2020); Ardanuy et al. (2020); Ayoola et al. (2022); Zhang and Bethard (2023) and as a classification problem, trying to map a location mention directly to one of an \(N\times N\) grid of tiles covering the Earth's surface Gritta et al. (2018); Cardoso et al. (2019); Kulkarni et al. (2021). The most successful approaches encode not just the mention and ontology entry names, but also context around the mention and information from the ontology such as population Gritta et al. (2018); Ayoola et al. (2022); Zhang and Bethard (2023). Many neural architectures have been considered, including convolutional Gritta et al. (2018); Kulkarni et al. (2021), recurrent Cardoso et al. (2019), and transformer networks Ayoola et al. (2022); Zhang and Bethard (2023).

In contrast to these approaches, we predict geographical attributes (e.g., enclosing state) and use those to deterministically select an ontology entry.

## 3 Proposed Methods

The problem of geocoding can be formalized as defining a function \(f(m|T,M,E)=\hat{e}\) where \(T\) is the text of a document, \(M\subset T\) is the location mentions in the document, \(E\) is the set of geographical database entries, \(m\in M\) is the mention under consideration, and \(\hat{e}\in E\) is the entry predicted by \(f\) for \(m\). In our paradigm for geocoding, we formulate \(f\) to first predict the country, state, and feature of \(m\), next query the ontology with \(m\) to find candidate entries, then select the entry that violates the fewest constraints implied by the predicted attributes as the prediction \(\hat{e}\). Formally:

\[\hat{C}_{m},\hat{S}_{m},\hat{F}_{m} =\textsc{AttributePredictor}(m,M)\] \[\hat{E} =\textsc{CandidateGenerator}(m,E)\] \[f(m|T,M,E) =\textsc{Constrainer}(\hat{E},\hat{C}_{m},\hat{S}_{m},\hat{F}_{ m})\]

where \(C_{m},S_{m},F_{m}\) are the lists of predicted countries, states, and feature classes for \(m\). The Attributepredictor (see section 3.1) is a novel formulation of geographical text classification, the Candidategenerator (see section 3.2) is the best ranking system from prior work, and the Constraint-based algorithm.

### Attribute Predictor

This function predicts the country, state, and feature class of \(m\). It is formulated as a text classification model, based on a novel input prompt coupled with a masked language modeling objective. The prediction targets are defined as:

**Feature Class**: is one of the nine types defined by GeoNames: \(A\), Administrative boundaries (e.g., countries, states, provinces); \(P\), Populated places (e.g., cities, towns, villages); \(U\), Undersea features (e.g., oceanic ridges, trenches), etc.
**State**: is the canonical name of one of the 3871 first-order administrative divisions in GeoNames, such as states, provinces, or regions.
**Country**: is the canonical name of one of the 252 countries in GeoNames.

Figure 1: The architecture of our model: GEOgraphical normalization by Predicting Attributes to Constrain Ontology Entries (GeoPLACE). The figure shows how GeoPLACE normalizes a mention of _Paris_.

 

We implement prediction of these targets as:

\[Z ={}_{\textsc{transformer}}(\texttt{Toinput}(m,M))\] \[\hat{C}_{m} =\mathrm{softmax}(Z_{c}W_{c})\] \[\hat{S}_{m} =\mathrm{softmax}(Z_{s}W_{s})\] \[\hat{F}_{m} =\mathrm{softmax}(Z_{f}W_{f})\]

where Toinput discards all of \(m\)'s context \(T\) except for the location mentions \(M\) and produces text of the form [CLS] This document discusses these location mentions: \(m_{1}\), \(m_{2}\), ..., \(m_{|M|}\) in which \(m\) is [MASK] located in [MASK] of [MASK] [SEP]1; \(f\), \(s\), \(c\), are the indexes of the three [MASK] tokens; \(W_{c},W_{s},W_{f}\in\mathbb{R}^{N\times H}\) are the learnable parameters of the three classification heads; \(N\) is the size of the transformer tokenizer's vocabulary; and \(H\) is the size of the transformer's contextualized representations. We add new tokens to the transformer's tokenizer to ensure that every country, state, and feature class is a single token in the classifier output, e.g., making _United States_ a single token. This single-token prediction approach compares favorably to a multi-token sequence-to-sequence prediction approach, as shown in section 4.

Footnote 1: This prompt dramatically reduces the size of the input while still providing most of the critical document-level information for disambiguating toponyms

The model is trained on the labeled data in the toponym datasets with cross-entropy loss:

\[L=C_{m}log(\hat{C}_{m})+S_{m}log(\hat{S}_{m})+F_{m}log(\hat{F}_{m})\]

where \(C_{m}\), \(S_{m}\), and \(F_{m}\) are one-hot vectors of size \(N\) representing the true country, state, and feature class for mention \(m\). At prediction time, we constrain the outputs of the softmax to the subset of the vocabulary appropriate for each prediction type. For example, when the model predicts the word for the country [MASK], only the 252 country names are allowed to be non-zero.

We also pre-train (before the fine-tuning) on data we synthesize from the GeoNames ontology. We collect all entries that are cities, states, or countries (filtering out those with less than 100 population) or are in the feature classes H (stream, lake), L (parks, area), or T (mountain, hill, rock). To construct a synthetic training example for a selected entry \(e\), we use the prompt format of Toinput, with the canonical name of \(e\) as the mention \(m\) and the canonical names of a randomly selected set of entries in the same country as \(e\) as the other document mentions \(M\).

### Candidate Generator

We adopt the candidate generator of Zhang and Bethard (2023), which outperformed prior candidate generators and some end-to-end systems. It uses Lucene to index GeoNames entries by their canonical and alternative names, selects entries for a mention by applying a series of searches including exact string matching and character 3-gram matching, and sorts the resulting entries to place most populous countries at the top of the list.

```
Input:a list of candidate entries, \(\hat{E}_{m}\)  top 3 predicted countries, \(\hat{C}_{m}\)  top 3 predicted states, \(\hat{S}_{m}\)  top 3 predicted feature classes, \(\hat{F}_{m}\) Output:selected candidate entry \(\hat{e}\)
1DefScore\((x,L)\):if\(x=L_{0}\)thenreturn2 else return\(x\in L\)
2DefEntKey\((e)\):\(c\leftarrow\textsc{Country}(e)\)
3\(s\leftarrow\textsc{State}(e)\)
4\(f\leftarrow\textsc{Feature}(e)\)
5\(key_{1}\gets\textsc{Score}(c,\hat{C}_{m})\cdot\textsc{Score}(s,\hat{S}_{ m})\)
6\(key_{2}\leftarrow(c\in\hat{C}_{m})\cdot(s\in\hat{S}_{m})\cdot\textsc{Score}( f,\hat{F}_{m})\) return\((key_{1},key2)\) returnMax\((\hat{E}_{m},\textsc{key}=\textsc{EntryKey})\)
```

**Algorithm 1**Constrained Entry Selection

### Constraint

Figure 2 illustrates how our constrained sorts the output of the candidate generator (entries) using the output of the attribute predictor (countries, states, and feature classes). Algorithm 1 formally defines this process. We define the Score of a prediction as 2 if it was the top ranked prediction, 1 if it was the second or third ranked prediction, and 0 otherwise. Entries are then sorted by the product of the country and state Scores, with the Score of the feature class used to break ties. Intuitively, if the attribute predictor predicts C and S as the most probable country and state, then the constraint will rank entries from GeoNames that are within country C and state S higher than other entries. We use a stable sort, so candidates that are assigned the same score retain their population-based sorting from the candidate generator.

```
Input:a list of candidate entries, \(\hat{E}_{m}\)  top 3 predicted countries, \(\hat{C}_{m}\)  top 3 predicted states, \(\hat{S}_{m}\)  top 3 predicted features, \(\hat{F}_{m}\) Output:selected candidate entry \(\hat{e}\)
1DefScore\((x,L)\):if\(x=L_{0}\)thenreturn2 else return\(x\in L\)
2DefEntKey\((e)\):\(c\leftarrow\textsc{Country}(e)\)
3\(s\leftarrow\textsc{State}(e)\)
4\(f\leftarrow\textsc{Feature}(e)\)
5\(key_{1}\leftarrow\textsc{Score}(c,\hat{C}_{m})\cdot\textsc{Score}(s,\hat{S}_{ m})\)
6\(key_{2}\leftarrow(c\in\hat{C}_{m})\cdot(s\in\hat{S}_{m})\cdot\textsc{Score}(f, \hat{F}_{m})\)
7return\((key_{1},key2)\) returnMax\((\hat{E}_{m},\textsc{key}=\textsc{EntryKey})\)
```

**Algorithm 2**Constraint Based on the Classifier

### Model selection

Before evaluating on the test sets, we performed model selection on the development sets.

For the attribute predictor, we explored a small number of learning rates (1e-6, 2e-6, 5e-6, 1e-5) 

and number of epochs (10, 20, 30, 40).The best learning rate and number of epochs was selected based on accuracy on the attribute prediction task (not on the full geocoding task).

For the constraint, we explored different ways to define \(key_{1}\) and \(key_{2}\) and different ways to define the number of predictions to consider in the container. See appendix A.2 for details.

Additional implementation details of our models are in appendix A.3.

## 4 Experiments

We conduct primary experiments on three toponym resolution datasets: Local Global Lexicon (LGL; Lieberman et al., 2010), a collection 588 news articles from local and small U.S. news sources; GeoWebNews Gritta et al. (2019) a collection of 200 articles from 200 globally distributed news sites; and TR-News Kamalloo and Rafiei (2018) a collection 118 articles from various global and local news sources. All datasets use as their ontology GeoNames, a crowdsourced database of almost 7 million entries that contains geographic coordinates (latitude and longitude), alternative names, feature class (country, city, river, mountain, etc.), population, elevation, and positions within a political geographic hierarchy. See appendix A.1 for statistics of the datasets.

We adopt the train, development, and test splits and evaluation metrics of prior work Zhang and Bethard (2023). We refer the reader to that paper for details, but briefly, _accuracy_ measures how often the correct database entry was predicted, while _accuracy@161km_, mean error distance_, and _area under the curve_ all give some partial credit for predicting entries that are wrong but geographically close to the correct entry.

We compare to the state-of-the-art geocolors:

**ReFinED** is an end-to-end Wikipedia-linking model that matches transformer-generated embeddings for tokens in the text to embeddings of ontology entries via dot products Ayoola et al. (2022). ReFinED was originally trained on Wikipedia, but Zhang and Bethard (2023) leveraged the existing links to GeoNames IDs to fine-tune it for toponym resolution. It is the Wikipedia-linking model with the best reported performance on our evaluation datasets.

**GeoNorm**Zhang and Bethard (2023) uses Lucene to index and generate candidate entries from the ontology, applies a transformer network jointly over the mention and each candidate entry to predict a single entry, and applies a two-stage process to first resolve countries and states and use them as context to resolve other mentions.

**GeoPLACE (+seq2seq)** is a variant of our model that replaces our masked language modeling objective with a generative objective, asking the model to directly produce is <feature-type> located in <state> of <country>. See appendix A.3 for prompting details.

## 5 Results

The top of table 1 compares our model to the existing state-of-the-art on LGL, GeoWebNews, and

Figure 2: Illustration of the constrained applied to _Paris_ in the context _It’s a northeast Texas thing, not just a Paris thing... Dallas media stations reported the same message as a hoax as early as Wednesday night.._



TR-News. (See appendices A.4 to A.6 for comparisons against other models and results on other datasets.) GeoPLACE outperforms prior work by large margins (more than 30% error reduction) on LGL and TR-News, while achieving similar performance on GeoWebNews.

Table 2 presents a qualitative analysis of errors encountered by GeoNorm [15] and our latest state-of-the-art model, GeoPLACE. The first row displays an example where GeoNorm falls short while GeoPLACE excels. This can be attributed to GeoPLACE's ability to accurately predict the countries, states, and feature codes of toponyms in the text prior to resolution.

The second row portrays an instance where our most proficient model, GeoPLACE, experiences a failure. This occurs because predicting feature codes with the aid of a masked language model proves to be more challenging compared to predicting countries and states. Thoroughly resolving this problem is likely to necessitate improvements in the prediction performance for all types of geographical metadata.

The bottom of table 1 shows an ablation of our model. Pre-training on synthesized data provides small but consistent gains across all datasets. The Seq2Seq approach yields worse performance than our masked language modeling approach both when fine-tuning BART-large and when using GPT3 in zero-shot mode.

We release our model for English geocoding under the Apache License v2.0, for off-the-shelf use at [https://github.com/clulab/geonorm](https://github.com/clulab/geonorm).

## 6 Conclusion

We introduced a new paradigm for geocoding where we predict implicit geographical attributes and use those to deterministically constrain the set of valid ontology entries. Our approach leads to large error reduction over the current state-of-the-art on the LGL and TR-News datasets.

\begin{table}
\begin{tabular}{l r r r r r r r r r r r} \hline \hline  & \multicolumn{7}{c}{LGL (test)} & \multicolumn{7}{c}{GeoWebNews (test)} & \multicolumn{7}{c}{TR-News (test)} \\ \cline{3-18} Model & Acc & A161 & Err & AUC & Acc & A161 & Err & AUC & Acc & A161 & Err & AUC \\ \hline ReFinED [9] & .576 & - & - & - & .658 & - & - & .720 & - & - & - \\ ReFinED [15] & .786 & - & - & - & .782 & - & - & - & .858 & - & - & - \\ Candidate Generator [15] & .606 & .685 & 119 & .263 & .694 & .774 & 92 & .194 & .716 & .812 & 95 & .169 \\ GeoNorm [15] & .807 & .824 & 46 & .135 & **.828** & .862 & **55** & .114 & .918 & .933 & 34 & .057 \\ GeoPLACE (ours) & **.863** & **.894** & **21** & **.084** & .822 & **.878** & **57** & **.112** & **.947** & **.957** & **18** & **.038** \\ \hline GeoPLACE (-synthesized pre-training) & .851 & .886 & 24 & .093 & .809 & .864 & 63 & .123 & .904 & .922 & 20 & .062 \\ GeoPLACE (+seq2seq, +generative fine-tuned BART) & .633 & .696 & 111 & .250 & .704 & .776 & 92 & .191 & .727 & .812 & 95 & .167 \\ GeoPLC (+seq2seq, +generative zero-shot GPT-3) & .733 & .795 & 80 & .176 & .719 & .811 & 85 & .171 & .830 & .869 & 63 & .115 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance on the test sets. Higher is better for accuracy (Acc) and accuracy@161km (A161). Lower is better for mean error (Err) and area under the error distances curve (AUC). We do not report distance-based metrics for ReFinED as it does not make predictions for all mentions. The best performance in each column is in bold.

\begin{table}
\begin{tabular}{l l c c c c c c} \hline \hline Example & \multicolumn{7}{c}{Candidate} & \multicolumn{3}{c}{Rank} \\ \cline{2-7} Name & Pop. & Type & State & Country & GeoNorm & GeoPLACE \\ \hline
1 _But the Mt. Pleasant_ New London County & 274055 & ADM2 & Connecticut & United States & 1 & 2 \\ _News has reviewed legal_ New London & 27179 & PPLL & Connecticut & United States & 2 & 3 \\ _documents.....we writes,_ New London & 7172 & PPLL & Wisconsin & United States & 3 & 4 \\ _as do my efforts to in-_**New London** & 1882 & PPLL & Iowa & United States & 4 & 1 \\ _sure New London is a safe community._ & & & & & & & \\ \hline
2 _John-Paul Delaney (18),_ Tipperary & 159553 & ADM2 & Munster & Ireland & & 1 \\ _is charged with assault,_**Tipperary** & 4979 & PPLL & Munster & Ireland & & 2 \\ _assant causing harm and_ Tipperary & 0 & HMSD & Western Australia & Australia & & 3 \\ _theft of a mobile phone_ Tipperary & 0 & HMSD & New South Wales & Australia & & 4 \\ _at Main Street, Tipperary,_ & & & & & & & \\ _on the same date._ & & & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Examples of predictions from GeoNorm [15] and our new SOTA model, GeoPLACE. Target location mentions are underlined. Human annotated ontology entries are in bold. (ADM2 represents a county, PPL represents a city, HMSD represents a residence specific to Australia and New Zealand) 



## 7 Limitations

The possible space of prompts is large, and while our novel location-based prompt worked well with our masked language modeling approach, it did not work well for generative models like BART. It is possible that more intensive exploration of alternative prompts could bring the performance of these generative models up to the performance of our masked language modelling approach. We also only explored zero-shot approaches for GPT-3, and though full fine-tuning BART did not yield acceptable performance, it is possible that few-shot approaches or fully fine-tuning GPT-3 would.

GeoPLACE is limited by its training and evaluation data, which covers only thousands of English toponyms from news articles, while there are many millions of toponyms across the world. It is likely that there are regional differences in GeoPLACE's accuracy that will need to be addressed by future research.

GeoPLACE is currently limited to geocoding. To apply this approach to other entity linking problems, one would need to identify the attributes that help constrain the search from the ontology, and then explore a few definitions of keys as we have in section 3.4. This would be an interesting area for future research.

## Acknowledgements

This work was supported by the National Science Foundation under Grant No. 1831551. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

## References

* C. Ardanuy, K. Hosseini, K. McDonough, A. Krause, D. van Strien, and F. Nanni (2020)A deep learning approach to geographical candidate selection through toponym matching. In Proceedings of the 28th International Conference on Advances in Geographic Information Systems, pp. 385-388. Cited by: SS1.
* Z. Ashktorab, C. Brown, M. Nandi, and A. Culotta (2014)Tweedr: mining twitter to inform disaster response.. In ISCRAM, pp. 269-272. Cited by: SS1.
* T. Ayoola, J. Fisher, and A. Pierleoni (2022)Improving entity disambiguation by reasoning over a knowledge base. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 2899-2912. External Links: Link Cited by: SS1.
* B. Technologies (2012)Cartographic location and vicinity indexer (clavin). External Links: Link Cited by: SS1.
* P. Bhargava, N. Spasojevic, and G. Hu (2017)Lithium NLP: a system for rich information extraction from noisy user generated text on social media. In Proceedings of the 3rd Workshop on Noisy User-generated Text, Copenhagen, Denmark, pp. 131-139. External Links: Link Cited by: SS1.
* A. Barbara Cardoso, R. Martins, and J. Estima (2019)Using recurrent neural networks for toponym resolution in text. In EPIA Conference on Artificial Intelligence, pp. 769-780. Cited by: SS1.
* J. A. de Bruijn, H. Moel, B. Jongman, J. Wagemaker, and J. CJH Aerts (2018)TaggS: grouping tweets to improve global geoparsing for disaster response. Journal of Geovisualization and Spatial Analysis2 (1), pp. 2. Cited by: SS1.
* G. DeLOzier, J. Baldridge, and L. London (2015)Gazetteer-independent toponym resolution using geographic word profiles. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, pp. 2382-2388. Cited by: SS1.
* N. Freire, J. Borbinha, P. Calado, and B. Martins (2011)A metadata geoparsing system for place name recognition and resolution in metadata records. In Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, pp. 339-348. Cited by: SS1.
* M. Gritta, M. Taher Pilehvar, and N. Collier (2018)Which Melbourne? augmenting geoco-ging with maps. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 1285-1296. External Links: Link Cited by: SS1.
* M. Gritta, M. T. Pilehvar, and N. Collier (2019)A pragmatic guide to geoparsing evaluation. Language Resources and Evaluation, pp. 1-30. Cited by: SS1.
* C. Grover, R. Tobin, K. Byrne, M. Woollard, J. Reid, S. Dunn, and J. Ball (2010)Use of the edinburgh geoparser for georeferencing digitized historical collections. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences368 (1925), pp. 3875-3889. Cited by: SS1.
* A. Halterman (2017)WordCAI: full text egop-arsing and event geocoding. The Journal of Open Source Software2 (9). Cited by: SS1.



Simon I Hay, Katherine E Battle, David M Pigott, David L Smith, Catherine L Moyses, Samir Bhatt, John S Brownstein, Nigel Collier, Monica F Myers, Dylan B George, et al. 2013. Global mapping of infectious disease. _Philosophical Transactions of the Royal Society B: Biological Sciences_, 368(1614):20120250.
* Hosseini et al. (2020) Kasra Hosseini, Federico Nanni, and Mariona Coll Ardanov. 2020. DeezyMatch: A flexible deep learning approach to fuzzy string matching. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 62-69, Online. Association for Computational Linguistics.
* Kamalloo and Rafiei (2018)Ehsan Kamalloo and Davood Rafiei. 2018. A coherent unsupervised model for toponym resolution. In _Proceedings of the 2018 World Wide Web Conference_, pages 1287-1296.
* Karimzadeh et al. (2013) Morteza Karimzadeh, Wenyi Huang, Siddhartha Banerjee, Jan Oliver Wallgrun, Frank Hardisty, Scott Pezanowski, Prasenjit Mitra, and Alan M MacEachren. 2013. Geotxt: a web api to leverage place references in text. In _Proceedings of the 7th workshop on geographic information retrieval_, pages 72-73.
* Kulkarni et al. (2021) Sayali Kulkarni, Shailee Jain, Mohammad Javad Hosseini, Jason Baldridge, Eugene le, and Li Zhang. 2021. Multi-level gazetteer-free geocoding. In _Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics_, pages 79-88, Online. Association for Computational Linguistics.
* Lieberman and Samet (2011) Michael D Lieberman and Hanan Samet. 2011. Multi-faceted toponym recognition for streaming news. In _Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval_, pages 843-852.
* Lieberman and Samet (2012) Michael D Lieberman and Hanan Samet. 2012. Adaptive context features for toponym resolution in streaming news. In _Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval_, pages 731-740.
* Lieberman et al. (2010) Michael D Lieberman, Hanan Samet, and Jagan Sankaranarayanan. 2010. Geotagging with local lexicons to build indexes for textually-specified spatial data. In _2010 IEEE 26th international conference on data engineering (ICDE 2010)_, pages 201-212. IEEE.
* Liu et al. (2021) Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021. Self-alignment pretraining for biomedical entity representations. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4228-4238.
* Martins et al. (2010) Bruno Martins, Ivo Anastacio, and Pavel Calado. 2010. A machine learning approach for resolving place references in text. In _Geospatial thinking_, pages 221-236. Springer.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auil. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of NAACL-HLT 2019: Demonstrations_.
* Speriosu and Baldridge (2013) Michael Speriosu and Jason Baldridge. 2013. Text-driven toponym resolution using indirect supervision. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1466-1476.
* Tateoasian et al. (2017) Laura Tateoasian, Rachael Guenter, Yi-Peng Yang, and Jean Ristaino. 2017. Tracking 19th century late blight from archival documents using text analytics and geoparsing. In _Free and open source software for geospatial (FOSS4G) conference proceedings_, volume 17, page 17.
* Tobin et al. (2010) Richard Tobin, Claire Grover, Kate Byrne, James Reid, and Jo Walsh. 2010. Evaluation of georeferencing. In _proceedings of the 6th workshop on geographic information retrieval_, pages 1-8.
* Wang and Hu (2019) Jimin Wang and Yingjie Hu. 2019. Enhancing spatial and textual analysis with eupeg: An extensible and unified platform for evaluating geoparsers. _Transactions in GIS_, 23(6):1393-1419.
* Wang et al. (2019) Xiaobin Wang, Chunping Ma, Huafei Zheng, Chu Liu, Pengjun Xie, Linlin Li, and Luo Si. 2019. Dom\(\_\)nlp at semeval-2018 task 12: A pipeline system for toponym resolution. In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 917-923.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Fantowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jenrite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Dzrane, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Zhang and Gelernter (2014) Wei Zhang and Judith Gelernter. 2014. Geocoding location expressions in twitter messages: A preference learning method. _Journal of Spatial Information Science_, 2014(9):37-70.
* Zhang and Bethard (2023) Zeyu Zhang and Steven Bethard. 2023. Improving toponym resolution with better candidate generation, transformer-based reranking, and two-stage resolution. In _Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)_, pages 48-60, Toronto, Canada. Association for Computational Linguistics 

Appendix

### Dataset details

The number of toponyms and articles in each of the splits of each of the datasets is shown in table A1.

### Constraint Variants

For the constraint, we explored three different ways to define \(key_{1}\) and \(key_{2}\).

**alg3**: defines \(key_{1}\) and \(key_{2}\) as in alg. 1.
**alg2**: allows scores to range from 0 to the length of the list, rather than just from 0 to 2. It defines:

\[key_{1}\leftarrow\textsc{RIndex}(c,\hat{C}_{m})\cdot\textsc{ Hindex}(s,\hat{S}_{m})\] \[key_{2}\leftarrow(c\in\hat{C}_{m})\cdot(s\in\hat{S}_{m})\cdot \textsc{RIndex}(f,\hat{F}_{m})\]
**Def**\(\textsc{RIndex}(x,L)\colon\textbf{if $x\not\in L$ then }0\)****

**else**\(|L|-lst.index(val)\)
**alg1**: prioritizes matching the first country, and also allows scores to range from 0 to the length of the list. It defines:

\[key_{1}\leftarrow(c=\hat{C}_{m_{0}})\cdot\textsc{ RIndex}(s,\hat{S}_{m})\] \[key_{2}\leftarrow(c\in\hat{C}_{m})\cdot(s\in\hat{S}_{m})\cdot \textsc{RIndex}(f,\hat{F}_{m})\]

We also explored four different ways to define the number of predictions to consider in the constrained.

**top3**: Only the top 3 countries, states, and feature classes are considered
**top4**: Only the top 4 countries, states, and feature classes are considered
**top5**: Only the top 5 countries, states, and feature classes are considered
**top553**: The top 5 countries, top 5 states, and top 3 feature classes are considered

Table A2 shows that there were not large differences between the key algorithms nor the number of prediction algorithms in terms of accuracy, but alg3 top3 performed slightly better. Table A2 also shows that pre-training on synthesized data consistently helped on LGL and GeoWebNews but led to small drops in performance on TR-News.

### Implementation details

We adopt the candidate reranker of Zhang and Bethard (2023). We implement the attribute predictor with the PyTorch2 v1.7.0 APIs in Huggingface Transformers v2.11.0 Wolf et al. (2020), using bert-base. We train with the AdamW optimizer, a learning rate of 5e-6, a maximum sequence length of 256 tokens, and a number of epochs of 40. When training, we use one NVIDIA A100 GPU with 40G memory and a batch size of 64. The total number of parameters in our model is 112M and the training time is about 0.15 hours.

Footnote 2: [https://pytorch.org/](https://pytorch.org/)

When synthesizing data from the geographical ontology for pre-training, most of the hyperparameters are same with finetuning except that batch size is 32 and training epochs is 10.

When using a generative sequence-to-sequence objective instead of a masked language modeling objective, we utilize bart-large with the PyTorch v2.0.0 APIs in Huggingface Transformers v4.11.3 Wolf et al. (2020) and FAIRSEQ v0.12.2 Ott et al. (2019). We train with the AdamW optimizer, a initial learning rate of 1e-5, a learning rate scheduler type of polynomial, a maximum sequence length of 1024 tokens, and the steps of training of 40000. When training, we use one NVIDIA A100 GPU with 40G memory and a batch size of 8. During evaluation, we use beam search with a beam size of 5. The total number of parameters in our model is 406M and the training time is about 1.3 hours. We use one model to generate only one attribute, when we generate the country name, we use the prompt [CLS] This document discusses these location mentions: \(m_{1}\), \(m_{2}\), ..., \(m_{|M|}\). Which country is START \(m\) END located ?, the prefix prompt for output generation is \(m\) is located in. When we generate the state name, we use the prompt [CLS] This document discusses these location mentions: \(m_{1}\), \(m_{2}\), ..., \(m_{|M|}\). Which state is START \(m\) END located ?, the prefix prompt for output generation is \(m\) is located in. When we generate the feature class, we use the prompt [CLS] This document discusses location mentions: \(m_{1}\), \(m_{2}\), ..., \(m_{|M|}\). Which feature class does START \(m\) END belong to ?, the prefix prompt for output generation is \(m\) belong to

### EUPEG results

We also report results using the Extensible and Unified Platform for Evaluating Geoparsers (EUPEG; Wang and Hu, 2019). This platform evaluates not geocoders, but geopparsers, where a model must both detect locations and match them to ontology entries. So we couple our geocoder with the best location detection model on EUPEG, the StanfordNER system 

This platform reports several metrics that are incomparable across systems. Accuracy, accuracy@161km, mean error, and area under the error distances curve are all calculated only over locations that were detected, so that a model that detects only 1% of locations but matches 100% of them to their correct ontology entries would get perfect values for these scores, while a model that detects 100% of locations and matches 90% of them to their correct ontology entries would score lower. We nonetheless report these incomparable metrics as EUPEG provides no alternative. EUPEG results are shown in table A3.

### Recall of Geographical Attributes Prediction

Table A4 shows the performance of the geographical attribute prediction classifiers alone, i.e., as classifiers rather than as components in a geocoding system. We report recall@3 since the constrained considers the top 3 predictions of the attribute predictor. Performance across all datasets and all classifiers is 0.84 or higher.

### Full table of Test Performance

Table A5 compares GeoPLACE to other systems that, due to space limitations, we could not include in table 1.

 



\begin{table}
\begin{tabular}{l c