HeGeL: A Novel Dataset for Geo-Location from Hebrew Text
Year: 2023

Abstract:
The task of textual geolocation — retrieving the coordinates of a place based on a free-form language description — calls for not only grounding but also natural language understanding and geospatial reasoning. Even though there are quite a few datasets in English used for geolocation, they are currently based on open-source data (Wikipedia and Twitter), where the location of the described place is mostly implicit, such that the location retrieval resolution is limited. Furthermore, there are no datasets available for addressing the problem of textual geolocation in morphologically rich and resourcepoor languages, such as Hebrew. In this paper, we present the Hebrew Geo-Location (HeGeL) corpus, designed to collect literal place descriptions and analyze lingual geospatial reasoning. We crowdsourced 5,649 literal Hebrew place descriptions of various place types in three cities in Israel. Qualitative and empirical analysis show that the data exhibits abundant use of geospatial reasoning and requires a novel environmental representation.1

heading: 1 Introduction and Background
text: Textual Geolocation Identification, a crucial component of Geographic Information Retrieval (GIR), is the task of resolving the location, i.e., coordinates of a place, based on the reference to it in a text. It requires a combination of language and environmental knowledge. On top of the usual non-spatial linguistic challenges in Natural Language Understanding (NLU), such as named entity recognition (NER), anaphora resolution, bridging anaphora, etc., the textual geolocation task presents geospatial challenges that require multimodal processing and grounding (Ji et al., 2022; Fried et al., 2022; Misra et al., 2017; Qi et al., 2020; Paz-Argaman et al., 2020).
∗Equal contribution. 1For data and code see https://github.com/OnlpLab/HeGeL
Proper names, such as ‘Rabin Square’, also known as named entities in Natural Language Procesing (NLP), and as rigid designators in formal semantics (Kripke, 1972), can be easily grounded based on a Gazetteer or a simple map. However, geolocating linguistic terms that involve spatial expressions without the explicit mention of a proper name still present an open challenge. This interpretation challenge includes the understanding and resolution of (at least): (i) definite descriptions, such as ‘the school’ (ii) geospatial terms, such as cardinal directions; ‘east of’; and (iii) geospatial numerical reasoning; ‘two buildings away from the pharmacy’. To address these and other challenges, we need to both ground entity mentions to their corresponding physical entities in the environment, and to reason about geospatial relations expressed between entities — these two processes being closely intertwined.
To do so, we need a corpus for the geolocation task that maps rich geospatial place descriptions to their corresponding location coordinates. However, current corpora for geolocation are based on naturally-occurring open-source resources, such as Wikipedia articles (Eisenstein et al., 2010; Wing and Baldridge, 2011; Han et al., 2012; Wing and Baldridge, 2014; Wallgrün et al., 2018), which are not spatially oriented, i.e., the description of locations is implicit or absent in the corresponding text. Subsequently, the accuracy of retrieval is fairly low (around 100 km).
Furthermore, all geolocation datasets previously studied in NLP are in English, with a dearth of corpora for low-resource languages, in particular, for morphologically rich languages, such as Hebrew. To understand the geolocation challenges and build models that do various spatial reasoning tasks, English cannot be our sole focus (Baldridge et al., 2018). Hebrew, a Semitic morphologically rich language is notoriously difficult to parse (Tsarfaty et al., 2020, 2019). Moreover, resources that are
7311
available for Hebrew NLP research focus on traditional tasks, such as Part-of-speech (POS) tagging, syntactic parsing, etc; and lack corpora for understanding and reasoning in real-world situations.
In this work we present HeGeL, a novel dataset for Hebrew Geo-Location, the first ever Hebrew NLU benchmark involving both grounding and geospatial reasoning. To create HeGeL, we crowdsourced 5,649 geospatially-oriented Hebrew place descriptions of various place types from three cities in Israel. We designed our task based on a realistic scenario of human place description, relying on people’s memory of the world, rather than, e.g., using a map (Anderson et al., 1991; Paz-Argaman and Tsarfaty, 2019). Crucially, relying on environmental cognition results in various levels of geospatial knowledge (Siegel and White, 1975) that are manifested in the descriptions and the geospatial reasoning that is required to resolve their location (Hayward and Tarr, 1995). To avoid the much simpler task of grounding proper named entities, we explicitly restricted the use of proper names in the description of the place and adjacent landmarks.
Unlike the text-based navigation task (MacMahon et al., 2006; Chen et al., 2019; Ku et al., 2020; De Vries et al., 2018; Thomason et al., 2020), which requires representing an agent’s current perspective, reflecting its route knowledge, we show that the HeGeL task requires a full-environment representation, thus, capturing complex geospatial relations among multiple physical entities. Through a thorough linguistic and empirical analysis, we demonstrate the characteristics and challenges associated with Hebrew place descriptions,
showing that HeGeL serves both as a challenging NLU benchmark and as a corpus for geospatial cognition research.

heading: 2 The HeGeL Task and Dataset
text: This work addresses the task of geolocating places on a map based on natural language (NL) geospatial descriptions that are given in a colloquial language and based on participants’ memory of the environment (i.e., cognitive map). The input to the HeGeL task is as follows: (i) an NL place description of the whereabouts of the place, and (ii) a map with rich details of the environment (e.g., physical entities names, geospatial relations, and attributes). The output is a pair of coordinates (x,y) specifying the physical location of the place described in the text. Figure 1 shows an example of a place description from HeGeL translated from Hebrew.
To simplify the crowdsourcing task and encourage participants’ engagement, we frame the data crowdsourcing process as the well-known game, the treasure hunt task (Kniestedt et al., 2022), in which the instructor-participant is required to describe in writing the location of the treasure, a known place in the city, to a different followerparticipant who then needs to locate it on a map. Thus, the online assignment is divided into two tasks: the instructor’s writing of place descriptions and the follower’s validation. To avoid preconceived notions as to the ‘correct’ way to describe a place, we first presented the participants with the task of writing a place description, and once completed, the validation task was given.2
2Appendix A includes additional data collection details.
We hereby provide the details of the two UI tasks:
(i) Task 1. Writing a place description In this task we requested participants to describe in a freeform text the location of a place known to them, to a third party who might not be familiar with the whereabouts of that place. To collect place descriptions based solely on people’s memory, we did not visualize the area of the place, e.g., on a map. Instead, we ensured that the participants are well familiarized with the place by asking them to state how familiar they are with the place on a scale of 1-5. If this score was 1 or 2, we presented the participant with a different place to describe. To ensure diverse human-generated textual descriptions, places were chosen based on their type, position/location in the city (places were spread across the city), geometry, size, and context. To avoid the use of proper names, we developed a rule-based methodology to make sure that the explicit name of the goal (place) or of the nearby landmarks (< 100 meters) will not appear explicitly in the description. The original description was saved, and the participants were asked to input another description without the above names.
(i) Task 2. Place description validation To verify that a person who reads the text description will understand where the treasure is hidden, i.e., geolocate the place, we developed a map-based retrieval task. The participant in the follower role was asked to read the crowdsourced textual description and mark its location on the map, i.e., where the treasure is hidden. For marking the location, we implemented an interactive online map based on OpenStreetMap (OSM),3 which allows the participants to move and zoom-in to precisely pin the described place on the map. The map supports the cognitive process needed to ground mentioned entities to physical entities, reason about the geospatial relations, and locate the described place. To familiarize participants with the interactive map tool and task, they had to first pass a simple map marking test, and only then they could start task 2 of reading place descriptions (given by other participants), marking place locations on the map, and rate the clarity of the textual description on a scale of 1-5.
Appendix B presents a display of the online assignment’s UI translated from Hebrew to English.
3OSM is a free, editable, map of the whole world, that was built by volunteers, with millions of users constantly adding informative tags to the map.
Target Selection and Retrieval Errors The treasure-hunt task we devised included 167 places in the three largest cities in Israel: Tel Aviv, Haifa, and Jerusalem. These three cities are differently shaped, and show different physical, morphological and topographic features, which potentially affect the legibility and imageability of urban components, and therefore also on place descriptions. These differences can be expressed in the use of various physical features and prepositions, e.g., frequent use of the physical object ‘landmark’ and the prepositions ‘above’ or ‘below’ in hilly terrains that characterize Haifa and Jerusalem.
To assess the quality and interpretability of the place descriptions, we calculate the shortest Euclidean distance between the coordinates of the goal’s (physical element) shape (polygon, line or point), and the location marked by the ’follower’ on the map (task 2); we term this distance as retrieval error. To determine the agreement rate among human participants, each textual place description is validated by at least two participants. To ensure that we work with descriptions that can be geolocated, we set a hard distance threshold of 300 meters, based on analysis of the descriptions’ clarity score that we had conducted on a prior (held-out) development corpus we collected for the task.

heading: 3 Data Statistics and Analysis
text: The resulting HeGeL dataset contains 5,649 validated descriptions paired with their coordinates on a map. The locations are divided among three cities: 2,142 in Tel Aviv, 1,442 in Haifa, and 2,065 in Jerusalem. 1,833 participants completed the writing task, inserting in total 10,946 place descriptions, and 2,050 participants completed 12,655 validation tasks. The dataset is balanced, with about 33 descriptions per place.
Figure 2 shows a Venn diagram representing the relation of the three sets of city-based vocabularies (formed from unique lemmas produced by More et al. (2019) lemmatization tool). The intersection of the three cities contains only 15.07% of the entire vocabulary (the union of the three cities’ vocabularies). The shared language is not focused on city-specific terms, such as ‘Knesset’. Instead, it includes rich spatial terms, such as ‘between’, modified prepositions such as ‘next to’, and nondefinite entities, such as ‘street’. From the Venn diagram we also conclude that almost half of the lemmas of the three vocabularies, corresponding
to the three cities, contain city-specific lemmas: 48.6%, 40.65%, and 49.3% for Tel Aviv, Haifa, and Jerusalem, respectively. As such, HeGeL enables a city-split setup, training on one city and testing on a different unseen city, where city-reserved named entities present an out-of-vocabulary (OOV) challenge for models trained on another city.
Table 1 shows an analysis of the linguistic phenomena manifested in the HeGeL dataset, demonstrating the spatial knowledge and reasoning skills required for solving the HeGeL task. We analyzed the frequency of the five types of elements in a city defined by Lynch (1960), along with the three types of spatial knowledge defined in Siegel and White (1975), and other spatial properties. The frequent use of cardinal directions, as well as the use of sur-
vey knowledge, suggests that any NLP model built to deal with the HeGeL task should not only represent a local view of the goal, or possible routes, but also take into consideration the full region, and mimic people’s map-like view of the environment. Therefore, unlike navigation tasks where only the agent’s current perspective is represented in the model, this task requires full representation of the environment.
We further perform a quantitative analysis of word tokens and lemmas that appear in HeGeL, depicted in Table 2. Overall, the HeGeL dataset contains a large vocabulary of 9,207 unique tokens and 6,663 unique lemmas. There are mentions of physical entities, but as we limited the mentions of named-entities of the described place and landmarks adjacent to it; these are relatively rare, and are mostly references to prominent city landmarks. Also, as most place descriptions are not route-based descriptions, there are only few verbs used in the descriptions. Prepositions, on the other hand, are abundant.
In Table 3, using a one-way analysis of variance (ANOVA) test, we found a significantly (p<0.05) different distribution between place type descriptions and the following features: number of named entities, number of verbs, human verification retrieval error, and clarity score.

heading: 4 Experiments
text: We create a zero-shot (ZS) city-based split, such that we train on one city and test on another. The train, development, and test sets correspond to the descriptions collected in Tel Aviv, Haifa, and Jerusalem, respectively. We evaluate different baseline models for the geolocation task on the HeGeL dataset. We use three evaluation metrics based on retrieval error: mean, median, and task completion (TC) accuracy – the percentage of place descriptions located within the 300 meters threshold. We provide three baselines for the HeGeL task.
We first assess a brute-force NER approach; i.e., we test whether recognizing named entities in the text and retrieving their corresponding coordinates is sufficient for solving the HeGeL task of geolocation. To this end, we used Google Maps API and produced two baseline models: (i) Google Maps API Query — we queried the API with the full raw text descriptions as input, with no prepossessing; and (ii) Oracle NER — we queried all 1-5 n-grams against Google Maps API and retrieved the closest geolocation to the goal.
In our second approach, we employ a dualencoder model. One encoder encodes the text using a Hebrew Monolingual pre-trained encoder, AlephBERT (Seker et al., 2022), which produces a 768- dimension vector representation of the text. The other encoder processes the environment, which is represented as a graph based on OSM data. Each point of interest in the graph is connected to an S2Cell4, which contains its geometry and is based on S2-geometry. These S2Cells are encoded using a random-walk algorithm to produce a 64- dimensional vector for each cell. These vectors are then passed through a linear layer to produce 768-dimensional vectors. We calculate the cosine similarity score between the text and environment vectors and use it to align the respective representations via maximization of the cosine similarity score with a cross-entropy loss over the scores.
4S2Cells are based on S2-geometry (https://s2geometry.io/), a hierarchical discretization of the Earth’s surface (Hilbert, 1935).
Performing an ANOVA test, we found a significantly (p<0.05) different distribution between place type descriptions and the retrieval error of the Oracle NER. The mean retrieval error of the Path and Node place types were the lowest in both human verification and Oracle NER. This suggests that both of these place types are easier for humans to geolocate.
The results in Table 4 show that our task is not solvable with adequate resolution by the Google Maps API. The human performance provides an upper bound for the HeGeL task performance, while the simple Google Maps API Query provides a lower bound. The Google API model’s low performance suggests that NER and the Gazetteerbased methods in and of themselves are insufficient to handle the HeGeL task successfully, and that geospatial reasoning is necessary. The Dualencoder’s low performance on the ZS split suggests that OOV is a major challenge. The few-shot (FS) split shows an improvement of the model after finetuning on additional samples from the test-region (FS 20% and 80%). This suggests that a possible solution for the city-split setup might be dataaugmentation via generating grounded descriptions for the tested region – an approach we reserve for future research.

heading: 5 Conclusion
text: The contribution of this paper is threefold. First, we present the first geolocation benchmark with Hebrew place descriptions. Second, to the best of our knowledge, this is the only crowdsourced geolocation dataset, thus, eliciting explicit geospatial descriptions, allowing for better retrieval resolution. Finally, our analysis shows that the dataset presents complex spatial reasoning challenges which require novel environmental model representation.
Limitations
While we aim for our HeGeL crowdsourcing methodology to be applicable to other languages, and in particular low-resource languages, the UI design and our analyses require knowledge of the intended language, as well as familiarity with the regions where it is spoken. Moreover, as our methodology relies on people’s familiarity with the places, it limits the cities chosen for the task and the participants that could take part, restricting the demographics of the participants accordingly. In addition, relying on people’s memory of the environment causes many of the descriptions to be too vague for humans to geolocate, thus, many of the descriptions were disqualified during the validation process as they could not have been resolved. The relatively low percentage of place descriptions that were successfully validated, raises the costs of collecting such a dataset.

heading: Acknowledgements
text: This research is funded by a grant from the European Research Council, ERC-StG grant number 677352, and a grant by the Israeli Ministry of Science and Technology (MOST), grant number 3- 17992, for which we are grateful.

heading: A Data Collection Details
text: We used the services of an Israeli surveying company to distribute the assignment to native Hebrewspeakers participants in Israel only. The survey company was charged with distributing the assignments to a balanced set of participants in terms of their demographic and geographic characteristics (e.g., an equal number of males and females). All participants were given full payment, nonrespective of whether they correctly completed the task.
The first page the participants viewed contains a disclosure about the assignments being part of academic research and the purpose of the assignments. The assignment protocol was approved by a behavioral review board. This approval was also presented to the participants on the initial screen. Also, the participants were required to read an informed consent form and sign an agreement box.

heading: B Participant Interface
text: The tasks are performed via an online assignment application, depicted in Figures 3-5.

heading: C Experimental Setup Details
text: The cross-entropy loss function was optimized with Adam optimizer (Kingma and Ba, 2015). The hyperparameter tuning is based on the average results run with three different seeds. The Learning rate was searched in [1e-5, 1e-4, 1e-3] and a 1e-5 was chosen. The S2cell level was searched in [13, 15, 17] and 13 was chosen. Number-of-epochs for early stopping was based on their average learning curve.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: limitation section
7 A2. Did you discuss any potential risks of your work? The data doesn’t contain private or sensitive information
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? The model is a standard encoder-decoder run for around 10 minutes with early stopping on 1 GPU.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? in the appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 2

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? in the appendix

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? in the appendix

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? in the appendix

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: in the appendix

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: in the appendix

heading: Future Work
text: This suggests that a possible solution for the city-split setup might be dataaugmentation via generating grounded descriptions for the tested region – an approach we reserve for future research