Into the Unknown: Generating Geospatial Descriptions for New Environments (2024)

Abstract: Similar to vision-and-language navigation (VLN) tasks that focus on bridging the gap between vision and language for embodied navigation, the new Rendezvous (RVS) task requires reasoning over allocentric spatial relationships (independent of the observer’s viewpoint) using non-sequential navigation instructions and maps. However, performance substantially drops in new environments with no training data. Using opensource descriptions paired with coordinates (e.g., Wikipedia) provides training data but suffers from limited spatially-oriented text resulting in low geolocation resolution. We propose a large-scale augmentation method for generating high-quality synthetic data for new environments using readily available geospatial data. Our method constructs a grounded knowledge-graph, capturing entity relationships. Sampled entities and relations (“shop north of school”) generate navigation instructions via (i) generating numerous templates using context-free grammar (CFG) to embed specific entities and relations; (ii) feeding the entities and relation into a large language model (LLM) for instruction generation. A comprehensive evaluation on RVS, showed that our approach improves the 100-meter accuracy by 45.83% on unseen environments. Furthermore, we demonstrate that models trained with CFG-based augmentation achieve superior performance compared with those trained with LLM-based augmentation, both in unseen and seen environments. These findings suggest that the potential advantages of explicitly structuring spatial information for text-based geospatial reasoning in previously unknown, can unlock data-scarce scenarios.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 2259–2273 August 11-16, 2024 ©2024 Association for Computational Linguistics
Similar to vision-and-language navigation (VLN) tasks that focus on bridging the gap between vision and language for embodied navigation, the new Rendezvous (RVS) task requires reasoning over allocentric spatial relationships (independent of the observer’s viewpoint) using non-sequential navigation instructions and maps. However, performance substantially drops in new environments with no training data. Using opensource descriptions paired with coordinates (e.g., Wikipedia) provides training data but suffers from limited spatially-oriented text resulting in low geolocation resolution. We propose a large-scale augmentation method for generating high-quality synthetic data for new environments using readily available geospatial data. Our method constructs a grounded knowledge-graph, capturing entity relationships. Sampled entities and relations (“shop north of school”) generate navigation instructions via (i) generating numerous templates using context-free grammar (CFG) to embed specific entities and relations; (ii) feeding the entities and relation into a large language model (LLM) for instruction generation. A comprehensive evaluation on RVS, showed that our approach improves the 100-meter accuracy by 45.83% on unseen environments. Furthermore, we demonstrate that models trained with CFG-based augmentation achieve superior performance compared with those trained with LLM-based augmentation, both in unseen and seen environments. These findings suggest that the potential advantages of explicitly structuring spatial information for text-based geospatial reasoning in previously unknown, can unlock data-scarce scenarios.

heading: 1 Introduction
text: The ability to extract locations and paths from natural language descriptions of spatial information holds immense significance. This capability proves
crucial in daily and disaster response scenarios, aiding the billions globally lacking formal addresses (UPU, 2012; Abebrese, 2019; Hu et al., 2023), and enhancing Geographic Information Retrieval (GIR), particularly leveraging web-based resources (Spink et al., 2002; Sanderson and Kohler, 2004).
Echoing the vision-and-language navigation (VLN) task’s goal, of bridging the gap between visual perception and natural language (NL) instructions for embodied agents (Ku et al., 2020), the recently introduced Rendezvous (RVS) navigation task (Paz-Argaman et al., 2024) seeks to achieve a similar connection, but specifically be-
2259
tween map representations and natural language NL. While VLN focuses on navigating within an environment based on visual cues and sequential instructions, RVS emphasizes the ability to utilize map information and non-sequential, often allocentric language descriptions to reach a specific target location. This shift from vision-centric instructions to map-aided guidance presents unique challenges, including reasoning about multiple spatial relationships simultaneously, inferring implicit actions from language, and navigating without explicit verification or step-by-step instructions.
However, there is a substantial gap between current models and the human performance on the RVS task, particularly in new environments that lack human annotated data. One approach to address this issue is to leverage naturally-occurring open-source data such as Wikipedia. However, these sources lack direct spatial information, which can result in models’ low performance (Solaz and Shalumov, 2023). Synthesizing data using large language models (LLMs) is a common method for addressing data scarcity in NLP (Yoo et al., 2021; Edwards et al., 2021). However, for multimodal scenarios requiring precise spatial relationships, accurately generating such data without introducing errors or “hallucinations” (i.e., spurious relationships or entities), which severely undermines the performance on the underlying downstream task, remains a significant challenge.
We propose a method for generating high-quality synthetic data for new environments using opensource geospatial data (Figure 1). Our method constructs a grounded knowledge-graph of the environment, capturing spatial relationships between entities. By sampling these entities, abstract shapes like ‘blocks’ (implicitly derived from street relationships), and relations (e.g., ‘the garden is next to a restaurant’), we generate navigation instructions by either (i) creating a large amount of templates via a generative context-free grammar (CFG), in which we embed the precise entities and relations sampled, or (ii) feeding the entities and relation into an LLM which generates the instructions.
Extensive evaluation on the RVS dataset demonstrates the clear advantage of our CFG-based method compared to the LLM approach. When navigating unseen environments, our method achieves a remarkable 9.1% absolute increase in 100-meter accuracy and a substantial 39-meter decrease in median distance error. Overall, our method helps
close the human-AI performance gap in unseen environments by 45.83% in 100m accuracy and a decrease of 1,183m in median distance error. For the seen environment, our method results in an absolute improvement of 19.56% in 100m accuracy and a decrease of 151m in median distance error.

heading: 2 The Task
text: The Rendezvous (RVS, Paz-Argaman et al., 2024) task evaluates a system’s ability to follow humangenerated, colloquial language navigation instructions within a dense urban environment depicted by a map. The system is provided with three inputs: (i) Detailed Map as Knowledge Graph: a comprehensive map of the environment represented as a knowledge graph. This graph encodes spatial relationships between landmarks and other relevant features. (ii) Explicit Starting Point (Geo-coordinates): the starting location specified as a latitude and longitude coordinate pair. (iii) Navigation Instruction: a natural language instruction describing the target location’s relative position to landmarks and the starting point. This instruction leverages colloquial language typically used in navigation scenarios. The RVS task demands the system to process these inputs and generate the goal location’s coordinates within the defined map boundaries.

heading: 3 Proposed: Relational Augmentation
text: Our augmentation method aims to generate natural language location descriptions that are both accurate and well grounded. To that end, we leverage OpenStreetMap (OSM).1 It involves three stages: (i) sampling paths; (ii) calculating spatial relations between entities; and (iii) generating instructions based on the spatial relations calculated in stage (ii). We use the OSM-based graph provided in RVS for the first and second parts and provide two methods for generating instructions based on the spatial relations between entities calculated in that part.

heading: 3.1 Sampling paths
text: To generate RVS-like samples accurately, we follow the RVS sampling protocol. We randomly sample small entities for the end point (the entity’s shape has a maximum radius of 100 meters). For the start point, we randomly select an entity that (i) is within 200–2000 meters of the end point; (ii) has a name or type tag (e.g., a bookshop). This
1OpenStreetMap is a user-updated map of the world – http://www.openstreetmap.org
information allows us to refer to the entity in the instruction by its definite description and not by its proper name. Finally, we pick a route that is the shortest distance between the start and end points.

heading: 3.2 Calculating Spatial Relations between Entities
text: Leveraging the Open Street Map (OSM) graph, we identify prominent landmarks relevant to the sampled paths for inclusion in navigation instructions. However, not all paths necessarily contain all types of landmarks and features. For instance, if the navigation ends in a dead-end street, landmarks beyond the leading path might not be relevant to the task. This section details the criteria for landmark selection and the spatial relations computed with respect to the start and end points, and the path.
Picking landmarks We pick three different types of landmarks with a certain spatial relation that will be referenced in the instruction: (i) landmarks close to the end point (within 100 meters from it); (ii) landmarks along the route; and (iii) landmarks that are on the same street as the goal location but beyond the route, such that if the agent keeps on walking beyond the goal, it will reach that landmark (“beyond landmark”). Priority for landmark selection is given to those with the highest level of external recognition, as determined by the following hierarchy: has a Wikipedia or Wikidata2 link, is a brand, is a tourism attraction, is an amenity, is a shop. We randomly select all landmarks from the most prominent level found. If in the area there are multiple landmarks of the same type, we group the landmarks according to their type and quantity (e.g., ‘two book shops’). If a landmark is far (over 200 meters) from the end point and it has a proper name, we can use its proper name in the text generation, e.g., ‘the Empire State Building’. If it is near (less than 200 meters) the end point, we will always use the indefinite name
2Wikidata is Wikipedia’s free, open, and interconnected knowledge base. https://www.wikidata.org/
of the landmark, e.g., ‘a bookshop’.
Calculating spatial relations The objective is to determine the spatial relationship between landmarks and the end point. There are several ways to describe the relationship between two entities, such as the number of blocks between them. The spatial relations calculated are (i) allocentric relations, i.e., cardinal directions, between landmarks, start and end points. Cardinal direction is calculated by the bearing θdegrees between two points (longitude, latitude), (x1, y1) and (x2, y2): θradians = (tan−1 (sin(λ2 − λ1) cos(ϕ2),
cos(ϕ1) sin(ϕ2)) − sin(ϕ1) cos(ϕ2) cos(λ2 − λ1)) +360◦) mod 360◦)
θdegrees = θradians ⋅ 180
◦
π (1)
Where λ1 = x1⋅π180◦ , λ2 = x2⋅π 180◦ , ϕ1 = y1⋅π180◦ and ϕ2 = y2⋅π180◦ . Bearings θdegrees fall into different ranges, each with a corresponding cardinal direction (e.g., ‘North-West’). (ii) Egocentric relations between landmarks and end point to the path, e.g., ‘on the right side’. To calculate egocentric relations, we rely on two key angles (Eq. 1): the bearing of the path itself θpdegrees, and the bearing of the shortest imaginary line connecting the path to the landmark’s point θldegrees:
∆θ l−p = (θldegrees − θpdegrees) mod 360◦ (2)
{‘RIGHT’, if ∆θl−p < 180◦ ‘LEFT’, otherwise
‘LEFT’ and ‘RIGHT’ indicate the landmark’s position relative to the path. (iii) The number of blocks and intersections the agent must pass through to reach the end point. (iv) the end point’s egocentric and allocentric position on the block, e.g., ‘middle of the block’ and ‘north-east corner of the block’.
Allocentric position on the block involves determining the bearing (Eq. 1) of the path along the block and mapping it to cardinal direction as in (i).

heading: 3.3 Data Generation
text: Based on the sampling and spatial relations’ calculations we use two methods to generate the instruction: via templates created with a CFG (Chomsky, 1956), and via prompting an LLM — for enlarging the vocabulary and the style of the text.
CFG-based Method Here the key idea is using a CFG to generate templates that can then be adapted according to the sampled data. The CFG we design requires defining terminal symbols (lexical elements), nonterminal symbols, and production rules. The nonterminals contain the main parts of what a path description contains, such as descriptions around the goal, along the path, what to avoid, and so on. The terminals contain optional variations, for example, the verb for the agent to proceed can be ‘go’, ‘walk’, and so on. The grammar creates templates that are processed into instructions, as demonstrated in Figure 2. For a given sampled path, we randomly pick a template that contains all the landmark categories and spatial relations calculated for the path. We then generate the instruction by replacing the variables in the chosen template with the corresponding landmarks and spatial relations (e.g., ‘NUMBER_INTERSECTIONS’ will be replaced with the actual number of intersections the agent should walk).
Prompting LLMs Using the aforementioned template-based instructions, we prompted an LLM to ‘rephrase the subsequent navigation instruction, ensuring it explains how to travel from the starting position to the destination: Navigation Instruction’, where the Navigation Instruction is an instruction based on the CFG generation process. For example, based on the example in Figures 1 and 2 we get the following sentence: ‘Head northwest from St. Vincent de Paul Church for 2 intersections. The garden is next to a fast-food restaurant.

heading: 4 Experimental Setup

heading: 4.1 Evaluation
text: We follow the RVS evaluation metrics: (i) 100m accuracy; (ii) 250m accuracy for coarse-grained evaluation; (iii) mean absolute error distance (MAE); (iv) median absolute error (Med.AE); (v) maximum absolute error (Max.AE); and (vi) area under the
curve (AUC) of the error distance. Here are the formulas for evaluating set S with metrics (iii-vi):
MAE(S) = 1∣S∣ ∑s∈S dist(loc(s), approx(s)) (3) Med.AE(S) = {dist(loc(s), approx(s))∣s ∈ S}⌊∣S∣/2⌋ (4) Max.AE(S) = max({dist(loc(s), approx(s))∣s ∈ S}) (5) AUC(S) = ∫ ∞ 0 (log dist(loc(s),approx(s))+ ∣s∈S)↑ds
logHmax⋅(∣S∣−1) (6) Where = 1e − 5 and Hmax = 20, 037 ⋅ 103,
approximately the maximum haversine distance.

heading: 4.2 Models for evaluation
text: T5-model We test our augmentation method with T5 model, a transformer-based encoderdecoder model designed with a text-to-text format (Raffel et al., 2020). Both encoder and decoder utilize multi-head, multi-layer self-attention mechanisms (Vaswani et al., 2017). Given input sequence text X = (x1, ..., xN) and a starting point ps, the encoder encodes the instruction and the starting point’s representation such that E l = (el1, ..., elN , elps) where l ∈ L, representing the L hierarchical encoded layers. The output of the final encoder layer is a sequence of hidden vectors H = (h1, ..., hN , hps). The decoder generates output tokens sequentially, predicting the probability p(pt∣p1∶t−1, H) = softmax(Wo⊗ h′t) of token pt at step t, based on the previous outputs and hidden state h′t. Importantly, the model is trained with a pre-defined high-level path P that guides the generation process. This path starts at the starting point, traverses through prominent landmarks ordered by their direction relative to the goal, and eventually reaches the goal itself.
Non-learning LANDMARK Baseline Predicts the location of a prominent landmark (defined in Sec. 3.2) in the map within a radius of 1 kilometer.

heading: 4.3 Data
text: RVS The RVS (Paz-Argaman et al., 2024) dataset serves as a human-level benchmark for the purpose of evaluating the ability to follow
allocentric navigation instructions based on a map. It consists of English navigation directives, each paired with a start and end point. The data is divided into four distinct sets: (i) Training-set – containing 7,000 instructions from Manhattan; (ii) Seen-city development-set – containing 1,103 instructions from Manhattan; (iii) Unseen-city development-set – containing 1,023 instructions from Pittsburgh; (iv) Test-set – containing 1,278 instructions from Philadelphia.
The following datasets are all synthetically generated. We generated 200,000 instructions per dataset for each region, with the exception of AugWikiGeo:
Aug-CFG Data created with the CFG method described in Section 3. The CFG method created 194,721 templates with 15 production rules. The vocabulary contains only unique 111 tokens. Table 1 shows that this method produces shorter instructions than the RVS, but with more entities in each instruction. Figures 1 and 2 show an example of an instruction generated based on this method.
Aug-CFG-Allocentric Data created with a CFG including templates with allocentric spatial relations between entities. E.g., ‘the school is north of the bar’. The data contains 69,720 templates.
Aug-CFG-Egocentric Data created using a CFG which contains templates with egocentric spatial relations between entities, e.g., ‘the bar on your right’. The data contains 112,640 templates.
Aug-CFG-Minimal This dataset leverages a minimal set of templates (64) to represent the full range of entities and spatial relationships found in the Aug-CFG data. We constructed Aug-CFGMinimal using a greedy selection process. We began by selecting the first template. Subsequently, we added templates that introduced new spatial features not covered by the existing set. This process continued until we reached a final set of 64 templates that comprehensively capture all possible spatial features.
Aug-Prompt Data generated by prompting PaLM2 (Anil et al., 2023) with the prompting method described in Section 3.3 Table 1 shows that this method produces longer instructions than the CFG-based method but with fewer entities.
Aug-WikiGeo To facilitate a comparison with standard open-source location augmentation methods, such as Wikipedia-based (Krause and Cohen, 2023), we generated the comprehensive Wikigeo dataset by consolidating data from Wikipedia (pages and backlinks), Wikidata (pages), and OpenStreetMap (entities). The data for Manhattan (Manh., 255,663 samples), Pittsburgh (Pitt., 27,401 samples), and Philadelphia (Phila., 52,367 samples) are generated based on the same regions as in the evaluation. Table 1 shows that this method produces shorter instructions and fewer entities than the CFG-based and prompting-based methods.
3We use PaLM2 ‘models/text-bison-001’ - for more details, see https://developers.generativeai.google
Aug-Dummy To assess the contribution of the text augmentation, as opposed to just learning all possible paths from a known starting point, we use an augmentation where the text does not convey any spatial details of the location. To create the non-spatial text, we used PaLM2 by requesting it to rephrase the sentence “Meet me here” and got a total of 31 versions. Path sampling followed the method described in Section 3.1.

heading: 5 Results

heading: 5.1 Analysis of Quantitative Results
text: Table 2 shows the results of our experiments over RVS’s seen-city development set (Manh.), unseencity development set (Pitt), and the unseen-city test set (Phila.). For the seen environment (Manh.), training on synthetic data only (line 5) outperforms human-annotated data (line 3). The gap is small, but if the model is first trained on synthetic data and then on human data (line 7), the gap is a 65% ratio in 100m accuracy and a 110m lower in Med.AE.
Training on real human data from other regions (lines 10, 17) fails to translate to unseen environments like Pittsburgh and Philadelphia. However, injecting region-specific synthetic data (line 12) dramatically boosts performance: 46.14% higher 100m accuracy and 987m lower Med.AE. WikiGeo’s emphasis on local features sacrifices spatial relational understanding, leading to lower performance on all development sets (lines 4, 11). However, its superior max distance estimation (up to 2km) and lower distance error over models trained on human-annotated data from a different region (lines 11 vs. 10), suggest a strong ability to learn path boundaries based on localized information.
Aug-Prompt, while generating instructions with richer language and stylistic diversity compared to Aug-CFG’s template-based approach, exhibits a marked decrease in performance (lines 6, 13 vs. lines 5, 12). Sampling 20 instructions from Aug-
Prompt, we found that in two cases, the type of goal was emitted, and in five cases, the model ‘hallucinated’ – adding incorrect spatial relations. This indicates a potential trade-off between linguistic complexity and the fidelity of spatial information within generated instructions.
Table 3 examines variations of CFG-based augmentation. Consistent with the emphasis on allocentric spatial relations in RVS, training on AugCFG-Allocentric (lines 2, 7) surpasses Aug-CFGEgocentric (lines 3, 8) in both development sets. However, results are mixed regarding whether training on Aug-CFG-Allocentric is better than training on Aug-CFG. In the seen environment, Aug-CFGAllocentric (line 2) outperforms Aug-CFG (line 1) in accuracy but underperforms in error distance. In the unseen-environment (Pittsburgh), the opposite is true – Aug-CFG-Allocentric (line 7) underperforms Aug-CFG (line 6) in accuracy and overperforms error distance. This inconclusive evidence suggests potential value in investigating a single data approach for tasks with varying demands, such as RUN’s reliance on egocentric relations.
Despite capturing identical spatial relations, Aug-CFG-Minimal’s limited stylistic variations and vocabulary size due to fewer templates hinder its performance compared to Aug-CFG across both seen (line 1 vs. line 4) and unseen (line 6 vs. line 9) sets. This suggests that the mere presence of accurate spatial relations may not be sufficient for optimal model learning, potentially due to insufficient exposure to diverse linguistic contexts and syntactic structures.
Training on Aug-Dummy teaches the model only the optional paths from a starting point, as the instruction is a dummy one. The poor results achieved over Aug-Dummy in both seen and unseen environments (lines 5, 10), prove that training on Aug-CFG allows the model to learn spatial relations from the instructions. The Aug-Dummy for
Pittsburgh results are better as Manhattan is much denser in entities than Pittsburgh, thus, it contains more optional paths from the starting point.

heading: 5.2 Data Quantity Impact
text: Figure 3 shows four graphs of T5 performance trained on different amounts of AUG-CFG data. The results reveal a quality-quantity trade-off influencing T5’s performance. In seen cities (a, b), 7,000 high-quality human annotations (green lines) outperform 7,000 synthetic AUG-CFG samples (red lines) in 100m accuracy (27.92% vs. 8.93%) and Med.AE (231m vs. 744m), indicating that the human-annotated RVS data possesses substantially higher quality than the synthetic data. The steep curve of the RVS train-set (green lines) compared to the mild curve of AUG-CFG (red lines) further reinforces this conclusion. However, this trend flips with 200K synthetic samples, showcasing the power of quantity over quality when data is abundant. This suggests ample data helps the model grasp the environment and spatial relations. Additionally, while both RVS train-set (green line) and AUG-CFG (red line) demonstrate strong performance, the combined AUG-CFG + RVS train-set (blue line) exhibits a sustained upward trend, consistently surpassing the green and red lines in both accuracy and distance error. This further indicates that augmenting with a large amount of data can
potentially enhance performance beyond even highquality human annotations. This approach offers a promising solution for the data scarcity challenges often encountered in NLP geospatial tasks.
In the unseen-city split (c, d), the RVS train-set (green line) exhibits steady improvements in accuracy and error distance despite originating from a different region. This demonstrates that highquality data, even when geographically distinct, can benefit the model. However, the significant performance gap between the RVS train-set (green) and AUG-CFG (red) lines, even with equal data quantities, reaffirms the importance of regionallyspecific data. While the RVS train-set (green) slope nears AUG-CFG (red line), its accuracy improvement remains significantly lagged (moderate slope), suggesting the model primarily learns general directional understanding rather than fine-grained spatial reasoning. Furthermore, the combined AUGCFG + RVS train-set (blue) offers only marginal gains over AUG-CFG alone (red), indicating that the benefits of high-quality non-regional data diminish when paired with large-scale augmentation.

heading: 5.3 Distribution Analysis
text: Figure 4 reports the performance of the various augmentation methods through cumulative distribution functions (CDFs). These CDFs show the percentage of inferences with error distances below
a specific meter value (x-axis), effectively capturing accuracy across all error values and exposing underlying error distributions. Notably, CFG-based methods (AUG-CFG and its variants) exhibit a remarkably similar distribution, characterized by a sharp accuracy ascent up to approximately 200m. Aug-Dummy’s curve (yellow) resembles a ray emanating from the origin, suggesting a strong correlation with the path distribution assimilated during training. Aug-WikiGeo’s low-resolution nature is evident in its CDF (pink). While it starts slow (2̃00m) and climbs faster than the yellow line, its limited accuracy holds it back. It only catches up to the CFG-based methods at 1̃500m, but then fizzles out (plateaus) before they even reach their peak.

heading: 6 Background and Related Work
text: Text-based navigation tasks constitute a multimodal challenge (Antol et al., 2015; Paz-Argaman et al., 2020; Ji et al., 2022) demanding the integration of language comprehension and environmental knowledge. This environment can be either indoor, commonly represented through the agent’s visual perception as it navigates (Anderson et al., 2018), or outdoor. For outdoor environments, representation falls into three main categories: (i) Visual Representation — similar to indoor settings, real-world imagery can be employed, with agents learning the environment through street-view like exploration (Anderson et al., 2018); (ii) Map-based Representation — agents navigate based on map perception, as seen in studies utilizing maps as the primary input (Anderson et al., 1991; Paz-Argaman et al., 2024); and (iii) Hybrid Representation — a combination of visual and map information (Vasudevan et al., 2020; de Vries et al., 2018).
Realistically, we would like to learn to navigate based on text in new environments, ones that our
models did not train on before. Furthermore, environments constantly change, such that we need to reacquaint our models with these changes in order for them to stay relevant (Zhang and Choi, 2021).
Synthetic data generation, facilitated by LLMs, has become a prominent approach in NLP to mitigate data (Sahu et al., 2022; Stylianou et al., 2023). This technique is particularly attractive due to its adaptability to various tasks and its ability to generate substantial volumes of data, allowing for robust model training and improved performance. Data generation has also been applied to multimodal tasks, demonstrating performance close to that of human-annotated data (Bitton et al., 2021).
Previous works on vision and language navigation tasks (VLN) tried to tackle the data scarcity issue by generating synthetic data with a generative model trained with human annotated data (Fried et al., 2018; Zhu et al., 2020; Majumdar et al., 2020). However, the learned distribution was limited to the environment the model was trained on. ENVEDIT (Li et al., 2022b) tackled the VLN task by generating new environments and synthetic navigation instructions in order to teach models to generalize to new environments. The new environments created differ in style, appearance, or the configuration of the objects. However, a lack of new objects hinders unseen object handling. Kamath et al. (2023) leveraged image-to-image synthesis via a Generative Adversarial Network (GAN) architecture (Koh et al., 2023) to create new viewpoints for existing environments and subsequently generate synthetic instructions for these environments.
Several prior works have tackled unseen environments without employing data generation: (i) Entity abstraction – Paz-Argaman and Tsarfaty (2019) propose learning spatial language independent of specific entities by linking abstracted entities to the agent’s perception. However, this approach is limited to simple navigation tasks where the agent has limited perception (i.e., line of sight). It struggles with complex tasks like RVS, which require allocentric spatial reasoning. (ii) cross-lingual augmentation – Li et al. (2022a) leverage spatial data across different languages for augmentation. However, it still requires some environment-specific annotations, even if not in the target language. Additionally, geolocation tasks utilizing open-source datasets like Wikipedia often rely on grid-based representations, suffering from spatially unoriented descriptions. This leads to significant retrieval er-
rors exceeding tens of kilometers, limiting their accuracy (Wing and Baldridge, 2011). (iii) Graphbased representation – representing the environment via a graph and learning the connection between environment and language (Paz-Argaman et al., 2023, 2024). This approach lacks promising results due to challenges in encoding complex spatial relationships within a graph format. Schumann and Riezler (2021) trained a neural network to generate synthetic navigation instructions based on OpenStreetMap representations. However, these instructions were limited to step-by-step, local lineof-sight guidance. Furthermore, the model’s performance was evaluated on unseen paths within the same city it was trained on.
This work tackles the problem of generating synthetic instructions for unseen environments, exemplified by the RVS dataset (Paz-Argaman et al., 2024). We propose two techniques: an LLMbased approach for its adaptability, and a CFG rulebased approach for increased precision, highlighting the trade-off between data-driven efficiency and human-crafted accuracy.

heading: 7 Conclusion
text: This work presents a novel data-augmentation solution for spatial NLP tasks. Leveraging spatial relation extraction it generates a vast, albeit slightly less refined, dataset compared to human annotations. This quantitative advantage unlocks superior performance, as evidenced by a 44.54% absolute improvement in 100m accuracy and a 1,170m reduction in median absolute error distance on unseen environments in the RVS dataset. These results demonstrate the effectiveness of our approach in handling novel environments with sparse or no human data. Moreover, our CFG-based augmentation method offers adherence to correct spatial relations, control over the content, and interpretability, over LLMs in spatial descriptions and could serve as a future tool for evaluation, detection, and mitigation of artifacts such as ‘hallucinations’.
Limitations
Data Dependence This paper presents a promising data augmentation method for enhancing NLP tasks like outdoor geolocation and navigation. However, its effectiveness hinges on accessing comprehensive geospatial data, which can be difficult to find in indoor (Anderson et al., 2018; Jain et al., 2019; Nguyen et al., 2019; Thomason et al., 2020;
Qi et al., 2020; Ku et al., 2020) and virtual environments (MacMahon et al., 2006; Yan et al., 2018; Misra et al., 2018; Shridhar et al., 2020; Kim et al., 2020). Furthermore, the reliance on open-source data introduces the risk of incompleteness or regional unavailability.
Rule-based Scalability vs. LLM-Generated Artifacts In addition, the leading augmentation method (CFG-based) used in this study is rulebased, which, while offering control, preciseness, and interpretability, also requires careful rule design. Crafting effective rules can be timeconsuming, laborious, and require substantial expertise in the specific domain. This complexity can limit the scalability and adaptability of the method to new situations or contexts. Furthermore, encoding all necessary knowledge into explicit rules can be challenging, especially for complex domains. This can limit the method’s ability to capture subtle nuances or unforeseen situations. This approach also stands in contrast to the prevailing augmentation methods, which are grounded in LLMs (Schick and Schütze, 2021; Wang et al., 2022). Using LLMs to generate data for augmentation represents a promising avenue but requires further research to address issues like hallucinations, which are particularly critical in spatial descriptions. Therefore, rule-based methods such as our CFG-based method, which are not vulnerable to hallucination problems, could serve as a valuable tool for future research exploring the evaluation, detection, and mitigation of such artifacts in LLM-generated text. Furthermore, by comparing augmentations generated using both methods on specific tasks, we might gain insights into the types of artifacts introduced by LLMs and how rule-based methods can help mitigate them.
Limited Perception: Lack of Visual Cues Our study primarily relied on map-based knowledge for navigation tasks, which deviates from how humans navigate in natural settings. Real-world navigation often involves integrating both visual cues and spatial knowledge acquired from maps, a complexity not fully captured in our current approach. While the StreetLearn dataset (Mirowski et al., 2019) offers Google Street View imagery for the test environments over both the Manhattan and Pittsburgh regions in the RVS setup, we did not leverage this visual information. Future research could extend the scope of our study by integrating visual perception and map-based knowledge into the text
generation approach for augmentation.
Bridging the Human-AI Performance Gap In spite of progress made in this research, a substantial gulf still separates current models’ performance from human performance in the RVS task. Even with our augmentation methods, current models lag behind human performance by 47.55% and 42.15% in 100-meter accuracy for seen and unseen environments, respectively. Bridging this gap presents a critical challenge and an exciting opportunity for future research, potentially unlocking novel avenues for pushing the boundaries of this task.

heading: Acknowledgements
text: This research has been funded by the European Research Council (ERC), grant number 677352 and by a grant from the Israeli Science Foundation (ISF) number 670/23, for which we are grateful. The research was further supported by a KAMIN grant from the Israeli Innovation Authority, and computing resources kindly funded by a VATAT grant and via the Data Science Institute from BarIlan University (BIU-DSI). We are also grateful for the additional support provided by a Google grant.

heading: A Data Generation
text: Our CFG rules define navigation instructions by combining five key elements in various orders:
• Goal Description: This specifies the target location.
• Main Path: It outlines the primary route using landmarks.
• Approaching Goal: This details how to get close to the target.
• Goal Landmarks: It describes landmarks relative to the final position.
• Off-Path Awareness: This identifies elements to avoid while navigating.
Table 4 demonstrates five navigation instructions and their corresponding CFG-templates.

heading: B Models
text: We also test on the T5+GRAPH presented in RVS, which is a T5 model incorporating an OSM-based graph representation of the environment.
B.1 The Graph Representation
A location can be represented by its position (where the location is) or by its semantics (what is present at the location, e.g., ‘a bar’). Semantic knowledge is crucial for grounding mentioned entities to their physical references in the environment. To this end, we aim to represent the semantics via the RVS map-graph. We use the RVS map-graph and connect each node to its corresponding S2-cells. As the S2-geometry is a hierarchical structure, we allow for multiple levels of S2-cells connections. Also there are edges between neighboring S2-cells at a given level (see bottom part in Figure 5). To learn an embedding for each S2-cell in the environment, we compute random walks on the graph using node2vec algorithm (Grover and Leskovec, 2016). Following Yu et al. (2021), we use linear projection to cluster the graph embeddings into K categories using the k-means algorithm with cosine similarity distance. A new token is assigned to each category and added to the tokenizer’s vocabulary. We perform multiple clusters and pass the graph’s tokens with the instruction’s tokens to the transformer encoder.
B.2 Experimental Setup Details
The Graph Embedding The graph was constructed using three levels of S2-Cells: 15, 16, and 17. At level 16, each sub-graph consisting of four neighboring S2-Cells was fully connected. All S2-Cells in the graph were linked to their parent
S2-Cell based on the S2-geometry’s hierarchy (i.e., level 17 S2-Cells were connected to level 16 S2Cells and level 16 S2-Cells were connected to level 15 S2-Cells). Extracted entities from OSM and Wikidata were linked to the smallest level 17 S2Cell that encompassed their geometry. The node of the entity included additional data such as their geometry, type and name of entity. Random walks on the graph were performed using node2vec (Grover and Leskovec, 2016).
For both T5-base models we use a pre-trained ‘T5-Base’ model from Hugging Face Hub, which is licensed under the Apache License 2.0. The T5 model was trained on the Colossal Clean Crawled Corpus (C4, Raffel et al. (2020)). The crossentropy loss function was optimized with AdamW optimizer (Loshchilov and Hutter, 2017). The hyperparameter tuning is based on the average results run with three different seeds. We used a learning rate of 1e-4. The S2-cell level was searched in [15, 16, 17, 18] and 16 was chosen. The number of clusters for the quantization process was searched in [50, 100, 150, 200, 250] and 150 was chosen. We used 2 quantization layers. Number of epochs for early stopping was based on their average learning curve. We used the following parameters for the node2vec algorithm: an embedding size of 1024, a walk length of 20, 200 walks, a context window size of 10, a word batch of 4, and 5 epochs.

heading: C Additional Results
text: Table 5 demonstrates the performance of T5+GRAPH with CFG augmentation. The T5+GRAPH model’s performance is improved by the CFG augmentation in both seen and unseen environments (lines 2 vs 4, and lines 8 vs 10). In the seen environment, T5+GRAPH with CFG augmentation achieves higher scores than T5 with CFG augmentation (lines 3 vs 4). However, there is no clear evidence to suggest which model performs better when using CFG augmentation data in both seen and unseen environments. This finding suggests that the explicit spatial knowledge offered by CFG augmentation data provides an easier path to learn spatial relations, making the graph information in T5+GRAPH redundant or even detrimental.

heading: Future Work
text: Moreover, our CFG-based augmentation method offers adherence to correct spatial relations, control over the content, and interpretability, over LLMs in spatial descriptions and could serve as a future tool for evaluation, detection, and mitigation of artifacts such as ‘hallucinations’. Limitations
Data Dependence This paper presents a promising data augmentation method for enhancing NLP tasks like outdoor geolocation and navigation. However, its effectiveness hinges on accessing comprehensive geospatial data, which can be difficult to find in indoor (Anderson et al. , 2018; Jain et al. , 2019; Nguyen et al. , 2019; Thomason et al. , 2020;
Qi et al. , 2020; Ku et al. , 2020) and virtual environments (MacMahon et al. , 2006; Yan et al. , 2018; Misra et al. , 2018; Shridhar et al. , 2020; Kim et al. , 2020). Furthermore, the reliance on open-source data introduces the risk of incompleteness or regional unavailability. Rule-based Scalability vs. LLM-Generated Artifacts In addition, the leading augmentation method (CFG-based) used in this study is rulebased, which, while offering control, preciseness, and interpretability, also requires careful rule design. Crafting effective rules can be timeconsuming, laborious, and require substantial expertise in the specific domain. This complexity can limit the scalability and adaptability of the method to new situations or contexts. Furthermore, encoding all necessary knowledge into explicit rules can be challenging, especially for complex domains. This can limit the method’s ability to capture subtle nuances or unforeseen situations. This approach also stands in contrast to the prevailing augmentation methods, which are grounded in LLMs (Schick and Schütze, 2021; Wang et al. , 2022). Using LLMs to generate data for augmentation represents a promising avenue but requires further research to address issues like hallucinations, which are particularly critical in spatial descriptions. Therefore, rule-based methods such as our CFG-based method, which are not vulnerable to hallucination problems, could serve as a valuable tool for future research exploring the evaluation, detection, and mitigation of such artifacts in LLM-generated text. Furthermore, by comparing augmentations generated using both methods on specific tasks, we might gain insights into the types of artifacts introduced by LLMs and how rule-based methods can help mitigate them. Limited Perception: Lack of Visual Cues Our study primarily relied on map-based knowledge for navigation tasks, which deviates from how humans navigate in natural settings. Real-world navigation often involves integrating both visual cues and spatial knowledge acquired from maps, a complexity not fully captured in our current approach. While the StreetLearn dataset (Mirowski et al. , 2019) offers Google Street View imagery for the test environments over both the Manhattan and Pittsburgh regions in the RVS setup, we did not leverage this visual information. Future research could extend the scope of our study by integrating visual perception and map-based knowledge into the text
generation approach for augmentation. Bridging the Human-AI Performance Gap In spite of progress made in this research, a substantial gulf still separates current models’ performance from human performance in the RVS task. Even with our augmentation methods, current models lag behind human performance by 47. 55% and 42. 15% in 100-meter accuracy for seen and unseen environments, respectively. Bridging this gap presents a critical challenge and an exciting opportunity for future research, potentially unlocking novel avenues for pushing the boundaries of this task