FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks

Authors: Tanawan Premsri, Parisa Kordjamshidi
Primary area: datasets and benchmarks
Keywords: Spatial language, Evaluation benchmark, Frame of reference

Abstract:
Spatial cognition is one fundamental aspect of human intelligence. A key factor in spatial cognition is understanding the frame of reference (FoR) that identifies the perspective of spatial relations. 
However, the AI research has paid very little attention to this concept.
Specifically, there is a lack of dedicated benchmarks and in-depth experiments analyzing large language models' (LLMs) understanding of FoR.
To address this issue, we introduce a new benchmark, **F**rame **o**f **R**eference **E**valuation in **S**patial Reasoning **T**asks (FoREST)  to evaluate LLMs ability in understanding FoR.
We evaluate the LLMs in identifying the FoR based on textual context and employ this concept in text-to-image generation. 
Our results reveal notable differences and biases in the FoR identification of various LLMs. 
Moreover, the bias in FoR interpretations impacts the LLMs' ability to generate layouts for text-to-image generation. 
To improve spatial comprehension of LLMs, we propose Spatial-Guided (SG) prompting, which guides the model in exploiting the types of spatial relations for a more accurate FoR identification. 
The SG prompting improves the overall performance of FoR identification by alleviating their bias towards specific frames of reference.
Eventually, incorporating the FoR information generated by SG prompting in text-to-image leads to a more accurate visualization of the spatial configuration of objects.

Decision: reject
Reviewer scores: 5, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730103818541

thread_type:
official_review

<thread_object>
creator: reviewer_bebn

<content>
summary:
Research on spatial perception capabilities in large language models is a key direction for optimizing their generation abilities. This paper introduces a benchmark for understanding frames of reference (FoR) and evaluates different LLMs to test the spatial perception capabilities. Additionally, it uses diffusion models to conduct experimental visualizations that simulate this understanding. The benchmark provides guidance on designing prompts that enhance spatial reasoning, contributing to improved text-to-image accuracy.

strengths:
This paper focuses on a core shortcoming in large language models' text comprehension-spatial understanding, presenting a benchmark for evaluating spatial perception abilities. It summarizes potential situations in spatial perception using existing evaluation metrics. Downstream tasks incorporate text-to-image generation experiments with diffusion models to visualize the difference in spatial understanding capabilities of different LLMs. The paper is well-structured, with straightforward explanations of the methods, and employs vivid examples to enhance understanding.

weaknesses:
However, this paper lacks logical coherence in its descriptions of various cases. The selection of content for the four FoR classes seems random, raising the question of whether there could be a clearer division for categorizing different spatial reasoning tasks. It remains unclear if the four cases can comprehensively cover all possible spatial reasoning tasks, and more citations are needed to support your claim. The presentation of experimental results is also insufficient; for instance, the two images in Figure 3 fail to follow the principle of controlling variables, rendering the conclusions unconvincing. Furthermore, the experimental analysis is inadequate; while the comparison between LLaMA3-8B and LLaMA3-70B results is noteworthy, the spatial understanding of LLaMA declines as parameters increase, varying across C-Split cases. This raises the question of what insights researchers can draw from this benchmark to adjust datasets to maintain or even improve spatial understanding performance. Addressing this issue is essential to the benchmark's purpose.

questions:
I would look forward to seeing more experimental results, particularly on how different large language models perform on the benchmark. For example, how do the latest models like Qwen2 or Molmo perform in different cases? Additionally, it’s intriguing that spatial understanding may decline as parameter count increases—what could be the underlying reasons? I also noticed that the performance decreases with the use of CoT and 4-shot settings, which is puzzling. What might be causing this effect?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730218537417

thread_type:
official_review

<thread_object>
creator: reviewer_wwk9

<content>
summary:
This paper introduces a new benchmark called FoREST to evaluate LLMs spatial ability in understanding "Frame of Reference". Additionally, it proposes Spatial-Guided prompting to improve the FoR classification task and layout generation in text-to-image generation.

strengths:
1. Spatial ability of LLMs are an important research topic yet less explored. A scientific benchmark would contribute to this area.
2. This work conducts various experiments with a range of LLM models and provide in-depth analysis. It also verifies the proposed prompting method in text-to-image task, adding its value to real world applications.
3. The paper is well organized and written.

weaknesses:
1. The dataset is pure synthetic and constructed by a limited number of textual templates. I have concerns about the FoR classification task given the template "<locatum> <spatial relation> <relatum> <perspective>". It seems hard to disentangle this task with linguistic and common-sense reasoning of LLMs. For example,  LLMs are able to determine whether the perspective is intrinsic or relative by analyzing perspective template, and analyze topology template to determine whether the locatum is external or internal. Both of them don't necessitate understanding the underlying spatial configuration under a specific perspective. On the contrary, the text-to-image task indeed requires the model to interprete the spatial configuration and transform the perspective to camera's.
2. Again, since the dataset is synthetic and constructed by textual templates, the inductive bias might be leveraged by SG-prompting. 
3. Lack of full prompts of different settings, such as few-shot, CoT, SG-prompting, text-to-layout and SG to layout.

questions:
1. Is SG-prompting zero-shot or few-shot? In T2I task, what are examples mentioned in line 290?
2. In table 4, it's abnormal to see GPT-3.5 outperforms GPT-4o in A-split. And why do GPT family models do exceptionally well in EI and II C-split, yet relatively bad at ER and IR C-split? It would be interesting to dive into these observations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730566068506

thread_type:
official_review

<thread_object>
creator: reviewer_m8vc

<content>
summary:
The paper presents the FoREST benchmark, aimed at testing large language models' (LLMs) understanding of frames of reference (FoR) in spatial reasoning. FoR refers to different perspectives (intrinsic, relative, and absolute) used to describe spatial relationships. The benchmark assesses LLMs' ability to identify FoR in ambiguous and clear spatial contexts and perform text-to-image generation accordingly. Results show that LLMs have biases in FoR interpretation, impacting spatial layout accuracy. They also introduce Spatial-Guided prompting to improve FoR comprehension and performance in related tasks.

strengths:
- **Novel Perspective**: Introduces an innovative approach to assessing spatial perception in large models, focusing on frames of reference (FoR).
- **Theoretical Support**: Draws on established spatial language literature to support the motivations and foundational concepts of FoR.
- **Insightful Analysis**: Offers valuable insights into both FoR identification and text-to-image mapping.

weaknesses:
No dataset or code provided

questions:
Could you make the datasets and code available?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730604898908

thread_type:
official_review

<thread_object>
creator: reviewer_xdew

<content>
summary:
This paper proposes a new Frame of Reference (FoR) comprehension task for LLMs, where the models need to identify the perspective category based on a given spatial narrative. For this task, a new benchmark, named FoREST is generated. Using this benchmark, the paper identifies the inability and biases of various LLMs in solving this task, and proposes a new prompting technique to guide LLMs in identifying key information from the textual input thus enhancing their performance on this task. This paper also shows how this ability can be utilized for text-to-image generation under specific spatial guidance, highlighting the potential application value of this work.

strengths:
The paper proposes a novel task that can potentially reveal the ability of LLMs in understanding spatial concepts.

weaknesses:
1. It is unclear if the proposed Spatial-Guided prompting technique helps “reduce FoR bias in LLMs”, as claimed in the abstract in the paper, or just clarify the category terms that LLMs are tasked to identify. Since the FoR classes (external intrinsic, external relative, etc) are technical terms in cognitive studies that do not appear commonly in the internet data used for training LLMs, a clear and intuitive explanation of the terms is naturally important for solving this task. However the definition of the terms provided to the LLMs is formal and not intuitive. For example, it does not define clearly what does “the referenced object’s intrinsic directions” mean. This is only explained to some extent in the Spatial-Guided prompting examples, such as “the car has direction”. What if the concepts are explained in plainer and more intuitive language? Such as:

$\textit{"External intrinsic: The spatial description of an item A relative to another item B, where (1) A is not contained by B; (2) The spatial relationship description is relative to the B’s facing direction, if B has one (Example: a horse, a car. Counterexample: a box.)"}$

2. Why understanding FoR is an important problem is not articulated adequately. Since, according to the introduction in this paper, this task is more commonly seen cognitive linguistics study rather than AI or related fields, more discussion on the potential application of FoR understanding ability of LLMs can help readers better understand the motivation. The text-to-image task shown in the paper is a great application, but it is on specifically designed command types. Can this ability be potentially applied for other embodied AI or robotic tasks that require strong spatial understanding capacity?

questions:
Does the temperature setting impact the bias of LLMs on this task? The paper sets the sampling temperature to 0, and claims the bias of LLMs by showing that they more frequently produce external classes under ambiguous queries (that correspond to multiple correct categories) in Figure 4. It is possible that a low temperature limits the diversity of LLMs response.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732920210790

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comment related to SG vs CoT

comment:
We want to clarify the confusion about our proposed SG approach and CoT approach.
We would want to remind you that our proposed prompting approach can also be called CoT since we only try to specify the importance concept for reasoning to identify FoR.
The main question is how to explain the reasoning steps for identifying FoR. 
The innovation of our approach is to characterize the important spatial concepts involved to help improve FoR comprehension. 
We try to have the baseline CoT that does not focus on these explicit concepts, and we want to show that including these concepts (type of spatial relations, topology, distance, and direction in addition to properties of the relatum) in SG promoting (variation of CoT) helps model for better FoR identification.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a frame of reference (FoR) comprehension task for LLMs. The paper specifically focuses on understanding the perspective of spatial relations. For example, can LLMs distinguish between "a cat is to the right of the car from the car's perspective" vs "a cat is to the left of a car from my perspective". The task is posed as a multi-class classification task. The authors test different prompting strategies for this task, and find that textual descriptions of topological relations (inside vs not), frame of reference and distances (far) can help the models perform better on the task. 

Understanding spatial relationships is an important skill and the paper further LLM's limited understanding of spatial relationships. Using text-to-image for understanding spatial understanding of LLMs 

The reviewers actively engaged with the authors during rebuttal, but several of the reviewer's concern remained. Specifically, reviewers are concerned that in this instantiation of the task, it is unclear if it tests the spatial understanding of LLMs or just linguistic understanding. Additionally, reviewers believe that further investigation and prompt engineering is required to make the experiments more sound.