reviews: 
Comments: - Investigates how vision-language models (VLMs) represent and reason about spatial relations, particularly when spatial language is ambiguous due to differing frames of reference (FoRs). It focuses on evaluating the robustness, consistency, and flexibility of VLMs in handling such ambiguities.

- The authors introduce a new evaluation protocol called the COnsistent Multilingual Frame Of Reference Test (COMFORT), which systematically assesses VLMs' spatial reasoning capabilities. The evaluation includes tasks with synthetic 3D images and text descriptions of spatial relations, with tests conducted in 109 languages across 170 regions worldwide.

- The study reveals that while VLMs show some alignment with English conventions for resolving spatial ambiguities, they struggle with: (A) Consistency and robustness in spatial reasoning, (B) Flexibility in adopting multiple FoRs, (C) Cross-linguistic and cross-cultural understanding of spatial relations, with English conventions often dominating over those of other languages.
Strengths: - Unlike existing benchmarks, which typically focus on spatial language understanding within a single frame of reference or specific cultural context, this work broadens the scope by emphasizing ambiguity in spatial reasoning due to different Framework of References and cross-linguistic/cross-cultural diversity.

- The paper is well-structured, with clear explanations of key concepts such as frames of reference and spatial reasoning.

More importantly, 

- The paper’s contributions are highly significant in the context of both vision-language research and broader discussions on AI alignment with human cognition. By focusing on spatial reasoning, a core component of human cognition, this work addresses a critical gap in the evaluation of VLMs.

- The cross-lingual evaluation across diverse cultural backgrounds provides valuable insights into the limitations of current VLMs in handling non-English spatial conventions. This has implications for the development of more globally applicable models, as the dominance of English-centric FoRs could hinder the usability of VLMs in non-English contexts.
Weaknesses: - While the introduction of the COMFORT benchmark is a novel contribution, the idea of testing spatial reasoning in vision-language models is not entirely new. Several previous benchmarks, such as those mentioned in the paper, have evaluated spatial reasoning capabilities in VLMs. The novelty of this paper lies primarily in its focus on the frame of reference ambiguities, but the paper does not sufficiently clarify how this focus significantly advances beyond the existing benchmarks. But I agree that the frame of reference is a rather novel take on the same topic.

- Although the paper provides valuable insights into how English conventions dominate spatial reasoning in multilingual models, the discussion around cultural and linguistic biases remains relatively high-level.

- The paper identifies that most models tend to default to egocentric relative frames of reference, which aligns with English spatial conventions. However, it does not deeply investigate whether this bias toward egocentrism is inherently problematic across all contexts or if there are scenarios where this behavior is acceptable or even preferable. 

+ (not required) It would be beneficial to include a dedicated section discussing the practical relevance of the findings in LLM applications. Is there an area the the findings of this paper can seriously affect?
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3