reviews: 
Comments: **Main contributions**: 
The paper has two main contributions: 
1. It presents COMFORT, a framework for evaluating VLMs' understanding of spatial frame of reference expressions, which relate two objects to one another (e.g. "Is the basketball *to the right of* the car?"). Expressions are evaluated using a rendered image containing the two objects and which is used to query the VLM with (as a binary Yes/No question) for whether or not the relation holds for the image. 

2. It uses COMFORT to show that a set of contemporary VLMs have distinct preferences for frame of reference parameters. Specifically, the paper presents evidence for the preference of English language frame of reference parameters in VLMs. 

**Framework Structure**:  
Expressions in COMFORT vary across two main conditions:
1. Expressions which include two objects with a semantic "front" (e.g. such as the "front" of a person being the side with the person's face). In this scenario the spatial expression must be reasoned about in terms of an anchor frame of reference (i.e. either object A's, object B's, or the camera's frame). The selection of the frame of reference will determine which direction in the image corresponds to "left of", "back of" etc. 

2. Expressions which include objects without a semantic "front". In COMFORT, this is instantiated by scenarios with two balls. In such scenarios, a coordinate transformation convention must be assumed for determining what is "right", "left", "back", etc. Different languages have different standard conventions, and so this condition is set up to test this. 

Preferences are measured by evaluating the [Yes/No] probability for a given spatial expression in relation to a ground truth *region of acceptability*, a continuous region of rotation of the second object around the center object (see Figure 1c for reference) in which a spatial expression may hold true. This region of acceptability allows for not only for accuracy to be measured but also for the dynamics of the probability to be measured as functions of distance from the center of the region of acceptability -- intuitively, the score of "right of" should decrease smoothly as the object is rotated from 0 to 90 degrees from the center of the region. Specifically, the paper presents. 
These measures probe the robustness of model scores (e.g. evaluating std across object variations unrelated to space like color or shape) as well as the consistency (are scores symmetric at both equivalent angles from the acceptability region's center? Do they change smoothly as angle is varied?). 

**Results**
The paper uses the region of acceptability based error measures to show that VLMs generally skew towards preferring egocentric frames of references (condition 1) and the reflected coordinate transformation convention. Additionally, the paper also shows that these preferences cannot be reliably changed when the prompt explicitly defines the frame of reference to be used, or when a language (other than English) with different known preferences is used to prompt the model.

**Recommendation** 
This is a solid paper which I believe presents a unique contribution that stands to better capacitate the community in vetting the spatial understanding of VLMs. I have noted a few suggestions below under Weaknesses to improve the presentation of the paper. With that said, I think it is already at a good quality and I recommend acceptance.
Strengths: **Strong Backing in Theory**: The paper's framing and setup in sections 1 through 3 (which motivates the problem statement and framework design choices) is a joy to read. Motivating statements and scoping within existing work is very clear and compelling. 

**Conclusions Well Supported**: The experimental setup is quite thorough and shows clear trends which the authors use to justify their claims. A majority of models are clearly shown to have FoR preferences based on the metrics *cos* and *hem* error functions defined, with the BLIP family of VLMs being a notable exception. I particularly appreciated that the paper additionally presents a compelling explanation for the BLIP discrepancy by showing that these same models score poorly on an evaluation of object hallucinations. 

**Novelty**: In my understanding I believe that the presented framework is novel based on its continuous definition based on regions of acceptability. This seems to me to be an important distinction namely because these spatial relationships are known to lie within continuums and are not discrete. 

**Clarity**: The paper's writing is generally clear, and I found the organizational structure helpful for understanding the concepts introduced.
Weaknesses: **Cross-lingual Experiment Presentation**: I think some further refinement/elaboration of the cross lingual experiments (Section 4.5) could help strengthen the paper. Although the paper does state upfront that it spends most of the content focused around English experiments, the treatment of the section still felt a little too short for the emphasis that is placed in the introduction for this being a core piece of the framework (e.g. the M in COMFORT stands for Multilingual). Specifically I think a couple of details could be expanded upon. 

First, the results in Figure 8 are a little ambiguously presented. Is the world map mapping the preferences of GPT-4o, or is it mapping ground truth human preferences? I believe the former, but the wording in both the figure and the paper makes it sound like these are ground truth preferences. 
Second, for ground truth preferences, I'm wondering how these are obtained? -- I'm assuming this is pulled from the linguistics literature, but it is not explicitly stated and appears like a key detail to me. I would appreciate it if the authors could clarify both of these points. 

Lastly, although the results presented clearly show a preference for the same reference frames as English, I think they could be strengthened by also showing a notion of what the expected preferences would be if the VLM were to be adhering to the conventions of the language being used. For example, I think this could be done by (a) adding a second map showing a visualization of ground truth human preferences, or (b) adding the bold/underline convention from the table in Figure 8 to Table 10.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4