Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?
Authors: Yichi Zhang
Jiayi Pan
Yuchen Zhou
Rui Pan
Joyce Chai
Venue: EMNLP 2023 Main (2023)
Keywords: vision-language model
visual illusion
grounding
TL;DR: A study on Vision-Language Models' comprehension of visual illusions.

Abstract: Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world. However, known as visual illusions, human's perception of reality isn't always faithful to the physical world. This raises a key question: do VLMs have the similar kind of illusions as humans do, or do they faithfully learn to represent reality? To investigate this question, we build a dataset containing five types of visual illusions and formulate four tasks to examine visual illusions in state-of-the-art VLMs. Our findings have shown that although the overall alignment is low, larger models are closer to human perception and more susceptible to visual illusions. Our dataset and initial findings will promote a better understanding of visual illusions in humans and machines and provide a stepping stone for future computational models that can better align humans and machines in perceiving and communicating about the shared visual world. The code and data are available at [github.com/vl-illusion/dataset](https://github.com/vl-illusion/dataset).

Grounding Visual Illusions in Language:
Do Vision-Language Models Perceive Illusions Like Humans?
Yichi Zhang
Jiayi Pan"
Yuchen Zhou'
University of Michigan
University of California, Berkeley
The New School
zhangyic@umica edu
jiayipan@berkeley.co edu
zhous9256neeschool.cco
Rui Pan
Joyce Chai
Northwestern University
University of Michigan chai jy@umi edu
rui rai.pan llogg northwestern.ed.
Abstract
Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world. However, known as visual illusions, numan's perception reality isn't t always faithful to the physical world. This raises a key question: do VL have the similar kind of illusions as humans do, or they faithfully learn to rep- resent reality? To investigate this question, build a dataset containing five types of visual illusions and formulate four tasks examine visual illusions in state-of-the-arr Vs.Ms. Our findings have shown that although the overall alignment low, larger models closer to human perception and more susceptible to vi- sual illusions. Our dataset and initial findings will promote better understanding of visual illusions in humans and machines and provide a stepping stone for future computational mod- els that can better align humans and machines perceiving and communicating about the shared visual world. The code and data are available at
(a)
(b)
Figure I: The checker shadow illusion (Adelson, 1995).
darker than square B. However, the reality is, the color pixels of and B are exactly the same, as shown in Figure 1(b). This example demonstrates that while the physical attributes of A and are the same, from humans' eyes, they may look different, which may further influence how language is used to describe these objects.
Motivated by human visual illusion phenomena. recent years have seen an increasing amount of work in machine visual illusions (Gomez-V Villa et al., 2019, 2020; Hirsch and Tal, 2020; Sun and Dekel, 2021; Lonnqvist et al., 2021). These pre- vious works were solely based on vision, for ex- ample, evaluating how the internal representation from computer vision model can be used proxy of stimulus compared to human's 's stimulus shift under illusions. Most previous experiments were conducted a case-by-case manner, without addressing general behaviors through systematic investigation.
1 Introduction
It's well established that human perceptual systems are susceptible to visual illusions, which de- fined as "consistent and persistent discrepancies between physical state of affairs and its represen- tation in consciousness" (Day, 1984). Figure shows a well-known example the checker shadow illusion (Adelson, 1995). Here, cylinder on the checker board creates shadow on the board. Hu- man viewers are directed look at the two squares A and B shown in Figure (a)). To most normal- sighted people, they will perceive that square A
Different from these previous works, this paper studies visual illusion from context of language communication. Language comprehension and language production are tightly linked how we perceive the visual world. Back Figure (a)) when two people are observing the fig- ure together, due to their likely shared illusion, the
angle, in the
WWrr done while the author was an undergraduate re- scarch assistant at the University of Michigan.

---

expression "the darker square" will lead the same reference of square A. But when a human com- municates with machine, will the machine also interpret "the darker square" as square A? Given the rise of large vision-language models (VLM), it's important to understand whether these VLM models have similar tendency to visual illusions, and to what degree they may align with human vi- sion. The answers to these questions will further impact the alignment of the grounded language communication between humans and machines.
2 Related Work
Human Visual Illusion Visual illusions hu- mans are instances where human subjective per- ceived properties, such as color or size, deviates from their true physical characteristics (Carbon, 2014). This underscores the fact that the human brain doesn't 't perfectly replicate physical features; rather, integrates contextual information and prior knowledge to form the perceptual experiences (Car- bon, 2014).
Visual illusions can affect human behavior in multiple ways. Research has shown that human tion cannot resist visual illusions (Gentilucci al., 1996; Franz, 2001; Carey, 2001), so language comprehension and language production. Such findings catalyze inquiries regarding the capabil- ity of models to comprehend language instructions based on human perceptions and align them with human intent.
To address these questions, we created a new visual illusion dataset covering five different cate- gories from the cognitive literature. Based on the dataset, we created a benchmark, Grounding Vi- sual Illusion in Language (GVIL), which consists of four subtasks: Same-Difference Question An- swering (SameDiffQA), Referential Question An- swering (RefQA), Attribute Question Answering (AttQQA), and Referential Localization (RefLoc) to assess machines' alignment with the human un- der visual illusions. We specifically evaluated four state-of-the-art vision-language models: Unified- 10 (Lu et al., 2022), OFA (Wang et al., 2022), LLaVA (Liu et al., 2023) and InstructBLIP (Dai et al., 2023). Our results have shown that these four models mostly do not align with human vision illusions, especially for QA-based tasks. However, for the RefLoc task, these models (especially ones with larger parameters) have demonstrated an im- pressive alignment with humans.
Machine Visual Mluusoo. Previous studies have significantly advanced our ability to examine visual illusions by providing systematic data and tools. These efforts include the introduction of tools for calculating and generating illusory images system- atically (Hirsch and Tal, 2020; Fan and Zeng, 2023), the development of open-source software with a parametric framework for controlled illusion gen- eration (Makowski et al., 2021), and the proposal of a framework synthesizing new visual illusions using automatic differentiation techniques (Gomez- Villa et al., 2022). With the goal of evaluating machine visual illusions, prior research (Gomez- Villa al., 2019, 2020; Afifi and Brown, 2019; Benjamin et al., 2019) has also demonstrated that convolutional neural networks trained on ImageNet or low-level vision tasks can be misled by certain visual illusions, similar to human responses. These works have formed a foundation for scalable and reproducible research on machine illusions.
To our knowledge, this is the first work that takes language into consideration to study machine vi- sual illusion. There are two main contributions of this work. First, this investigation provides an ini- tial understanding of the alignment between human and machine visual illusions. Such understanding will enable techniques for a better communicative grounding in situated language communication and help develop more reliable embodied agents in the future. Second, unlike using internal representa- tions to explain illusions, language can be used proxy to demonstrate whether and how ma- chine illusions match or mis-match with the hu- man illusion. This benchmark will not only facili- tate computational models for better human agent alignment, but also provide tools for scientific un- derstanding of visual illusion in both humans and machines.
Unlike prior research focusing exclusively on vision models, our study introduces a novel and unique angle by presenting the first dataset offering natural language annotations for the evaluation of machine- processed visual illusions. This work in- tends to bridge the current gap and facilitate future evaluations of lasguael models concerning their alignment with human visual illusions. This novel perspective illuminates future improvements in human- mumann alignment and promotes the cru- cial role of human language as the interaction inter-

---

face with machines.
Color Constancy
Foundation Vision-Language Models. Recent advancements in foundational vision-language models (VLMs) have shown impressive results across broad spectrum of tasks (OpenAI, 2023; Wang et al., 2022; Lu et al., 2022; Alayrac al., 2022; Radford al., 2021). These models, acting user interfaces, interact with users through both language and visuals, necessitating a deep under- standing of human intent and an alignment with hu- man values to make them more useful. While pre- vious research has primarily focused on language- based uni-modal alignment problems (Ouyang et al., 2022; Kosinski, 2023), our work offers fresh perspective. Centered the intersection of VLM's perception capability and human cognitive biases, we investigate to what degree they can un- derstand humans and align with human intentions under visual illusions.
ship on left still red after applying blue filter, the blue ship on the right still looks blue af- applying red filter, though RGB colors of both ships are the
Color Assimilation
The two circles have the same color, while the left looks red (due to its neigh- bor/foreground) and the on the right looks orange.
the
Color Contrast
The two grey circles have the same color, while the one on the left looks lighter and the one on the right looks darker.
Geometrical Relativity
The two orange circles have the same size, while the one on the left looks smaller and the one on the right looks bigger.
3 The Grounding Visual Illusion in Language (GVIL) Benchmark
Geometrical Perspective
The two people have same height, while the one the left looks shorter and the one the right looks taller.
To facilitate our investigation, we created a bench- mark for evaluating machine visual illusions in the context of grounded communication. This bench- mark is built upon a set of images with visual illu- sions. Each image consists of two objects which may look different humans but actually iden- tical in their pixels. This setup has two advantages. First, the definition of illusion clear and non- ambiguous, thus easy to measure whether the machine has a similar illusion as humans. Sec- ondly, the multi-object setup naturally supports the evaluation of language grounding, such as evalu- ating whether the machine can select the object expression grounds to under the illusion (i.e., square A is what "the darker square" grounds to in (a))
Table Example illusion from each category and the corresponding explanations.
wc only included distortions among the four cat- egories in Robinson's s illusion classification in or- der to fix the need for a comparative assessment. The illusions we used to generate our data include Delboeuf (Delbocuf, 1865), Ebbinghaus, and Jas- illusions (Jastrow, 1892) for relativity, and Müller-Lyer (Müller-Lyer, 1889) and Ponzo illu- sions (Ponzo, 1911) for perspective distortion. The following explamations give an overview of the hu- man perception phenomena underlying each cate- gory:
3.1 Data Collection
Our dataset encapsulates five distinct types of illu- sions, each reflecting different elements of human physiological and cognitive processes (Gregory, 1997; Kitaoka, 2010; Robinson, 2013). Table dis- plays sample of each illusion type, along with detailed description.
Color Constancy refers to phenomenon where the color of an object remains constant perceptu- ally, despite changes in lighting conditions.
Color Assimilation shows how two adjacent color patches influence each other's perceptual appearance, reducing the perceived color differ- ence.
These illusions can be categorized into two broad areas: color and geometric illusions. For color illu-
sions, we adopt the classifications of color con-
Color Contrast. The perceived color of an ob-
stancy, assimilation, and simultancous contrast (MacEvoy, 2005). terms geometric illusions,
ject shifts opposite to the color of its surround-
ings

---

Category
#Root
#image
#Instance
Color Constancy
96
Color Assimilation
5
34
544
Color Contrast
3
30
480
Geometrical Relativity
320
Geometrical Perspective
2
10
160
Total
16
100
1600
Table 2: Dataset statistics.
Figure 2: Data augmentation examples for the Ebhing- haus illusion.
IMGi (illusion-free)
IMG2 Gillusion-iddue
Geometrical Relativity refers the distortion in the perceived shape or size of an object due to the influence of surrounding oobjects.
Q: Are the two balls the same color or different?
Geometrical Perspective reflects the tendency of our visual system to perceive perceptually dis- tant objects as larger than nearby ones of the same size.
IMG1:
Same
Same
Same
Different
IMG2: Different
Different
Same
For each illusion type, we first collected several root images from the literature (Todorović, 2020) and online r We manually identify at- tributes that can be changed without impacting the effect of illusion (e.g., the color of objects in geo- metric illusions, the position of objects in color illusions), and edit them create more illusion instances of the same type, to enrich the number of images in our dataset. We show some augmentation examples Figure
Humanlike
No-Illusion
N/A
Figure 3: Illustration of the SameDiffQA setup. For each instance, the model is asked about its perception of object property across two images, illusion-free and one illusion-induced. For valid illusion evaluation, the model must initially identify identical properties in the illusion-free image.
illusions. As shown in Figure 3, each question concerns pair of images (IMG1 and IMG2). One image (IMG1) illusion-free where two objects (blue balls) are identical color. The other image (IMG2) is induced with an effect illusion where two balls appear in different colors (blue on the left and green on the right) although their pixels are the same as in IMG1. The model is tasked to answer whether two objects the same color for each of the images. From human's 's perspective, the an- swer would be "Same" IMG1 and "Different" IMG2 due to the visual illusion. If the model gives the answer 'Same" to IMG1 and "Different" to IMG2, then the answers align with human's answers and therefore the model is considered "human-like" If the model gives "Same" to both images, it implies that the model is faithful to reality and does not per- ceive the same illusion humans do. If the model answers "Different" to IMG1, it means lacks ba- sic ability correctly perceive reality and these cases are considered not applicable to our illusion evaluation.
The statistics of our dataset shown in Table 2. Note that since this dataset is only used for the eval- uation purpose, i.e., to assess machine's alignment with human in visual illusion, we chose quality over quantity. The dataset is modest in size each instance carefully selected (or augmented) based on cognitive literature. Nonetheless, our infras- tructure also designed foster the continuous development and augmentation this dataset, al- lowing for community contributions, for instance. It will become an important resource to not only support better understanding of machine/human visual illusion, but also facilitate the adaptation of computational models visual illusions.
3.2 Benchmark Tasks
We define four vision-language tasks targeting dif- ferent model capabilities.
Same-Different Question Answering (SameDif- fQA) aims at evaluating the ability of recognizing
hhttps: httpse//mmcchheeeeeeee de/ot/

---

While SameDifJQA focuses on the detection of the presence of illusions, we design three tasks to examine how well machines align with humans when communication happens under the influence of illusions. Since it is reported that models tend to take shortcut by giving an answer purely based on the text question without looking at the image (Goyal et al., 2017), we propose paired test to reduce the evaluation bias. As shown in Figure each instance comes with two images: one original illusion image (IMG1) and one image IMG2 that flips the objects from the original image (IMG1) in a way that will also invert the answer to the question. Specifically, evaluate the following three as- pects:
IMG1 (original)
IMG2 (flipped)
Which ball looks blue, left or right?
RefQA
IMG1: Left
IMG2: Right
Is the ball on the left blue or green?
AttrQA
IMG1: Blue
IMG2: Green
Localize the blue ball in the image.
RefLoe
IMG1:
IMG2:
Referential Question Answering (RefQA) tests the human-likeness of referring to objects under the illusion. the question, the object of interest is referred to by property affected by the illusion, and the machine is asked to select the object from two options, e.g., select either left or right for the ball that looks blue, in IMG1 and IMG2.
Humanlike
Unlike
Figure 4: Illustration of the RefQA, AurQA and RefLoc setups. We flip the illusion image wherein the grounding outcome should also be inverted, to create pair images for each test. Model success requires accurate identification in both original and flipped versions to align with human responses. Matching human answers signals the model's capability to interpret illusions in a humanlike way, while a mismatch indicates otherwise.
Attribute Question Answering (AurQA) tests the human-likeness to describe the attribute of objects under the illusion. The question is about describing a visual attribute (e.g. color) influenced by the illusion of a selected object, and we provide two answer options to select from.
Referential Localization (RefLoc) tests the of localizing the referred object un- der the illusion. Given a referential expression that makes sense humans under the effect of illusion (but may not be faithful reality), the model needs predict a bounding box for the object the expres- sion referring to. For each referential query, we consider the machine's response to be humanlike only when the pair of responses from the two im- ages both match with those from human's 's. This enforces that humanlike response from machines has to be grounded in the illusion image.
corresponding annotations are verified by at least three human annotators from our team.
Experimental Setup
Vision-Language Models. To be evaluated on all of the four tasks in GVIL, the model has to be equipped with the visual question-answering skill and the object localization skill simultaneously. Among few candidates, choose two of the-art models, the Unified-1O (Lu al., 2022) and OFA (Wang al., 2022), both of which are trained on a wide range of vision-language tasks, and achieve impressive performance with a strong zero-shot inference capability on unseen data. Ad- ditionally, we select two recent works that adapt large language models to understand visual images: the LLaVA (Liu et al., 2023) and InstructBLIP (Dai et al., 2023). These models are interesting in- spect as they have shown a highly conceptual under- standing of the nuance in images, such as the capa- bility of interpreting jokes, which may also use- ful interpreting illusions. For each of the afore-
To create this benchmark, we annotate the col- lected illusion images with natural language ques- tions and the corresponding answers that humans will give under illusions. To support the study of language grounding, we also annotate the referring expressions for each of the objects with the cor- responding bounding box, where the expressions are formed under illusions. We provide several paraphrases for all the language annotations to help the evaluation be more robust to models that are sensitive to language forms. All the images and

---

Humanlike
No-Illusion
N/A
80
80
67.2
OFA
60
43.0
52.8
44.0
51.0
LLave
60
50.5 2
48.849.8
40
36.5
34.0
40
20
19.5
15.0
20
4.2
3.2
1.5
Tiny (33M)
Base (93M)
Large (470M)
Huge (930M)
Vicuna-78
Vicuna-13B
Unified-10
80
76.2
InstructBLi
80
60
56.0
60
43.0
40
36.6
28.8
40
26.830.2
22,8
21.5
20
8.2
8:2
15.5
8.5
20
0
Smail (71M)
Base (241M)
Large (776M)
XL (2925M)
Vicuna-7B
Vicuna-138
Figure 5: Results of SameDiffQA. The number shows the percentage of the answers. Each cluster represents the distribution over humanlike, no-illusion and N/A answers from a model. The green and red line correspond to the linear regression of humanlike rate and no-illusion rate across all the model sizes. Except for OFA-Large Unified- 1O-Large, InstructBLIP-I1 13B, the differences between the humanlike rate and the no-illusion rate are statistically significant P < 0.005. Details are Table Appendix
mentioned models, there exists range of variants in different sizes: OFA- (Tiny, Base, Large, Huge), Unified-1O- (Smalll,, Base, Large, XL), LLaVA- (Vicuna-7B, Vicuna-13BB 133), InstructBLIP- (Vicunaa 7B, Vicuna-13B). This allows to study the im- pact of size variations on model's s understanding of visual illusions.
RQ3: does the degree of alignment between VLMs and human responses vary across dif- ferent categorics of illusions?
We highlight several of our findings across this three questions below.
5.1 Illusion Recognition
Metrics. Through the experiments, we keep track of the Humanlike rate to measure the alignment be- tween humans and VLMs, which is the percentage of examples where the machine gives exactly the same answers as humans. For the SameDiffOA task, we also compute the No-Illusion rate, which cor- responds to the percentage of examples where the machine consistently considers the objects as the same under both illusion and illusion-free settings. For examples where the model fails to identify the objects as the same in the illusion-free image or produces nonsense answers to the questions, mark them Not Applicable (N/A) and exclude them from the illusion recognition assessment.
The results of SameDiffQA are shown in Figure 5. Relative proportions of "humanlike," "no-illusion." and "not applicable (N/A)" responses are repre- sented as green, orange, and grey bars respectively for each model, which all together account for 100%. First of all, we notice large percentage of responses, across all models, fall under the N/A category. This suggests that these models often cannot even tell that the objects are identical in the illusion-free image, underscoring the need for im- provement in standard vision-language reasoning capabilities beyond the scope of illusion contexts. Given the high proportion of N/A responses, one might question the benchmark's 's adequacy reli- ably reflecting a model 's tendency towards either "humanlike" or "no-illusion". Excluding the N/A responses, we employed a x² 2 and found that 9 out of 12 models would reject the null hypothesis which posits that the "humanlike" or "no-illusion" responses are uniformly distributed. In other words, these models do behave randomly. Refer Ap- pendix for more details. Such findings indicate that, despite certain limitations in their capabili- ties, our dataset and experimental design effectively
5 Results Analysis
From our experiments, we are interested in investi- gating the following research questions:
RQI: to what extent do VLMs recognize the presence of illusions similar to humans?
RQ2: how much do VLMs align with humans when communication happens under the influ- ence of illusions?

---

Constancy Assimilation
Contrast Relativity
Perspective
RefOA
60
0.3
2.8
3-8
5.2.5.2
20.12
20
SamfDiffQA
22.22
RefO
61
AttrQA
OFA
Unifled-10
LLavA
I-BLIP
15
Unified-I0
263
AttrQA
12
11.2
9.5
16
21.3
20
24
2318
2.22.0
2.0
SameDiiIQA
ReIQA
ALLIQA
RefLoc
OFA
Unified-10
LLaVA
I-BLIP
LLaA 138
60
50
RefLoc
40
39.2
30.8 3200
20
30
SameDifiQA
RefQA
AttrQA
20
Instructins 139
6,8
OFA
Unified-10
2 8
25.
Figure 6: Humanlike rate on RefQA, AttrQA and RefLoc. Each bar represents different model size, arranged in ascending order from left to right. Note that LL and InstructBLIP cannot predict object bounding boxes thus do not have the RefLoc results.
SameDifrQA
Re(QA
ALLIQA
Figure 7: Humanlike rates of the largest model of each family, with finer- meer-grained human likennss scores on each illusion category.
Task
Model
Pearson coeff.
p-value
An emerging trend reveals that "humanlike" scores tend increase as the model scales, whereas "no- illusion" responses tend to decline. This finding suggests a positive correlation between model scale and mucan-m alignment under illusions. We hypothesize that this observed trend could be at- capa-
OFA
0.689
0.311
SimeDiffOA
UnifiedIO
0,940
0.059*
OFA
0.946
0.054
RefOA
UnifiedIO
0.977
0.022**
OFA
0.957
0,043
AttOA
UnifiedIO
0.853
0.146
OFA
0.933
0.066*
RefLoc
UnifiedIO
0,960
0.039
tributed to the enhanced wattern-recognin bilities of larger models. These models, arguably, are better suited to discern and assimilate patterns present data generated by humans, which may have been shaped by the influence of illusions. Con- sequently, it's plausible to assume that these models are progressing towards a more humanlike compre- hension of illusions.
Table 3: Pearson's 's correlation malysis between the hu- manlike rate and model size. Statistically significant results with p < 0.05 and p < 0.1 are marked with ** and 8, respectively.
gauge illusion recognition in the assessed VLMs.
When examining cases where responses are ap- plicable for testing illusion recognition, we observe that the majority of models are more likely to fail in recognizing illusions (35.4% on average) than producing humanlike responses (15.6% on aver- age). This discrepancy is most pronounced in the case of InstructBLIP, where the model predomi- nantly offers 'no-illusion' answers. Conversely, the IO XL model stands the only model exhibiting a significant propensity towards human- like illusion recognition. further investigation of the underlying reason that causes this discrepancy would be interesting further work.
5.2 Communication Under Illusion
The results of RefQA, AtrQA, and RefLoc exper- iments are shown in Figure offering insights into the alignment between machine and human responses under the influence of visual illusions. We find that all the VLMMs encounter significant challenges in responding to questions presented un- der the influence of visual illusions in both RefQA and AurQA. As result, the models obtain a maxi- mum humanlike response rate of only 14.0% and 11.2% for ResQA and AurQA, respectively. Inter- estingly, models exhibit much stronger alignment in the localization task, with the highest alignment of 44.5% achieved by Unified-IO XL. This indi- cates that the learned object localization skill aligns
To illustrate how the scores evolve with model size, we plot regression lines of "humanlike" (green) and "no-illusion" (red) rates, respectively.

---

Attn. the humanlike choice under illusion
Attn. the humanlike choice illusion
Attn. on the counter-humanliee choice under illusion
Attn. on the counter-humanlike choice w/o illusion
60
53.5
48.7
45.1
47.4 43.3
46.244.8
39.4
36.0 35.234.3
19.919.9
19.920.0
20
Tiny
Base
Large
Huge
Figure Attention weight distribution of OFA models for the RefLLoc task.
5.3 Delving into Illusion Categories
We provide a more granular analysis by examin- ing each type of illusion, presenting the human- like rates for each category Figure 7. The re- sults depicted here are sourced from the largest model within each model family, namely Unified- IO Huge, OFA Huge, LLaVA ficuna-13B, and InstructBLIP Vicuna- 13B. Our observation reveals that the perspective category demonstrates the high- est degrec of alignment between machines and hu- mans. On the contrary, color constancy illusions emerge the category with the least congruity in relation to human responses.
Figure 9: Visualization of the attention maps generated by the OFA-Large model for the RefLoc task. In each row, the imput image shown on the left, and the atten- tion map for the referential expression "smaller orange ball" is shown on the right. The attention maps surround- ing the object of interest are highlighted for enhanced visibility.
5.4 Understanding the Cause of Illusions
To gain insight into model predictions under the in- fluence of illusions, analyze the attention distri- butions of OFA models in the RefLoc task. Specif- ically, we compute the attention weight from the localization query (e.g., "the smaller orange ball") to the object representation of either a "humanlike" or "counter-hue hhunttrr-humaniike perception under illusions. As depicted by the dark blue and light blue bars in Figure 8, as the model size increases, attention weights lean more towards the humanlike selec- tion. This trend is consistent with the humanlike rate observed for the RefLoc task in Figure 6. To determine if this bias stems from the illusion, we also calculate attention weights for images without the illusion inducer (represented by orange bars). These weights are nearly equally distributed across both objects, suggesting that the illusion does in- deed influence the model's attention and biases its predictions similarly to human perceptions.
better with humans under illusions compared the visual question answering skills. Research into the underlying reason behind this difference might be an interesting future direction.
Notably, we find a positive correlation between scaling up models and the increase of human- like rate across different models and tasks, which echoes our earlier observations from the SameD- iffQA experiment. To verify the statistical signifi- cance, we conducted Pearson's correlation analysis for OFA and UnifiedIO models?. As shown in Ta- ble 3, 6 of the tested scenarios showed significant or moderately significant positive correlations, with Pearson coefficients exceeding 0.9. Such results un- derscore the potential of larger models to enhance the human- machine alignment of responses across different tasks and illusion contexts.
Figure shows an example using the attention visualization tool (Aflalo et al., 2022). The first image displays the original illusion image, with two orange balls of identical size while the left ball seems smaller. The second image is devoid
IInstructBI IP and I.aVV were excluded since at least three data points are needed for the test.

---

of the illusion inducer, while the third image arti- ficially enlarges the right orange ball. Attention maps corresponding to the "smaller orange ball" query' are shown adjacent to each image. In the original illusion, the model predominantly focuses on the left ball, aligming with human observations. Without the illusion inducer. the query becomes ambiguous, leading to a dispersed attention map. However, when actual size difference is present, the model's attention decisively shifts to the cor- rectly perceived smaller ball on the left. A compar- ison of these attention maps highlights that while illusions can steer the model's s attention similarly to humans, its effect is less pronounced than when a real disparity exists.
Gaining Insights from Illusions. Exploring the effects of visual illusions can offer fresh per- spectives to comprehend the intricacies of vision- language models. Visual illusion, some way, is similar to various types of values shared by our human society, but not shared among today's Al models. Given the rapidly growing applications of large Al models, it's important to identify and understand various aspects of alignment between these models and humans. Vision illusion is only an example among many possibilities for future studies.
Limitations
This work is only the initial attempt to the question and there are many limitations which we think of exciting future research directions. First of all, al- though our experiments yields some interesting em- pirical findings, it is not clear why different forms of tasks (e.g., QA-based tasks vs. Refloc) lead to a stark contrast in the results. As these findings may have implications in future technology that adapt to visual illusions, more in-depth understanding of these behaviors will be needed in the future. Sec- ond, our benchmark is currently small in size. It lays an infrastructure for this work. Future efforts to collect more data to form centralized reposi- tory will be desired for studying visual illusions both humans and machines. Third, our investiga- tion is only based on a manually collected dataset for our intellectual curiosity. The construction of this dataset has the limitations that the effect of visual illusions not validated by wider range of human subjects other than the authors. While has motivation in improving situated language communication with embodied agents the physi- cal world, how visual illusions play in perceiving and communicating about the real physical world remains an interesting question.
6 Discussion and Conclusion
We introduce GVIL, the first dataset facilitating a systematic evaluation of machine visual illusion via natural language. Evaluating four distinct series of state-of-the-art vision-language model families across varying scales, observe notable align- ment between these models and human perceptions during object localization in the context of illusions. Interestingly, this alignment tends to strengthen with increased model size. Conversely, many mod- els face challenges in mirroring human perspec- tives within visual question question-answerine answering tasks. Our preliminary observations underscore the need for further discussions in two directions:
Assessment of Vision-Language Models in the Realm of Visual Illusions. Vision-language models have demonstrated commendable prowess in both visual and language understanding. Yet, notable gap persists in assessing their performance the presence of visual illusions. Given that such illusions are intrinsic to human perception, overlooking this facet may contribute to misalign- ment between human and Al interpretations during real-world engagements. While our study unveils certain trends, like the correlation between model human-model model alignment, making defini- tive assertions non-trivial due to the discrepancy in model architectures and their training datasets. Through GVIL, we aspire to catalyze further re- search that addresses visual illusion VLMs.
Ethics Statement
The data are collected and annotated by the au- thors without the involvement any other human subject. Data samples are selected from a wide literature search on the subject of visual illusions.
size and human-
A Acknowledgements
This work was supported by NSF I1S-1949634 and the DARPA PTG program HR001 12220003. The authors would like to thank the anonymous review- ers for their valuable comments and suggestions.
WW use the second last layer of the OFA large model, as the overall attention score of this layer is the highest. Atten- tions from all the heads are averaged.

---

References
Alex Gomez- Villa, Adrián Martin, Javier Vazquez- Corral, Marcelo Bertalmio, and Jesús Malo. 2022. On the synthesis of visual illusions using deep gener- ative models. Journal of Vision, 22(8):2-2.
Edward H Adelson. 1995. Checkershadow illusion.
Mahmoud Afifi and Michael S Brown. 2019, What else can fool deep learning? addressing color constancy errors on deep neural network performance. Pro- ceedings of the IEEE/CYF International Conference on Computer Vision, pages 243-252.
Alexander Gomez- Villa, Adrian Martin, Javier Vazquez- Corral, and Marcelo Bertalmío. 2019. Convolutional neural networks can be deceived by visual illusions. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 12309- 12317.
Estelle Aflalo, Meng Du, Shao- Yen Tseng, Yongfei Liu. Chenfei Wu, Nan Duan, and Vasudev Lal. 2022. Vi-interpret: An interactive visualization tool for interpreting vision-language transformers. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21406-2144 21415.
Alexander Gomez- Adrian Martin, Javier Vazquez- Corral, Marcelo Bertalmio, and Jesús Malo, 2020. Color illusions also deceive cmns for low-level vision tasks: Analysis and implications. Vision Research, 176: 156-144.
Cann-Battstte Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, al. 2022. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204.1 14198.
Yash Goyal, Tejas Khot, Douglas Summers- Stay. Dhruv Batra, and Devi Parikh. 2017. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913.
A Benjamin, Cheng Qiu, Ling-Qi -i Zhang. K Kording, and Stocker. 2019. Shared visual illusions between humans and artificial neural networks. In 2019 Con- ference on Cognitive Compulational Neuroscience, volume 10, pages 2019-1299.
Richard Gregory. 1997. Visual illusions classified. Trends in cognitive sciences, 1(5):190-194.
Elad Hirsch and Ayellet Tal. 2020. Color visual illu- sions: A statistics-based computational model. Ad- vances Neural Information Processing Systems, 33:9447-9458.
Claus-Christian Carbon. 2014. Understanding human perception by human-made illusions. Frontiers in human neuroscience, 8:566.
Joseph Jastrow. 1892. Studies from the laboratory of ex- perimental psychology of the university of wisconsin. ii The American Journal of Psychology, 4(3):381- 428.
David P Carey. 2001. Do action systems resist visual illusions? Trends in cognitive sciences, 5(3):109- 113.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. In- structblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500.
Akiyoshi Kitaoka. 2010. A brief classification of colour illusions. Colour: Design & Creativity. 5(3):199.
Michal Kosinski. 2023. Theory of mind may have spon- taneously emerged in large language models. arXiv preprint 232.02300..0088...
RH Day. 1984. The nature of, perceptual illusions. In- terdisciplinary Science Reviews, (1):47-58.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. arXiv preprint 2304.22304.0848)
Franz Joseph Delboeuf. 1865. Note sur certaines illu- sions d' optiqque: Essai d'une théorie psychophysique de la maniere dont I'aeil apprécie les distances et les angles. Bulletins de I' l'Académie Royale des Sciences, 19:195-216.
Ben Lonnqvist, Alban Bornet, Adrien Doerig, and Michael Herzog. 2021. comparative biology ap- proach to dnn modeling of vision: A focus on differ- ences, not similarities. Journal of Vision, 21(10): 17- 17.
Jinyu Fan and Zeng. 2023. Challenging deep learn- ing models with image distortion based on the abut- ting grating illusion. Patterns, 4(3).
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Unified- io: A unified model for vision, language, and multi- modal tasks. arXiv preprint arXiv:2206.088
Volker H Franz. 2001. Action does not resist visual illusions. Trends in cognitive sciences, 5(11):457- 459.
Bruce MacEvoy. 2005. Handprint: : Color vision.
Dominique Makowski, Zen J Lau, Tam Pham, W Paul Boyce and SH Annabel Chen. 2021. A para- metric framework to generate visual illusions using python. Perception, 50(1 1):550-965.
Maurizio Gentilucci, Sergio Chieffi, Elena Daprati, M. Cristina Saetti, and Ivan Toni. 1996. Visual illu- sion and action. Neuropsychologia, 34(5))369-376.

---

FC Lyer 1889. Optische urteilstänschungn. Archiv für Physiologie Suppl, pages 263-270.
piece of strong evidence that our dataset and exper- imental setup can support the evaluation of illusion recognition for the tested VLMs.
OpenAl. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin. Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744,
Model
#HL
#NI
x²
p-value
OFA-TTmy
129.45
c0.001
OFA-Base
78
20.64
<0.001 ***
OFA-Large
58
1.72
0.190
OFA-Huge
60
136
29.47
<0.001
UnifiedOO-Small
33
221
139.15
<0.001
UnifiedIO-Bass
33
62
9.38
0.002
Unified[O-Large
115
103
0.66
0,416
Mario Ponzo. 1911. Intorno ad alcune illusioni nel campo delle sensazioni tattili sull. Archives Itali- enney Biologie.
UnifiedIO-XL XI
142
34
66.27
<0.001**
LLaVA-7B
13
202
12.89
<0,001
LLaVA- 1 3B
6
195
177.72
<0,001 #:34
InstructBIIIP- 7B
91
223
55.49
<0.001 R*0
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
InstructBLIP 13B
107
121
0.86
0.354
Table 4: Chi-square test for the SameDiff Task (Figure 5). #HI and #NT denote the number of humanlike illu- sionary answers and no- answers, respectively. Statistically significant results with < 0.001 and p 0.05 marked with *** and **, respectively.
James Outram Robinson. 2013. The psychology of vi- sual illusion. Courier Corporation.
Eric D Sun and Ron Dekel. 2021. Imagenet-traine, deep neural networks exhibit illusion-like response to the scintillating grid. Journal of Vision, 21(1) 1):15- 15.
Dejan Todorović, 2020. What are visual illusions? Per- ception, 49(1 1):1 111112881199 1199.
Peng Wang. An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Ofa: Unifying ar- chitectures, tasks, and modalities through simple sequence-to-sequence learning framework. In Inter- national Conference on Machine Learning, pages PMLR.
A Statistical Analysis for Illusion Recognition
We conducted statistical analysis to rigorously validate that our experimental setup is able to reveal model's inclination towards either "humanlike" "no-illusion", notwithstanding the high prevalence of N/A samples. Specifically, we applied X -test to the model predictions omitting the N/A samples. The null hypothesis posits that the humanlike and no-illusion samples are uniformly distributed, i.e., the model behaviors randomly. As shown in Table 4, a significant proportion of the models reject the null hypothesis. Out of the 12 models tested, 8 models rejected the null hypothesis with a p-value 0.001, and 9 models rejected the null hypothe- sis with a p-value < 0.01. These figures strongly suggest that most models perform better than what would be expected by chance alone, which a


Reviews:
[{"review_id": "iCLwzLhPPL", "reviewer": "Reviewer_2Xp9", "summary": "", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing this work is “intriguing” and “holds promise for inspiring future work”. We appreciate all valuable feedback and questions about evaluation results. We believe we have addressed the reviewer’s concerns by providing detailed statistical analysis results. We welcome more questions during the discussion period if any part of our response is not clear. \n\n> **The size of the proposed dataset is limited. This raises concerns about the reliability of the evaluation findings.**\n- We acknowledge the current size of the data is on the small side, however it’s sufficient to support several initial findings (see the results next).  We want to emphasize that GVIL, while limited in its size, is the first-of-its-kind dataset that was carefully curated to facilitate study of visual illusions by VLMs. The diverse set of illusions included in the dataset ensures that it's not just the quantity but also the quality and complexity of data that provides value, especially as a first step to initiating this research direction.  Nevertheless, we aim to expand the dataset in our future work and make a platform available for the community to contribute to data expansion.  \n\n> **Concerns about whether the high ratio of N/A samples in the SameDiffQA task (Fig. 5) will affect the conclusion's validity.**\n- We perform a statistical analysis to validate that our experimental setup is able to show the model’s behavior towards either humanlike or no-illusion, even with a high N/A ratio. In particular, we do a Chi-square test among the samples that exclude the “N/A” samples. The null hypothesis is that the humanlike or no-illusion samples are uniformly distributed, i.e., the model behaviors randomly. As shown in Table, 8 out of 12 models reject the null hypothesis with a p-value <0.001, while 9 out of 12 reject with a p-value <0.01. This suggests that most models do not merely perform at chance level. This indicates strong evidence that our dataset and experimental setup can support the evaluation of illusion recognition for the tested VLMs even with limited capability of answering these questions. We will add this statistical analysis in the revision. \n\n|        Model      | #Humanlike | #No-illusion |Chi-square|  p-value|\n|-----------------------|--------------|-----------------|---------------|------------|\n| OFA-tiny             |      17      |        172      |    129.45   |   <0.001 |  \n| OFA-base           |      78      |        146      |    20.64     |  <0.001 |    \n| OFA-large           |      58      |        73        |    1.72       |  0.190    |\n| OFA-huge           |      60      |        136      |    29.47     |   <0.001 |\n| UnifiedIO-small   |      33      |        221      |    139.15   |   <0.001 |\n| UnifiedIO-base   |      33       |        62        |    9.38       |    0.002  |\n| UnifiedIO-large   |      115     |       103       |    0.66       |  0.416    |\n| UnifiedIO-xl         |     142     |        34        |     66.27     |  <0.001  |\n| LLaVA-7B            |     13       |        202      |    12.89      |  <0.001  |\n| LLaVA-13B          |     6         |        195      |    177.72    |  <0.001  | \n| InstructBLIP-7B   |     91       |        223      |    55.49      |   <0.001 |\n| InstructBLIP-13B |     107     |        121      |    0.86        |    0.354  |  \n\nTable 1. Chi-square test for the SameDiff Task (Figure 5). \n\n&nbsp;\n\n**Additional statistical test for the correlation between human likeness and model sizes.** \n- To validate our second main claim that the model’s human likeness under visual illusions increases with their size, we also add a Pearson’s correlation analysis on all the four tasks for OFA and UnifiedIO. Note that we need at least 3 data points to perform the test so InstructBLIP and LLaVA are not applicable here. As shown in Table 2, under 6 out of 8 setups we observe a significant or moderate significant result for a positive correlation (pearson coefficient >0.9). This supports our finding of positive correlation between scaling up models and the increase of humanlike rate across different models and tasks. \n\n|        Task             |   Model    |  Pearson coefficient   |  p-value |\n|-----------------------|--------------|------------------------------|-------------|\n| SameDiff QA      |      OFA    |             0.689              |  0.311    |\n| SameDiff QA      | UnifiedIO |             0.940              |  0.059*  |\n|          Ref QA      |      OFA    |             0.946              |  0.054*  |\n|          Ref QA      | UnifiedIO |             0.977              |  0.022** |\n|          Attr QA      |      OFA    |             0.957              |  0.043** |\n|          Attr QA      | UnifiedIO |             0.853              |  0.146    |\n|          Ref Loc     |      OFA    |             0.933              |  0.066*   |\n|          Ref Loc     | UnifiedIO |             0.960              |  0.039**  |\n\nTable 2. Pearson's correlation analysis between the humanlike rate and model size.   **: p < 0.05   *: p<0.1\n\n&nbsp;\n\n> **Lacks an analysis of the underlying reasoning of machine illusions**\n- We fully acknowledge the reviewer's point regarding the importance of a deeper analysis of the underlying behaviors of the VLMs. In Section 5.4 and Figure 8, we attempted to address this question by showing the visualized attention map of an OFA model under different situations. We observe that the model’s attention weight can be biased towards a humanlike manner with the presence of illusion inducers, which explains its output to some degree. While understanding the precise 'why' behind these behaviors is crucial, we intended to set the stage for this exploration. We hope the preliminary results shown in our paper and the resources that will be released soon can inspire future research to understand the problem better. \n\n> **Figure 6: \"14.0\" is out of the figure**\n- Thanks for pointing out this. We will correct the figure in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Exploring the behavior of VLMs in the context of machine-generated visual illusions is intriguing and holds promise for inspiring future research.\n- Overall, the paper is well-written. The comprehensive experiments and analysis can illustrate some behaviors of prominent VLMs.", "reasons_to_reject": "- The size of the proposed dataset is limited. This raises concerns about the reliability of the evaluation findings.\n- Statistical tests were not carried out. Fig. 5 illustrates that the N/A (not applicable) sample rate is approximately 50%, suggesting that the evaluated VLM struggles to comprehend and answer questions accurately. This leaves room for chance-level responses, human-like or not. While the authors introduce a paired test (flipping the sample), I find it insufficient. I recommend the authors evaluate with more augmented samples and perform statistical tests to further validate their conclusion.\n- While this paper sheds light on the behavior of certain VLMs, it lacks an analysis of the underlying reasons for these behaviors.", "questions_to_authors": "see weakness", "paper_topic_and_main_contributions": "This paper examines the performance of Vision-and-Language models (VLMs) in addressing questions involving visual illusions. A new dataset, GVIL, comprising five types of visual illusions, is assembled. Multiple VLMs are tested on four tasks to showcase their behavior. The results reveal that these models closely match human performance in object localization, yet face challenges in aligning with humans for visual question-answering scenarios.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 6: \"14.0\" is out of the figure"}, "extra_text": {}}, {"review_id": "4mPsbmuLzL", "reviewer": "Reviewer_EWKP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing that visual illusions of VLM is an interesting problem and this paper makes a focused contribution. We appreciate the reviewer’s feedback on the evaluation results. We believe our response has sufficiently addressed the reviewer’s concern. We will be more than happy to answer more questions if any part of the response is not clear. \n\n> **Significant amount of the VLMs’ output is N/A which indicates the incapability of VLMs on more foundation tasks and make the result and analysis less convincing.**\n- First, we acknowledge that the current VLMs are far from being perfect on some foundation tasks as indicated by the high N/A ratio. However, the goal of our work is to (1) define the problem of machine visual illusion in the context of VLM; and (2) create a first-of-its-kind dataset that can support future work on visual illusion. Although the SOTA models we applied are not reliable or robust enough, the problem itself and the dataset are not specific to these models. As the field progresses and new models are introduced at an unprecedented rate, this work is to set the foundation/stage where future models can be evaluated upon.    \n- Second, even with the existing models, this work has generated some key findings. The high ratio of N/A samples in the SameDiffQA task does not affect the validity of visual recognition evaluation.  Here we show statistical analysis for the results in Figure 5. We did a Chi-square test among the samples that excluded the “N/A” samples. The null hypothesis is that the humanlike or no-illusion samples are uniformly distributed, i.e., the model behaves randomly. As shown in Table, 8 out of 12 models reject the null hypothesis with a p-value <0.001, and  9 out of 12 reject with a p-value <0.01. This suggests that most models do not merely perform at chance level. This indicates strong evidence that our dataset and experimental setup can support the evaluation of illusion recognition for the tested VLMs even with limited capability of answering these questions. We will add these analysis results in the revised version. \n\n|        Model      | #Humanlike | #No-illusion |Chi-square|  p-value|\n|-----------------------|--------------|-----------------|---------------|------------|\n| OFA-tiny             |      17      |        172      |    129.45   |   <0.001 |  \n| OFA-base           |      78      |        146      |    20.64     |  <0.001 |    \n| OFA-large           |      58      |        73        |    1.72       |  0.190    |\n| OFA-huge           |      60      |        136      |    29.47     |   <0.001 |\n| UnifiedIO-small   |      33      |        221      |    139.15   |   <0.001 |\n| UnifiedIO-base   |      33       |        62        |    9.38       |    0.002  |\n| UnifiedIO-large   |      115     |       103       |    0.66       |  0.416    |\n| UnifiedIO-xl         |     142     |        34        |     66.27     |  <0.001  |\n| LLaVA-7B            |     13       |        202      |    12.89      |  <0.001  |\n| LLaVA-13B          |     6         |        195      |    177.72    |  <0.001  | \n| InstructBLIP-7B   |     91       |        223      |    55.49      |   <0.001 |\n| InstructBLIP-13B |     107     |        121      |    0.86        |    0.354  |  \n\nTable 1. Chi-square test for the SameDiff Task (Figure 5)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem about visual illusion of VLM is interesting.\n\n2. The paper makes a focused contribution.", "reasons_to_reject": "Significant amount of the VLMs’ output is N/A which indicates the incapability of VLMs on more foundation tasks and make the result and analysis less convincing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. Constructed a visual illusion dataset for five different type of visual illusions\n2. Proposed probing tasks for color and geometric related visual illusion and evaluated four VLMs: Unified-IO, OFA, LLaVA and InstructBLIP\n•\tSameDiffQA to detect the existence of illusions.\n•\tRefQA to test illusion object reference. \n•\tAttrQA to test whether VLM can recognize the attribute of the object.\n•\tRefLoc to test whether VLM can locate the object.\n3. Proposed below metrics:\n•\tHumanlike Rate\n•\tNo-Illusion Rate\n•\tNot Applicable (N/A)", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cnxpPnQF8H", "reviewer": "Reviewer_Zh5s", "summary": "", "questions": "Will you release the code and dataset?\n\nIt is not really a question, more of a comment, so no need to address this in the rebuttal if space is limited:\n\nVLMs are trained on images and text where the text has been generated by humans alongside the image? Isn’t it expected for the models to show a certain degree of visual illusions because they have been learning to describe images like humans and give answers human expect? I am surprised to see that they produce little humanlike responses. Maybe this is correlated with their poor performance in general, that most of the time (L410) they do not even recognize the object, so recognizing the “ambiguitiy” of a visual illusion might require even more capabilities that the models do not have? This thought seems to align with what you write in L472.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the contribution of this paper is “substantial”. We appreciate the reviewer’s valuable feedback and questions which call for more in-depth studies in the future. \n\n> **Will you release the code and dataset?**\n- Absolutely. We will release the dataset and software after the anonymity period.   \n\n> **VLMs are trained on images and text where the text has been generated by humans alongside the image? Isn’t it expected for the models to show a certain degree of visual illusions because they have been learning to describe images like humans and give answers human expect? I am surprised to see that they produce little humanlike responses. Maybe this is correlated with their poor performance in general, that most of the time (L410) they do not even recognize the object, so recognizing the “ambiguity” of a visual illusion might require even more capabilities that the models do not have? This thought seems to align with what you write in L472.**\n- We appreciate the reviewer for pointing this out. One of the driving-force of conducting this study is our curiosity about whether a VLM that is competitive on common vision-language benchmarks is better at aligning with humans even when they are deceived, or better at faithfully representing the actual reality. From the trend of an increased human likeness when the model becomes larger, it seems that the model does learn to mimic more human behaviors when fitting the data produced by humans. Regarding the low ratio of human-like answers, just as you mentioned, it could be that the models are too poor to output robust and self-consistent answers, in that sense not humanlike. Another possibility is that the models may not be trained on many illusion images, thus struggling in those out-of-distribution samples. These are interesting questions which will require more in-depth studies in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The contributions of this paper are substantial:\n\n1.\tIntroduction of a visual illusion dataset for VLMs including visual illusion from 4 categories.\n\n2.\tTesting 4 SOTA VLMs.\n\n3.\tClearly written, visualised and well motivated paper.\n\nThe research question is interesting and the experiments are well executed to address it.", "reasons_to_reject": "This paper makes it really hard to find a reason to reject, as its contribution is substantial and the methodology is sound.", "questions_to_authors": "Will you release the code and dataset?\n\nIt is not really a question, more of a comment, so no need to address this in the rebuttal if space is limited:\n\nVLMs are trained on images and text where the text has been generated by humans alongside the image? Isn’t it expected for the models to show a certain degree of visual illusions because they have been learning to describe images like humans and give answers human expect? I am surprised to see that they produce little humanlike responses. Maybe this is correlated with their poor performance in general, that most of the time (L410) they do not even recognize the object, so recognizing the “ambiguitiy” of a visual illusion might require even more capabilities that the models do not have? This thought seems to align with what you write in L472.", "paper_topic_and_main_contributions": "This paper tests whether VLMs are also prone to visual illusions like humans are. It finds that indeed, they are to a small degree.\nFor this, the authors assemble a new VL dataset testing for visual illusions from 4 categories in a QA fashion and 4 recent test VL models on it.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Very well written."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers judged this work to be sound technically and also reproducible. The paper is well written and clear. The study itself was found to be creative and novel, with important results. The major limitation is the small size of the dataset and the fact that the VLM used this work would respond with "N/A" is a large number of trials.
Decision: Accept-Main