IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models
Authors: Haz Sameen Shahgir
Khondker Salman Sayeed
Abhik Bhattacharjee
Wasi Uddin Ahmad
Yue Dong
Rifat Shahriyar
Venue: COLM (2024)
Keywords: Vision Language Model, Optical Illusions, Dataset, Benchmark, Hallucination
TL;DR: IllusionVQA dataset tests Vision Language Models' comprehension and localization of optical illusions, revealing their limitations compared to human performance.

Abstract: The advent of Vision Language Models (VLM) has allowed researchers to investigate the visual understanding of a neural network using natural language. Beyond object classification and detection, VLMs are capable of visual comprehension and common-sense reasoning. This naturally led to the question: How do VLMs respond when the image itself is inherently *unreasonable*? To this end, we present IllusionVQA: a diverse dataset of challenging optical illusions and hard-to-interpret scenes to test the capability of VLMs in two distinct multiple-choice VQA tasks - comprehension and soft localization. GPT4V, the best performing VLM, achieves 62.99\% accuracy (4-shot) on the comprehension task and 49.7\% on the localization task (4-shot and Chain-of-Thought). Human evaluation reveals that humans achieve 91.03\% and 100\% accuracy in comprehension and localization. We discover that In-Context Learning (ICL) and Chain-of-Thought reasoning substantially degrade the performance of Gemini-Pro on the localization task. Tangentially, we discover a potential weakness in the ICL capabilities of VLMs: they fail to locate optical illusions even when the correct answer is in the context window as a few-shot example.

Published as a conference paper at COLM 2024
Illusion VQA:: A Challenging Optical Illusion Dataset for Vision Language Models
Haz Sameen Shahgir Khondker Salman Sayeed', Abhik Bhattacharjee', Wasi Uddin Ahmad?, Yue Dong', Rifat Shahriyar'
Bangladesh University of Engineering and Technology,,
AWS AI L University of California Riverside? sameen20800gmaill com, salkhon@50egmail.c com
Abstract
The advent of Vision Language Models (VLM) has allowed researchers investigate the visual understanding of a neural network using natural language. Beyond object classification and detection, VLMs are capable of visual comprehension and common-sense reasoning. This naturally led to the question: How do VLMs respond when the image itself is inherently unreasonable? To this end, we present IllusionVQA. a diverse dataset of chal- lenging optical illusions and of VLMs in two distinct multiple-choice VQA tasks comprehension and soft localization. GPT4V, the best performing VLM, achieves 62.99% accu- racy (4-shot) on the comprehension task and 49.7% on the localization task (4-shot and Chain-of-Thought), Human evaluation reveals that humans achieve 91.03% and 100% accuracy in comprehension and localization. We discover that In-Context Learning (ICL.) and Chain-of-Thougee substantially degrade the performance of Gemini-Pro in the localization task. Tangentially, we discover a potential weakness in the ICL capabilities of VLMs: they fail to locate optical illusions even when the correct answer is in the context window fho-shot example.
scenes to test the ca capability
reasoning
1 Introduction
Optical illusions are characterized by visual percept that arguably differs from reality. Illusions come in a wide variety; their categorization is difficult because the underlying cause is often unclear (Bach & Poloschek, 2006; Gregory, 1997a). Figure 2 shows examples that are distinct from each other but all fall under the umbrella of optical illusions. Figure 2a shows an impossible object where- "Parts of the object (in this case the elephant's 's legs) become the background, and vice versa." " (Shepard, 1990). Figure 2b depicts a 3-by-3 grid where some cells appear pale-yellow, but upon closer inspection, it is evident that all cells are perfectly white. Figure 2c shows a realistic 3D drawing of white cube with a cup positioned as resting as if on top of the fictional cube. Although all three examples are arguably optical illusions, they have little in common. Despite the challenges in optical illusion classification, Gregory (1997b) proposed four main classes based on appearance ambiguities, distortions, paradoxes, and fictions. Based on this framework, Figure 2a is a paradox since the object depicted cannot truly exist. Figure 2b is a distortion since none of the regions is pale yellow, although it might appear so. Figure 2c is a Trompe-l'aell illustration and its classification as an optical illusion is contentious (Wade & Hughes, 1999), although it might fall under the aforementioned fiction category.
The cause of our perception of optical illusions is a rich field of research in Cognitive Psy- chology (Helmholtz, 1948; Gregory, 1968; Gentilucci et al., 1996). Motivated by similarities
Equal Contribution
t The work does not relate to the author's 's position at Amazon.
The code and datasets are available at https:/ //github comm//sebuett/pp/lusssiooogg
1

---

Published as a conference paper at COLM 2024
Hidden
Deceptive Design
Size
Illusion
Angle Illusion
Scal-S
Systhoic Doncratinn
Bench
VQA
Bllaion
Co
Size
Color
Size
including
Color
impossible Ooject
Quantion Type
Binary (Yes/No)
Binary (Yss/No)
Muliple choics
Number of
avatage
Miscellaneous
Anarago Length (cher.)
25
25
55.4
GPT4V
GVILL HallueiorBesch byy editing oolor, shick, crientaiio IlluinnVVQ only the lusa image,, stsoring dlisrsll..
syntheiic magts genersted
Upside CirrreSppiral
Pustlive Negaalve Space
(4-shot)
GVE. Hallision
GPT4V
VGA
(4-shot)
Figure 1: Left: Comparison of human and VLM performance in Illusion Comprehension. Right: Comparison of TllusionVQA with prior illusion datasets GVIL (Zhang et al., 2023) and HallusionBench (Liu et al., 2023a).
Q: What is unusual about this line drawing of elephant?
A. The elephant has five six legs
B. The elephant using trunk as fifth leg
C. The elephant is merging with the background in some regions
D. The elephant has six legs while the rest of its body normal
(a) Shepard elephant Shepard (1990)
Q: How many pale yellow regions are in this image?
A. Four, the four corner regions Zero, all the regions white
Five, the center and the four corner regions
D. One, only the center Nine, the regions are pale yellow
(b) Watercolor Illusion Pinna et (2001)
Q: What unusual about this image?
A. The left cup is miniature and the right cup is normal-sized cup The white cube a highly realistic 3D drawing while everything else real
C. The left cup is highly realistic 3D drawing while everything else is real
The left cup and the white cube it is resting
highly realistic 3D drawings right cup is absurdly large and possibly highly realistic 3D drawing
(c) A trompe l'oeil art, not traditionally considered as optical illusion in the cognitive sciences.
Figure 2: Examples of optical illusions Illusion ClusionVQA-Conee Www.-AAAA-mmmeeeneniiiio
in human visual cognition and artificial neural networks (Cichy et al., 2016; Fickenberg et al., 2017), researchers have explored if and how artificial neural networks perceive optical illusions (Gomez-Ville et al., 2019; 2020; Sun & Dekel, 2021; Lonnqvist et al., 2021; Hirsch & Tal, 2020). However, the scope and generalizability of these works were limited by the need for a case-by-case analysis of the activations and gradients of the model in response to individual optical illusions. As remedy, Zhang et al. (2023) was the first to explore optical illusions through small VLMs (up to 13 billion parameters) and to investigate their internal beliefs about optical illusions through natural language.
2

---

Published as a conference paper at COLM 2024
Unlike prior work, curate challenging optical illusions from the Internet that span 12 distinct categories inherited from cognitive psychology studies. We craft detailed question- answer pairs and several incorrect options designed to probe the abilities of VLMs. We frame the problem as a standard visual question-answering task (VQA) and evaluate state- of-the-art VLMs such as GPT4V (Achiam et al., 2023) and Gemini-Pro (Team et al., 2023) in addition to smaller open-source models. Our detailed categorization and extensive human evaluation allow for fine-grained comparison of human and VLM cognition of optical illusions along different axes. Our contributions are summarized as follows.
1. We introduce IllusiooVQQ., novel dataset designed rigorously test the ability of VLMs to locate and comprehend challenging optical illusions.
2. We com comprehensively test a wide range of open-source and closed-source VLMs. Our experiments reveal that GPT4V is the most capable model on illusion comprehen- sion and localization but still lags significantly behind human-level performance.
3. Through experiments, we discover that state-of-the-art VLMs can locate ordinary objects accurately, but struggle with optical illusions. Furthermore, these models exhibit inconsistencies when evaluated through In-Context Learning and Chain-of- Thought reasoning.
2 Related Work
Impossible Object
Real Scene
Size Illusion
Hidden Illuion
Deceptive Design
Angle Illusion
Color Illusion
Edited Scene
Upside Down
Postive Negative
Circle Spiral
Miscellaneous
Figure 3: Categories in Illusion VQA--Vmpreee-naaeeeeece
Refer to Appendix D for details.
Researchers have shown that convolutional neural networks trained on ordinary images are susceptible to certain optical illusions, similar to humans (Gomez-Villa. et al., 2019; 2020; Afifi & Brown, 2019; Benjamin et al., 2019; Sun & Dekel, 2021). However, most previous experiments have focused on specific categories of optical illusions and their methodologies do not generalize to all types.
Zhang et al. (2023) is the first to probe how VLMs perceive optical illusions through natural language. The authors collected 16 root images of optical illusions and created 100 variants through manual editing. They ask the model a Yes/ No question to test whether a VLM has been "fooled" by an optical illusion. They tested multiple open-source VLMs (up to 13B parameters) and discovered that larger VLMs are more susceptible to optical illusions. Liu et al. (2023a) introduces HallusionBench, benchmark for testing VLM susceptibility to hallucination and optical illusions which includes 72 base images of optical illusions mostly derived from Zhang et (2023). Bitton-Guetta al. (2023) tests VLM comprehension on weird images, e.g. "A lit candle inside sealed bottle". They generate 500 such images using Text-to- Image generative models (Ramesh et al., 2021; Rombach et al., 2022) and employ human annotators to create descriptions of why the image is weird. They found that GPT3 with oracle image description achieves 68% accuracy, while humans achieve 95%.
Prior research has yielded multiple tools for synthetic optical illusion generation. Hirsch & Tal (2020) and Fan & Zeng (2023) automatically produce color-related and grating-related optical illusions, respectively. Gomez- Villa et al. (2022) explores creating illusions using
3

---

Published as a conference paper at COLM 2024
GANs. The primary limitation of synthetic illusion generation is the limited variety in the illusions they can create, especially compared to the wide spectrum of optical illusions discovered in the literature. As such, relying solely on existing synthetic illusion generation algorithms to evaluate VLMs severely limits the scope of the evaluation.
In contrast to prior works, we collect images of optical illusions from the internet to ensure data diversity and test VLMs along multiple axes. We manually create a multiple-choice question-answe -ansernnnn (QA) dataset where each question is crafted to have only one un- ambiguous answer while all other options appear plausible but are incorrect. Beyond comprehension, we test how well VLMs can locate geometrically impossible objects in an image. Our experiments probe VLM's dependence on language priors (Goyal et al., 2017) and feature entanglement (Tang et al., 2023) in the context of optical illusions.
3 The Illusion VQA Dataset
IllusionVQA is Visual Question Answering (VQA) dataset with two sub-tasks. The first task tests comprehension on 435 instances 12 optical illusion categories. Each instance consists of an image with an optical illusion, question, and 3 to 6 options, one of which is the correct answer. We refer to this task as Illusion VQA--Coprrehensioon The second task tests how well VLMs can differentiate geometrically impossible objects from ordinary objects when two objects are presented side by side. The task consists of 1000 instances following a similar format to the first task. We refer to this task Illusion VOA--SfttLoocalzztiin section details image collection and filtering, followed by QA generation and classification for the Comprehension task and procedural QA generation for the Soft-Localization task.
This
3.1 Image Collection and Filtering
We scraped more than 3500 images of optical illusions from multiple online repositories. We manually inspected each image to ensure that they were optical illusions. Since VLMs are trained on web-scraped data, dataset contamination image datasets. Liu et al. (2023a) notes that GPT4V recognizes all the illusion cases and knows their names. We opted for web data over synthetic generations capture the rich diversity of real-world optical illusions. To mitigate the risk of data contamination, we filter the dataset using GPT4V Specifically, we asked GPT4V to describe the image and inspect its response. We did not include an optical illusion in Illusion describe it accurately. In some cases, GPT4V was able to detect the presence of an optical illusion but could not describe it (e.g. it mistook an Impossible Cube (Penrose & Penrose, 1958) as a Necker Cube (Rosenholtz, 2011)). We included such cases in Illusion This leaves with 374 images of high-quality optical illusions, all of which pass the internal filter checks of the latest GPTTVV and Gemini-Pro APls. The source for each image is included in our dataset repository. We highlight few examples of filtered-out images in Appendix I.
major concern for web-scraped
if GPT4V could detect and
3.2 QA Generation for Illusion VQA-Comprehension
We pose the comprehension task in a standard multiple-choice VQA setting similar to (Gurari et al., 2018; Goyal et al., 2017). In some examples, this involves simply realizing that there is an illusion (Figure 2a). In other cases, we ask VLMs specific questions about the image to probe their understanding of illusions (Figure 2c).
We construct suitable questions that probe VLM's understanding of optical illusions beyond simply detecting their presence. Each question in Illusion has one correct answer and multiple wrong options. We write the questions and the correct answers and include definitively incorrect alternate options, rather than under-specified or ambiguous. We use the following methods and heuristics to generate the wrong choices:
1. We include the most likely misinterpretation of an optical illusion as an alternate whenever possible (e.g. the "Five, the center and the four corner regions" option in Figure 2b).
2. We prompt VLMs with the image, the question, and incorrect descriptions generated by different models as alternate options.
4

---

Published as a conference paper at COLM 2024
Q: Which object geometrically impossible?
The left object
B. The right object
C. Both objects
D. Neither object
A. The left object
B. The right object
Both objects
D. Neither object
Figure 4: Examples of IllusinnVQA-Soff-LLocliiaat
3. Since VLMs are prone to ignoring visual input (Goyal et al., 2017; Jabri et al., 2016), we include incorrect descriptions generated by VLMs when prompted with only the question and without the image.
4. We manually inspect and edit each incorrect option to ensure they are challenging, given the question and the image.
We create multiple questions for an image when appropriate and ensure that the questions are distinct and not simply rephrased versions of one another. We use the Perspectives AP2 to filter out toxic content in the questions, choices, and answers. Altogether, we created 439 question-answer pairs from the 374 carefully selected optical illusion images. Through rigorous filtering and question-answe -assween- generation techniques, we ensured every instance in the Illusion VQA dataset is highly challenging yet unambiguous. Our dataset includes 30 images similar to those present in the GVIL (Zhang al., 2023) and HallusionBench (Liu et al., 2023a) but differs in the specific question asked. Furthermore, our dataset evaluates VLMs through multiple-choice QA instead of Boolean QA in GVIL and HellusionBench.
3.3 Illusion VllusionVAA-Commme VQA-Comprehnnsion Classification
We adapt the classification methods proposed by Gregory (1997b) and Bach & Poloschek (2006) to our specific dataset and classify 12 classes shown in Figure 3. We opt for an appearance-baset classification that better fits our goal of studying VLMs in black-box setting, without access to weights, gradients, and activations. There is no clear consensus on classifying optical illusions, so some overlap between classes was unavoidable. In particular, the Deceptive Design category could be considered a subset of the Real Scene category and the same goes for Edited Scene for the Impossible Objects category. However, the images were visually distinct enough to warrant separate categorization. Moreover, the Deceptive Design, Edited Scene, and Upside Down categories are not generally considered as optical illusions in Cognitive Science. We include these images because they often considered optical illusions colloquially. We discuss the relevant literature on optical illusion and its classification in Appendices A and C.
3.4 Procedural QA Generation for Illusion VQA-Soft-LLcaaizattio
We procedurally combine images of "Impossible Objects" (Figure 3) to test if VLMs can locate illusions within a scene. Although VLMs have shown impressive capability in generating bounding boxes (Bai et al., 2023; Wang et al., 2023), GPT4 and Gemini-Pro were not designed for this task. We opt for an easier localization task by stitching together two images of geometrically impossible objects horizontally and prompting the VLMS to locate the impossible object (Figure 4). We refer to this task as Soft-Localization since the VLMs need only respond with "left" "right" (as opposed Precise Localization where a model outputs exact bounding box coordinates)
hhtpp://wwwwpprppptvvpp..cm//
5

---

Published as a conference paper at COLM 2024
To evaluate the abilitt of VLMs Soft Localization, we curate 40 diagrams of geometrically impossible objects and 20 diagrams of ordinary geometric objects. We further group ordinary and impossible objects that look similar shown in Figure 4. We generate 250 images for each of the four possible answers for 1000 images in total, prioritizing similar-looking pairs. As result, Illusion -oO--Sot-LLcaalizttio consists of 1000 label-balanced samples.
Soft localization of geometrically impossible objects challenging for VLMs because requires precise detection of contours and advanced geometric and spatial reasoning. We conduct control experiments for soft localization and verify the ability of VLMs at this task with ordinary, yet moderately challenging objects in the Appendix F.
4 Experimental Setup
Since open-source VLMs do not support multiple image input, we evaluate open-source VLMs in the 0-shot and closed-source VLMs in both 0-shot and 4-shot settings. For IllusionVQA-onee Comppo/hOO,,,mmmmeeennnnioooo we selected one sample from the four most common il- lusion categories as few-shot example. The Illusion Ulluion/QQ--Cmmreehh -Compreheennin test set contains 435 instances (370 images). Due to the inherent difficulty in describing optical illusions in natural language, we did not attempt Shain-of-Thought (CoT) evaluation on the compre- hension task. For VossionVVA-Sooon Soft-Localization we selected one instance of each of the four alternate choices as a few-shot example. We provide a standardized reasoning template (see Appendix E) for each example in the case of 4-shot+CoT evaluation. We use the default API arguments and generation parameters.
We resize all images to 512 pixels for all models while maintaining the aspect ratio. For the soft-localization task, we apply grayscale transform since the image color is not related to the geometry of an object. We used the latest versions of GPT4V and Gemini-Pro available at the time of writing (March 2024). We included updated (July 2024) results from GPT4o, Saude-3.5-Sonnee and other VLMsin Appendix J. We tested three open-source VLMs, namely InstructBLIP (Dai et al., 2024), LLaVA-1.5 (Liu et al., 2023b), and CogVLM (Wang et al., 2023).
Human Evaluation We employed three human evaluators and provided them with all samples from the comprehension task, well as 200 randomly chosen samples from the localization task. This was facilitated via a WebUI that tracks the time spent on each question in the background. Further details can be found in Appendix G.
Results
5.1 IllusioNVQA-Compreeenst
Table shows that the performance of VLMs on illusion comprehension lags significantly behind that of humans in all categories. The larger models outperform the smaller VLMs in most categories, and GPT4V outperforms other VLMs in all but three categories.
Human Performance The multiple-choice setting of Illusion VQA significantly eases the difficulty in describing illusions. As a result, our three evaluators achieve > 85% accuracy which increases to 91.03%. with majority voting and random tie breaks. The best-performing VLM, GPT4V with 4-shot outperforms human evaluators only two categories, namely Size and Color. We detail evter-evaluator agreement and response times in Appendix G.
GPT4V outperforms other VLMs. GPT4V leads other VLMs in both 0-shot and 4-shot set- tings by a wide margin, even though falls far short of human performance. We conducted further case analysis into GPT4V's output. GPT4V leads in ten of the twelve Illusion VQA categories and shows competitive performance in the remaining two. Categories such as Real-Scene, Size, Color, and Deceptive Design test VLM's perception of depth, color, and challenging scenes; skills that are particularly relevant when using VLMs as autonomous robots in the real world (Zitkovich et al., 2023; Wake et al., 2023). GPT4V maintains sub-

---

Published as a conference paper at COLM 2024
Class
0-shot
4-shot
Human
I-BLIP
LLaVA
Cog
Gemini
GPT4V
Gemini
GPT4V
Impossible Object
34.22
43.28
44.03
56.72
55.22
56.72
58.96
98.51
Real-Scene
64
26.56
42.19
34.38
46.88
57.81
46.88
54.69
98.44
Size
19.57
13.04
58.70
52.17
69.57
63.04
Hidden
45
44.44
42.22
42.22
42.22
51.11
48.89
46.67
100
Deceptive Design
37.84
43.24
45.95
64.86
67.56
72.97
94.59
Angle Illusion
26
30.77
38.46
30.77
53.85
69.23
50
84.62
84.62
Color
30.43
26.09
30.43
17.39
69.57
17.39
82.61
60.87
Edited-Scene
21
42.86
61.90
42.86
66.67
71.43
66.67
80.95
100
Upside-Down
7
42.86
71.43
71.43
57.14
71.43
57.14
71.43
100
Pos.-Neg. Space
57.41
42.86
71.43
85.71
57.14
71.33
85.71
100
Circle-Spiral
33.33
0.00
16.67
33.33
50
33.33
33.33
66.67
Miscellaneous
36.84
42.11
42.11
52.63
42.11
57.89
42.11
89.47
Total
435
34.25
40
38.16
51.26
58.85
52.87
62.99
91.03
Table 1: Performance of VLMs on IlluionVQAA-Comprehensitt
Categories where accuracy has improved using 4-shot prompting are underlined. 'Human' performance refers to the aggregated performance based on the majority vote of three human evaluators.
stantial leads in all four categories and is fairly resistant to challenging real scenes, with 10.93% lead over Gemini-Pro.
Small VLMs Among the three small VLMs (#parameters 14 17B), LLaVA-1.5 and CogVLM show similar performance in most categories while InstructBLIP shows the largest variation in different categories, performing significantly better (>5%) in comprehending Size and Circle-Spiral illusions while performance significantly worse in other categories.
In-Context Learning (ICL) We tested GPT4V and Gemini-Pro using 4-shot learning and observed marginal improvements in overall accuracy in However, the improvement was not consistent between different categories. For example, 4-shot GPT4V accuracy was lower than O-shot in three categories, namely Real-Scenc, Hidden Illusion, and Circle-Spiral Illusions. Gemini-Pro shows similar accuracy drops but in different categories. It is possible that ICL is not ubiquitous strategy because few- shot examples introduce additional language priors that bias the VLMs towards incorrect answers (Goyal et al., 2017; Jabri et al., 2016).
VQA--Coprehhenioon
Performance of VLMs on Ordinary Vertically-Flipped Images The Upside- category for images that depict two more intertwined entities depending on vertical orientation. As shown in Table 1, the smaller open-source VLMs perform competitively in this category while significantly lagging behind closed-source VLMs in most others. To validate this discrepancy, we curated a subset of VQA-v2.0 (Goyal et al., 2017) and tested LLaVA-1.5 and Gemini-Pro on both unedited and vertically flipped images. Table 2 shows that Gemini-Pro's accuracy drops by 15.5% on flipped images, while LLaVA-1.5's accuracy only drops by 6.5%, confirming our findings in Table
VLM
Orientation Normal Normal
Accuracy
VA-155
80
Gemini-Pro
79.5
VL1.5-
Vertically Flipped
73.5
Gemini-Pro
Vertically Flipped 64
Table 2: Performance of VLMs on a subset 200 instances from VQA-v2.0 (Goyal et al., 2017). The model outputs were manually evaluated.
5.2 Illusion Allsiion@QA-Sof-Looaliaaat
Table shows that the localization of optical illusions a significant challenge for VLMs but quite straightforward for humans. Small open-source VL.Ms, particularly, show close
7

---

Published as a conference paper at COLM 2024
VLM InstructBLIP
Prompt Type Accuracy
O-shot
24.3
0-shot
24.8
CogVLM
0-shot
28
0-shot
40
GPT4V
4-shot
46
4-shot CoT
49.7
O-shot
43.5
Gemini Pro
4-shot
41.8
4-shot + CoT
33.9
Human
100
Table 3: Performance of VLMs on VAASionnQQ-Soff-LLooooo
Human performance
denotes the majority-vote performance of three human evaluators on 200 random instances.
to random performance (25%). GPTV and Gemini-Pro fare somewhat better but fall short of human performance. On 200 randomly selected examples, all three human evaluators achieved perfect accuracy. Even with ICL and CoT, the performance of GPT4V in the MlusionVQA-Sie Sott--oof-LLooaizatioo dataset remains subpar, achieving only 49.7% accuracy This unexpectedly poor performance motivated us to conduct additional experiments, presented in Appendix F.
Large VLMs Can Localize Ordinary Objects But Not Illusions We created a new dataset for soft localization using different brands of sports cars and asked the VLM "Which car is [BRAND]?" with the same four options as Figure 4). This task remains moderately challenging for VLM since sports cars of different brands share the same basic shape and differ in details. Although smaller VLMs struggled with this task, both GPT4V and Gemini-Pro achieved 100% accuracy. GPT4V and Gemini's stellar performance in soft localization of ordinary objects makes their unexpectedly low performance on VIlusionVQA- Soft Localization surprising. This is likely because differentiating geometric illusions from ordinary objects requires spatial reasoning (a known weakness VLMs (OpenAI, 2023)) instead of feature extraction. We present the details regarding the dataset and the model performance in Appendix F.
In-Context Learning and Chain-of-Thoupee and Chain-of-Thought (CoT) (Wei et al., 2022) are widely used techniques that boost LLM performance. Research has shown that these techniques are effective in vision language tasks (Yang et al., 2023), but are still understudied.
In-context learning (ICL) (Brown et al., 2020)
Table 3 shows that Gemini-Pro performs better than GPT4V (+3.5%) when evaluated without ICL or COT. However, GPT4V shows the highest accuracy overall when evaluated with shot+CoT (+9.7% gain) while Gemini-Pro's performance degrades substantially (-9.6%). We hypothesize this is because GPT4V is likely the larger model and exhibits stronger emergent substantial number of instances in which the 0-shot evaluation yielded the correct answer, while 4-shot+CoT reasoning failed to do This implies that ICL and CoT are not universally applicable to IllusionVQA Wtw.VVAAAAOS-Loooiiiittttt
ICL abilities (Wei et al., 2023). As illustrated in Figure 5, there
Failure on In-Context Examples To probe the weakness of VLMs, we conducted additional experiments in which we asked GPT4V and Gemini-Pro with 4 examples (4 shots) and presented the same examples as questions. In this case, the exact image, the question, and the correct answer are in the context window the VLM. Surprisingly, neither GPT4V nor Gemini-Pro can successfully answer all questions. This highlights puzzling limitation of ICL for VLMs where models fail even when the correct answer is in the context which points to over-dependence on language priors (Goyal et al., 2017; Jabri et al., 2016) and visual information all but ignored. This phenomenon persists all the combinations we tried. We provide the prompt and result for one such combination in Appendix H.
Random guessing would yield 25% accuracy rate.
8

---

Published as a conference paper at COLM 2024
shot
A-T shot
A-hattCoT
4-hatt
6-sho-Ciat
shoc
4-shot
112
166
83
175
46
487
35
218
33
174
70
170
120
34
239
66
35
74
220
O-shot
S-shot
0-shat
(a) GPT4
(b) GPT4
(c) Gemini-Pro correctly ans.
(d) Gemini-Pro incorrectly ans.
correctly ans.
incorrectly ans.
Figure 5: Venn Diagrams showing the agreement between prompting techniques. There are instances where ICL and CoT cause the VLMs to answer incorrectly.
Discussion
6.1
IllssoonReeisstatt VLMs for Robotics
Vision Language Models (VLMs) are increasingly being deployed in real-world applications, particularly in robotics (Zitkovich et al., 2023; Wake et al., 2023), where they play a crucial role in enabling robots to navigate and interact with their environment. Embodied Al need to demonstrate consistent performance across a wide range of real-world situations, including those involving deceptive designs, optical illusions and obscured objects.
The Illusion VQA dataset has the potential to function as complete stress test to assess the robustness and adaptability of embodied VLMs. By assessing the performance of VLMs on this challenging dataset, researchers can pinpoint areas of weakness and enhance the models' capability to navigate perceptually ambiguous scenarios.
6.2 Thinking Fast and Slow Comparing VLM and Human Response Times
Kahneman (2011) popularized the concept of two modes of thinking: 'System 1', which is fast and instinctive, and 'System 2', which is slower, more deliberate, and logical. Due to their autoregressive architecture, current state-of-the-ar LLMs are primarily capable of 'System 1' thinking (Bubeck et al., 2023). Approximating 'System 2' reasoning with LLMs has motivated significant research efforts such as -hainhof-Thought (Wei et al., 2022), Tre--of-Thought (Yao et al., 2024), and other related methods.
Understanding and locating optical illusions requires deliberate and logical 'System 2' thought processes. Human evaluators spent an average of 14.99 seconds on each question from Illusion Salooo/A---ooomeesiiiiiii
and 5.68 seconds on Since GPT4V and Gemini-Pro APIs don't reveal exact inference times, we report API response times 4 In the 0-shot (4-shot) setting, GPT4V takes an average of 1.81 (3.27) seconds, and Gemini-Pro takes 3.82 (4.59) seconds regardless of the task. While API response times significantly overestimate model inference times, they are still notably faster than the time human spends deliberating on each question.
7 Conclusion
We demonstrate that current state-of-the-art VLMs struggle with understanding optical illusions and largely falter in identifying geometrically impossible objects. We find that GPT4V maintains substantial leads in most illusion types and that there is a noticeable performance gap between open-source and closed-source VLMs. We validate the effective- ness of few-shot learning for GPT4V and Gemini-Pro in the comprehension task but show that few-shot learning and Chain-of-Thougeee reasoning are not universally effective in the localization task. Additionally, we discover that GPT4V and Gemini-Pro fail to correctly answer illusion localization questions even when the correct answer is provided in the context. We release our dataset and evaluation code to stimulate further research.
"This includes image-upload times and internal load balanbagn delays, likely making API response times much slower than internal model inference times.

---

Published as a conference paper at COLM 2024
Limitation
Our current study presents valuable initial findings; however, acknowledge several limitations that present avenues for future research.
1. Scale of the dataset: The number of examples in the MalssionVOA-CCoppeheeeee
dataset is relatively modest due to the rigorous filtering process applied to the initial candidate set of over 3500 images scraped from the Internet. Despite extensive ef- forts, we could not find additional optical illusions that met our inclusion standards. One potential avenue for expanding the dataset in the future could be the use of synthetic optical illusions. However, the current capabilities of image generation models in creating novel, high-quality optical illusions are limited, making this approach challenging for the time being.
2. Evaluation Scope: Due to the OpenAl API rate limits (500 requests/day), we were unable to evaluate GPT4V in multiple evaluation runs. It is possible that more advanced evaluation techniques (Lei et al., 2024) or task-specific fine-tuning of open-source VLMs with sufficient data might yield improvements IllusionVQA. Soft-Localization.
3. Open-ended Question Answering: We primarily refrained from investigating open-ended QA due to the subpar performance of our baseline multiple-choice VQA setup. The complexity of open-ended generation is further heightened by the need for additional human evaluators to serve as judges.
Acknowledgement
We thank the contributors and the owner of moillusions.com for curating and hosting collection of high-quality optical illusions online. We thank Michael Bach for allowing us to gather optical illusions and their descriptions from his excellent repository. Lastly, we appreciate the efforts of our human evaluators who diligently evaluated the illusionVQQA dataset.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXio:2303.08774 2023.
Edward H. Adelson. Checker shadow illusion, 1995. URL https://persci. mit.edu/ gallery//hhekkershhabvw
Mahmoud Afifi and Michael S Brown. What else can fool deep learning? addressing color constancy errors deep neural network performance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 243-252, 2019.
Anthropic. Claude 3.5 sonnet, 2024.
URL
claude-3-$5 3-tie--3---oneett
Michael Bach and Charlotte M Poloschek. Optical illusions. Ado Clin Neurosci Rehabil, 6(2):
20-21, 2006.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar Xiv:2:230 129666 2023.
Ari Benjamin, Cheng Qiu, Ling-Qi Zhang, Konrad Kording, and Alan Stocker. Shared visual illusions between humans and artificial neural networks. In 2019 Conference Cognitive Computational Neuroscience, volume 10, pp. 2019-1299, Cognitive Computational Neuroscience, 2019.
10

---

Published as a conference paper at COLM 2024
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Ischannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXio preprint (XXiv:240.00722 2024.
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt,, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! vision-and-language benchmark of synthetic and compositional images. In Proceedings of the IEEE/CVF Interna- fional Conference on Computer Vision, pp. 2616-2627, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arv preprint ar arXiv:2303.1271, 2023.
Claus-Christian Carbon. Understanding human perception by human-made illusions.
Frontiers in human neuroscience, 8:566, 2014.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far to gpt-4v? closing the gap to commercial multimodal models with open-source suites. ar Xiu preprint arXio:2404.16821, 2024.
Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, and Aude Oliva. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific reports, (1):27755, 2016.
Bevil R. Conway, Akiyoshi Kitaoka, Arash Yazdanbakhsh, Christopher C. Pack, and Mar- garet S. Livingstone. Neural basis for a powerful static motion illusion. Journal of Neuroscience, 25663)65611-5666, 2005. doi: 105533/NEUROSSI1084000000 https://www www.:oscerrocccccooommom /etp:eiiorrrgoooeet///////////
URL
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Michael R.W. Dawson. Cognitive Impenetrabiliity pp. 1-3. Springer International Publishing, Cham, 2017. ISBN 978-3-319-47829-6. doi: 10.1007 /91999999--19977799666666 1596-1. URL https://doi. org.//...rg/1.... 1tt.://r00....11111133--------10
Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. See- ing it all: Convolutional network layers map the function of the human visual system. Neurolmage, 152:184-199 2017.
Jinyu Fan and Yi Zeng. Challenging deep learning models with image distortion based the abutting grating illusion. Patterns, 4(3), 2023.
James Fraser. new visual illusion of direction. British Journal of Psychology, 2(3):307, 1908.
Maurizio Gentilucci, Sergio Chieffi, Elena Daprati, M Cristina Saetti, and Ivan Toni. Visual illusion and action. Neuropsychologia, 34(5))369-376, 1996.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 580-587, 2014.
11

---

Published as a conference paper at COLM 2024
Alex Comez-Villa, Adrián Martin, Javier Vazquez-Corral, Marcelo Bertalmío, and Jesús Malo. On the synthesis of visual illusions using deep generative models. Journal of Vision, 22(():2-2, 2022.
Alexander Gomez- Villa, Adrian Martin, Javier Vazquez-Corral, and Marcelo Bertalmío. Convolutional neural networks can be deceived by visual illusions. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12309-12317, 2019.
Alexander Gomez-Ville Vonlz, Adrian Martín, Javier Vazquez-Corral, Marcelo Bertalmío, and Jesús Malo. Color illusions also deceive cnns for low-level vision tasks: Analysis and implications. Vision Research, 176:156-174 1744, 2020.
Yash Goyal, Tejas Khot, Douglas Summers Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the JEEE conference computer vision and pattern recognition, pp. 6904-6913, 2017.
Richard L. Gregory. Knowledge in perception and illusion. Philosophicaal Transactions of the Royal Society of London. Series B: Biological Sciences, 352(11599:1121-1127, 1997a.
Richard L Gregory. Visual illusions classified. Trends in cognitive sciences, 1(5):190-194, 1997b.
Richard Langton Gregory. Perceptual illusions and brain models. Proceedings of the Royal Society of London. Series B. Biological Sciences, (11244)) 1968.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEF. conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Uri Hasson, Talma Hendler, Dafna Ben Bashat, and Rafael Malach. Vase or face? neural correlate of shape-selective grouping processes in the human brain. Journal of cognitive neuroscience, 13(6):744-753, 2001.
Hermann von Helmholtz. Concerning the perceptions in general, 1867. Readings in the Itistoyy of psychology, 1948.
Elad Hirsch and Ayellet Tal. Color visual illusions: A statistics-based computational model. Advances in neural information processing systems, 33:9447-9458, 2020.
Allan Jabri, Armand Joulin, and Laurens Van Der Maaten.
Revisiting visual question answering baselines. In European conference on computer vision, pp. 727-739. Springer, 2016.
Charles H Judd. The muller-lyer illusion. The Psychological Review: Monograph Supplements, 1905.
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Gaetano Kanizsa. Subjective contours. Scientific American, 234(4):48-55) 53, 1976.
Xuanyu Lei, Zonghan Yang, Xinrui Chen, Peng Li, and Yang Liu. Scaffolding coordinates to promote vision-language coordination in large multi-modal models. Xiv preprint 1rXio:2402.12058, 2024.
Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. arXiv preprint ar arXio22300.1556. 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint ar 2r31o:2310.03744, 2023b.
Ben Lonnqvist, Alban Bornet, Adrien Doerig, and Michael H Herzog. A comparative biology approach to dnn modeling of vision: A focus on differences, not similarities. Journal of vision, 171) 2021.
12

---

Published as a conference paper at COLM 2024
OpenAI. Vision openai api, 2023. URL https://platformm. http:sss/ppaattoo.popo....o.oooo wtt.ppaa...cmooc///suuueeeeeeeee vision.
OpenAL. Gpt 4o, 2024. URL http:///pppaaaacccccn/eeeeeeeelr
Lionel S Penrose and Roger Penrose. Impossible objects: a special type of visual illusion.
British Journal of Psychology, 1958.
Baingio Pinna, Gavin Brelstaff, and Lothar Spillmann. Surface color from boundaries: a new 'watercolor 'illusion. Vision research, 41(00))269-2267, 2001.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to -o-iiaage generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEÉE/CVE conference computer vision and pattern recognition, pp. 10684-10695, 2022.
Ruth Rosenholtz. What your visual system sees where you are not looking. In Human Vision and Electronic Imaging XVI, volume 7865, pp. 343-356. SPIE, 2011.
R.N. Shepard. Mind Sights: Original Visual Illusions, Ambiguities, and Other Anomalies, with Commentary on the Play of Mind in Perception and Art. W.H. Freeman and Company, 1990. ISBN 9780716721345.. URL https://books: wwwooooooo..o.oo...eoeoooooooooe com bod/bookssidt1vvtlggga
Eric D Sun and Ron Dekel. Imagenet-trained deep neural networks exhibit illusion-like response to the scintillating grid. Journal of Vision, 21(11):15-15, 2021.
Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Türe. What the daam: Interpreting stable diffusion using cross attention. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3644-5659, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Rean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arviv preprint arXiv:2312.11886, 2023.
Nicholas J Wade and Patrick Hughes. Fooling the eyes: trompe l'oeil and reverse perspective. Perception, 28(9):1115-1113 1999.
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Gpt-4v(ision) for robotics: Multimodal task planning from human demonstration. ArXiv, abs/2311.12015, 2023. URL https://api: semanticscholar wwwpoolaarrrrrroo//oeooeoeeoo wrwp/oorpsSD:265222222 2652950111.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogylm: Visual expert for pretrained language models, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. thain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, al. Larger language models do in-context learning differently. arXiv preprint ar XxXv:2303.00846, 2023.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). aixiv preprint ar Xio:2309.17421. (1):1, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at COLM 2024
Yichi Zhang, Jiayi Pan, Yuchen Zhou, Rui Pan, and Joyce Chai. Grounding visual illusions in language: Do vision-language models perceive illusions like humans? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 5718-5728, 2023.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, al. Rt-2: Vision-languaag--ction models transfer web knowledge to robotic control. In Conference on Robot Learning, pp. 2165-2183. PMLR, 2023.
Friedrich Zöllner. Ueber eine neue art von pseudoskopie und ihre beziehungen zu den von plateau und oppel beschriebenen bewegangsphänomenen. Annalen der Physik, 186(7): 500-523, 1860.
Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7748-6758, 2023.
14

---

Published as a conference paper at COLM 2024
Supplementary Material: Appendices
A Preliminaries
This section covers the relevant cognitive science literature regarding optical illusions.
Optical Illusion An optical illusion refers to a visual perception phenomenon in which the visual system misinterprets reality, resulting in a disparity between what is perceived and what exists. Researchers have proposed that optical illusions mainly arise from discrepancies in a person's perceptual knowledge and conceptual knowledge (Gregory, 1997b; Carbon, 2014). As concrete example, our fast perceptual knowledge perceives Figure 2a as an elephant, yet upon closer inspection, we understand that the diagram does not align with our preexisting concepts regarding geometry.
Figure 6: Left: Kitaoka's Throwing cast nets' (Conway et al., 2005), an example of cognitive impenetrability. Even though we know it is static image, the illusion of motion persists. Right: Schroeder stairs (Rosenholtz, 2011), an example of an ambiguous or bistable image. The drawing may be perceived either staircase leading from left to right downward or as the same staircase only turned upside down.
Cognitive impenetrability a property of the basic information processes of cognition. If process is cognitively impenetrable, then the operation of this process cannot be altered by changing the contents mental representations (i.e., an agent's beliefs, desires, or goals). This is because the process is "wired in and its operation must be explained by
appealing to neuroscience (Dawson, 2017). Select visual illusions (Figure 6 Left) exhibit this phenomenon because the illusion persists even if we know or made aware of the illusion.
Ambiguous or Bi-stable Images are images that create ambiguity between or more distinct image forms. Classic examples of ambiguous images include Rubin's 's Vase (Hasson et al., 2001), Schroeder stairs (Figure 6 Right), and Necker's s Cube (Rosenholtz, 2011) among many others.
B Object Localization with VLMs
Object Detection (Girshick et al., 2014) is core computer vision task which models output the bounding box coordinates of objects in an image. Although pure vision models such as DETR(Zong et al., 2023) are state of the art in object detection, there has been considerable progress in object detection using pretrained VLMs (Wang et al., 2023; Bai et al., 2023). However, most VLMs do not natively support object detection and perform significantly worse than vision-only models. Consequently, we opt for simpler task, soft localization, where the VLMs are prompted output the approximate position of objects (i.e. left side or right side) instead of exact bounding box coordinates.
15

---

Published as a conference paper at COLM 2024
C Literature Review of Illusion Classification
Classifying optical illusions has posed significant challenges, with no definitive consen- sus reached within the scientific community. We review two prominent categorization approaches proposed by Gregory (1997b) and Bach & Poloschek (2006) as precursors to our categorization of IllusionVQA.
Gregory (1997b) proposed a four-fold categorization of optical illusions based on their underlying causes: physical, physiological, knowledge-based, and rule-based. Physical illusions, like those induced by fog or lens distortions, directly interfere with image for- mation. Physiological illusions, such as scintillations from pressure on the eye, arise from disruptions within the visual system. Knowledge-baset illusions, exemplified by inter- preting 2D sketches as 3D objects, leverage prior knowledge to misinterpret sensory input. Finally, rule-based illusions, such as Kanizsa triangles (Kanizsa, 1976), exploit the visual system's 's tendency to complete patterns or infer missing information, leading to illusory percepts. Gregory (1997b) proposes complementary classification of optical illusions based on appearance and rooted in natural language, namely ambiguities (Figure 6), distortions (Figure 2b), paradoxes (Figure 2a), and fictions (Figure 2c). We adopt this classification scheme for Illusion V 'QA in section 3.3.
16

---

Published as a conference paper at COLM 2024
D
IllssionVQA-CCopprheension Categories and Question Types
Class
Description
Example
Impossible Ob-
134
Objects that cannot exist in 3D space.
Penrose Triangle (Penrose & Pen- rose, 1958)
ject
Real-Scene
64
Hard interpret due to forced perspective, unnatural poses, objects being obscured, etc.
Size
46
Two objects with the same dimensions look different or vice versa.
Müller-Lyer illu- sion (Judd, 1905)
Hidden
45
(Silhouettes of) certain objects are present somewhere without being immediately ob- vious.
Deceptive sign
De-
37
Items have been painted to appear some- thing else.
Trompe-I' art (Wade & Hughes, 1999)
Angle Illusion
26 Parallel lines look curved or angled, and vice versa.
Zöllner illusion (Zöllner, 1860). Checker Shadow Illusion (Adelson, 1995)
Color
23
Different colors or shades appear the same, and vice versa.
Edited-Scene
21 Edited to depict something impossible al- though it may appear normal on cursory view.
Upside-Down
Two intertwined entities depending on ver- tical orientation.
Positive-
7 Images that create ambiguity between two or more distinct forms represented by two more colors.
Rubin's vase (Has- son et al., 2001)
Negative Space
Circle-Spiral
Circles appear as spirals due peculiar color or background pattern.
Fraser (Fraser, 1908)
Spiral
Miscellaneous
19
Total
435
Table Types of Illusions in Illusion IllusionVQAA-CCoppeeenninon Test Split
Major Question Types
Count
"What is unusual about this image?"
114
"Deccribe this image."
80
"What hidden in this image?"
22
Image-specific Questions
219
Table Question Types in IllnsoonVQACComppeheersioo Test Split.
17

---

Published as a conference paper at COLM 2024
E Few-shot and Chain-of-Thought Evaluation Prompts
O-shot Instruction
You'l 11 be given an image, an instruction, and some options. You have to select the correct one. Do not explain your reasoning. Answer with only the letter that corresponds to the correct option. Do not repeat the entire answer.
4-shot Instruction
You't 11 be given image, instruction, and some options. You have to select the correct one. Do not explain your reasoning. Answer with only the letter that corresponds the correct option. not repeat the entire answer. Here are few examples: {few-shot examples) Now you try it.
4-shot+CoT Instruction
You'll 11 be given image, instruction, and choices. You have to select the correct one. Reason about the choices in the context of the question and the image. End your answer with 'Answer': (letter_of_comm ofoooooorccccccoiiccee without the curly brackets. Here are few examples: (few-shot examples) Now you try it.
CoT Reasoning Template
The left object is a (description) which is (possiille/imoossible),, The right object is (description) which is (possible/imposiilee)
F Soft Localization with Ordinary Objects
Since the performance of VLMs on the Soft Localization task of IllusionVQA had been unexpectedly poor, we created simple auxiliary dataset to test the capability of VLM in locating ordinary objects. Specifically, we use the images of different brands of sports cars and ask the model locate certain brand. Since the two objects are visually similar, the task remains moderately challenging.
Q: Which car is Lamborghini?
A. The left car
B. The right car
C. Both cars
D. Neither car
Q: Which car a Porsche?
A. The left car
B. The right car
C. Both cars
D. Neither car
Figure 7: Soft localization of ordinary objects.
Figure 7 shows samples from this new dataset. We collect 400 instances for this task and test all VLMs in the 0-shot setting. Table shows that while small VLMs perform poorly this task, GPT4V and Gemini-Pro achieve perfect accuracy. This confirms that large VLMs can locate objects in scene.
18

---

Published as a conference paper at COLM 2024
VLM
Prompt Type Accuracy
InstructBLIP
0-shot
30
LL.
0-shot
29.25
CogVLM
0-shot
49
Gemini-Pro
0-shot
100
GPT4V
0-shot
100
Table 6: Accuracy of localizing ordinary objects.
G Human Evaluation
Pause
What is unusual about the monument?
The monument is actually composed pyramids.
The pillars depict an impossible goomelty/triden..
It is floating without touching the floor.
Nexti
Figure 8: WebUl presented to human evaluators.
We employed three nonauthor undergraduate students from the author's institution to evaluate how well humans do on comprehension and localization optical illusions. The evaluation was carried out without Internet access and the evaluators were not informed in advance that the task was related optical illusions. We presented each evaluator with WebUl (Figure 8) and with the following instructions.
Instructions to Evaluators: 'Each page will have question and an image of optical illusion followed by some options. Select the option that best answers the question. Illusions can be misleading, choose the option that reflects how entities in the image truly are, not how they might appear. Zoom in or out to get better view, if necessary. Each response is being timed internally, so use the "Pause" button when required."
All three evaluators were presented with the questions in the same order. Table shows that the individual accuracy of the evaluators is comparable. The pairwise Cohen's 's Kappa
19

---

Published as a conference paper at COLM 2024
Evaluator
Accuracy
Avg. Response Time
1
87.13
16.46
12.95 86.67
3
92.87
15.57
Table 7: Human performance and response time on Illusion Ilaasoo
between the evaluators are 0.808 0.796 and 0.773, indicating substantial level of inter- evaluator agreement On
all evaluators achieved perfect accuracy on the selected subset of 200 instances and the average response time was 5.68 seconds.
H Limits of In-Context Learning For Optical Illusions
Few-shot Examples:
Q: Which object is geometrically impossible?
A. The left object
B. The right object
C. Both objects
D. Neither object
C. Both objects
D. Neither object
В. The right object
A. The left object
GPT4-Vision Answers:
1/4 Correct
o
A. The left object
B. The right object
B. The right object
B. The right object
Gemini- Pro- Vision Answers
3/4 Correct
C. Both objects
B. The right object
B. The right object
A. The left object
Figure 9: GPT4V and Gemini-Pro consistently fail to locate geometrically impossible objects even when the correct answer is in context (4-shot).
TTo determine the agreement metric, we considered each question-option pair as singular question with yes/mn no answer which denotes whether evaluator has selected that option as the answer or not.
20

---

Published as a conference paper at COLM 2024
I Examples of Images Removed During Preprocessing
GAME HRO
THE NEW SEASON
MARCH 31
(d) Printed shadow de- sign
(a) Unclear edited
(b) Unclear edited
(c) Pareidolia
Figure 10: Examples of images we filtered out during rre-processing. These images were found in online archives of optical illusions and are colloquially considered illusions.
J Updated Illusion VOA--oomremmoonnnsiio Results
Class
O-shot
4-ahot
IntemVL12
Pali
Gemini
Claude 3.5
GPTTV
GPT4o
Gemini
Claude 3.5
GPTIV
GPT
Impossible Cloject
134
49.25
32,09
56.72
64.93
35.22
63.43
56.72
65.45
58.96
61.94
98.51
Real-Scene
Size
46
43:48
15.22
15.65
50.00
$8.70
45.65
52.17
90.45
69.57
93.47
63.04
Hidden.
Deceptive Design
37
37 84
32.43
64.86
70:27
20.27
72.97
67.57
87 57.57
72.97
78 38
94,59
Angle Ullusion
26.92
Color
26.09
34.78
17.39
65.22
69:57
52.17
17.39
96.9%
92.61
78.26
60.87
Edited Scene
66.67
42.86
66.67
61.90
71.43
66.67
71.43
90.95
85.71
100
Upside-Doee Down
65.71
42.86
57.14
85.71
71.43
71.43
57.14
95.71
71.43
42.86
100
Pos.Neg, Space
42.86
42.86
71.43
57.34
71.43
71.43
85.71
71.43
100
Cirde-Spiral
0.00
0.00
33.33
33.33
50.00
50.00
33.33
50.00
33.35
30.00
66.67
Miscelaanens
19
42.11
31,58
52.63
47.37
42.11
52.63
57 89
52
42.11
63
89.47
Total
435
45.06
31.26
51.26
39.08
38.85
62.53
$2.87
56.44
62.99
67.12
91.03
results updated to include state-of-the-art VLMs as of July 2024. Added evaluation of InternVLM22 (Chen et al., 2024), PaliGemma-3B (Beyer al., 2024), GPT-4o (OpenAI, 2024) and Claude 3.5 Sonnet Anthropic (2024). Categories where accuracy has improved using 4-shot prompting are underlined.
Table 8:
21


Reviews:
[{"review_id": "HLCVzZqmXy", "reviewer": "Reviewer_XYvp", "summary": "This paper introduces IllusionVQA, a dataset for testing the ability of VLMs to locate and comprehend challenging optical illusions. The dataset consists of two tasks IllusionVQA-Comprehension and IllusionVQA-Soft-Localization. It is an interesting and novel problem and the community will be benefited from this perspective.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer XYvp**\n\nThank you for appreciating our work on the “Soft Localization” benchmark which probes the VLM’s understanding of 3D geometry instead of commonsense reasoning or chart/graph understanding covered in previous studies. \n\n**Concern 1: “Potential to scale up the dataset is limited.”**\\\nWe agree that scaling up the IllusionVQA-Comprehension dataset is challenging due to the challenges mentioned in our Limitations section. Still, there is room for some enlargement using image manipulation such as image flipping, masking, color editing, etc but at the cost of dataset diversity. GVIL [1] expanded 16 root images to 100 to robustly test human-VLM agreement and HallusionBench [2] expanded 72 images to 144 to probe parametric memorization in VLMs. Adapting IllusionVQA-Comprehension images to use cases where diversity is less of a concern is a possible direction for future research.\n\n[1] Zhang et. al. - Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans? [EMNLP 2023]\\\n[2] Liu et al. - HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination & Visual Illusion in Large Vision-Language Models [CVPR 2024]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The dataset includes a diverse set of 12 distinct categories of optical illusions.\n\n2. The proposed problem is relatively hard for current models. The authors evaluate a wide range of open-source and closed-source VLMs. Experiments show that VLMs can locate ordinary objects accurately but struggle with optical illusions. \n\n3. The paper introduces a novel \"soft localization\" task that tests VLMs' ability to differentiate geometrically impossible objects from ordinary objects, which has not been explored in previous studies.", "reasons_to_reject": "1. Potential to scale up the dataset is limited. The dataset size is relatively small, but it is totally understoodable, due to the difficulty to find additional high-quality optical illusions that met the inclusion criteria, which limits the scale of the dataset. Although the authors mentions synthetic optical illusion generation could be a potential way, but current image generation models have limited capabilities to follow such instructions.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ae9nWljiSd", "reviewer": "Reviewer_9wgj", "summary": "This paper presents a novel dataset aimed at evaluating Vision Language Models (VLMs) on their ability to understand and localize optical illusions, featuring two principal tasks: comprehension and soft localization. These tasks test VLMs' proficiency in interpreting complex visual phenomena that are typically misleading to human vision. The dataset has been rigorously tested on various state-of-the-art VLMs, including GPT4V and Gemini-Pro. While these models generally perform well in standard object recognition, they exhibit significant difficulties with optical illusions, especially in tasks demanding intricate visual and spatial reasoning.\nOverall, this paper is well-structured and clear, with methodical descriptions of experimental methods, dataset construction, and results, enhancing its accessibility and understanding. However, this paper lacks detailed descriptions of data sources, raising concerns about its diversity and applicability for practical VLM applications.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer 9wgj**\\\nThank you for your valuable feedback. We appreciate the opportunity to clarify your concerns and improve our work based on your suggestions.\n\n**Concern 1: *“This paper lacks clear descriptions of the data sources for its optical illusion images, raising concerns about the diversity and representativeness of the dataset.”***\\\nThe majority of our dataset is from two online sources. 216 out of 374 images in IllusionVQA-Comprehension are from an image-sharing website about optical illusions. Most of the “colloquial” illusions in IllusionVQA ( real-scene, edited-scene, hidden, deceptive-design) are from this source. A further 27 images are from a second website maintained by an optical illusions expert. This website was the primary source for formal optical illusions (color, angle, and size categories). The remaining 131 images are from secondary sources such as Pinterest and Imgur. \n\nWe have **refrained from specifically mentioning the two primary website names because we contacted the website administrators** and obtained permission to collect images. We will acknowledge their invaluable contributions, clarify data sources, and include all URLs in the final versions of our manuscript and dataset.\n\n**Concern 2: *“There is no detailed methodology on ensuring data comprehensiveness, which is crucial for a varied visual category like optical illusions.”***\\\nWe scraped all relevant online repositories and collected more than 3500 images, which we filtered down to 374. We encountered repetitions or trivial variations when we attempted to collect more images. Likewise, procedurally generated optical illusions are not diverse enough to warrant inclusion in our dataset. To the best of our knowledge, ours is the most comprehensive illusion dataset available. We respectfully invite you to review Section 3.1 for details on filtering and our Limitation section for a detailed explanation of the challenges involved in expanding the dataset further.\n\n**Concern 3: *“The study does not discuss whether the illusions are representative of real-world scenarios, which would be vital for practical applications of VLMs. “***\\\nWe believe that certain categories such as real scenes, edited scenes, deceptive design, and hidden illusions are particularly relevant to real-world applications of VLMs. The ability to comprehend challenging but real image compositions and deceptive designs is crucial for VLM applications in robotics, as we highlight in Section 6.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The IllusionVQA dataset introduces a unique challenge to the field of VLMs by focusing on optical illusions. This dataset differs from traditional VQA datasets by specifically testing the models' comprehension and spatial reasoning abilities through its two main tasks: comprehension and soft localization. These tasks assess VLMs' capability to interpret and localize elements within images that are inherently deceptive, extending beyond basic object recognition to probe deeper into advanced visual processing.\nInitial testing on the IllusionVQA dataset highlights significant shortcomings of contemporary VLMs, including GPT-4V. The underperformance compared to near-perfect human results underscores the current gap in VLMs' ability to handle complex visual inputs.These findings point to a pressing need for advancements in VLM architectures and training methods.", "reasons_to_reject": "This paper lacks clear descriptions of the data sources for its optical illusion images, raising concerns about the diversity and representativeness of the dataset. There is no detailed methodology on ensuring data comprehensiveness, which is crucial for a varied visual category like optical illusions. Furthermore, the study does not discuss whether the illusions are representative of real-world scenarios, which would be vital for practical applications of VLMs. Addressing these issues could enhance the dataset’s utility and relevance, providing a stronger benchmark for VLMs in practical environments.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6pehoNs9AW", "reviewer": "Reviewer_L4KJ", "summary": "This paper introduces a new benchmark for evaluating the performance of vision-language models under visual illusions through standard multiple-choice VQA tasks. The authors measure a wide range of model performances under different settings, such as the inclusion of In-Context examples and Chain-of-Thought prompting.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer L4KJ**\\\nThank you for your constructive review. We appreciate the opportunity to clarify our decisions regarding dataset curation and annotation methodology.\n\n**Concern 1: Data Contamination and GPT4V bias.**\\\nWe fully acknowledge the risks of data contamination and bias when using web-sourced data and have taken steps to mitigate these concerns during the dataset curation process. We opted for web data over synthetic generations to capture the rich diversity of real-world optical illusions. Certain types of illusions, such as impossible objects, challenging real-world scenes, or deceptive designs, cannot be generated synthetically. IllusionVQA-Comprehension covers 12 distinct categories, making it larger and more comprehensive than previous datasets focused solely on color and size illusions.\n\nTo mitigate contamination, we removed any optical illusions that the state-of-the-art GPT-4V model could answer correctly zero-shot without being provided options.  This filtration step ideally removes illusions for which GPT-4V memorized the correct explanations. Since GPT-4V was the best-performing VLM  in our pilot experiments, its zero-shot capability was the best proxy for measuring potential data contamination. Without such model-based filtration, there would be no reliable way to assess and mitigate contamination risks. Notably, this implicitly biases the dataset *against* GPT4V since only illusions it could not comprehend were included in the dataset. \n\n**Concern [2.1]: *“the most likely misinterpretation of an optical illusion\" (the one humans perceive?)”***\\\nYes! We refer to human misperception in that case. We also include zero-shot wrong answers generated by VLMs as alternate options as outlined in Section 3.2.\n\n**Concern [2.2]: *“The authors assume that the model should not experience illusions…”***\\\nThat’s an excellent observation! We fully agree that there is no right answer for how VLMs should interpret optical illusions. As such, following HallusionBench [1],  we used \"objective reality\" as the benchmark for IllusionVQA. We did not include any images where we could not deduce objective reality solely from the images (Please refer to Appendix I for concrete examples). We likewise instructed human evaluators to answer according to objective reality (Please refer to Appendix G).\n\n\n[1] Liu et al. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination & Visual Illusion in Large Vision-Language Models [CVPR 2024]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The mined dataset of 374 examples can be a useful resource for the community. \n2. The study on the effects of in-context examples and chain-of-thought prompting reveals interesting phenomena, such as in-context examples not always being helpful.", "reasons_to_reject": "1. Data contamination and bias in the dataset: GPT-4V is used as a filter in the dataset creation process, which likely results in a dataset biased towards GPT-4V's capabilities. Additionally, the majority of the data is sourced from the web, which likely comes with data contamination issues and is already part of many models' pre-training data. In comparison, prior works like GVIL/Hallusionbench use synthetic data to mitigate this problem.\n2. The authors assume that the model should not experience illusions, which is still a topic open for debate. As discussed in GVIL, it remains an open question whether we want models to \"experience the same kinds of illusions as humans do, or to faithfully represent reality?\" This paper directly uses \"the most likely misinterpretation of an optical illusion\" (the one humans perceive?) as a negative example, presupposing a specific stance on this issue.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vn8SA93MJh", "reviewer": "Reviewer_7WKq", "summary": "This paper introduce IllusionVQA, a novel dataset designed to rigorously test the ability of VLMs to locate and comprehend challenging optical illusions. The authors comprehensively test a wide range of open-source and closed-source VLMs and find some interesting results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "**Dear Reviewer  7WKq**\\\nThank you for your review. We appreciate the opportunity to respectfully highlight the contributions of our work and the differences from other datasets, and we hope this addresses your concerns. We would kindly invite further discussion, as we believe our dataset is quite different from existing benchmarks. \n\n**Concern [1]: *“The only difference is that Hallusionbench is binary answer while IllusionVQA is multiple choice questions.”***\\\nWhile HallusionBench is an excellent investigation of VLM capabilities, there are major differences in the dataset, methodology, and overarching goal of our work.\n\n**1. Investigation Goal:** HallusionBench aims to answer *“when and why VLMs hallucinate”* while our work investigates *“how well VLMs can deduce objective reality from images of optical illusions”*. The authors of HallusionBench did not differentiate between optical illusions and other image types such as text and posters during evaluation.\n \n**2. Dataset Size and Diversity**: IllusionVQA-Comprehension (374 images and 12 categories) is much larger and more diverse than HallusionBench-Illusions (72 root images and 2 categories).\n\n**3. Data Contamination:** The authors of HallusionBench note that GPT-4V recognized the optical illusions in all 72 root images in their dataset. We filtered out any image that GPT-4V was able to describe without options, thus largely mitigating dataset contamination.\n\n**4. Human Evaluation:** HallusionBench aims to test VLM hallucination, so they do not perform a controlled human evaluation on their dataset. In contrast, we employed three human evaluators to establish that VLMs are still far behind humans when deducing objective reality in most illusions.\n\n**5. Soft Localization:** Soft-localization of geometrically impossible objects is a significant and novel part of our paper’s contribution. Determining which object is geometrically impossible is a complex task that requires a deep understanding of 3D geometry and isometric perspectives.  In Table 3, we show that all VLMs struggle with soft-localization while humans achieve perfect accuracy. \n\n**6. Advanced Prompting Techniques:** To the best of our knowledge, we are the first to test the effect of In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting on optical illusion tasks for VLMs. We discover that ICL and CoT are not universally effective and degrade the performance of Gemini-Pro on IllusionVQA-Soft-Localization  (Please see Table 3).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset is interesting and challenging enough. The results are good.", "reasons_to_reject": "The paper is very similar to hallusionbench. The only difference is that Hallusionbench is binary answer while IllusionVQA is multiple choice questions.", "questions_to_authors": "Please list more difference with Hallusionbench.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a new multiple choice VQA task, IllusionVQA, which
challenges models in understanding unusual images of various
categories.  A wide range of models perform poorly compared to humans
in both "explain why the image is unusual multi-choice"
("comprehension") and "pick which of these two objects is unusual"
("soft localization") settings. They will release their curated corpus.

Weaknesses included:

- Dataset size (374 images, XYvp: I will note that it would have been
  nice for the authors to include some notion of variance in their
  accuracy measures)

- a potential bias because GPT-4 was used during data curation (L4KJ);

- similarity to hallusionbench (7WKq);

- details of the dataset collection could have been expanded (9wgj);

I read over the responses the authors raised when addressing these
shortcomings, and am convinced by the responses.

Overall, the case for accepting IllusionVQA is clear: a task trivial
for the sighted human annotators, on which machines perform poorly, is
a valuable benchmark for the community to examine in more detail. And,
the fact that this dataset is split into various illusion categories
(compared to prior work) provides additional capacity for error
analyses in the future (e.g., it appears that some categories of
illusions are easier/harder than others for models).

[At least one review was discounted during the decision process due to quality]
Decision: Accept