言語処理学会 第30回年次大会 発表論文集（2024年3月）

Find–the–Common: Benchmarking and Assessing InductiveReasoning Ability on Vision-Language Models
Shi Yuting1 , Wei Houjing1 , Jin Tao1 , Zhao Yufeng1 , and Naoya Inoue1,2
1 JAIST 2 RIKEN{s2210096,houjing,morgan,yfzhao,naoya-i}@jaist.ac.jp

Abstract
Recent advances in Instruction-ﬁne-tuned Vision and Language Models (IVLMs) have revolutionized the landscapeof integrated vision and language understanding. However, Inductive Visual Reasoning—a vital skill for textimage understanding—remains underexplored due to theabsence of benchmarks. So, in this paper, we introduce
Find–the–Common (FTC): a new vision and language task
for Inductive Visual Reasoning. In this task, models are
required to identify an answer that explains the common
attributes across visual scenes. We create a new dataset
for the FTC and assess the performance of several contemporary approaches including implicit reasoning, symbolicreasoning, and implicit-symbolic reasoning with variousmodels. Extensive experiments show that even state-ofthe-art models like GPT-4V can only archive with 48%
accuracy on the FTC, for which, the FTC is a new challenge for the visual reasoning research community. Our
dataset is available online.

1Introduction

Instruction-tuned Vision Language Models (IVLMs),such as MiniGPT-4 (1), InstructBLIP (2), LLaVA (3), Visual ChatGPT (4), and GPT-4V (5) have been demonstrated with excellent performance on Vision and Languagetasks (6; 7; 8) and also show the strong zero-shot generalization ability to unseen tasks, such as writing HTMLs
based on a hand-drawing sketch and explaining the implicit
meaning of memes (1; 3).
Assessing VLMs involves various tasks testing perceptual skills like object recognition and complex analyses
including counting and reasoning. Notable benchmarks
like Compositional Visual Reasoning (9) evaluate compositional reasoning, Visual Spatial Reasoning (3) focuses on

Q: What is the common regularity between four 3Dscenes?

a. The yellow object on the far right among all yellow objects
is a cylinder.
b. The red sphere is in the forefront.
c. The cube on the far left among all cubes is green. ✓d. The object farthest away is purple.

Figure 1: Example of the Find–the–Common task. Given
four 3D scenes and a 4-choices question, the task is to perform inductive reasoning to identify the correct statement
describing the common regularity between the 3D scenes.
The choices consist of one correct choice (c) two wrong
choices (a, d), and a decoy choice to fool models (b).
spatial understanding, and Visual Commonsense Reasoning (10) tests knowledge beyond visuals. Despite theircomprehensiveness, these benchmarks do not fully address deductive reasoning, where models derive conclusions from given premises, often employing commonsense
knowledge. Another important type of reasoning is inductive reasoning, which aims to generalize a group ofﬁnite observations to induce general rules in a bottom-up
fashion (11). In the context of vision processing, we candeﬁne such competencies as Visual Inductive Reasoning,which requires understanding multiple visual scenes andthen reasoning out common conclusions from those diﬀerent scenes. We argue that visual inductive reasoning has
been underexplored despite its importance, which raises
the following question: Given a set of visual scenes, canVLMs identify a common rule describing these diﬀerent
scenes?
To address this issue, we propose a novel benchmark,termed Find–The–Common (FTC): a new task that required understanding and reasoning across 3D visual― 834 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

scenes and text reading comprehension. An example isTable 1: Attributes of objects and their values.

shown in Fig. 1. Given four 3D scenes and a multiplechoice question, the task is to perform inductive reasoning
to identify the correct statement describing the common
regularity between the 3D scenes. Our contributions canbe summarized as follows:• We introduce a novel benchmark Find–the–Commonfor evaluating the inductive visual reasoning capabilities of IVLMs (Sec. 2.1, Sec. 2.2). Our dataset consists of 353 instances, each of which provides four 3D
scenes consisting of 2-6 objects and a four-choices
question, including a decoy choice that is partially

AttributesValues

ColorShapeSizePosition

Red, Green, Blue, Yellow, PurpleCube, Sphere, Cylinder, Cone
Small, Medium, Large(1,1), (1,2), ..., (7,7), (8,8)

Question and Answer.

An answer is a set of choices

𝐶 = {𝑐, 𝑤 1 , 𝑤 2 , 𝑑} represents the regularities within{𝑆1 , 𝑆2 , 𝑆3 , 𝑆4 }. Four choices {𝑐, 𝑤 1 , 𝑤 2 , 𝑑}, comprising
one correct choice 𝑐, two wrong choices 𝑤 1 , 𝑤 2 , and one
decoy choice 𝑑. The decoy choice is designed to be partially true for given scenes, ensuring models examine alltrue in scenes. Our dataset is available online.
• We adapt three approaches for evaluation: (i) Implicit Reasoning, (ii) Symbolic Reasoning , and (iii)
Implicit-Symbolic Reasoning with the current latest

scenes to determine the correct choice.
Given a question 𝑄, models aim to identify an answer 𝑐 ∈ 𝐶 that holds true across all the provided scenes

state-of-the-art IVLMs. Our extensive experiments

𝑆1 , 𝑆2 , 𝑆3 , 𝑆4 . An example is given in Fig. 1.

show that InstructBLIP，LLAVA, and GPT-4V reveal

2.2

Dataset Creationthat even state-of-the-art IVLMs struggle to solve theFTC task (Sec. 4). Our detailed analysis also ﬁnds thatGPT-4V exhibit tendencies to “hallucinate”, despite
their demonstrated proﬁciency in text-based inductive

adopt a two-step approach: (1) Answer Generation: generate choices based on set pre-deﬁned linguistic rule, and(2) Scene Generation: generate scenes that satisfy thecorrect choice and do not satisfy the wrong choices.

reasoning.

2

To generate 3D scenes and multiple-choices answers, weBenchmark: Find–the–CommonAnswer Generation.

We construct 13 linguistic tem-plates with attribute placeholders at three levels:

2.1

Task Formulation• Four one-attribute template (e.g., “All objects are[Shape].”).

The benchmark focuses on the task of visual inductive reasoning, which involves extracting principles from speciﬁc• Six two-attributes template (e.g., “All objects are[Color] [Shape].”).

instances. Considering the challenges brought about by• Three three-attributes template (e.g., “The [position]

the complexity and nuances in real photographs, we ini-[color] is [shape].”).

tiate our research from simpliﬁed 3D object scenes. Thisapproach allows us to construct inductive reasoning tasks

We then randomly select a template and ﬁll in the place-with clarity. By omitting extraneous details, we can con-holders with diﬀerent values. For [position], we use rela-centrate more on evaluating the reasoning performance oftive choices like “on the far left”, “on the far right”, “fore-IVLMs. More speciﬁcally, we formulate the FTC task asfront”, and “farthest away” for diversity.
Scene Generation. We generate two types of 3D

follows: Given an input instance 𝑍 = (𝐼, 𝑄, 𝐶) consistingof an image 𝐼, a question 𝑄 with respect to a four-choiceanswer 𝐶.

scenes: (i) one scene satisfying 𝑐 but not 𝑑, 𝑤 1 , 𝑤 2 , and(ii) three scenes satisfying 𝑐, 𝑑 but not 𝑤 1 , 𝑤 2 . We formu-

Each image consists of four 3D scenes 𝐼 =late this as a constraint satisfaction problem using Answer{𝑆1 , 𝑆2 , 𝑆3 , 𝑆4 }, each scene containing 2-6 objects. Eachobject has four attributes including Color, Shape, Size

Set Programming (ASP)(12), a logic-based framework forand Position, and each attribute corresponds with one

swer set corresponds to one scene conﬁguration. We deﬁnevalue. We defer to Tab. 1 for further details.

predicates like shape(X, S) i.e., object X has shape S andImage.

such problems. We create an ASP program where each an-encodes rules as ASP rules. An example of ASP is shown― 835 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

in table4 in Appendix. To generate scenes that satisfy (or2(c)). Speciﬁcally, in addition to generated scene descrip-

do not satisfy) these rules, we use integrity constraints,

tions, 3D scenes are also provided to VLMs for predicting

such as:the ﬁnal answer. We showcase how it works on GPT-4V• To satisfy the rule “All objects must be red”:

in this work. The goal is to assess the models’ proﬁciency:- not r all SH(red).

in integrating visual and textual information to deduce the• To not satisfy the rule “All red objects must be cube”:

correct option.

:- r all CL are SH(red, cube).
We then randomly sample one answer set using clingo1）
and convert it back to a scene conﬁguration. Scenes are
rendered using pyrender2）. The ﬁnal dataset comprises
353 instances.

2.3

Dataset Quality

To ensure that our visual inductive reasoning problems
are consistently solvable by humans, we conduct a human evaluation study. We randomly sample 100 instances
and ask two graduate school students to solve them. Theinter-annotator agreement between these evaluators indicates Cohen’s Kappa (13) of 0.92, indicating almost agreement. The accuracy scores of the two evaluators are 0.98
and 0.88, respectively. An evaluator with an accuracy of0.88 is frequently fooled with decoy choices, which resultsin lower accuracy.

3ApproachFigure 2: Baseline Reasoning

We employ three approaches to assess the zero-shot generalization capability of IVLMs on visual inductive reasoning:
Implicit Reasoning.

Experiment

In this section, we present our experimental setup andTests the models’ ability to iden-tify common rules among scenes via utilizing visual perception (Fig. 2(a)). The task involves selecting the correct
answer from the given four options by reasoning from thedistinct four 3D scenes.
Symbolic Reasoning.

4results, focusing on evaluating three state-of-the-art Integrated IVLMs using the baseline approaches outlined
in Sec. 3, comparison with human evaluation is also included. Also, we build ground truth experiments to create
a controlled environment that isolates the models’ logical

Evaluates the models’ capac-reasoning faculties.

ity to convert visual data into textual descriptions for logicalreasoning (Fig. 2(b)). Initially, four scenes are fed together
into IVLMs to generate their scene descriptions. Thenthese descriptions, along with four options and a question,
are processed by a Large Language Model (LLM) to produce the correct choice.

4.1

Setting

To evaluate models on our Find–the–Common benchmark,we randomly sampled 50 instances and tested them by using
three baseline approaches discussed in Section 3, using one
or more of three types of Visual Language Models (VLMs):

Implicit-Symbolic Reasoning.

Can be seen as acombination pipeline of the above two approaches, where(i) InstructBLIP (2), a smaller IVLM; (ii) LLAVA, which

the four 3D scenes are provided to models twice (Fig.

and (iii) GPT-4V(ision)3）, an extremely large VLM. We1） https://potassco.org/clingo/
2） https://github.com/mmatl/pyrender

combines a vision encoder with Large Language Models;3） https://openai.com/research/gpt-4v-system-card― 836 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

use Accuracy for evaluation. Further details of prompts

scriptions: Automatically generated descriptions by VLMs

and model parameters, can be found in the Appendix. Ad-compared to actual scenes show a noticeable accuracy gap,

ditionally, we use Chain-of-Thought (CoT) prompting (14)around 40% regardless of the method used. A manual re-without any few-shot demonstrations (henceforth, Zero-view of GPT-4V CoT-generated descriptions showed 82%

shot CoT) in implicit reasoning.

(41/50) cases of object hallucination, suggesting potential

4.2

misinterpretations in complex visual information process-Results and Analysising.
3) Language module ability in VLMs aﬀects reason-

Table 2: Accuracy of diﬀerent models and approaches
ApproachSymbolic

Implicit

Implicit-Symbolic

Model
GPT-4V
InstructBLIP
LLaVA
GPT-4VGPT-4V (CoT)InstructBLIP
InstructBLIP (CoT)LLaVA
LLaVA (CoT)Human
GPT-4Ving: InstructBLIP and LLAVA struggle with COT under-

Accuracy
48.0%
12.0%
44.0%
42.0%
24.0%
30.0%
91.5%
46.0%

standing, with LLAVA capable of generating, albeit inaccurately, JSON scene descriptions. In contrast, GPT4V demonstrates stronger COT understanding and JSONﬁle generation. GPT-4’s 74% accuracy in the SymbolicGround truth (Table 4) indicates its proﬁciency in processing JSON-formatted textual information for linguistic
inductive reasoning tasks.

5The performance of models for three baselines is shown in
Table 2. InstructBLIP and LLAVA, as smaller IVLMs, facechallenges, achieving only 21% and 30% accuracy respectively in the Implicit baseline, indicating limited accuracy
compared to the human baseline. GPT-4V achieves 44%accuracy, showing improvement but still lagging behind
human performance. These results suggest limited zeroshot generalization capability in current visual instruction
ﬁne-tuning methods for visual inductive reasoning.

Conclusion
In our Find–the–Common benchmark test, the eval-uation of IVLMs’ visual inductive reasoning capabilities
revealed key ﬁndings. Even advanced models like GPT-4V,
while making progress in visual tasks, still face signiﬁcantchallenges compared to the human baseline, particularly inobject detection and scene interpretation. This highlights
the need for improvement in the ﬁeld of visual inductivereasoning.
Our study also underscores the importance of accurate
scene information in enhancing model performance andthe crucial role of eﬀective interaction between images andTable 3: Ground Truth Approach
ModelAccuracy

text prompts in increasing accuracy. Additionally, smaller

Implicit-Symbolic
Symbolic-Ground truth
Implicit-Symbolic-Ground Truth

46.0%
74.0%
92.5%

models like InstructBLIP and LLAVA show deﬁciencies
in handling complex reasoning tasks, indicating a need forfurther optimization in model design.

To analyze the reasoning capabilities of VLMs further,
we conducted two experiments, replacing VLM-generated
scene descriptions with ASP-generated scene parameters.
Using GPT-4V as the VLM and GPT-4 as the LLM, we

All of these ﬁndings inspired the following for our future work. First, given the challenges with multiple visual
scenes and hard-to-perceive objects, reﬁning the datasetwith a well-designed hierarchy complexity will better evaluate VLMs’ adaptability and comprehension. Then, con-observed:
1) Including image information signiﬁcantly enhancesmodel performance: In Table 4, the Implicit+Symbolic +Ground Truth approach shows 92.5% accuracy, a 16.5%
increase over the 76% in the Symbolic baseline groundsidering the object hallucination tendencies observed, apotential direction for improvement in the future lies intraining regimes that emphasize precise visual reasoning
over textual inference.

truth, indicating images provide essential background information or visual cues.
2) A gap exists in reasoning accuracy using scene de-― 837 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number 19K20332.

Additionally, we thank Surawat

Pothong and Tien Dang Huu, members of RebelsNLU labfor their support.

References[1]Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding withadvanced large language models. arXiv preprint arXiv:2304.10592,2023.
[2]Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong,Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and StevenHoi. Instructblip: Towards general-purpose vision-language models
with instruction tuning, 2023.
[3]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visualinstruction tuning, 2023.
[4]Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, ZechengTang, and Nan Duan. Visual chatgpt: Talking, drawing and editing
with visual foundation models. arXiv preprint arXiv:2303.04671,2023.
[5]Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching
Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary
explorations with gpt-4v(ision), 2023.
[6]Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu,and Enhong Chen. A survey on multimodal large language models.
arXiv preprint arXiv:2306.13549, 2023.
[7]Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng
Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlmehub: A comprehensive evaluation benchmark for large visionlanguage models. arXiv preprint arXiv:2306.09265, 2023.
[8]Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, MengdanZhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng,et al. Mme: A comprehensive evaluation benchmark for multimodal
large language models. arXiv preprint arXiv:2306.13394, 2023.
[9]Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li FeiFei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic
dataset for compositional language and elementary visual reasoning,
2016.
[10]Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From
recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision andpattern recognition, pp. 6720–6731, 2019.

[11]Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik
Cambria. Are large language models really good logical reasoners?
a comprehensive evaluation and beyond, 2023.
[12]Ilkka Niemelä. Answer set programming: A declarative approach tosolving challenging search problems. In 2011 41st IEEE International Symposium on Multiple-Valued Logic, pp. 139–141, 2011.
[13]M.L. McHugh. Interrater reliability: the kappa statistic. Biochem
Med (Zagreb), Vol. 22, No. 3, pp. 276–282, 2012.
[14]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei
Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought
prompting elicits reasoning in large language models. Advancesin Neural Information Processing Systems, Vol. 35, pp. 24824–24837, 2022.

― 838 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 4: ASP program example
Template: All objects are [shape]:
Rule: r all SH(S) :- shape(X, S) : obj(X)Template: A [color] object exists, and all [color] objects are [shape]:
Rule: r all CL are SH(C, S) :- r CL exists(C), shape(X, S) : color(X, C).

Table 5: Prompt examples for various baseline approaches
Implicit Baseline Approach
What is the common regularity between four 3D scenes? Choose one correct answer from the following choices:(a) The green cube is on the far left.
(b) A blue object exists.
(c) The purple cylinder is farthest away.
(d) The object in the forefront is cone.
Implicit Baseline Approach (COT)
What is the common regularity between four 3D scenes? Choose one correct answer from the following choices:(a) The green cube is on the far left.
(b) A blue object exists.
(c) The purple cylinder is farthest away.
(d) The object in the forefront is cone.
Let’s think step by step.
Symbolic Baseline Approach
Step 1:
Please analyze the provided image with 4 scenes of objects on a ﬂat surface...

Table 6: Hyperparameter Settings for Various Models
ModelHyperparameters

InstructBLIP

num beams = 5 max new tokens = 500, min length = 10, top p = 0.9, repetition penalty= 1.5, length penalty = 1.0, temperature = 1

GPT-4Vtemperature = 0.7, max tokens = 100, top p = 1.0, frequency penalty = 0.0, presence penalty = 0.0

LLAVA

do sample = True, temperature = 0.2, max new tokens = 1024, use cache = True, stopping criteria = [stopping criteria]

GPT-4

temperature = 0.7, max tokens = 1000, top p = 1.0, frequency penalty = 0.0, presence penalty = 0.0― 839 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).