SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The dataset has a balanced distribution of question types. The basic questions account for  $20\%$  of the dataset, the two-operator compound questions account for  $30\%$ , the three-operator compound questions account for  $40\%$ , and the conditional logical questions account for  $10\%$ . In particular, the LoRA dataset includes  $4\%$  SWRL rule-based conditional logical reasoning questions, and these can be divided into seven different categories as shown in Table 4.
Logical Reasoning Category	Example	Answer	Inference Steps
Hypothesis replacement	To cook spicy noodles, if we do not have chilli, can we use any other items in the image instead?	spicy sauce	6
Functional comparison	To boil or fry eggs, which tools in the image can be used?	pot or pan	3
Similarity	To cook fried eggs with rice noodles, what alternative ingredients can replace rice noodles for a similar dish?	pasta	5
Avoiding	Which food in the image should be avoided when cooking a meal for vegetarians?	meat	3
Non-essential	Which foods in the image are not required to cook spicy food for vegetarians?	shrimp, salmon	3
Missing	What tool is missing to cook vegetable noodles?	pot	3
Dissimilarity	Which vegetables have at least two different nutrients compared to the one between apple and pear?	capsicum, onion, garlic	5
Table 4: SWRL Rule-Based Conditional Logical Reasoning Questions.
4.2 Logical Reasoning Complexity Analysis
To objectively evaluate the complexity of logical questions, we formally define the concept of question difficulty [12], based on two criteria: the logical-syntactic complexity of the question and the complexity of the reasoning needed for the answer.
Firstly, the Question Syntax Complexity is defined by evaluating the quantity and diversity of logical connective operators used, with each additional connective or negation incrementally contributing to the complexity. Secondly, the Answer Inference Complexity is determined by the level of difficulty in deducing the correct answer through logical inference steps queried to a knowledge base. A more comprehensive elaboration of these definitions is provided in Section B of the supplementary material.
The logical problems in the LoRA dataset span a wide and diverse range of complex logic, divided into three levels of increasing difficulty from simple to complex. Level 1 comprises basic Description Logic questions based on nine foundational operators, such as conjunction, disjunction, negation, restriction, and others. For example: "What are the names of the vegetables that are green in color in the picture and usually the leaves are the edible part?", Level 2 includes questions that randomly combine two or three different types of logical operators, selected from the nine foundational operators in Level 1. For example: "Which foods in the picture are solid and do not necessarily contain fat but are rich in protein?", Level 3 involves rule-based logical questions categorized into seven different categories based on Description Logic, where reasoning complexity increases with intricate rules, such as: "If I want to boil or fry eggs, which tools in the image can be used respectively?".
LoRA Ontology Types
Question Types
Figure 3: Dataset Statistics: Ontology types, question types, and logical reasoning types.
Logical Reasoning Types
4.3 Question and Answer Types Analysis
Question Types: We categorize the question types in the LORA dataset into seven groups shown in Figure 3 based on the head of each question template: What, Which, How many, Is there ..., Are there ..., If ... then, and Why ...? The distribution of each question type in the dataset is shown in Figure 3, which also presents the data distribution based on ontology types and the question distribution based on logical reasoning types. The lengths of the logical questions range from eight to thirty words.
Answer Types: The answers within the dataset fall into four distinct categories: yes/no responses; individual or multiple objects (e.g., pumpkin, cheese); numerical values; and a small proportion of specialized responses, including answers to 'why' questions that incorporate relations and attributes. Examples of such specialized responses include "because it has juicy taste". or instances where the answer is "None". The length of the answers ranges from two words to a maximum of 23 words, varying based on the answer type.
Supplementary material Section A offers a more comprehensive data analysis, encompassing the analysis of the ontology, question types, and the distribution of question logic in LoRA. Section A.3 of the supplementary material provides examples of question logical complexity and answer inference complexity in LoRA, ranging from 3 to 9 required inference steps.
5 Baseline Experiments and Analysis
5.1 Baseline Experiments
The VQA dataset was evaluated and analyzed via different VQA baseline models and large vision-and-language models. The evaluation results are shown in Table 5. In our methodology, we categorize the queries according to their intricacy, on a scale of 1 to 9. A score of 1 signifies a relatively straightforward query, while a score of 9 implies the necessity for more than nine logical deductions to infer the answer. In addition, a study of human performance was conducted, details of which can be found in Supplementary Section E.2.
Ablation Study. In order to carry out an ablation study and evaluate the capacity of baseline models to provide accurate answers without image data, question data, or external knowledge, we examined three models: a "deaf" CNN [4] model, a "blind" LSTM [16] model, and a CNN+LSTM model that made predictions based on visual and linguistic features, but without a knowledge base. The performance of all three models was found to be unsatisfactory, underscoring the importance of integrating visual, question, and knowledge base information for effective VQA reasoning.
End-to-End Training Baseline.
We trained MAC network [19] end-to-end on the LoRA dataset, inputting both visual and question language features to estimate the accuracy for each type of logical question. The MAC network struggled with rule-based and complex logical questions compared to simple ones, resulting in  $64.6\%$  accuracy.
Fine-tuning Baseline. Moreover, we trained and fine-tuned the VisualBERT model [24], based on the transformer architecture, on the LoRA dataset. Visual features were extracted using a pretrained detectron2 model[34], while BERT[8] was used for question features. These image and language features were fed into vision-and-language fusion layers to predict final answers. Despite
the significant improvement in performance brought by the transformer of VisualBERT over other models, it displayed a decrease in its ability to answer complex questions as reasoning complexity increased, from  $85.1\%$  to  $57.8\%$ .
Zero-shot Baselines.
We evaluated the zero-shot performance of cutting-edge large vision-and-language models (VLMs), specifically, MiniGPT4 [39], Multimodal-GPT [13], InstructBLIP [7], and Multimodal Chain-of-Thought (MMCoT) [38] across three model types: in-context learning, instruct-tuning, and zero-shot CoT, using our LoRA dataset. We used the questions from the LoRA dataset as the prompts, with the accompanying images serving as the input images for these models for zero-shot experiments. Each question in the dataset not only guided the model in the direction of the answer but also implicitly highlighted the logical construct being evaluated. Our results revealed that all four models exhibited basic VQA capabilities, managing to answer questions that do not require much logical reasoning. They could also handle simple negation questions, but inconsistently. However, limitations were evident. The models occasionally ignore the visual information in the image and generate answers solely based on textual knowledge, especially MiniGPT4. Furthermore, they faltered in providing comprehensive responses involving multiple objects, often only listing one or two. Moreover, they are inconsistent and tend to fabricate answers that are often incongruent with the image content. Among the four models, InstructBLIP outperforms the other models with an average accuracy of  $41.2\%$ , but it performs poorly on complex logical questions that require more than six inference steps, achieving only  $30.5\%$  accuracy. MiniGPT4 struggles with negation questions that involve invisible attributes and complex logical questions involving multimodal inference. MMCoT exhibited suboptimal performance in VQA logical reasoning tasks, producing reasoning chains with logical inconsistencies.
Few-shot Baseline.
Furthermore, we evaluated Multimodal Chain of Thought based on the few-shot setting. We provided the logical operator that constructed each question as the prompting context to the model when asking it to answer each question. Using the logical operators of the questions as prompts improved the performance by  $5.87\%$  compared to not using them. Further analysis shows that logical prompts can help multimodal models stimulate logical thinking better.
5.2 Error Analysis
The following analysis investigates the different behaviors and performance of VQA models and multimodal LLM models, delving deeper into the underlying reasons behind the observed model behavior, and elucidates the potential factors that contribute to the strengths or weaknesses of a particular model.
5.2.1 MAC and VisualBERT's Challenges in Complex Logical Reasoning
The MAC network and VisualBERT model, despite their groundbreaking advancements in VQA, primarily rely on deep neural networks to derive implicit representations from the provided data. Deep learning models, by their very nature, often lack the inherent capacity to reason in an explicit, step-by-step manner that might be required for intricate logical constructs. Instead, these models tend to predict answers based on the patterns and associations they have learned. While this approach works efficiently for numerous tasks, it may not provide the rigorous, step-by-step logical reasoning required for our dataset, especially when questions increase in complexity.
5.2.2 Multimodal LLMs Baselines' Challenges in Complex Logical Reasoning
Building on our observations from the zero-shot and few-shot experiments, we further analyzed challenges faced by large vision-and-language models (VLMs). We observed that large vision-and-language models demonstrate deficiencies in handling logical constructs, especially with operators like negation and multiple logical operators appearing in a question. For example, when presented with negation-centric queries, these models tend to produce keyword-driven positive responses, overlooking inherent logical relationships. Such tendencies underscore that, despite expansive training data, these models lack genuine "logical understanding".
Complexity of Inference Steps	Logical Operators	CNN	LSTM	CNN+LSTM	MAC	VBert	MGPT4	MMGPT	InstBLIP	MCoT	Humans
1	C, D	46.3	45.2	65.3	76.2	85.1	47.9	45.7	48.2	30.2	88.2
1	R	42.2	33.4	64.1	69.7	83.2	41.3	40.9	45.7	33.2	77.6
2	¬C, ¬D	43.0	31.8	62.1	72.6	80.6	40.7	39.5	44.5	29.6	78.4
2	¬R	35.78	39.65	61.3	70.5	80.2	34.5	40.2	43.6	28.4	76.5
2	C ⊆ D	35.64	38.25	60.2	68.4	78.3	43.2	41.0	45.4	29.7	72.9
3	C ∩ D
C ∪ D	36.8	38.2	61.3	69.2	80.2	45.7	44.2	45.6	32.5	82.2
D ∈ {D, P ≥ n, P ≤ n}
4	¬C ∩ D
¬C ∪ D	36.2	35.6	60.2	67.8	75.3	41.8	38.5	42.9	26.5	75.6
D ∈ {D, P ≥ n, P ≤ n}
4	C ⊆ D, R1 ∩ R2	31.3	30.8	59.6	65.2	74.3	41.2	42.5	43.9	27.6	78.2
C ⊆ D, R1 ∪ R2
5	C ⊆ D, ¬R1 ∩ R2	28.7	29.3	53.2	59.7	68.8	40.1	39.7	41.7	26.9	70.6
C ⊆ D, ¬R1 ∪ R2
¬C ⊆ D, R1 ∪ R2
4-5	C ∩ D ∩ E	31.2	30.6	56.7	62.3	75.4	40.2	40.7	41.5	27.5	73.5
C ∪ D ∪ E
C ∪ D ∪ E
5-6	¬C ∩ D ∪ E	29.3	28.5	50.2	58.6	62.8	32.5	30.9	35.7	26.3	72.8
¬C ∪ D ∪ E
6	C ⊆ D, R1 ∩ R2 ∩ R3	29.7	28.6	51.3	57.6	68.5	31.5	30.2	34.4	22.6	75.6
C ⊆ D, R1 ∪ R2 ∪ R3
6-7	C ⊆ D, ¬R1 ∩ R2 ∩ R3 ≥ n	22.6	23.9	42.3	54.5	61.8	29.7	29.7	30.5	20.3	65.8
C ⊆ D, ¬R1 ∪ R2 ∪ R3 ≤ n
C ⊆ D, ¬R1 ∪ R2 ∪ R3 ≤ n
complexity(φ) + 2	if ..., then C, R → D	29.8	28.7	48.6	52.3	57.8	30.6	30.2	32.5	21.6	83.6
Learning	train set	train set	train set	train set	train set	zero-shot	zero-shot	zero-shot	few-shot	humans
Table 5: Evaluation of Baseline Models over Varying Levels of Logical Complexity, based on Inference Steps and Question Syntax Complexity (Accuracy in %). Model names: VBert = VisualBert, MGPT4 = MiniGPT4, MMGPT = Multimodal-GPT, InstBLIP = InstructBLIP, MCoT = Multimodal CoT.
Furthermore, large vision-and-language models often ignore visual cues, leaning heavily towards text-driven answers. Given their significant training on textual data, the VLMs might inherently prioritize textual patterns over visual cues when formulating responses.
In this paper, we propose LoRA, a novel VQA dataset that challenges the complex and formal logical reasoning abilities of VQA models and large vision-and-language models. LoRA consists of 200,000 multimodal complex logical problems. It covers a wide and diverse range of logical reasoning types, divided into three levels of increasing difficulty. We have also provided automated scripts for generating logical questions and images, enabling researchers to extend and customize the dataset. We have conducted a comprehensive and systematic evaluation of several state-of-the-art VQA models and large vision-and-language models on LoRA, revealing their strengths and weaknesses in different logical reasoning tasks. Our results show that there is still a large gap between the current models to solve multimodal logical reasoning problems, especially for complex logical reasoning questions. We hope that LoRA will serve as a valuable benchmark and a source of inspiration for future research in multimodal logical reasoning.
We would like to thank the anonymous reviewers for their valuable comments and suggestions. Additionally, our thanks go to Zhaofeng Yuan, Ross Zarghami, Lihan Li, Ben Brown, and Chris Ell for helpful discussions and suggestions. We also appreciate the support from the Blender Community, with special thanks to James Hurlock, Anna Trofimova, Chris Lee and John Joestar for assisting us with Blender.
[34] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detector2. https://github.com/facebookresearch/detectron2, 2019.
[35] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. CLEVRER: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019.
[36] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic VQA: Disentangling reasoning from vision and language understanding. Advances in Neural Information Processing Systems, 31, 2018.
[37] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. arXiv preprint arXiv:2304.00685, 2023.
[38] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.
[39] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.00685, 2023.
[40] Yeyun Zou and Qiyu Xie. A survey on VQA: Datasets and approaches. In 2020 2nd International Conference on Information Technology and Computer Application (ITCA), pages 289-297. IEEE, 2020.