SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3 shows the zero-shot results on the downstream tasks. For localized reasoning tasks, we first observe that scaling the visual encoder size (CLIP ViTB-16 vs ViT-L-14x336; BLIP-2 ViT-L vs ViT-G) in general improves the performance. CLIP outperforms BLIP-2 on VCR tasks but fall short on Shlerock and VisualCOMET. After applying localized symbolic knowledge distillation (LSKD) to BLIP-2, there is a consistent improvement over the BLIP-2 model on all downstream tasks (5.4% on VCR Q  $\rightarrow$  AR, 10.2 on Sherlock Comparison, 1.3% on VisualCOMET Acc@50).
For non-localized reasoning tasks, we observe a similar pattern. Interestingly, applying LSKD improves the performance of BLIP2 model further across all the tasks (AOKVQA, SNLI-VE, Visual7W) over the vanilla model, despite these tasks not being the primary target domain. This demonstrates that the advantages of distilling models with localized reasoning can be transferred to high-level visual commonsense tasks, thanks to the visual precision and enhanced reasoning abilities learned from the generated knowledge corpus.
Influence of Critic Filtering on Downstream Tasks How does the process of critic filtering influence the performance of downstream tasks? Keeping the size of the selected statements the same at  $\sim 300\mathrm{K}$ , we select qualified knowledge statements with varying prediction thresholds. We also compare with training on the LLaVA-instruct dataset which similarly prompts an LLM (GPT-4) to generate complex questions using ground truth verbalizers [34]. Figure 4 presents the resulting performances at these diverse thresholds across different datasets. Compared to LLaVA, we observe that localized knowledge statements without filtering does not show any improvement for the downstream model, while any thresholding over 0.2 is consistently better than LLaVA across all datasets. For tasks that demand relatively moderate commonsense, such as VCR  $Q \rightarrow A$  and Sherlock Comparison, increasing the threshold consistently improves the model performance. For
Localized	Non-Localized
Dataset	Size	Annotator	VCR Q → AR	Sherlock Comparison	VisualCOMET Acc@50	AOKVQA Mult. Choice	SNLI-VE Classification	Visual 7w Telling QA
Zero-Shot	NA	NA	28.0	19.5	39.0	68.0	33.4	77.1
Sherlock [19]	300K	Human	34.6	30.5	39.7	67.2	38.6	70.1
VisualCOMET [41]	1.2M	Human	31.8	25.3	50.2	68.5	35.6	70.8
LLAVA-Instruct [34]	150K	GPT-4	28.1	26.9	37.5	71.0	42.6	79.5
LSKD (Ours)	150K	ChatGPT	33.3	28.6	39.7	69.6	38.0	75.9
LSKD (Ours)	1M	ChatGPT	33.4	29.7	40.3	68.9	40.3	79.5
Table 4: Ablations of BLIP-2 ViT-G trained with varying sources of visual-knowledge corpus annotated by humans and machines. We break down to visual reasoning tasks that require localized reasoning and those do not. Critic filtering is applied to the LSKD corpus (Ours).
tasks requiring a higher degree of commonsense such as VCR QA  $\rightarrow$  R and VisualCOMET Hit@50, the performance increases until a certain threshold and then fluctuates. We speculate that a more grounded critic model could potentially mitigate this fluctuation, and we intend to investigate this in our future work. Overall, our findings suggest that higher thresholds (i.e., more critical filtering) tend to yield superior quality generations, thereby enhancing the performance on downstream tasks.
3.3 Human vs Machine Annotated Corpus
Can training on machine annotated corpus result in competitive performance with human annotations? In Table 4 we compare the performance of BLIP-2 ViT-G trained on existing human-annotated corpora with our machine-annotated corpus across various scales. First, we found that increasing the size of our training corpus (150K vs 1M) leads to consistent improvement across all tasks, indicating a promising trend of scaling law for synthetic training data. Regardless of the size, training on our dataset yields considerable benefits over the zero-shot model on localized reasoning tasks.
Next, we observe that training on human annotated corpus vastly improves the performance of their relative tasks (e.g. training on VisualCOMET boosts performance from 39.0 to 50.2). However, this can lead to inferior results on other visual reasoning tasks than the zero-shot counterpart. For instance, the performance on Visual7W drops from 77.1 (Zero-shot) to 70.1 (Sherlock) and 70.8 (VisualCOMET). This suggests that human-designed datasets may limit task generalization due to their lack of diversity. Interestingly, we see that training the model our full LSKD corpus (1M) leads to uniform gains over the zero-shot model across the tasks, and even outperforms the human annotation corpus for the non-localized tasks as well. This shows that machine-annotated datasets, when curated and scaled adequately, can indeed rival or even surpass the performance of models trained on human-annotated corpora.
We directly compare training on ours and the LLaVA dataset. Regardless of our dataset scale, we observe that  $\mathrm{LSKD} +$  filtering wins over training on the LLaVA corpus on localized reasoning benchmarks, even when using a less powerful teacher model (ChatGPT vs GPT-4). This suggests that our creation of a new localization corpus is crucial to support the model with grounded reasoning. On the other hand, LLAVA wins on non-localized reasoning tasks as they are aligned with the nature of training corpus. We thus observe that the appropriate application of the corpus can be task-dependent, and adopting a selective approach towards generating the corpus may result in significantly enhanced performance across various benchmarks.
3.4 Localized Reasoning with Generative Models
We extend LSKD to train generative models that can refer and talk about highlighted regions in image. We finetune BLIP-2 FlanT5 and Mini-GPT4 and prompt them to answer questions from the VCR data. As there is no baseline zero-shot model that can reason about regions to answer questions, we make a direct comparison of the student LSKD model to the teacher LLM with access to verbalizations. We ask annotators on Amazon Mechanical Turk (AMT) platform to run head-to-head comparisons (with ties) on three criteria, if the answer delivers: 1) visually correct details, 2) informative and interesting information, and 3) content that sounds plausible. Finally, they select their overall preference. We take the majority vote of 3 annotators, and disregard the instance if there is no clear majority.
Model	Correctness	Informativeness	Plausibility	Overall
ChatGPT w/ Vebalizers	34.7	33.9	39.6	45.0
BLIP-2 (FlanT5XXL-11B) + LSKD	31.7	41.0	30.2	41.2
Tie	33.7	25.1	30.2	13.1
ChatGPT w/ Vebalizers	29.8	31.7	36.8	40.6
Mini-GPT4 (Vicuna-13B) + LSKD	34.3	53.0	34.2	49.1
Tie	35.9	15.3	30.0	10.3
Table 5: Human evaluation of generative models with LSKD vs Chat-GPT with verbalizers. Humans are asked to choose the better generation or tie if they share the same quality.
Table 5 shows the human evaluation results. We observe that the LSKD generally wins in informativeness over ChatGPT, but not in plausibility. We see a conflicting pattern in correctness and overall preference, where Mini-GPT4 is equipped with a more powerful language model that outperforms the teacher model while BLIP-2 falls short. Unlike previous language-based distillation where a relatively weak student model can outperform the teacher [58, 3], we see that a strong student model may be required to outperform the teacher LLM in the multimodal domain.
Qualitative Results Figure 5 presents a comparative analysis of question-answering with rationale results on VCR samples generated by ChatGPT, LLaVA [34] and 0urs. Both Ground Truth (GT) and 0urs consistently identify the correct entities, with 0urs model often providing broader context, which is uncertain on rare occasions (e.g. “likely the bride”). On the other hand, ChatGPT predominantly focuses on observable actions or states as described in the text context, occasionally resulting in the misidentification of the visual entities and their relations. In the third example in Figure 5, “waiting for someone” focuses on the observable state “standing still”, missing visual detail such as a cave, holding a flame, and surrounding context. LLaVA, in contrast, generally provided a broad overview while identifying a specific visual entity in most cases. However, it often struggled to accurately identify specific entities within the complex scene (e.g. “holding a wine glass” in Figure 5(1), “cigarette” in Figure 5(3)). Compare to LLaVA, 0urs often aligned closely with GroundTruth and incorporated both actions and inferred knowledge in its answer. Overall, 0urs delivers a more nuanced and contextually-rich response.
Knowledge Distillation Recent research [1] has extensively explored the use of language models as knowledge bases, highlighting their potential in reasoning, explainability, and consistency, which can enhance downstream tasks by distilling knowledge from LMs. [15] demonstrated how knowledge augmentation explicitly from knowledge bases and implicitly from GPT-3 improved open-domain multimodal tasks. [33] showed that overgeneration with GPT-3 from exemplars to filter, as well as reviewed by humans, is a new and reliable way to create an NLI dataset with human and AI collaboration. This setup also has the advantage of bringing forth cultural internalization via human collaboration [9]. Previous works have explored knowledge distillation in the multimodal domain by prompting the teacher LLM with human-annotated verbalizations [34, 74, 10]. Our work is different in that it generated localized commonsense descriptions and the knowledge generation framework can operate a scale without the need for aligned descriptions.
Filtering [2] filters the generated sentences using a classifier trained on original sentences and a set of generated sentences. [55] used the same technique to filter out synthetic data created, which is of low quality. Large language models can be used to refine the commonsense knowledge retrieved from web contents by filtering the data generated from these models [39]. They perform a consolidation step that filters topical and relevant assertions based on OpenIE.
Multimodal commonsense reasoning requires more profound real-world knowledge, potentially spanning logical, causal, and temporal relationships between concepts. For example, elements of causal reasoning are required to answer the questions regarding images in VCR [66] and VisualCOMET [41], while other works have also introduced datasets with video and text inputs to test for temporal reasoning (e.g., Social-IQ [65], MovieQA [51], MovieFIB [36], TVQA [28]). Benchmarks for multimodal commonsense typically require leveraging external knowledge from knowledge bases [49] or pretraining paradigms on large-scale datasets [35, 68].
Figure 5: Qualitative examples comparing ChatGPT (the teacher model), LLAVA trained for complex visual reasoning [34], and ours. Each person referenced in the question has been assigned a unique number with a bounding box and their mention has been marked with a corresponding color. Any errors within the generated results are emphasized with a red highlight.
Region Understanding with Multimodal alignment Capturing elements across modalities that have a common meaning and is exemplified by tasks such as visual coreference resolution [25, 42], visual referring expression recognition [8], multimodal question answering [19, 66], and cross-modal retrieval [13, 43]. Alignment between modalities is challenging since it may depend on long-range dependencies, involves ambiguous segmentation (e.g., words or utterances), and could be either one-to-one, many-to-many, or not exist at all. Resources for fine-grained alignment include Visual Genome [26] and dense captioning [21], diverse reasoning [63]. Recent methods have adopted either generative or retrieval-based methods for alignment: generative methods create localized verbalizations of the region of interest [72, 14, 21, 70], while retrieval aims to select the most accurate caption for the region of interest despite possibly given only coarse-grained paired data of captions for entire images [5, 18].
We present LSKD, a method for sampling localized commonsense knowledge from a large language model. With the help of supervised critic model aligned with human judgements, we create a diverse, reliable 1M localized commonsense corpus. Training on the resulting corpus supports models that can accept region references as input, which allows users to interact with specific parts of images by "pointing;" all without having to write out a referring expression explicitly. We show that training on our corpus improves the zero-shot performance of vision-language models for tasks requiring regions-as-input. Making the critic model more critical by strict thresholding improved performance further. We present a state of the art zero-short performance with our approach opening avenues for visual commonsense models with our localized commonsense knowledge corpus.
[64] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[65] Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In CVPR, pages 8807–8817, 2019.
[66] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720-6731, 2019.
[67] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16354–16366, 2022.
[68] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-23651, 2021.
[69] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
[70] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Lianian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16803, 2022.
[71] Bolei Zhou, Aditya Khosla, Ågata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. ArXiv, abs/1610.02055, 2016.
[72] Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. More grounded image captioning by distilling image-text matching model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4777-4786, 2020.
[73] Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaogian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. ArXiv, abs/2303.06594, 2023.
[74] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
[75] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images, 2016.