RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Dataset.
MSCOCO Lin et al. ( 2014 ) is a large-scale image recognition, segmentation, and captioning dataset.
Here, we randomly sample 2,000 images with annotations about contained objects and human-labeled captions from its validation set as our evaluation dataset.
For computing the CHAIR metric on MSCOCO, we follow the settings in Rohrbach et al. ( 2018 ) which only considers 80 objects appearing in the MSCOCO segmentation challenge.
Models.
The evaluated LVLMs basically consist of three parts: a visual encoder, an alignment model, and a large language model.
All the above models have been tuned on collected visual instruction data.
A detailed comparison ( e.g., backbones and trainable components) of these LVLMs is shown in Table 7 .
We also collect the evaluation results of smaller VLPMs, i.e., OSCAR Li et al. ( 2020 ) , VinVL Zhang et al. ( 2021 ) , BLIP Li et al. ( 2022b ) and OFA Wang et al. ( 2022 ) from Dai et al. ( 2023b ) as baseline results.
Figure 1: Cases of object hallucination in LVLMs. Bold objects are ground-truth objects in the annotations and red objects are hallucinated objects by LVLMs. The left case is from the traditional instruction-based evaluation method, and the right cases are from three variants of POPE.; Cases of object hallucination in LVLMs. Bold objects are ground-truth objects in the annotations and red objects are hallucinated objects by LVLMs. The left case is from the traditional instruction-based evaluation method, and the right cases are from three variants of POPE.
(a) Hallucination times of top ten frequently appearing objects, whose frequencies decrease from right to left.; (b) Hallucination times of top ten objects co-occurring with “dining table”, whose frequencies decrease from right to left.; Hallucination times of frequently appearing/co-occurring objects in MSCOCO.
Figure 3: Overview of the POPE pipeline. Given an input image, POPE first extracts ground-truth objects in the image either from human annotations or with the help of automatic segmentation tools like SEEM. Then, POPE conducts negative sampling for nonexistent objects in the image under Random / Popular / Adversarial settings. Finally, the ground-truth objects and nonexistent objects are formulated into question templates to poll LVLMs.; Overview of the POPE pipeline. Given an input image, POPE first extracts ground-truth objects in the image either from human annotations or with the help of automatic segmentation tools like SEEM. Then, POPE conducts negative sampling for nonexistent objects in the image under Random / Popular / Adversarial settings. Finally, the ground-truth objects and nonexistent objects are formulated into question templates to poll LVLMs.
Results of CHAIR on VLPMs and LVLMs. I_{1} denotes “ Generate a short caption of the image ” and I_{2} denotes “ Provide a brief description of
the given image ”. Len refers to the average length of generated captions. The results of VLPMs (OSCAR, VinVL, BLIP, and OFA) are collected from Dai et al. ( 2023b ) . The best results in each block are denoted in bold.
Results on MSCOCO that quantify the correlations between the appearing/co-occurring frequency of objects and the hallucination times of LVLMs.
Results of LVLMs under three evaluation settings of POPE on the validation set of MSCOCO. Yes denotes the proportion of answering “Yes” to the given question. The best results in each block are denoted in bold.
Evaluation results of LLaVA on POPE and CHAIR with different prompt templates.
SEEM-based POPE results of LVLM on MSCOCO. F1 Score (Truth) are the results of POPE using ground-truth annotations, which are copied from Table 3 . The best results in each block are denoted in bold.
Evaluation results of LVLMs on POPE and VQA. For VQA tasks, we report the VQA score on A-OKVQA and Accuracy on GQA. For POPE, we copy the result under the random setting from Table 11 .
Comparison of the evaluated LVLMs. VE, AN, LLM stand for Visual Encoder, Alignment Network and Large Language Model, respectively. denotes frozen and denotes trainable . The fine-tuning of LLM in MultiModal-GPT and mPLUG-Owl is implemented by LoRA.
The accumulated proportions of the hallucination times of all 80 COCO objects. We arrange all objects by their frequency of occurrence.
The accumulated proportions of the hallucination times all objects that co-occur with dining table . We arrange all objects by their co-occurrence frequency with dining table .
The HR C result of chair and car .
SEEM-based POPE results of LVLMs and BLIP on A-OKVQA and GQA. The probing objects are selected from the segmentation results of SEEM on these datasets. The best results in each block except BLIP are denoted in bold. We employ the BLIP fine-tuned on VQAv2 from LAVIS Li et al. ( 2022a ) .
MSCOCO caption results.