Don't Fight Hallucinations, Use Them: Making Use of LVLM Hallucinations to Quantify the Image Realism via NLI
Authors: Elisei Rykov, Kseniia Petrushina, Kseniia Titova, Alexander Panchenko, Vasily Konovalov


Reviewer paper summaries: This paper addresses the challenge of quantifying image realism with AI by introducing a novel method that assesses image realism using Large Vision-Language Models (LVLMs) and Natural Language Inference (NLI). This approach leverages the hallucinations generated by LVLMs when extracting atomic facts from images provided that defy common sense to quantify the realism of the image, as the extracted atomic facts from these images by LVLSs could be taken as a mix of accurate facts and erroneous hallucinations. This approach uses the pairwise entailment scores among facts to produce a reality score, identifying contradictions and signaling images that violate common sense. The authors propose three NLI score aggregation approaches for comparison and they validate the effectiveness of their approach with ablation studies on WHOOPS! dataset.
Summary of strengths: 1. This paper proposes a novel idea to utilize the hallucination that may exist in the images provided that helps to assess the realism of the image. The idea is novel and the problem statement is good.
2. The experiment results are sound with ablation studies, comparison with baselines, and discussions over different NLI strategies.
Summary of weaknesses: 1. A figure illustrating the pipeline of this approach is necessary in this paper, otherwise the overall pipeline is not clear to readers. This pipeline figure is better to be included in the main paper. 
2. The authors need to provide more details on how they generate the atomic facts, like some examples of prompts and LVLM outputs on the atomic fact generation, as the atomic facts from the given image are hard to ground.
3. In Section 5.2, the authors need to provide more details about the weights ($w_{con}$, $w_{neu}$, and $w_{ent}$), like whether they are provided heuristically or computed from some mechanisms.
4. An experiment evaluating the performance of all three NLI strategies with discussions over results is required in the main paper.
5. More discussion on the related work is required and the current Section 2 needs to be re-organized. 
6. More showcases are needed. Besides, the authors need to provide some illustrations or insights about why their approaches achieve the expected performance in the showcase section.
Comments, suggestions and typos: 1. The prompt examples listed in Section 6 could be moved into the appendix.
2. It might be better to use a heatmap or other format to present the result in Appendix A.
3. Figures given in Appendix B should have a larger font.
Overall assessment scores: 2