Do Vision Encoders Truly Explain Object Hallucination?: Mitigating Object Hallucination via Simple Fine-Grained CLIPScore
Authors: Hongseok Oh
Wonseok Hwang
Venue: Accepted by TMLR ()

Abstract: Recently, Large Vision-Language Models (LVLMs) show remarkable performance across various domains.
However, these models suffer from object hallucination.
In this work, we study object hallucination primarily in a discriminative, retrieval-style evaluation setting (OHD-Caps), rather than in free-form caption generation.
This study revisits the previous claim that the cause of such hallucinations lies in the limited representational capacity of the vision encoder. 
Our analysis implies that the capacity of the vision encoder is not necessarily a major limiting factor in detecting object hallucination.
Based on this insight, we propose Fine-grained CLIPScore (F-CLIPScore), a simple yet effective evaluation metric that enhances object-level granularity by incorporating text embeddings at the noun level.
Evaluations on the OHD-Caps benchmark show that F-CLIPScore significantly outperforms conventional CLIPScore in accuracy by a large margin of 39.6% without additional training.
We further demonstrate that F-CLIPScore-based data filtering reduces object hallucination in LVLM (4.9% in POPE accuracy after alignment pretraining).
Our code is publicly available at https://github.com/abzb1/f-clip

Published in Transactions on Machine Learning Research (01/2026)
Do Vision Encoders Truly Explain Object Hallucination?? Mitigating Object Hallucination via Simple Fine-Grained CLIPScore
Hongseok Oh Department of Artificial intelligence University of Seoud
cru05190uos. uc.kr
Wonseok Hwang
wonseok.hwwnng@uos oc.kr
Department of Artificial intelligence Universily of Scoul
Reviewed on OpenReview: https: optp//vveeervvee.../ net/ forum? id= JTua
Abstract
Recently, Large Vision-Tanguage Models (LVLMs) show remarkable performance across various domains. However, these models suffer from object hallucination. In this work, we study object hallucination primarily in a discriminative, retrieval-style evaluation setting (OHD-Caps), rather than in free-form caption generation. This study revisits the previous claim that the cause of such hallucinations lies in the limited representational capacity of the vision encoder. Our analysis implies that the capacity of the vision encoder is not nec- essarily major limiting factor in detecting object hallucination. Based on this insight, propose Fine-grained CLIPScore (F-CL.PScorre), simple yet effective evaluation metric that enhances object-level granularity by incorporating text embeddings at the noun level. Evaluations on the OHD-Caps benchmark show that F-CLIPScore significantly outperforms conventional CLIPScore in accuracy by a large margin of 39.6% without additional training. We further demonstrate that F-CLIPScore-bert based data filtering reduces object hallucination in LVLM (4.9% in POPE accuracy after alignment pretraining). Our code publicly avail- able at https://gitthn httpg/tt...cccbbbbaoooooeee com/abzbl/f-clip
1 Introduction
Recent studies identify Large Vision-Language Models (LVLMs) loading approach for vision-language integration (Liu et al. 2023 Wang et al. 2024a Zhu et al. 2023; Chen et al. 2024). However, like hal- lucinations in large language models (L.LMss) (Ji et al. 2023 Zhang et al. 2023). LVLMs exhibit object hallucination, referring to nonexistent or misidentified objects that may undermine their reliability ęLi et al. 2023a; Liu et al. 2024b). While object hallucination often discussed in the context of caption generation, this paper focuses on discriminative vision vision- language models and retrieval-style evaluation protocols, where the goal to select or score candidate captions given an image.
Liu et al. (024cc built OHD-Caps, dataset designed to measure object hallucination in discriminative caption-selection setting. The dataset comprises 1.5k image-captions pairs where model needs to select the best caption that docs not show hallucinations. They found that CLIPScore (Hessel et al. 2021 achieved only 10 20% accuracy, and the further fine-tuning with the proposed objective function with the dataset improves the accuracy up to 80-90 80-90%. However, when they connected the OHD-Caps-traine CLIP to an LVLM and conducted full fino-tuning, the resulting accuracy sometimes drops, showing lower performance compared to original CLIP (for instance, 1st row of Table 4 in (Liu al. 2024c) shows the accuracy in POPE benchmarks [Li et al. 2023b) drops from 85.4% to 81.2%).

---

Published in Transactions on Machine Learning Research (01/2020)
CLIPScore (w/o training): A lady and two children in the street playing with a tennis racquet, a car nearby, and chair.
CLIPScore (trained): lady and two dogs in the park playing with frisbee.
F-CLIPScore (w/o training): lady and two children the street playing with tennis racquet.
Figure 1: A representative example from the OHD-Caps test set is shown. The original CLIP selects a sentence mentioning "children" and "tennis" but adds hallucinated objects. The OHD-Caps-trainde CLIP hallucinates "dog" and "frisbee" without introducing new content. In contrast, F-CLIPScore selects a sentence that preserves the original meaning without hallucinations.
Moreover, we found that OHD-Caps- Cap---as-trr CLIP someLinnes tends hallucinate by replacing existing ob- jects. When both CLIPScore and OHD-Caps trai-ned-- CLIP fail, but our Fine-grained CLIPScore (introduced in a later section) succoeds, a clear pattern emerges: CLIPScore adds nonexistent objects, whereas OHD-Cape trained CLIP replaces existing ones. This trend holds across COCO, Flickr30k, and NoCaps with rates of 56%, 58,, and 59%, respectively. A representative example is shown in Figure This suggests that object hallucination may stem from factors beyond the vision encoder's capacity.
To address this issue, we introduce Fino-grained CLIPScore (F-CLIPScore), a novel image-text correlation metric. F-CTIPScore leverages sentence parser like spaCy Honnibal al. (2020) and the forward pass of Vision-Language Model (VLM) like CLIP Radford al. (2021), offering an efficient way to evaluate the vision encoder's representational capacity. Our experimental results show that applying F-CLIPScore to the OHD-Caps test set improves accuracy by +39.6% without additional training. This indicates that the limited capacity of the vision encoder may not be the primary cause of object hallucination. Additionally, we verify that using E-CLIPScore for pretraining data curation in LVLMs enables the training of models with reduced hallucination, even with significantly fewer data samples. Notably, in LVLM pretraining, data filtering alone improved POPE accuracy by 4.9% compared to the baseline.
This study offers the following key contributions:
We introduce Fine- graenerained CLIPScore, novel evaluation metric that relies solely on forward prop- agation.
We provide evidence suggesting that, in discriminative evaluation, object hallucination is pri- marily explained by the vision encoder's representational capacity alone.
We demonstrate that F-CLIPScore enables more efficient LVLM training with reduced object hal- Iucination through pretraining data curation.
Our code is available at httpss//ggihubbccm//bbbb///////
2

---

Published in Transactions on Machine Learning Research (01/2026)
7-
Correct Incorrect
3.0
Correct Incorrect
3.0
Text Hallucination
Text w/ Hallucination
2,5
Density (%)
2.0
2.0
1.5
3-
1.0
0.5
0.5
0.4 (a) Cosine similarity of image embedding (b) Cosine similarity of text embedding
0.3
0.5
0.7
0.0
0.2225
(c) Cosine similarity of text embedding
Figure Histograms of cosine similarity between two embedding vectors: one from the original CLIP-L and the other from the OHD-Caps-trained CLIP-L. (a) The histogram from the vision encoders. Correct (blue) indicates the scores are from the examples where original CLIP-L predict the ground truth. The other examples are colored in orange. (b) The histogram from the text encoders. Same color scheme is employed. Measured only on ground truth text. (c) The cosine similarity distribution between text embeddings of text without object hallucination (purple) and with object hallucination text (green) for all samples.
2 Related Work
Object hallucination refers to cases in which the generated textual descriptions include objects that do not correspond to the given image (Liu et al.] 2024b). LVLMs generally consist of three components: vision encoder, an LLM, and an adapter (Liu et al., 2023). The structural characteristies of LVLMs contribute to object inllucination, which arises from multiple intertwined factors (Liu et. al. 2024b). While some studies argue that hallucinations can be mitigated by enhancing the decoding process the LLM (Manevich & Tsarfaty 2024 Wang et al. 2024b), others suggest that one of the causes lies in the limited representational capacity of the vision encoder (Liu et al.) (024c). Additionally, some research indicates that training the adapter with contrastive data is essential to reduce object hallucination (Jiang et al. 2024). Moreover, the
trained bias of the model has also been identified cause of hallucination (Hu et al. 2023 Liu et al. 2024a).
CLIPScore (Hessel et al. 2021 is a reference-free evaluation metric that assesses the consistency between an image and text caption by computing the cosine similarity between the embeddings generated by the vision encoder and text encoder of the CLIP model. Beyond its application measuring caption quality, several studies have also leveraged CLIPScore for data curation in the training of Vision LaiguaL Models (VLMs) (Schuhmann ct al. 2021 Gadre et al. 2023
A recent study utilized CLIPScore to evaluate object hallucination in Vision-Tanguage Models (VLMs) Liu et al. 2024ch. Their findings suggest that this phenomenon stems from the limited capacity of the vision encoder. In this study, we carefully reassess this claim and demonstrate that object hallucination is not necessarily caused by the limitations of the vision encoder alone.
3 Methods
3.1
Motivation
Based on our initial observation that OHD-Caps-trained CLIP occasionally does not yield better results (Section D. further investigate how fine-tuning affects the embedding vectors produced by vision and text encoders of CLIP. We compute the cosine similarity between two embedding vectors: one from the original CLIP-L and the other from the OHD-Caps-trained CLIP-L, using image-text pairs in the OHD- Caps test. set.
We first computed the distribution of cosine similarity between the image emboddings from CLIP-L and OHD- OHD-Caps-trained CLIP-L, focusing on the samples that were correctly predicted by CLIP-L. We then performed the same analysis on the samples that were incorrectly predicted by CL.P--I and compared the
3

---

Published in Transactions on Machine Learning Research (01/2026)
Vision Encoder
CLIPScore
BARROZ
Double decker bus with vase and people on the top floor on the road.
Text Encoder
Parser
Text Encoder
decker
road
Σ
F-CLIPScore
Figure 3: The graphical representation of F-CLIPScore.
two distributions. As shown in Figure 21 we observe that there is no substantial difference between the two distributions. We conducted the same analysis using ground-truth text embeddings of the pre-trained and fine-tuned models, and similarly found no significant. difference the cosine similarity distributions between the samples that were correctly and incorrectly predicted by CLIP-L (Figure pp). Two-sample t-tests yield p-values of 0.11 for a and 0.67 for b. In contrast, Figure 2 reveals significant changes in lext embeddings for captions with hallucination (green), and without hallucination (purple) highlighting the distinct adaptation of text representation. Two-sample t-tests yicld near zero p-value for c. These obscrvations suggest that OHD- Caps training induces more noticeable changes in the text representation space, particularly in discriminating hallucinated from non-hallucinated captions, whereas the vision representation space exhibits relatively minor changes.
3.2 Fine-grained CLIPScore
Motivated by these observations, we propose a simple metric called Fine-grained CLIPScore (F-CLIPScoree) which enhances the discriminative power of the VLM in order to utilize textual information with more granularity without additional training. F-CLIPScore first utilizes the spaCy parser Honnibal et al. (2020) extract nouns from given sentence. Then, evaluates the quality of an image caption by averaging the CLIPScore of the entire sentence and each individual noun (Figure 3)
Mathemalically, given image I and a caption C containing a total of N nouns, each denoted as ni F-CLIPScore is defined as Eq.
F-CT.ISScore CLIPScore(T,C) EN CLIPScore(I,m.)
(1)
N
4 Experiments
For evaluating OHD-Caps test set, F-CLIPScore only requires an additional parsing step (we used _ore _n__crre_web________eeeee_______ web_ssm of spaCy (Honnibal et al.) 2020)), which takes on average 170 ms per sentence and can be parallelized across samples. We note that this parser runs in lincar time with respect to the length of the sentence (Honnibal & Johnson 2015). For training CLIP with Eq. 2 on the OHD-Caps train set, we used batch size of 64 and learning rate of 1e-5, which required 3 hours
H100 GPU. For LLaVA
4

---

Published in Transactions on Machine Learning Research (01/2026)
OHD-Caps ACC (%. 1)
Metric
COCO
Flickr30k
NoCaps
w/o training
CLiPScore
22.6
22.6
12.4
F- R-CLIPScore
62.2
62.2
46.6
trained w/ OHD-Ioss
CLIPScore
79.8 #1.7
84.8 #1.6
84.0
=0.7
trained w/ F-CLIPScore loss
CLIPScore
80.5 11.8
84.8 #1.3
84.1
+1.3
t: We trained spenaa/cliprii---bge--acch with the train code from Liu et al. (021cc
Table 1: Accuracy on the OHD-Caps test set evaluated with OpenAl CLIP-L. For trained models, we report the average evaluation results over 10 runs with different random seeds.
pretraining, we employed an effective batch size 256 and a learning rate of 1e-3, which took hours on two H100 GPUs.
5 Results
By utilizing the F-CLIPScore, which directly leverages the image emboddings from the CLIP vision encoder without any training gradients, while only adding parsing step during forward propagation, we can efficiently gain insights into whether the issue of object hallucination arises from the limited capability of the vision encoder.
5.1 F-CLIPScore OHD-Caps
We evaluated the OHD-Caps test set, object hallucination assessment dataset, using the proposed F- CLIPScore. As shown in Table 1 evaluation results with OpenAl CLIP ViT-L Radford et al. (2021) indicate that F-CLIPScore outperformed the baseline model by up to 39.6% without additional training (row 4 vs. 5). However, it still performed 17.6% to 37.4% worse than trained models (row vs. 7). This may indicate that although CLIP vision encoders may not be the main cause of object hallucination, further training may enhance their capability.
To test whether F- C-CLee is orthogonal the previously proposed method Liu al. 2024c), modify the loss as
LeLir + LORD B (1 F-CLLPSSorr(4,.CC) C:))
where LOLIP is the contrastive loss proposed in Radford et al. 2222 and LOHD is the marginal loss proposed in Liu et al. (2024c). B denotes the batch size, while 1; and C2 are image and caption of the i-th positive pair. a is a hyperparameter that we set to 0.3. Applying F-CLIPScore as a loss improved performance by 0.7 percentage points on COCO and 0.1 percentage points on NoCaps (row 7 vs. 9). These results suggest that F-CLIPScore could be complementary component. VLM training.
We randomly replaced the caption embeddings into noun embeddings extracted from Wikipedia cor- pus (Davies] 2015) to examine how F-CLIPScore responds to such perturbations. As shown in Table 2 we observed the expected decline in accuracy as nouns were replaced. This indicates that the strong performance of F-CLIPScore, even without additional training, is not due to random noise, but rather stems from its ability to more effectively leverage fine-grained textual embeddings.
We experimented with alternative configurations of F-CLIPScore using verbs and noun phrases instead of nouns. As shown in Table 3 using verbs yielded little to no improvement, while noun phrases performed better than verbs but still fell short of the performance achieved with nouns.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Replacement Rate
0.2
0.8
1.0
COCO Acc mean std (%, 1)
38.9+2.55
35.9+1.4
31.0+1.7
23.5+2.0
19.9
Flickr30k Acc mean std (%, f)
37.5+1.55
33.0+1.2
26.2+2.1
22.0:t..6 18.9+3.0
NoCaps Acc mean std (%, f)
29.4 29.4119
24.242.3
21.242.0
16.241.7
16.540.4
Table 2: F-CLIPScore results the OHD-Caps test set with different random noun replacement rates (0.2 to 1.0). Each value shows the mean accuracy + standard deviation over five random seeds.
OHD-Caps ACC (%, f)
Metric
coco
flickr30k
nocaps
CLIPScore
22.6
22.6
12.4
F-CLIPScorey
23.6
24.8
15.8
F-CLIPScoreNP
54.2
57.8
39.6
I-CLIPScoreN
62.2
62.2
46.6
Table 3: Accuracy on the OHD-Caps test set evaluated with OpenAl CLIP-L. F-CLIPScoreN refers to the noun-based F-CLIPScore defined in Eq. [ F-CLIPScoreNP uses noun phrases instcad of individual nouns, and F-CLIPScorey replaces the nouns with verbs.
Our F- F-CLIPScore even outperforms the training-based methods (Zhang et al. 2024 Yuksekgonul et al. 2023 as shown in Table
demonstrates that F-CLIPScore consistently achieves gains across different scales and architectures of vision-language models.
Furthermore, Table 5
5.2
LVLM Pretrain Data Curation with F-CLIPScore
As shown in Section 5.1 F-CLIPScore was able to exhibit competent performance in detecting object hal- lucination without any training, and could further serve as an complementary method for training VLMs. We aimed investigate whether F-CI.PSccore can influence the pretraining process of LVT.M by acting as data filter between the vision encoder and the LLM. To this end, we applied F-CLIPScorre to filter the pretraining data for LLaVA (Liu et al. 2023), which consists of 558k samples, by removing the bottom x% of the alignment training set. based on F-CLIPScore.
As shown in Table training the aligmment model on the top 70% of data curated by F-CLIPScore resulted in a +4.9% improvement POPE accuracy compared to training on the entire dataset (row vs. 7). In contrast, using the OHD-Caps-train trained CL.IP for filtering yielded only marginal gains, and random filtering showed no improvement. These results may suggest that F-CLIPScore effectively captures object hallucination-rclttel
OHD-Caps Accuracy
COCO
Flickr30k
NoCaps
CLIP-B/32 (CLIPScore)
15.2
17.6
10.2
CECLIP Zhang el al. 2024
32.8
28,0
25.0
NegCLIP Yuksekgonul
2023
52.8
40.8
23.4
CLIP- /32 (F-CLIPScore)
56.0
53.2
43.0
Table 4: OHD-Caps accuracy across different methods.

---

Published in Transactions on Machine Learning Research (01/2026)
OHD-Caps Accuracy
COCO
Flickr30k
NoCaps
SigLIP VIT-L (CLIPScore) Zhai et al. 2023
48.6
38.4
30.2
SigLIP VIT-L (F-CLIPScore
69.0
64.2
54.2
EVA-CLIP VIT-L (CLIPScore) Sun
2023
38.6
31.8
22.6
EVA-CLIP VIT-L (F-CLIPScore)
69.6
64.8
55.4
CLIP VIT-B (CLIPScore)
15.2
17.6
10.2
CLIP VIT-B (F-CLIPScore)
56.0
53.2
43.0
CLIP VIT-I (CLIPScore)
22.6
22.6
12.4
CLIP VIT-L (F-CLIPScore)
62.2
62.2
46.6
CLIP VTT-H (CLIPScore)
36.8
31.4
20.6
CLIP VIT-H (F-CLIPScore)
57.6
46.6
Table 5: OHp-Caps accuracy across different vision-encoder architectures and sizes.
POPE Ace (t, %)
Filtering rate (%)
Filtering Method
20
30
40
50
60
Random
50.7
49.9
49.9
50.5
49.8
CLIPScore (w/o train)
52.2
47.3
50.0
53.1
50.0
F-CLIPScore (w/o train)
51.6
55.5
50.6
50.1
51.8
CL.PScoore (trained)
49.8
49.8
50.8
49.8
52.6
w/o filtering
50.6
Table 6: Accuracy on the POPE benchmark after LLaVA pretraining (vision-text alignment training before SFT). We use CLIP-L Radford et al.] (2021) as a vision encoder, and Llama 7B Touvron et al. (2023) as an LLM backbone. "trained" indicates the HHDD-Cpps-traing CLIP-L. "Random" " refers to the removal of x% of samples chosen at random.
quality, even in general-purpose datasets. This finding underscores the need to explore alternative causes of object hallucination beyond the capacity of the vision encoder. Although model performance shows a non-lincar trend with respect the filtering ratio, our experiments used fixed number of training epochs, which may account for this behavior. As Goyal et al. (2024) highlights the need to balance data quality with computational cost, identifying the optimal filtering ratio under such trade-offs remains an important direction for future research.
6 Conclusion
We introduce F-CLIPScore, simple yet effective metric for evaluating fine-grained image-caption alignment and addressing object hallucination in Vision-Languane Models. Unlike conventional CLIPScore, which relies solely on sentence-level embeddings, F-CLIPScore also incorporates noun-level embeddings. This refinement allows the model to better mitigate object hallucination without requiring additional training for the vision encoder. We validate F-CLIPScore by showing +39.6% accuracy in OHD-Caps benchmark. We also show that data filtering based on F-CLIPScore with reduced dataset. Our results suggest that the limitations of existing evaluation metrics, rather than the vision encoder itself, may contribute to object hallucination because they fail to accurately reflect the vision encoder's true capacity.
enhance LVLM performance in hallucination mitigation, even
Limitations
First, our study primarily targets discriminative evaluation and retrieval-style scoring (c.g., OHD-Caps, POPE) and does not directly evaluate hallucination in free-form caption generation. Investigating how noun- level scoring be integrated into generative decoding training objectives remains future work.

---

Published in Transactions on Machine Learning Research (01/2026)
While this study proposes method for analyzing and mitigating object hallucination using F-CLIPScore, it is subject to the following limitations. First, we were unable to conduct experiments on the Supervised Fine-TTuming (SFT) for Large Vision-Language Models (LVLMs), In the LVLM training pipeline, after the alignment pretraining phasc-when the vision encoder and LLM remain frozen- the SFT stage follows, in which these components are unfrozen and further trained. However, due to computational resource lim- itations, we did not fully explore the potential impact of F-CLIPScore during SFT. Future work could investigate ways to incorporate F-CLIPScore into the SFT process to enhance training effectiveness.
Second, our method faces linguistic constraints and challenges in multilingual generalization. This study employs the spaCy parser (Honnibal et al.) 2020) to extract nouns from text, technique that performs relatively reliably in well-structured languages such as English. However, parsing accuracy may vary across different languages, potentially leading to inconsistencies in F-CLIPScore computation. To address this, future research should explore the scalability of F-CLIPScore by evaluating its effectiveness on multilingual datascts and refining the parsing methodology for broader linguistic applicability.
Acknowledgments
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea govermment(MSTT) (RS-20055-33224855)
References
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Intern vl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24185 24198, 2024. doi: 10.1109/CVPR522733 2024.02283.
Mark Davies. The wikipedia corpus.
2015. Available online.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Ha- jishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Thirty-seventh Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openrrevier wwtp:r/www..e.eeeoooooo/////aaaa
Sachin Goyal, Pratyush Maini, Zachary C. Lipton, Aditi Raghunathan, and J. Zico Kolter. Scaling laws for data filtering- data curation cannot be compute agnostic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 22702 22702-22711, June 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference- evaluation metric for image captioning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 18.18653/v http//221...mmmmmmee. 2021. emnlp monnlp-min.... 595/
URL httphhttps/////
org/
Matthew Honnibal and Mark Johnson. An improved non con-monotonic transition system for dependency parsing. In Lluís Marquez, Chris Callison-Burch, and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1373 1378, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1 /D/1///1/1 1162. URL https://aclanttoooogy. org/D15-1108 1162/
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial-strength natural language processing in python. 2020. doi: 10.5281/zemood 12t.ooooo.11220 1212303.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. CIEM: Contrastive instruction evaluation method for better instruction tuning. In NewrIPS 2023 Workshop on Instruction. Tuning and Instruction Following, 2023. URL intts:://openrevier
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Suru. 55(12), March 2023. ISSN 0360-0300. doi: 10.1145/3571730 URL https:://di..rg// 1ttpe//./r//11111/3555/3 1145/3571730
Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model, 2024. URL bttbps://arrtie www.atg.////////oooo./........a/ 221. 06968
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292 222, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 11263//1112233.mmpmmmm/ https://aclanthol/ wwwph/oooooo...//2222.... emnlp-main. 20/
URL
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292 305, Singapore, December 2023b. Association for Computational Linguistics. doi: 100553/v1/022.mmm/l main. 20. URL https://aclanthoo/ org/ 2023. emnlp mapnm.mn......
Fuxiao Liu, Kevin Lin, Linjic Li, Jianfeng Wang, Yasor Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2024a. URL
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang. Liping Hou, Rongium Li, and Wei Peng. A survey on hallucination in large vision-language models, 2024b. URL https: //arxiv.org. /tgg/ba///02.0000////
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (ods.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 34892 34916. Curran As- sociates, Inc., 2023. URL http://ppocceiingg.neerrpppppppp wttp://ppeeeeeeeepppeeee/e////ee pttp.//aee///////////a/.. pdf
Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Aimin Zhou Investigating and mitigating object hallu- cinations in pretrained vision-language (CLIP) models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18288-18288 Miami, Florida, USA, November 2024c. Association for Computational Linguistics. doi: 10.18653/v1 http//222...mmpmmaaaii 1016. URL http:::/addatthoooogooo/////// org/ emn 1016/
Avshalom Manevich and Reut Tsarfaty. Mitigating hallucinations in large Vision-lanuuag models (LVLMs) via language-contrasto decoding (LCD). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguisties: ACL 2024, pp. 6008 6022, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1 1118633/1122244.ff /2/////22.2222s--//// 3c1.39. URL wwwp///2.../222200// findings-acl..3555
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 2021.
PMLR,
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 40iom-400m: Open dataset of clip-filtered 400 million image-text pairs, 2021. URL https::/aaxxvv.orreeee 02114
9

---

Published in Transactions on Machine Learning Research (01/2026)
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. orXiv preprinl arXiv:3303. 15389, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: https: //arxiv. org/abs /ttp://3300099233
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024a. URL https:://arat wwwpxiiw...goaae... http//bb/2200.....
Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision- language models with instruction contrastive decoding. In Lei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 15840 15853, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18633/v1/2024 findings-acl.oo acl.937. URL
www..//4............./ findings-acl.99 .977/
Mert Yuksekgomul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023. URL
net/ wtt.:/orrrmm----LLxXeeeee
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVP international conference on compuler vision, pp. 11975 11986, 2023.
Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting rn-rall and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogriition, pp. 13774 13784, 2024.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's 's song in the ai ocean: A survey on hallucination in large language models, 2023. URL https://aaxii: www.xii..rg//bbbb. arg//bs//200 2309. 01219
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models, 2023. URL httpp:///raaa org.aas//rg////////// 2304. 10592]
10

---

Published in Transactions on Machine Learning Research (01/2026)
A Qualitative Analysis of F-CLIPScore Filtering
We first observed the degrec of overlap between the F-CLIPScore and CLIPScore metrics when filtering the data. At filtering rates of 20, 30, 40, 50, and 60, the proportions of overlapping image caption pairs filtered by both metrics were 57%, 62%, 67%, 71,, and 76%, respectively. We then randomly sampled 10 amage-captinn pairs that did not overlap between the two metrics' filtered sets at the 30% filtering rate, which was the setting that yielded the best performance when using the F-CLIPScore metric. The results are presented in Figure 4 As shown in Figure | the images filtered by CLIPScore (upper row) appear to include some that should not have been filtered out, despite having normal and semantically correct captions. Although cases such as (4) and (h) involve repetitive lexical usage where word is repeated even if it correctly refers to an object present in the image-it itt is still reasonable for such captions to receive lower scores. However, it is unfortunate that these samples were filtered only by CLIPScore and not by F-CLIPScore.
On the other hand, the images filtered by F-CLIPScore (lower row in Figure 4 include cases such as (l), where even though there is repetitive lexical usage, the corresponding object in the image is actually a model rather than a designer, making the filtering reasonable. There were also examples like (n), where the captioned object is missing from the image, and (r), where the image shows personal debt chart for "Brainy" rather than "brain". These qualitative samples provide insights into how filtering with F-CLIPScore influenced the performance of the LVLMs.
Furthermore, we measured the CLIPScore and F-CLIPScore for each caption pair across the entire LaaVA-Pretran dataset and sorted them ascending order, such that a higher score corresponds a higher rank. We then sorted the samples by the difference between the two ranks (F-CLIPScore rank CLIPScore rank) and selected the top 10 samples with the largest positive values (i.c., those that CLIPScore rated higher than F-CLIPScore, shown in the upper of Figure 5) and the bottom 10 samples with the largest negative values (i.e., those that F-CLIPScore rated higher than CLIPScore, shown in the lower row of Figure
As shown in Figure 5 the samples on side (I), which CLIPScore rated higher than F-CLIPScore, contain many rentrendered images. Conversely, the samples on side (II), which F-CLIPScore rated higher than CLore,ore, tend to exhibit duplicated lexical usage. These findings are consistent with the previous observations, and such trade-off is expected to make determining appropriate filtering rate more challenging.
B Licenses for the datasets and models used
The LLaVA pre-training code is licensed under the Apache License, Version 2.0. LLaMA 2 7B is released under the LLaMA Community License. The LLaVA pretraining dataset is licensed under a combination of LAION, CC, and SBU licenses. The OpenAI CLIP model is distributed under the MIT License. COCO is available under the Creative Commons Attribution 4.0 License (CC BY 4.0). Flickr30k is provided under a custom license that permits use for non-commercial rescarch and/or educational purposes. NoCaps is distributed under the Creative Commons Attribution 2.0 License (CC BY 2.0). We used the datasets and models in accordance with their respective licenses.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Helel cel aRU
(a) a view of the crowd during the
(b) a group food vectors white background
(c) a pair of samsung charger stand with charg- ing charger the background
(d) an illustration of lord krishns marathi language
(e) the duie and duchess of bridge pose for a family portrait
2016 nba allstar
game madisou arema august
30 2018 new
505
(f) blue and yel- low jungle sticker
(g] cliff cliff is the island of maui canvus bag
(h) the brown shoe shown with two black rubber outsole
(1) garmin edge gps watch with heart rate and gps coordinates
(j) ayrmoors long length coat 8
(I) Image samples filtered by CLIPScore at the 30% filtering rate
MAGICO
RESHANTIE
dermastir's de lux ultra wax
(k)
(I) elegant
(ml) pdf manual for microphone microphones for alto telephone
(n) stainless steel sink with three drainers and two spout faucets
(0)
horizon TM
black designer designer gown with
magico mesh coil
spf
beautiful
(p) tv the stone beach on stock
upward galactica the chart shows complete series the number of complaints about brain disease
(q) battlestard
(s) the official documents of the commission for commissioning remediar- nation of power transmission
(t) telefish garden pitcher in new ras- tle pa, the bou- quet
photo
blu - ray
units.png
(II) Image samples filtered by F-CLIPScore at the 30% filtering rate
Figure 4: Randomly sampled 10 imago-caption pairs each from (I) samples filtered by CLIPScore and (IT) samples filtered by F-CLIPScore at the 30% filtering rate. Images overlapping between the two metrics were excluded.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Penns
ESCAPING
Around The House Giveaway 1nd111//1
Utiiniaaa
EWITEE QNA
ittemsht tasortunbes Chsage
(a) the image shows photo of money trap with text escaping the debt trap mondaysmoneyuk twitter chat
(b) elderly couple with check presenta- tion from the penms valley ambulance
(c) lake michi- gan skyline and a chicago skyline with the text the ultimate list of top in- teruship opportu- nities in
(d) nikwax tech wash for bike hel- mets, helmets and helmets
(e) knife. clochet, and round-up of the nobox household appliances gift
paid
michigan
Elabennan
A70
LEGACY WEBINANS
Ways
HOLIDAY MODEAN MOBILE GAMER GUIDE
MARU FINN OOLEET
COMMUNITY BANK NEEOS
(f) samsung a70 battery drain trst, testing and battery at home in days tery indicator
(g) girl lying a skatehoard with the words holiday modern mobile game guide front
(h) ing hands outside of plane with the title, 5 ways to market your pri- vate jet charter
shak-
laptop, phone, coffee cup, and keyboard with the words things every community bank
(i)
(i) calendar with the words legacy family webinars for holiday meal
needs
(I) Top 10 samples with the largest positive ranking differences
Klondike
CE
(k) solitaire
slexander alexander double breasted coat
(m) screeuplay screenplay with scriptwriting, handwritten
(n) illusion
250gb
solitaire
solita
lusion illusion illu- sion in black and white
540 internal asd
solitaire solitaire
solitaire solitaire
solitaire solitaire
sol
T.
Le mida)
(p)
fujiffllm
(q) the gap and
(r) the headstone
(8) old cassette
(t) the algorithm, with allusion
m7 with the fuji
the gap the
of robert dutton
tape recorder
105mm f2
gaps gap of the
gaps gap the gap of the gaps is
(IT) Bottom samples with the largest negative ranking differences
Figure 5: The figure shows the top and bottom 10 samples from the entire LLaVA-Pretree dataset, sorted by the difference between the F- F-CLIPScore rank and the CLIPScore rank. (I) represents samples that CLIPScore rated higher than F-CLIPScore, while (II) represents the opposite cases. Each caption is written below its corresponding image.
13


Reviews:
[{"review_id": "Hah2jdukdF", "reviewer": "Reviewer_yfEB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a F-CLIPScore that is meant to better capture the image-text correlation than the traditional CLIPScore. The results are presented on a set of datasets and demonstrate a higher accuracy score when this metric is used for training. The paper is clearly written and well presented, but more insights could be given on the results to strengthen the quality of the findings.", "claims_and_evidence_yn": "No", "claims_explanation": "The authors introduce the F-clip score which adds N additional scores to the original CLIPScore between the full caption embedding and the visual embedding by analysing also the noun-sub-part of the captions. While the idea is interesting and the paper is perfectly written and argumented, I find the results so far insufficient and would have appreciated a better coverage of the findings. \n\nMy main concern stems from the unseen bias that (1) encompasses. When we add N additional CLIP scores, we are essentially adding cosine similarities (not exactly as CLIPScore is w*max(cos(c,v),0)) but one can think in terms of proportionality) from a larger pool of vectors. This introduces a bias that comes just from sampling multiple times from the vector space. It is true that there is a normalization happening with N+1 at the denominator, but I am not sure that this is sufficient. In cases with semantically close terms (e.g. kitchen, stove, cupboard) the score is adding together terms that are coming from a dense region where they are likely to be close to each other (and hence have higher cosine similarity). This may be biasing the score higher not because the caption is better, but simply because there are more semantically close terms in it. \nWhen we compare a short and a long caption that are equally correct this may become even more evident, e.g. \"A mug in a kitchen cupboard next to a stove\" could potentially score lower than \"A mug or drinking cup tucked away in a kitchen cupboard or storage cabinet next to a stove, cooker, or oven\" only because of the higher density of words in it. \n\nOne could also argue that the ad-hoc re-encoding of the words may not be the most robust way to obtain embeddings, as single-word inputs belong to a distribution that is shifted from what the text encoder was trained on. On the contrary, one could directly act at the attention level and use masks to obtain embeddings that only refer to the nouns in the sentence. The comparison between the two approaches would be interesting to see.\n\nThe result in Figure 2 is, to me, one of the most interesting of this paper, but it is poorly documented. It would have been interesting to see more insights on this aspect, for example presenting statistical tests that confirm this hypothesis first, on which relies the rest of the paper.", "audience_yn": "Yes", "audience_explanation": "The topic is of relevance to the TMLR audience encompassing technical development and new approaches to developing models and understanding hallucinations in vision captioning models.", "requested_changes": "I would like to see more evaluations on:\n \n- sequence length variations in the absolute character count and on the number of nouns \n- two captions with the same semantic meaning but one augmented by synonyms for the nouns in it (so having different lengths)\n- a better analysis on the claims associated to Figure2, for example by adding statistical tests on the two distributions and visualising  the distributions on cases with and without hallucinations (the ones in c but also for the image embeddings). \n- In Table2 it would be good to see the distribution of removed samples, to verify if they are far from the ones kept during training (and if they actually contain hallucinations).", "broader_impact_concerns": "No impact statement needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1ewkD11bk2", "reviewer": "Reviewer_DLAr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of object hallucination in Large Vision-Language Models (LVLMs), where generated captions mention objects that are not present in the image. Prior work attributed this issue to the limited representational capacity of the vision encoder.\nHowever, the authors argue that this explanation may be incomplete. They observe that even after fine-tuning, hallucination persists, and in some cases. This raises doubts about whether vision encoder capacity is the main cause.\nTo address this, the authors propose F-CLIPScore, an extension of CLIPScore that incorporates noun-level granularity by averaging the similarity scores of the full sentence and its constituent nouns. Without requiring additional training, they showed that F-CLIPScore improves hallucination detection accuracy (up to +39.6% ).\nThey further explore two use cases: (1) using F-CLIPScore as a training loss, which yields marginal gains over prior methods, and (2) using it as a data filtering metric to remove low-quality samples before LVLM training.\nThe authors conclude that F-CLIPScore is a valuable inference-time metric, and that object hallucination may stem more from limitations in evaluation and data quality than from vision encoder capacity itself. They also claim that using F CLIPScore as a data filtering tool during LVLM pretraining can reduce hallucination.", "claims_and_evidence_yn": "No", "claims_explanation": "I have several comments from different angles, which I point them out below.\n\n------\n### **Concerns on embedding distribution analysis (Figures 2b and 2c)**\nThe authors analyze cosine similarity distributions between pre-trained and fine-tuned CLIP models in Figures 2a–2c, to show that fine-tuning induces more substantial changes in text embeddings for hallucinated vs. non-hallucinated captions (Figure 2c), as compared to correct vs. incorrect predictions (Figure 2b). I have some concerns about the statistical rigor and interpretation on this experiment:\n\n1. **Unequal statistical populations across figures:**\nI have a question regarding the populations compared in Figures 2b and 2c; are they different in sample size? In 2b, each image contributes only one prediction (the caption selected by CLIP), whereas in 2c, all captions for each image (including both hallucinated and non-hallucinated) are considered? If so, then, this creates a mismatch in sample sizes and statistical properties, which could explain the apparent distributional differences in Figure 2c? This is to determine whether the observed shift is an effect of fine-tuning or an artifact of population sampling.\n\n2. **Lack of formal statistical testing:**\nThe paper interprets distributional shifts visually (via histograms), but no statistical tests are provided to support claims about significant differences. I recommend including e.g. two-sample tests along with FDR correction to validate whether the observed shifts are statistically significant.\n\n3. **Missing analysis of between-group distances:**\nThe authors focus on within-group similarity (i.e., how much each group’s embeddings changed after fine-tuning), but do not analyze the between-group separation, that is whether hallucinated and non-hallucinated captions become more distinguishable from each other after fine-tuning (similarly between correct and incorrect cases and similarity between hallucinated and non-hallucinated captions). This seems more relevant to the claim that fine-tuning helps discriminate hallucinations than the current reported distributions based on the similarity within the groups. \n\n-----\n\n### **Concerns about data filtering and validation of F-CLIPScore-based selection**\n\nIn Section 5.2, the authors apply F-CLIPScore as a data filter to remove the bottom x% of training samples from the LLaVA alignment set to show improved downstream performance on the POPE benchmark. I have several concerns regarding the interpretation and validation of this filtering procedure:\n\n4. **Lack of verification of removed samples:**\nThe paper assumes that a low F-CLIPScore corresponds to “bad training examples” (e.g., hallucinated or misaligned image-caption pairs), but this is never verified. There is no inspection of the filtered-out examples to confirm their quality. Thus, the conclusion that these samples are undesirable is speculative.\n\n5. **Assumption of proxy label without justification:**\nF-CLIPScore is treated as a proxy for caption quality or hallucination, yet its reliability as such is not established beyond correlation with performance metrics. It remains unclear whether the metric accurately identifies hallucinated samples or simply filters out atypical and more challenging examples.\n\n6. **Potential removal of challenging or rare samples:**\nA critical concern I have is that F-CLIPScore may penalize semantically valid but rare, ambiguous, or challenging samples, which naturally score lower due to unusual content or composition. By removing these, the model may be trained only on easy or typical cases and overfits or reduces robustness. The observed performance gain might thus come at the cost of generalization.\n\n7. **Implicit contradiction with dataset assumptions:**\nThe alignment dataset is originally constructed under the assumption that image-caption pairs are usable for training. Without evidence that removed samples are indeed faulty, the filtering contradicts this assumption without clear justification.\n\nSo, using F-CLIPScore as an automated system for data filtering could be interesting, but the paper’s interpretation of results should be significantly strengthened by a qualitative and quantitative analysis of the removed data. Also, measuring model performance on challenging or rare categories is necessary to assess whether the filtering introduces a bias toward “easy” examples.\n\n---\n\n### **Other comments:**\n\n8. The paper frames its contribution around object hallucination in LVLMs, but the experiments are primarily focused on CLIP-based retrieval tasks. There is no evaluation of hallucination in generated captions, which is central to LVLM. This limits the strength of the claims regarding their method’s effectiveness in actual caption generation scenarios.\n\n9. F-CLIPScore seems to be effective at inference. But its contribution as a training objective is very limited based on current results. So. this claim in the paper that it \"**could further serve as an orthogonal method for training VLMs**\" does not seem to be accurate to me.\n\n10. In Table 1, I expect the authors to also report F CLIPScore values for the trained models and include standard deviations for the untrained cases for a fair comparison. Similarly, in Table 2, no standard deviations are provided, which makes it difficult to judge. As the CLIPScore (w/o training) already reaches 53.1, which is close to the best case of F CLIPScore (55.5), the absence of standard deviations prevents a meaningful statistical comparison between the methods.", "audience_yn": "Yes", "audience_explanation": "The paper explores an interesting direction toward understanding the object hallucination issue in LVLMs from the perspective of evaluation metrics, though it still requires further development to support its claims with stronger evidence and analysis.", "requested_changes": "Based on my comments, the following changes, clarifications, and improvements are required:\n\n- clarification for statistical populations in embedding analyses\n\n- add statistical tests (e.g. two-sample t-test) for embedding similarity distributions\n\n- include analysis of between-group separation\n\n- verify the quality of filtered-out samples\n\n- assess the impact of filtering on challenging or rare samples\n\n- reconsider the claim of orthogonality in training with FCLIPScore loss\n\n- evaluate hallucination in caption generation\n\n- report F-CLIPScore for trained models\n\n- report standard deviations for all results", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}, {"review_id": "GJgIS24cWu", "reviewer": "Reviewer_NwnV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Strengths:\nThis paper proposes a novel approach to enhance the anti-hallucination capability of visual encoders without additional training, by observing and analyzing the cosine similarity between the trained CLIP in OHD-Caps and the original CLIP across different modalities. The method demonstrates effectiveness and provides insights into modality interactions.\n\nWeaknesses:\nHowever, the origin and derivation of the F-CLIPScore calculation formula are not thoroughly explained, which may hinder understanding and reproducibility. In addition, the experimental validation is limited, as comparisons are only performed on the OHD-Caps benchmark. This raises concerns regarding the generalizability of the proposed method. It would be beneficial to include more comprehensive experiments on additional benchmarks, such as LLaVA or other relevant datasets, to further substantiate the effectiveness and robustness of the approach.", "claims_and_evidence_yn": "No", "claims_explanation": "The motivation for proposing the F-CLIPScore calculation formula in this paper is not sufficiently clear, and the derivation process is not well articulated, making it difficult to convince readers of the formula’s validity. In addition, the limited number of experiments makes it hard to demonstrate the generalizability of the proposed method.", "audience_yn": "Yes", "audience_explanation": "The method proposed in this paper enables rapid assessment of image-text matching without the need for additional training, which indeed has certain practical applications.", "requested_changes": "Please provide a detailed explanation of the motivation behind the improvements made to the OHD-Caps method, and clarify the underlying reasoning for proposing the F-CLIPScore calculation formula.\nIt is recommended to add more experiments to demonstrate the generalizability of the proposed method, and to present results on additional LLaVA-related benchmarks to show that the method does not negatively affect other benchmarks.\nFurthermore, it is necessary to validate the effectiveness of the method on other more state-of-the-art LVLM models.", "broader_impact_concerns": "This paper does not discuss the broader impact issues. It is recommended that the authors supplement the relevant content, including but not limited to: potential ethical risks arising from the application of the technology (such as privacy protection and algorithmic bias), societal impact (such as effects on vulnerable groups), legal compliance (such as whether data usage complies with laws and regulations), and environmental impact (such as energy consumption). Addressing these aspects will help improve the completeness and social responsibility of the paper.", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}, {"review_id": "IQw2GRy3Fa", "reviewer": "Reviewer_44XK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper challenges the prior claim that object hallucination in Large Vision-Language Models (LVLMs) primarily stems from the limited representational capacity of vision encoders. The authors introduce Fine-grained CLIPScore (F-CLIPScore), a simple, training-free evaluation metric that refines the standard CLIPScore by incorporating noun-level text embeddings (extracted via a parser like spaCy) alongside sentence-level embeddings. This enhances object-level granularity in assessing image-text alignment. Key contributions include: \n(1) Empirical evidence from cosine similarity analyses and failure mode patterns showing that vision encoder limitations are not the main cause of hallucinations; \n(2) Demonstrating that F-CLIPScore achieves a 39.6% accuracy improvement on the OHD-Caps benchmark without training, outperforming baselines and even some trained methods; \n(3) Showing that using F-CLIPScore for pretraining data curation in LVLMs (e.g., LLaVA) reduces hallucinations by 4.9% on POPE, with fewer data samples.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well-supported by a combination of motivational analyses (e.g., cosine similarity distributions in Figure 2 showing minimal vision embedding changes post-fine-tuning) and rigorous experiments. Evaluations on OHD-Caps demonstrate clear accuracy gains (e.g., Table 1: 39.6% improvement over baseline CLIPScore), with multiple runs for variance and comparisons to trained models. Data filtering results on POPE (Table 2) convincingly show benefits like 4.9% accuracy uplift, isolated from confounders via random and alternative filtering baselines. Robustness checks in the appendix (e.g., Tables 3-6) further validate the metric across architectures (e.g., CLIP variants, SigLip, EVA-CLIP), though some experiments still need to be supplemented.", "audience_yn": "Yes", "audience_explanation": "TMLR's audience, which includes machine learning researchers focused on vision-language models, multimodal AI, and hallucination mitigation, would find this relevant. The paper addresses timely issues in LVLM reliability, such as object hallucinations, and offers practical tools for efficient training and evaluation. Findings on reevaluating vision encoder roles and data curation could influence work in scalable ML systems, aligning with TMLR's emphasis on foundational and applied research in areas like representation learning and model evaluation.", "requested_changes": "- Add ablation or comparative experiments to test scenarios involving the extraction of words with other properties (e.g., verbs or adjectives), random word extraction, or individual word extraction separately.\n- Recommended: Add experiments or evidence to demonstrate that this noun-focused method does not affect the visual encoder's representation of other words and the overall sentence.\n- Minor: Fix minor typos (e.g., the bracket error in the citation of the 4th paragraph in the Introduction section).", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)