HICEScore: A Hierarchical Metric for Image Captioning Evaluation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:28:38

Abstract: Image captioning evaluation metrics can be divided into two categories: reference-based metrics and reference-free metrics. However, reference-based approaches may struggle to evaluate descriptive captions with abundant visual details produced by advanced multimodal large language models, due to their heavy reliance on limited human-annotated references. In contrast, previous reference-free metrics have been proven effective via CLIP cross-modality similarity. Nonetheless, CLIP-based metrics, constrained by their solution of global image-text compatibility, often have a deficiency in detecting local textual hallucinations and are insensitive to small visual objects. Besides, their single-scale designs are unable to provide an interpretable evaluation process such as pinpointing the position of caption mistakes and identifying visual regions that have not been described. To move forward, we propose a novel reference-free metric for image captioning evaluation, dubbed Hierarchical Image Captioning Evaluation Score (HICE-S). By detecting local visual regions and textual phrases, HICE-S builds an interpretable hierarchical scoring mechanism, breaking through the barriers of the single-scale structure of existing reference-free metrics. Comprehensive experiments indicate that our proposed metric achieves the SOTA performance on several benchmarks, outperforming existing reference-free metrics like CLIP-S and PAC-S, and reference-based metrics like METEOR and CIDEr. Moreover, several case studies reveal that the assessment process of HICE-S on detailed texts closely resembles interpretable human judgments. The code is available in the supplementary.

Main paper: 
 introduction: 
 Image captioning (IC) [47, 52] is a fundamental task in visionlanguage (VL) multi-modal learning, which aims to generate descriptions in the natural language given an image. In recent years, IC has attracted more and more attention from researchers since it has wide applications such as helping the visually impaired people [6] ,References:• A few bags laying around in a living room.• A living room with a couch, two tables and a painting.• A living room with a couch and a bottle of wine.• A white couch a brown table with a bottle and glass.• Black bags sitting on top of a white couch in a living room.A living room with a couch and a table.A living room with a couch with pillows and a table.The image showing a view of a living room. The room is furnished with a white sofa and a wooden coffee table. The walls are adorned with a patterned wallpaper, and a window with a curtain is visible. The overall ambiance is warm and inviting.The image showing a view of a living room. The room is furnished with a white sofa and a wooden coffee table. The walls are adorned with a white decorated painting and a window with a curtain is visible. The overall ambiance is warm and inviting. The evaluation scores that agree with human judgments are highlighted in green and disagree in red.Globaland it also serves as the foundation of other image-to-text tasks like Visual Question Answering [4, 53] and Video Captioning [7, 46] . Based on the encoder-decoder framework, various advanced IC methods have been proposed in the past few years, by extracting meaningful visual information [4, 48, 55] , considering effective VL feature alignment [17, 31, 52, 57] , and incorporating the knowledge from pre-trained large models [24, 25, 33] .In parallel with the improvement of IC models, increasing efforts have also been dedicated to automatic quality evaluation for generated captions. Besides evaluation, a good metric is also a good learning guidance that can in turn improve the performance of IC models [15, 50] .Traditional referenced-based metrics like BLEU [32] , METEOR [5] , CIDEr [45] , etc., are based on n-gram similarity between the generated caption and the human-written references, which focuses on exact phrases matching. However, despite being the most popular measurements for conventional image captioning models, these metrics still have two major limitations that hinder their potential applications in real-world scenarios. First, human-annotated references are time-consuming and only contain partial visual content. These metrics struggle to correctly identify the visual concepts not appearing in the limited references. As shown in Fig. 1 (a), METEOR and CIDEr prefer the brief caption 1 due to "pillows" exist in the image but are not included in the references. Secondly, recent sophisticated multimodal large language models (MLLMs [29, 65] ) are inclined to produce detailed captions describing rich details of visual objects. Meanwhile, these detailed captions often exhibit unique linguistic styles that differ from the references. Consequently, traditional metrics often experience a significant decline in performance when assessing those detailed captions due to the notable disparities in sentence length and word choice compared to the references.To break through the above limitations, with the help of Contrastive Language-Vision Pretraining (CLIP) model [33] , referencefree metrics CLIP-S [12] and PAC-S [36] are proposed by evaluating the similarity between the given image and the candidate caption using CLIP. Specifically, CLIP projects the original images and sentences to a shared embedding space to calculate cosine similarity between two modalities. Though free of time-consuming humanannotated references and robust in both diverse descriptive styles and sentence lengths, it is often observed that CLIP-S and PAC-S fail to detect hallucinations in the caption, and small objects in the image, as shown in Fig. 1(b ). This might be due to the fact that CLIP is trained on web-scale noisy image-text pairs [37] , and learns global representations for the whole image and text, ignoring local cross-modality relations among image regions and text phrases [62] . Besides, this inherent characteristic renders them incapable of performing human-like interpretable evaluation capabilities to pinpoint concrete textual mistakes or identify semantic visual regions that may have been omitted.In general, human experts typically evaluate the quality of image descriptions from two perspectives: i) Correctness. Are there any mistakes existing in the captions? If so, where do they occur? ii) Completeness. Has all visual content been well comprehensively described? What has been included and what has been left out?Building on these analyses, in this paper, we propose a novel reference-free metric for IC which is closer to human judgment, named Hierarchical Image Captioning Evaluation Score (HICEScore, abbreviated to HICE-S) as shown in Fig. 2 . Specifically, HICE-S is calculated by gathering the image-caption similarities globally and locally as follows. For global evaluation, we follow previous reference-free metrics and employ a pre-trained VL model to extract image and text representations and then compute global image-text similarity based on distances in the shared embedding space. For local evaluation, we first construct semantic image regions set and informative text phrases set to decompose the visual content and candidate captions, respectively. Subsequently, these two sets are used to measure local similarity by computing the description completeness of semantic regions (recall) and the correctness of text phrases (precision). Finally, HICE-S combines the global and local similarities as a fusion score for reference-free IC evaluation. Similar to previous reference-free metrics, HICE-S can also be easily extended to RefHICE-S when human references are provided. Except for image-text compatibility (ITC), RefHICE-S additionally computes hierarchical text-text compatibility (TTC) between references and captions. To evaluate the effectiveness of our proposed metric, we conduct extensive experiments on a series of benchmarks from different perspectives, including the correlation with human judgment, caption pairwise ranking, hallucination sensitivity, and system-level correlation.Our main contributions can be summarized as follows.•  
related work: 
 Image captioning evaluation methods can be roughly divided into two categories, reference-based and reference-free, according to whether a human-provided reference is available.As references are provided, such as the COCO dataset [28] containing five human-written annotations for every image, we can evaluate the IC performances by calculating the reference-caption similarity, i.e. , text-text compatibility (TTC). To this end, several n-gram matching-based metrics are proposed. Specifically, the initial n-gram-based metrics are borrowed from language generation tasks, such as BLEU [32] and METEOR [5] from machine translation; ROUGE [27] from document summarization. Further considering the unique property of IC task that different words in the caption have different importance for describing the given image (e.g. , verbs and nouns are always more important than articles), CIDEr [45] and CIDEr-D [45] use TF-IDF to assign weights on different n-grams.To overcome the limitations of these methods sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment, SPICE [3] and SoftSPICE [26] are proposed by considering the semantic propositional content defined over scene graphs.Recently, with the advance of pre-trained large models, some researchers [9, 16, 18, 20, 22, 23, 54] proposed to conduct assessment in the text-feature domain rather than the original word space. For instance, BERT-S [23] and its enhanced version BERT-S++ [54] use the pre-trained BERT [18] to transform word tokens into an embedding space for similarity calculation. Besides, ViLBERTScore [23] and TIGEr [16] are proposed to incorporate visual information into TTC calculation. However, as shown in Fig. 1 , the above referencebased metrics have a performance degradation when assessing detailed captions, due to the large gap in sentence lengths and visual granularity compared to the brief references.The aforementioned reference-based methods inevitably face that acquiring human-written references for various real-world applications can be prohibitively expensive [36, 41, 51, 64] , and it is often insufficient to compare generated captions with even multiple references for each image [12] . As a result, there is a large gap remaining between reference-based metrics and human judgments [10, 19] .To solve this problem, reference-free metrics are proposed based on a large VL model CLIP [33] , which is trained through the visuallanguage contrastive loss on a large web-scale image-text-paired dataset [37] . Specifically, CLIP-S [12] is the first reference-free IC metric by using pre-trained CLIP to calculate the image-caption similarity, i.e. , image-text compatibility (ITC). To further enhance CLIP-S, PAC-S [36] proposes to finetune the CLIP with positiveaugmented contrastive learning. Since CLIP-S and PAC-S employ the CLIP text encoder to obtain the sentence embeddings for the calculation of TTC, they can be extended to reference-based metrics by fusing the image-caption (ITC) and reference-caption (TTC) similarities, which are named RefCLIP-S and RefPAC-S, respectively. However, considering only the global evaluation which treats the image and the caption as a whole with CLIP model [33] , CLIP-S [12] and PAC-S [36] fail to detect hallucination in the caption and they are not sensitive to visual details in the image. To address finegrained evaluation, the InfoMetIC builds an object-token similarity score. Nevertheless, InfoMetIC is designed for regular brief captions where an object is generally described by a single token. To better represent local regions from detailed captions, our proposed HICE-S exploits a graph parser to decompose the descriptive texts into more informative local descriptions rather than a single word.To provide a more human-correlated evaluation metric, HICE-S considers hierarchical image-text compatibility evaluation, showing a superior ability to assess whether the caption expresses full visual details. It enables HICE-S to perform better for IC evaluations. 
hierarchical image captioning evaluation 3.1 problem formulation and preliminaries: 
 Given an image I and a candidate caption C for evaluation, our proposed reference-free metric HICE-S is developed for calculating a similarity score between them, denoted as HICE(I, C). Following CLIP-S and PAC-S, we also present a reference-based version RefHICE(I, C, R ℎ ) when human references R ℎ are available.Alpha-CLIP. Compared with the vanilla CLIP [33] which is trained via global image-text contrastive learning, Alpha-CLIP [42] is a region-aware enhanced CLIP with an auxiliary alpha channel. Specifically, the alpha channel takes a binary region mask as input to facilitate attention to specific regions. Thus, HICE-S steers Alpha-CLIP to evaluate region-text similarity, as:EQUATIONwhere A is the binary region mask fed into the alpha channel, C 𝑣 (•), C 𝑡 (•) are visual encoder and text encoder, respectively.cos (•, •) denotes the cosine similarity.Although achieving a human-like evaluation effect, the existing reference-free metrics only focus on the global representations of the whole image and text, but ignore the local relations among image regions and text phrases. As a result, they may ignore some visual details such as small objects. To overcome this limitation, as shown in Fig. 2 , HICE-S is developed on a hierarchical structure with both global and local similarity evaluations. In practice, the reference-free HICE-S primarily focuses on ITC (image-text: between I and C), while the reference-based RefHICE-S considers the combination of ITC and TTC (text-text: between R ℎ and C), simultaneously.As in previous works [12, 36] , we transform the image and text to the shared embedding space to compute similarity. (EQUATIONii) Global TTC: We use Alpha-CLIP text encoder C 𝑡 (•) to obtain the similarity between human references R ℎ and the caption C asgTTC(R ℎ , C) = cos (C 𝑡 (R ℎ ), C 𝑡 (C)).(3)For local evaluation, we firstly need to obtain local representations for images and texts, and then calculate their similarities. The illustration is shown in Fig. 3 .Local representations. We first utilize a pre-trained lightweight segmentation model and a textual graph parser to extract semantic visual regions and informative textual phrases. Then we feed them into the corresponding feature extractor C 𝑣 (•) and C 𝑡 (•) to obtain the local representations.i) Image-region representation set F 𝐼 : Recently, segment-anything models (SAM) [21, 58, 59 ] has shown powerful capabilities in zeroshot image segmentation field. Many recent works applied SAM to various image-related tasks like medical image analysis [30, 40, 61, 63] , object detection [2, 11, 43] , and 3D reconstruction [39] . In our work, we use its segment-everything mode to transform the original image to multiple semantic regions with corresponding binary region masks {A 𝑛 } 𝑁 𝑛=1 , where 𝑁 denotes the number of region masks. Then, the Alpha-CLIP visual encoder is employed to obtain the representation for image regions as 𝒇 𝐼,𝑛 = C 𝑣 (𝐼, A 𝑛 ), which can form a set of image-region representation as F 𝐼 = {𝒇 𝐼,𝑛 } 𝑁 𝑛=1 . ii) Text-phrase representation set F 𝐶 for captions C; F 𝑅 ℎ for human references R ℎ : Transforming the complete sentences to the text graph whose nodes are nouns while relations are verbs and preposition is effect for language understanding [26] . Motivated by this, given C (or R ℎ ), we use a textual scene graph parser to get a text graph having 𝑀 𝑐 (or 𝑀 ℎ ) subject-predicate-object triplets {𝑟 𝐶,𝑚 } 𝑀 𝑐 𝑚=1 (or {𝑟 𝑅 ℎ ,𝑚 } 𝑀 ℎ 𝑚=1 ). Each triplet is equivalent to a short sentence, which is fed into the Alpha-CLIP text encoder to get the representation for these triplets as 𝒇 𝐶,𝑚 = C 𝑡 (𝑟 𝐶,𝑚 ); 𝒇 𝑅 ℎ ,𝑚 = C 𝑡 (𝑟 𝑅 ℎ ,𝑚 ). Then, we obtain the sets of text-phrase representations for caption C as F 𝐶 = {𝒇 𝐶,𝑚 } 𝑀 𝑐 𝑚=1 . for human references R ℎ asF 𝑅 ℎ = {𝒇 𝑅 ℎ ,𝑚 } 𝑀 ℎ 𝑚=1 .Flickr8k-Expert [13] Flickr8k-CF [13] Composite [1] Kendall Local-similarity evaluation. The calculations of similarity between F 𝐼 and F 𝐶 for local ITC and F 𝑅 ℎ and F 𝐶 for local TTC are shown in Fig. 3 . Concretely, we use the cosine similarity to get a pair-wise similarity matrix that denotes the similarity between any two elements from two sets. Then, we use the harmonic mean hMean of precision P and recall R to measure the similarity between two sets. i) Local ITC: We calculate the harmonic mean of precision and recall scores between two sets F 𝐼 and F 𝐶 to get the local ITC asEQUATIONEQUATIONEQUATIONwhere precision P(I, C) represents the correctness of captions with respect to the images, while recall R(I, C) represents the completeness of captions which measures whether all semantic regions are well mentioned. i) Local TTC: In a similar way, we derive the local TTC between two sets F 𝑅 ℎ and F 𝐶 through the harmonic mean of the precision and recall scores, formulated as:EQUATIONEQUATIONEQUATIONFinally, after the combination of global and local evaluation from the ITC view, with gITC in (2), lITC in ( 6), and we introduce our proposed reference-free metric for IC asEQUATIONSimilar to RefCLIP-S [12] and RefPAC-S [36] as extensions of CLIP-S and PAC-S, respectively, when the human-provided references R ℎ are available, we can present a reference-augmented version of HICE-S, i.e. RefHICE-S, which achieves higher correlation with human judgment. Specifically, we just need to further consider the global TTC score, gTTC in Eq. 3 and local TTC score 𝑙TTC in Eq. 9.Then, same as Eq.10 to calculate the harmonic mean of those four items, we have the RefHICE -S(I, C, R ℎ ). EQUATION 
experiments: 
 In this section, we conduct comprehensive experiments including correlation with human judgments in Sec. 4.2, caption pairwise ranking in Sec. 4.3, object hallucination sensitivity in Sec. 4.4, and system-level correlation in Sec. 4.5, to demonstrate the superiority of our proposed metrics HICE-S and RefHICE-S. Furthermore, we conduct ablation experiments and present some discussion to investigate the roles of each part in our metric.Owing to the strong generalization capabilities of large models, HICE-S and RefHICE-S employ distinct pre-trained models to achieve different functionalities without further fine-tuning. We choose Alpha-CLIP-ViT-L/14 to compute global and local similarity for both ITC and TTC. In the local evaluation, we utilize a fast and lightweight segment-everything model mobileSAMv2 [59] to extract semantic image region masks. For the textual phrases extraction, we need to transform a sentence to several subject-predicate-object phrases, which is realized by TextGraphParser [26] .Compared metrics. Following previous works [12, 36] , we mainly compare our metrics with three types of metrics, one for referencebased and one for reference-free. i) n-grams-based metric is the reference-based one including BLEU [32] , METEOR [5] , ROUGE [27] , CIDEr [45] , and SPICE [3] . ii) embedding-based metric is another type of the reference-based one including BERT-S [23] , BERT-S++ [54] , LEIC [9] , TIGEr [16] , UMIC [22] , VilBERTScore [23] ,FOIL [38] Acc. ( 66.2 87.0 ROUGE [27] 54.6 70.4 METEOR [5] 70.1 82.0 CIDEr [45] 85.7 94.1 MID [20] 90.5 90.5 CLIP-S [12] 87.2 87.2 PAC-S [36] 89.9 89.9 HICE-S 93.1 93.1△ (+3.2) (+3.2)RefCLIP-S [12] 91.0 92.6 RefPAC-S [36] 93.7 94.9 RefHICE-S 96.4 97.0△ (+2.7) (+2.1)Table 3 : The accuracy results on the FOIL hallucination detection datasets [38] with one reference and four references, respectively.SoftSPICE [26] , and MID [20] and infoMetIC [14] . iii) CLIP-based metric is recently proposed reference-free one including CLIP-S [12] and PAC-S [36] as well as their reference-based extension RefCLIP-S and RefPAC-S. Our metric HICE-S belongs to the reference-free one and its reference-based extension is RefHICE-S.Given an image-caption pair, this task is to see whether the scoring of the metric is consistent with the human scoring (often contains more than one person and takes average). To this end, we conduct the experiment on three widely adopted human judgment datasets Flickr8k-Expert, Flickr8k-CF, and Composite [1, 13] . Following previous approaches [12, 36] , we compute both Kendall 𝜏 𝑏 and Kendall 𝜏 𝑐 correlation scores for each dataset.Dataset. The Flickr8k-Expert [13] contains three expert human rating scores for each image-caption pair, with 5,664 images and 17k expert annotations in total. The evaluation scores range from 1 to 4 and the higher the better. The Flickr8k-CF [13] is collected from CrowdFlower [44] with 48k image-caption pairs. The assessment on Flickr8k-CF is binary where "yes" ("no") represents that the candidate caption is (not) relevant to the image. Each imagecaption pair has more than three scores. Following [12, 36] , we use the mean proportion of "yes" annotations as the final score for each pair to compute the correlation with human judgments. The Composite [1] is composed of 12k human judgments with 3,995 images from Flickr8k [13] (997 images), Flickr30k [56] (991 images), and COCO [28] (2007 images). Each image-caption pair is evaluated with a human score, ranked from 1 to 5 where a higher score indicates the better matching between the caption and the image.Results. Results on Flickr8k-Expert, Flickr8k-CF, and Composite are reported in Table 1 . Our proposed metric achieves the best correlation results with human judgment compared to previous metrics. Specifically, on the Flickr8k-Expert, the HICE-S and RefHICE-S exhibit significant advantages over the SOTA score PAC-S [36] and RefPAC-S [36] , with more than 1.5 points gained in terms of both 𝜏 𝑏 and 𝜏 𝑐 . On the Flickr8k-CF and Composite, our metrics also perform better than previous models, in particular under the reference-free scenarios.Given an image with two captions, this task is to see whether the better caption selected by metric is consistent with the human choice.Dataset. Pascal-50S [45] contains 1k images collected from UIUC PASCAL Sentence dataset [34] and 4k sentence pairs, which are split into four groups, two correct captions written by human (HC); one incorrect and one correct caption by human (HI); two correct captions with one written by human and the other one by machine (HM); two correct captions generated by machine (HM). Each caption pair has 48 human judgments about which one better matches the image. The vote is based on the principle of majority rule, while a random selection will be operated in case of the equal number of votes. Following [36] , we compute the accuracy instead of correlation coefficients and select 5 references among the 48 human references to serve as ground truth to compute referencebased metrics (average over 5 random draws of references).Results. In Table . 2, we provide the accuracy results. Obviously, under the reference-free scenarios, our proposed HICE-S outperforms SOTA metrics, CLIP-S [12] and PAC-S [36] by a large margin, about 3.7 points improvements on average accuracy of four categories, as shown in the "Mean" column. Even compared to reference-based scoring methods, our HICE-S metric is still in the lead with 6 points better than CIDEr [45] , 2.8 points than RefCLIP-S [12] and 1.4 points higher than RefPAC-S [36] . Such an excellent performance can be attributed to the introduction of hierarchical evaluation designs, which are more in accordance with human criteria. Furthermore, our reference-based metric RefHICE-S also achieves superior accuracy results compared to previous methods.As mentioned by Rohrbach in [35] , current image captioning models are prone to generate objects that are not present in the images, aka, object hallucinations. Thus, it is critical to identify object hallucinations for IC evaluation Dataset. FOIL dataset [38] contains 32k images from COCO validation set [28] . Each image is accompanied by a correct-foil caption pair, where the correct sentence is collected from the original COCO annotations [38] while the foil sentence is generated by replacing a noun with a similar but incorrect description. Following previous works [12, 36] , the accuracy is derived as the percentage of higher scores assigned to the correct captionResults. The results are available in Table 3 under both one reference-available case and four reference-available case [12, 36] . Note that since the reference-free methods do not use reference, the performance is the same between the two columns. It can be observed that our proposed HICE-S and RefHICE-S outperform the existing best metric PAC-S [36] with a large margin, especially for reference-free evaluation. For reference-based cases, the improvement of RefHICE-S on "1 Refs" is higher, which is more difficult than "4 Refs", since reference is useful to find hallucinations [12, 36] .We also propose to evaluate the efficaciousness of HICE-S and RefHICE-S within image captioning models. Table 4 has shown the evaluation results on images from the COCO testing set [28] for both brief caption generations and detailed caption generations. The results about different evaluation metrics are exhibited, including BLEU-4 [32] (B@4), METEOR [5] (M), CIDEr [45] (C), CLIP-S [12] , PAC-S [36] , and our HICE-S.For conventional brief caption evaluation, we assess the caption generations of some SOTA methods including M 2 Transformer [8] , VinVL [60] , and BLIP-2 [24] . As we can see, both reference-based metrics (METEOR, CIDEr) and reference-free metrics (CLIP-S, PAC-S, and our proposed HICE-S) can effectively evaluate the previous image captioning methods and identify the best captioning model.For detailed caption evaluation, we leverage an advanced multimodal large language model LLaVA-1.6 [29] to generate detailed captions of different caption lengths with a prompt template "describe the image scene in {length} words". As we can see, CIDEr is unable to assess captions whose lengths are larger than 65 words, presenting nearly zero scores. HICE-S in Table. 4 : Ablation studies of our proposed HICE-S on Flickr8k-Expert dataset [13] , Pascal-50S [45] and FOIL dataset [38] . global and local denote global and local evaluation, respectively.longer the caption, the lower the precision score, and conversely, the higher the recall score. This demonstrates that although longer captions can express more visual content, they often tend to introduce more errors and hallucinations. To further explore the correlation with human judgments, we invite 5 human experts to assess the LLaVA captions from both correctness and completeness perspectives. Fig. 5 has shown the comparison results between human assessment and HICE local assessment. The high correlation demonstrates the effectiveness of our proposed hierarchical evaluation.To further investigate the roles of each part in our proposed metric, we conduct the ablation experiments on Flickr8k-Expert [13] , Pascal-50S [45] and FOIL [38] .Hierarchical evaluation. The first row of Table 4 displays the performance results of the individual global evaluation, while the second row shows the individual local evaluation results. The third row presents the combined results of global and local evaluations. When adopting a hierarchical evaluation, there is significant performance improvements on Flickr8k-Expert, Pascal-50S, and FOIL. Finally, our HICE-S achieves the SOTA results on all datasets by hierarchical evaluation.Interpretable evaluation process. As presented in Fig. 6 , HICE-S can perform human-like interpretable evaluation process. For correctness evaluation, HICE-S can effectively recognize correct phrases and incorrect phrases that are not included in the image content. For completeness evaluation, HICE-S can also identify the mentioned region and match textual phrases. 
conclusion: 
 In this paper, we propose a novel reference-free metric for image captioning evaluation, dubbed HICE-S. We introduce a hierarchical evaluation design, which includes the global evaluation, built on the image-level and the sentence-level representations, and the local evaluation, based on the region-level and the phrase-level features. The combination of global evaluation and local evaluation renders our HICE-S more sensitive to object hallucination and minor errors. Extensive experiments demonstrate that our proposed HICE-S and the reference-based RefHICE-S are superior to all previous metrics including reference-free and reference-based methods on correlations with human judgments, caption pairwise ranking, and hallucination detection.  
 
 Appendix:  


Decision: The paper initially received mixed ratings (2BA, BR, WR), which have then slightly moved towards positive after the rebuttal phase (3BA, 1WR). While the concerns raised from the negative reviewer are appropriate and sound, the paper still has value and is above the bar for an acceptance at ACM Multimedia. In particular, the usage of fine-grained semantic regions in an image captioning metric has never been proposed before, and the proposed metric beats the SoTA in terms of human correlation on different datasets. The number and kind of experiments and ablations performed is in-line with those done in recent image captioning metrics published in major venues (e.g. CVPR). Also, the quality of the paper and of the writing are above the bar. Nevertheless, I strongly encourage the authors to address the comment from all reviewers in the camera ready version of the paper. The paper, overall, can be accepted as poster.

Reviews: review: paper_summary: paper_summary: This paper propose a reference-free image captioning metric, called HICEScore. HICEScore considers two aspects: global and local information.  For global matching, authors compute similarity between the image and text embedding. For local matching, they extract triplets by mobileSAMv2 and TextGraphParser for images and text respectively.
strength_weakness: strength_weakness: 1. HICE includes a hierarchical score for image captioning evaluation: the global score for image-caption compatibility and the local score for region-phrase compatibility.

2. HICE can pinpoint incorrect textual mistakes and present unmentioned visual regions.

3. This paper is well-organized and well-written, making it easy to understand.
limitations: limitations: Dividing the evaluation of image and text matching into global and local levels is not innovative. InfoMetIC also uses the fine-grained information to evaluate image-caption pairs. From the results on open source benchmark, the effectiveness between HICE and InfoMetIC is very similar. One potential advantage of HICE is its ability to evaluate long captions. But Fig. 4 only shows the score of detailed captions generated by LLaVA-1.6 under multiple metrics, which does not demonstrate the advantages or disadvantages between HICE and InfoMetIC. Fig. 5 shows the correlation between human and HICE. This is not compared to InfoMetIC. I suggest that the author should compare more with the reference free method on detailed captions, and also publicly release the ratings of five human experts on LLaVA captions as a dataset.
paper_summary: paper_summary: For developing reference-free metrics for evaluating the quality of detailed image captioning, this work proposes a metric namely HICEScore by leveraging a segmentation model to detect visual region masks, a text model to extract graphs, and a vision-language model to calculate the cross-modal similarity. Without extra training, their metric achieves state-of-the-art correlation with human judgment on multiple benchmarks.
strength_weakness: strength_weakness: 1. A reference-free metric achieves SOTA correlation with human judgment without extra training.
2. They perform sufficient experiments on multiple benchmarks to verify the effectiveness of their model.
limitations: limitations: 1. InfoMetIC is a reference-free metric while wrongly classified as reference-based metrics in lines 635-662. Therefore, HICE-S doesn't achieve SOTA performance on Pascal-50S and Composite under reference-free scenarios, descriptions in lines 716-718 and 738-741. However, I still recognize the effectiveness of HICE-S because InfoMetIC should be fine-tuned on captioning datasets while HICE-S is not needed. Authors should give more discussion about the comparison with InfoMetIC about whether need extra training during experiments.

2. Both InfoMetIC and HICE-S consider global and local similarity. Since HICE-S is declared more suited for evaluating detailed descriptions, a direct correlation comparison of InfoMetIC and HICE-S with human judgment on detailed descriptions should be given. For example, authors can add extra two lines of InfoMetIC to Figure 5.
paper_summary: paper_summary: The paper introduces HICE-S, a reference-free metric for evaluating image captioning. The proposed of HICE-S aims to overcome the limitation of reference-based and CLIP-based metrics methods. It uses a hierarchical approach to assess both global image-text compatibility and local region-phrase matches, allowing for more detailed and accurate evaluations. HICE-S is shown to be highly effective in benchmarks, outperforming other metrics in correlating with human judgment and detecting inaccuracies in captions.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow.
2. This paper proposes a local evaluation strategy to better model the similarity between visual concepts and phrases with the help of segment anything. It also incorporates a method to aggregate hierarchical scores.
3. The author conducts extensive experiments across multiple datasets, which suggest the data generalizability of the proposed method.
limitations: limitations: 1. The contribution seems to be limited. The previous reference-free evaluations (e.g., InfoMetIC [1]) also consider the fine-grained region-word level similariy. And in this paper, the author uses a mask-phrase level similariy with the help of  segment anything. Are the performance improvment taken from the stronger detector (sam vs. bottom-up detector)?
2. The author state InfoMetIC is a reference-based method (L635-L662), however InfoMetIC is a reference-free method as stated in its paper.
3. The experiment evaluate recent MLLMs such as LLAVA-1.6 and BLIP-2. It may be beneficial to include results for GPT4V to enhance the reliability of the experimental findings.
4. The HICE-S rely on segment anything, which visual encoder is also widely used by some recent proposed MLLMs (e.g., DeepSeek-VL [2]). Does this reliance on segment anything make HICE-S evaluations potentially unfair towards these models?
[1] Hu, A., Chen, S., Zhang, L., & Jin, Q. (2023). InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation. arXiv preprint arXiv:2305.06002.
[2] Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., ... & Ruan, C. (2024). DeepSeek-VL: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525.
paper_summary: paper_summary: The paper introduces a hierarchical evaluation metric for image captioning, called Hierarchical Image Captioning Evaluation Score (HICE-S), that focuses on capturing more detailed descriptions. This novel approach aims to better align with human judgments and overcome the limitations of existing metrics that either rely too heavily on limited human-annotated references or fail to detect local textual hallucinations and small visual objects. Specifically, the authors develop a global and local similarity evaluation to better align with human judgments.  They demonstrate the effectiveness of the HICE metric across several datasets.
strength_weakness: strength_weakness: * The claim is clear and also well supported through a range of experiments. 
* The motivations are strong, grounded in real-world problems, and are driven by some problems observed in the currently used metrics.
* The ability of the model reported in Figure 6 is extremely interesting
limitations: limitations: * The paper uses the Alpha-CLIP-ViT-L/14 backbone, while some competitors, such as CLIPScore and PACScore, employ the CLIP ViT-B/32 backbone.  This discrepancy makes the comparison reported in the tables unfair.
It would be more fair to report in all the main tables the Clip ViT-L/14 or OpenClip ViT-L/14 CLIP-S and PAC-S scores.
* The model's use of a detector to obtain masks increases computational requirements as the number of objects present in the image grows.  It would be interesting to compare the time required to compute the metric when varying the number of detected masks.	
Moreover, for all the evaluation datasets (Flickr-Expert, CF, etc..) the proposed model requires the computation of masks to feed as input to the Alpha-CLIP backbone. The runtime required for the entire pipeline should be compared with the time needed to compute scores for other metrics
* Clarification is needed on whether both the Alpha-CLIP and mobileSam models are frozen.
* Minor: Figure 3 should be more readable. It is suggested to reduce the image I size to emphasize the segmentation mask and lighten the formula part for better clarity.