PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning
Authors: Yongil Kim
Yerin Hwang
Hyeongu Yun
Seunghyun Yoon
Trung Bui
Kyomin Jung
Venue: EMNLP 2023 Findings (2023)
Keywords: Image Captioning metric
Perturbation Robustness
TL;DR: Perturbation Robust Multi-Lingual CLIPScore

Abstract: Vulnerability to lexical perturbation is a critical weakness of automatic evaluation metrics for image captioning. This paper proposes Perturbation Robust Multi-Lingual CLIPScore(PR-MCS), which exhibits robustness to such perturbations, as a novel reference-free image captioning metric applicable to multiple languages. To achieve perturbation robustness, we fine-tune the text encoder of CLIP with our language-agnostic method to distinguish the perturbed text from the original text. To verify the robustness of PR-MCS, we introduce a new fine-grained evaluation dataset consisting of detailed captions, critical objects, and the relationships between the objects for 3,000 images in five languages. In our experiments, PR-MCS significantly outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages, while maintaining a strong correlation with human judgments.

PR-MCS:
Perturbation Robust Metric for MultiLingual Image Captioning
Yongil Kim!
Yerin Hwang² Hyeongu Yun'
Seunghyun Yoon
Trung Bui
Kyomin Jung!
'Dept. of Electrical and Computer Engineering, Seoul National University 'IPAI, Seoul National University, 'Adobe Research
(miles94, dpfls589, youaredead]@snu. ac. (syoon, buadbee com, kjungesnu.ac ac.
Abstract
Baseline
Ours
Originel (En)
Vulnerability lexical perturbation is crit- ical weakness of automatic evaluation met- rics for image captioning- This paper proposes Perturbation Robust Multi-Lingual LJIPScorePP-MCS) which exhibits robust- ness to such perturbations, novel reference- free image captioning metric applicable to mul- tiple languages. To achieve perturbation robust- ness, fine-tune the text encoder of CLIP with our language-agnostic method to distin- guish the perturbed text from the original text. To verify the robustness of PR-MCS, troduce a new fine-grained evaluation dataset consisting detailed captions, critical objects, and the relationships between the objects for 3, .00 images in five languages. In our ex- periments, PR-MCS significantly outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five lan- guages, while maintaining strong correlation with human judgments.'
Porturbed (En)
0.969
0.603
(+1.10%
(42.3%%
Original (De)
0.965
1.034
Perturbed (De)
(+..1%
(-49.00 (49.0%
a. white house with brick walls and car parked with trees
A brick walls with tree and and and a white house parked parked with
Hauseinfahrt, Garage der ein Auto steht, Steinpfeiler vor Hauseingang. mehrere Bäume
Drivewa garage with stone pllaars front of house entrance, several trees d. Hauseinfahrt, Garage in der ein Steinpfeiler steht, Auto vor vor einem Bäume, mehrere mehrere mehrere Hauseingang Drivoway, garage with stono pilaars frontirrnt trees, several several ansvent house antranna
in-
Substitution
Repelition
Figure 1: An example for perturbation robustness test, The baseline metric shows similar scores to both original and perturbed captions, but metric shows prominent score drop for perturbed captions indicating that pertur- bation is well detected.
1 Introduction
et al., 2002; Lin, 2004; Banerjee and Lavie, 2005; Vedantam et al., 2015) to reference-free met- rics (Lee et al., 2020, 2021; Hessel et al., 2021). I). Recently, CLIPScore (Hessel et al., 2021) has been proposed to leverage the large-scale pretrained vi- sion and language model CLIP (Radford et al.. 2021). In evaluating generated captions by com- puting cosine similarity between embedded vec- tors (i.e., image and text) using CLIP, CLIPScore achieves higher correlation with human judg- ments than traditional metrics.
Image captioning (Xu et al., 2015; Vinyals et al., 2015, 2016; Lu et al., 2017) is a multimodal task that automatically generates captions that describe the visual content of an image and integrates multi- ple disciplines of visual and textual modality. Im- age captioning is a natural language generation (NLG) task (Gatt and Krahmer, 2018), but the evaluation metric has different characteristics from other NLG metrics (Sai et al., 2022). Image cap- tioning metrics should be able to evaluate not only linguistic fluency and syntactic thoroughness but also semantic correspondence to visual content(Bal and An, 2018).
However, Sai et al. (2021) have revealed that cur- rent metrics are prone to failure in capturing lexical noise in generated captions. For example, when a perturbation is applied to original caption (c.g., removal or swap at the token level), existing im- age captioning metrics do not recognize the change and compute a score similar to that for the original caption case. This failure to capture lexical noise raises a critical question concerning the reliability
Evaluation criteria for image captioning have evolved from N-gram-based metrics (Papineni
Corresponding author.
All the datasets and code are available in https:// gi wwtthu..oo/yooo.. com/yongt -wiee-iii//---ee- MCS. Our dataset is distributed under the CC-BY-NC 4.0 license.

---

of the metric, as shown in the example in Figure 1. CLIPScore exhibits the same tendency in our anal- ysis, which reflects its vulnerability to perturbed texts. By extending CLIPScore to a multilingual setting, we observe that a multilingual CLIPScore exhibits the same limitations in multiple languages other than English, i.e., French, German, Spanish, and Japanese.
2 Related works
Image captioning metrics As with other natural language generation tasks, image captioning can be evaluated using various proposed metrics. BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) are representative image captioning metrics based on n-gram similarity with reference captions. Other widely used reference-based metries include CIDEr (Vedantam al., 2015), which weights n-gram similarity (Kondrak, 2005) through TF-IDF (Aizawa, 2003), and SPICE (Anderson et al., 2016), which evaluates captioning based on scene graphs. Recently, reference-based metrics using embedding similarity with reference captions based on model, such as BERTScore (Zhang et al., 2019), BERT-TBR (Yi et al., 2020), and VilBERTScore (Lee et al., 2020), have been introduced.
In this paper, we address this problem by propos- ing novel method for enhancing the perturba- tion robustness of CLIPScore. Our method to fine-tune the text encoder of CLIP with perturbed captions so that the text encoder can distinguish the perturbed text embeddings from the original text embeddings. The simplicity and effectiveness of method enable to apply it multiple languages without relying on human annotations. Using our method, we develop Perturbation -obuust Multilingual CLIPScore (PR-MCS), a perturbation- robust and language-agnostic metric for image cap- tioning.
CLIPScore Researchers have also proposed unreferenced image captioning metrics that evaluate generated captions by comparing them with original images that does not require ground- truth caption (Madhyastha et al., 2019; Kusner et al., 2015; Lee et al., 2021; Chen et al., 2020). CLIPScore (Hessel et al., 2021), which is also a reference-free metric, relies heavily on the CLIP (Radford et al., 2021) model, trained with 400 million image caption pairs using contrastive objective function that distinguishes original image-caption pairs from unmatched captions. The caleulated CLIPScore is the weighted value of cosine similarity between image embedding and text embedding encoded by the CLIP model. Despite the fact that CLIPScore exhibits a high correlation with human evaluation, it is limited in that it is an image captioning metric that applies only to English. In this study, we propose new multilingual image captioning metric developed by extending CLIPScore multilingual setting.
Furthermore, validate the robustness of PR- MCS against perturbations and its high human cor- relation, we introduce two newly created datasets: M-FineCap3k and M-CapEvallkk Currently, most image captioning datasets are limited to English, necessitating a machine translation (Bahdanau et al., 2014; Johnson et al., 2017) process for mul- tilingual experiments. However, this process relies on the performance of the translation model, which may result in lower evaluation reliability compared human-annotated labels. Hence, elicit image captions directly from human experts, tailored to the purpose of the datasets. Firstly, M-FineCap3k is designed as an image captioning dataset, cre- ated to generate fine-grained captions that are ap- propriate for the corresponding images. Secondly, M-CapEval k serves as benchmark dataset de- veloped for the purpose of measuring the human correlation with image captioning metries.
Finally, experimental results on five datasets, in- cluding M-FineCap3k, demonstrate that PR-MCS outperforms baseline metrics in capturing lexical noise in captions across all five languages consid- ered. In addition, the results of measuring hu- man correlation using M-CapEvallk reveal that PR-MCS exhibits strong alignment with human judgments. Therefore, we confirm that the pro- posed PR-MCS is useful and reliable image cap- tioning metric with perturbation robustness and a strong correlation with human judgments.
Perturbation Robustness In a recent study, Sai et al. (2021) selected various criteria for use in assessing how various NLG evaluation metrics per- form. In addition, perturbation was applied to mul- tiple image captioning factors assess the pertur- bation robustness of the image captioning metrics. Sai et (2021) provided a perturbation checklist of metrics for NLG tasks; we go further and present a novel metric that overcomes the limitations of

---

(Pre training]
mnnetuning]
Loss2
Language
EN
DE
ES
FR
JA
MSE
Loss1
Loss3
Metric
CLIPScore
0.270
CLIP Textual
ML-CLIP Tixtuua
ML-CLIP Textual
MI-CLIP Textual
iMCS
0.264
0.264
0.251
0.235
0.250
Visual
+MCS
0.268
0.265
0.253
0.236
0,252
FR JA
Original Caption
Perturbed Caption
Table I: Human Correlations with Multi-CapEval
The symbol denotes MCS metric with Fredrik Carls- son (2022), and the denotes an MCS metric with our pretrained text encoder.
Figure 2: Overall training procedure PR-MCS. We pre-train Multilingual text encoder with teacher learning. Then, we fine-tune the Multilingual text encoder.
uses image-caption pairs with weight given to the cosine similarity of embeddings created through visual and text encoders, respectively. The formula for image-caption input pair (I,c) in MCS is follows:
other metrics. We select some perturbation criteria from among those suggested by Sai al. (2021), designate them as target perturbations, and show that the CLIPScore cannot detect these perturba- tions in multiple languages. Even if the generated captions are corrupted, CLIPScore outputs simi- lar results for the original and corrupted sentences. This study proposes novel metric with perturba- tion robustness based on CLIPScore address its weaknesses multiple languages.
MCS(I,))
mar(0, cos
where V(I) is the visual embedding where the im- age is passed through the visual encoder and T(c) the text embedding where the caption is passed through the multilingual text encoder. The value of w is set to 2.5 as in the original CLIPScore.
3 Perturbation Robust Multi-Lingual CLIPScore
3.2 Human Correlations of MCS
An adequate evaluation of image captioning metrics requires assessing the correlation be- tween metric- seauri-geeerated caption scores and human- generated caption scores. However, to the best of our knowledge, no benchmark exists to evaluate im- age captioning metries across multiple languages. For languages other than English, machine trans- lation necessary, but this approach can lead incorrect results when machine translation mod- els attempt to correct sentences that have already been annotated with a low score. To tackle this problem, we created the M-CapEval benchmark by translating Caplivalkk (Lee et al., 2021), an image captioning metric evaluation set originally in English, into five languages (English, German, Spanish, French, and Japanese) with the assistance of native speakers for each language. Our transla- tion process ensured that the goodness or badness scores for each sentence were maintained. This benchmark can be leveraged for the quantitative evaluation of multilingual image captioning met- rics, and example of the M-CapEvallk is pro- vided in the Appendix C.
3.1 MultiLingual CLIPscore
We propose multilingual image captioning metric, Multilingual CLiPScore (MCS), over- come the limitation of CLIPScore that it can only be applied to English. Recently, Fredrik Carlsson (2022) proposed multilingual CLIP model appli- cable to various languages by learning the expres- sive power of CLIP's text encoder through teacher learning. Similarly, in this study, we pre-train the multilingual text encoder with teacher learning (Figure 2 left). In teacher learning, the multilin- gual text encoder learns the pre-trained CLIP text embedding of an English sentence so that the em- beddings of the sentence translated through the ma- chine translation model are similar. We use MSE loss as the teacher learning loss, and the formula is as follows:
L(I N Σ 2)2,
where t is the teacher embedding and is the stu- dent embedding. More details on multilingual tex- tual encoder pre-training can be found in the ap- pendix A.
Table 1 shows the Kendall tau-c (5.) value (Kendall, 1938) representing the human correla- tion for each language of the metrics. The MCS which uses our pretrained model as a backbone
We present new multilingual image captioning metric, MCS, using this model as a backbone. MCS

---

1.2
3.4 Perturbation- Multilingual CLIPScore (PR-MCS)
1.0
0.8
We introduce a novel language-agnose perturba- tion method that increases the robustness of MCS, This method of fine-tuning the multilingual text encoder is to add three losses to original CLIP loss. The CLIPScore is constructed through em- beddings of pre-trained CLIP without additional training. The CLIP loss LCLIP consists of in-batch contrastive loss using cross-entropy loss, and the implementation is the same as the pre- pre-trainn vee-training loss of CLIP, We construct a loss based on the con- trastive loss of CLIP to maintain the high correla- tion with the human judgment of CLIPScore.
0.6
0.4
0.2
0.0
English
German
French
Spanish Japanese
Figure 3: Scores of original captions (blue) and per- turbed captions (red) with repetitive lexical noises. In all cases, the perturbed captions show no differences from the original captions.
Then, train the text encoder by adding three additional losses for perturbation robustness. These losses aim to maintain the close relationship be- tween the image embedding and the original cap- tion while increasing the distance from the per- turbed caption. An (image, original caption, per- turbed caption) triplet is then used as input to fine- tune the text encoder through three losses, as shown in Figure 2 (right). The losses are as follows:
demonstrates a high correlation with human judge- ment similar CLIPScore in English. For other languages, can serve as baseline for multilin- gual image captioning metrics future research. Therefore, the MCS presented in this paper extends CLIPScore well to languages beyond English.
3.3 Vulnerability to lexical perturbation
We employed some of the perturbation criteria iden- tified by Sai et al. (2021) and checked the pertur- bation sensitivity of CLIPScore and MCS. One of the criteria, "Repetition", is perturbation in which words appear repeatedly the token level in the original caption (e.g.. "I am boy." + "I am a boy boy."). Figure 3 shows what score is given by baseline metrics when repetitive lexical noise is introduced. We randomly selected 3,000 sam- ples from the MSCOCO (Lin et al., 2014) dataset, translated them into four languages, and gave repet- itive lexical noise to the captions. The blue bars are the score for the original captions, and the red bars are the score for the perturbed captions. For English, CLIPScore is used a metric, and for other languages, MCS used for score extraction. The caption to which lexical noise is added is ex- pected to have a lower matching score with the image than the original caption. However, for all languages, the scores for the perturbed captions are not lower than those for the original captions. There are even cases in which the perturbed caption is given a higher score. Similar tendencies can be observed for other perturbation criteria well as "Repetition". These results confirm that CLIPScore and MCS are limited in that they are vulnerable to lexical perturbation and that a metric that is robust to perturbation is needed.
C 1 cos(V T(o))
(1)
C2 max(m, cos (V T()),
(2)
C3 max(m, cos coococossstc (o)), T(p)).
(3)
where (I, o. is the (image, original caption, per- turbed caption) triplet, V is a visual encoder, and T is text encoder.
Equation (1) is composed of the cosine embed- ding loss of the two representations needed to in- crease the similarity between the image embedding and the original caption. Since MCS is based on cosine similarity, the purpose of Eq. (1) is to obtain a higher score for the original caption. Equation (2) reduces the marginal cosine similarity of image em- bedding and perturbed caption embedding. Equa- tion (3) reduces the similarity between the original and perturbed caption embeddings. The margin is set to 0.1. These three losses are combined to obtain the final objective function follows:
L LCLI + d * Li X2 * L2
+ Z3 L3.
We develop PR-MCS by fine-tuning Multilingual CLIP using the proposed loss function.
P PR-MCS((,,) w maz cos (c)),,
where (c) the text embedding from the fine- tuned multi- lingull text encoder. w is also set to 2.5, as in the original CLIPScore and MCS.

---

Original Caption
Critical Objjects
white shirt, grey shorts, gollf green field
Deux hommes femme souriant fond noir
A man, wearing white shirt and grey shorts, playing green field with green and
Origimal
Two and women emiling black backgroond
Critical Objects
deux hommes
femme
the backgroud
men
green playing
background. green
Relationship
souriant
smiling
Jumble
shotts, and white the wearing man, and
Background
fond nair
golf shirt field
black background
Remsvvll
man,
shorts, playing green trees
Figure 4: An example of the proposed M- FineCap3k dataset (translation provided for explanation purpose).
A man, man, wearing whise white shirt and and gny shorts, shorts, playing playing polf on a green with green and blue the background.
Repetition
A [MASK] wearing [MASK] shirt [MASK] grey
Masking
[MASK] [MASK] [MASK] golt on [MASK] green with
4 M-FineCap3k Dataset
[MASK] [MASK] and [MASK] blue the background.
A mon, wearing golf and green field, playing
To evaluate the performance of PR-MCS, introduce new image captioning dataset, M- FineCap3k. Most existing image captioning datascts are limited to English (Young et al., 2014; Krishna et al., 2017). Therefore, a model for machine translation (MT) from English to other languages is essential to evaluate image captioning in various languages. However, translated evalua- tion set is highly dependent the performance of the MT model and is highly likely have an mnglish-language bias. In addition, translation is inarticulate in the target language because it difficult to reflect the unique characteristics of each language, such as word order and lexical choice (Zhang and Toral, 2019; Cao et al., 2020). Therefore, the results obtained using translated evaluation set achieve poorer agreement with human evaluation than those obtained using the English evaluation set. For these reasons, human-annotated image captioning evaluation set with a wide variety of languages is needed.
Substitution
shirt grey
green blue
the bockground.
Figure Example of perturbed captions FineCap3k in English. The critical objects shuffled for in-sentence substitution.
Fine-grained caption with critical objects The widely-used multi-lingual image captioning dataset, Multi30K(Ellion) et al., 2016, 2017), is also based on human annotations of images from Flickr30K (Young et al., 2014). However, the dataset is composed of brief sentences, rendering it challenging to execute diverse lexical perturbations. Morcover, it is only expanded to two languages, namely German and French. To address this, we create M-FiineCa33 with long, detailed captions of 20 words or more enhance the impact of human annotation and generate range perturbed captions for evaluation purposes. addition, we had human experts point to critical objects, backgrounds, and relationships to create perturbed captions effectively, As described above, the image captioning metric should also reflect the semantic correspondence of whether the caption captures information contained in visual content well. The critical object of the caption should point to the most important object of this visual content, so plays a key role in the comparison between embeddings. When perturbation is applied to this critical object, more powerful and effective perturbation is achieved.
Multilingual image captioning evaluation set We introduce a new human-annotated multilingual evaluation dataset, M-FineCap3k. We extended FineCapEval (Cho et al., 2022), which has only English captions, German, French, Spanish, and Japanese). Human experts viewed the images for each language and added captions directly. Each sentence generated directly by native speakers more fluent than translated versions. Moreover, M-FineCap3k can capture various cultural aspects that MT models cannot (Liu et al., 2021). Therefore, the reliability of evaluation in multilingual settings increases. An example of the dataset is shown in Figure Human annotators for each language created a caption, critical objects, backgrounds, and relationships for a given image.
five languages (English,
For instance, the well-known weakness of CLIP is that it does produce different results when the positions of critical objects in the sentence are changed. For example, the CLIP text embeddings of the two sentences "A blue car

---

MSCOCO
Flickrök
VizWiz
Multi30k
M-FineCap3k
:1 MCS
MCS PR-MOS
PR-MOS
(Faw-shot)
300
Figure Experiment result graph. The y-axis value represents the score drop of the perturbed caption as percentage difference compared to the original caption. The experiment is conducted with five datasets, and we report the average of the languages to confirm the results for each perturhation. As a result, it can be seen that PR-MCS is more robust than baseline metries for all perturbations across all datasets.
front of the white church" and "A white car in front of the blue church" are almost identical. To evaluate the robustness to this perturbation, we construct perturbation criteria using critical object information. The statistics of the dataset can be found in the Appendix D.
and multilingual text encoder. The first baseline is the MCS metric constructed using the multilingual CLIP text encoder implemented by Fredrik Carls- son (2022) the backbone. The second baseline is the MCS metric constructed using the multilingual text encoder trained by the teacher learning method described in Section 3.1.
5 Experiments
5.2 Perturbation configuration
We select the following five criteria to perturb the sentences in the fine-tuning and evaluation sets. The criteria below error types commonly found in model-generater captions. These criteria are part of the checklist proposed by Sai al. (2021). We select five orthogonal criteria. Each perturbation example is shown in Figure 5.
Our framework seeks to identify whether given metric can detect lexical noise in a generated cap- tion. Through exhaustive experiments, we evaluate whether the PR-MCS developed as described in this paper successfully distinguishes the perturbed caption from the original caption.
5.1 Experimental Setup
Repetition Repeated words are found in several model-generane captions. A well-known problem is that the transformer model is vulnerable because does not capture repetitive perturbation well at the embedding level. We give each word token a repeating perturbation with a probability 0.4.
Fine-tuming set We use MSCOCO, the dataset most widely used for image captioning, as the fine- tuming set to enhance the perturbation robustness of MCS. We use the training and validation split of the MSCOCO dataset described by Chen et al. (2020). The number of elements in the training set is 414k. Since only English captions exist in MSCOCO, captions are translated into four other languages using the MBART-50 (Liu et al., 2020) MT model.
Removal Among the sentences given low score in the evaluation dataset for the image captioning metric, such as Composite (Aditya et al., 2015) or Pascal50s (Vedantam et al., 2015), some word tokens removed, and incomplete sentences found. We configure perturbation by removing some tokens to reflect this noise. Each word token drawn with a probability 0.4.
Evaluation set To comprehensively evaluate the perturbation robustness of PR-MCS, we choose four evaluation sets: MSCOCO, VizWiz (Gurari et al., 2020), FlickrSk (Anitha Kumari et al., 2019), Multi30K, and M-FineCap3k. As the fine-tuning, MSCOCO, VizWiz, and Flickr8k, which have only English captions, are translated using the MBART-50 MT model.
Masking Masking is perturbation in which ran- domly selected tokens in the caption are replaced with [Mask] tokens. When lexical noise is given in units of tokens, the meaning of the corresponding token disappears, but unlike in the Removal case, the position is maintained. Position information can be critical in a reference-free metric based on a transformer model such as CLIPScore (Dai
Baseline metric As the baseline of the experiment, we use two MCS metries. As mentioned above, the MCS metric is configured using CLIP's visual

---

DE
JA
Eval Dataset
Metric
Perturbed
Perturbed
Perturbed
Perturbed
Perturbed
Original
Original
Original
Original
Original
Average
Average
Average
Average
Average
1.053
0:9813
MCS
L,0939
1,0899
1,8377
1.799
1,0366
MSCOCO
(-32275)
6696)
(-28839)
(-2.78%)
603)
0.299
0,5156
0.6643
0.6814
0,9202
PR-MCS
1,4177
1,3153
1.4127
1,0086
1.9988
(-78.66%)
-60.80%
(-52.97%)
(-51,63%)
(34402%%
1.0543
1.0455
L,0488
0:9879
MCS
1,0915
1.0779
1,0758
1.0747
1,0293
Flickerkk
(-34415)
(-3.000 (3.0
(-2.98%)
(-3.03%)
0256)
0.2402
0.2674
0,4275
0.4685
0.4463
PR-MCS
1,1484
1.1621
1.6681
1.6834
1.6279
(-79.09%)
(-76.99%)
(-74.37%)
(-71.66%)
(-72.58%)
1,0338
1,0251
1,0088
MCS
1.0626
1.0491
1,0509
1,0505
1.0414
VieWix
(-2.71%)
(-3.7206)
(-2,469)
(-3411%
(-3.425)
0.2944
0.415
0.6163
0.6257
0.6224
PR-MCS
0.9769
1.442
1.4526
1.4113
(-69.86%
(-58.06 %
(-57.26%)
(-56.939))
(-55.90%)
1,0148
1.0064
1,0247
MCS
1,6111
1.0625
1.0024
MoltEMO
(-4.56%)
(-3,3996)
(-3554%)
0.3041
0.3613
0.5114
PR-MCS
1,1008
1,095
1.5767
(-72.37%)
(-67.03%%
(-67.56 %
1,0372
1,0295
0,9856
MCS
1,0657
1,08035
1,0575
1,0877
0.963
(-2.675)
(-3.0096)
(-2.6598)
(-2.65%)
(2.34%)
0.5579
0.4611
0.8488
0.9426
0.6133
M-FinnCCu..k
PR-MCS
1,0499
0.7261
1,3634
1,3821
1.1551
(-46.51%)
(.36.50%)
(-37.74%)
(-31,80%)
(-4.991%)
PR-MCS
0.1584
0.1742
0.4391
0L3964
0.314
0.7125
0.589
L.4862
1.4292
1.4136
setting)
(-77.77%)
(-70.42%)
(-70.45%)
(-77.79%)
Table 2:
Experiement results table. Each values are represented using the average value for each perturbation. For five datasets, PR-MCS outperforms the baseline performance for all languages, and the performance has further increased after additional M-FineCap3k fine-tuning in few-shot settings.
et al., 2019; Devlin et al., 2018; Ramachandran et al., 2019; Wu et al., 2021). Therefore, even the [Mask] token does not appear the generated caption, we select Masking perturbation as the criterion, separate from Removal, to address the above case. Each word token is replaced with [Mask] token with probability of 0.4.
5.3 Perturbation robustness evaluation
We report the main results for all datasets and lan- guages in Figure 6 and Table 2. The robust eval- uation metric is expected to give lower scores to perturbed captions than to original captions.
Each graph of Figure 6 shows the experimental results for MSCOCO, VizWiz, Flickr8k, Multi30K, and M-FineCapEval by perturbation. Each point represents the average results for five languages for one perturbation. It shows how much score drop the perturbed caption has from the original caption. The green line indicates PR-MCS, and the blue and red lines refer the two baseline multilingual CLIPScores. The scores of the perturbed caption by baseline metrics do not differ much from those of the original caption for any perturbation methods. In some cases, the scores for the perturbed captions are higher than those for the original captions.
Jumble We generate perturbed samples using random-order permutation at the token level in the original reference caption. The model composing the metric can see all tokens of the sentence, including visual content, but considerable noise introduced into the position information.
Substitution Substitution involves changing the positions of key elements in a sentence. In the case of M-FineCap3k, substitution is performed using critical objects annotated by human experts. In the remaining datasets, nouns in the caption are extracted, and their positions are changed. The per- turbed caption includes all clements that exist in the original caption, but unlike in the Jumble case, it does not deform the grammatical structure at all. Detecting substitution noise well is the most chal- lenging task because it requires judging semantic correspondence to visual content perfectly.
However, our metric exhibits a significant score decrease for all perturbations compared to the orig- inal captions, which means that the metric clearly distinguish when the perturbation is applied,
In other words, our metric exhibits robustness for all perturbations in the evaluation dataset. In par- ticular, even in the cases of Repetition and Substi- tution, which are known to be challenging pertur-

---

bations, PR-MCS detects perturbations very well, while bascline metrics do not capture perturbations at all.
English
Other Langs
Multi-
Perturbation
Lingual
Robustness
CLIPScore
0.270
0.408
Table shows the score of the original caption and the average score of the perturbed caption given by the metrics for each of the four evaluation sets for each language. The result shows how much the percentage of the score decreased for each pertur- bation in comparison to the original caption. The results for the baseline metrics show that the score decrease for the average perturbation is very small, i.e., approximately 3%, relative the original cap- tion. It is difficult say that the metric can distin- guish the perturbed caption from the original cap- tion based on such slight difference. In contrast, the case of PR-MCS, the percentage decrease for the perturbed caption ranges from 50% to 70%. Clearly, our proposed method exhibits perturbation robustness in the metric score and can identify per- turbed captions through anomaly detection only with performance drop. In the cases of Vizwiz, Flickr8k, Multi30K, and M-FineCap3k, the perfor- mance outstanding even though the distributions are not trained in fine-tuning.
MCS
0.268
0.401
0.250
0.385
PR-MCS
0.267
0.425
0.250
0.376
Table 3: Correlations with human judgment for M- CapEvallk.
Dataset
Composite (EN)
Flickrök (EN)
Metric
BLEU-I
0.313
0.323
BLEU-4
0.306
0.308
ROUGE-L
0.324
0,323
METEOR
0.389
0.418
CIDEr
0.377
0.439
SPICE
0.403
0.449
PR-MCS
0.493
0.527
CLIPSCore
0.497
0.530
Table 4: Kendall Tau (Ta) correlations with human judg- ment for existing English datasets.
erage score, increases for all five languages. These results show that lexical noise in long sentences is more reliably captured by learning small number of samples with few-shot setting. The experimen- tal results for all languages and all perturbations for each dataset are provided in the Appendix H.
5.4 Few-shot setting for M-FineCap3k
As the results summarized in section 5.3 show, PR- MCS is much more robust to perturbation than the baselines in M-FineCap3k. However, the perfor- mance degradation for perturbed captions in M- FineCap3k is lower than for MSCOCO, VizWiz, Flickr8k, and Multi30K (e.g., -55.66% to -39.89% in average from MSCOCO). We attribute this to the distribution shift from the sequence length dif- ference between the MSCOCO fine-tuning set and the M-FineCap3k test set. VizWiz, Flickr8k, and Multi30K are composed of short captions, so there is not much difference in caption length between them and MSCOCO. Therefore, to check whether the distribution can be learned when some infor- mation about the M-FineCap3k 3K test set is pro- vided, perform additional experiments on M- FineCap3k with few-shot setting. We split M- FineCap3k into subsets proportioned 1:9 in size and use only 300 perturbed captions as the few- shot input.
5.5 Correlations with human judgement
Table 3 and Table shows that PR-MCS is an use- ful image captioning metric with high correlation with human judgment. The higher the Kendall tau- (Te) value and the Pearson correlation coefficient (p) (Benesty et al., 2009), indicators for viewing the correlation with human judgment, the better. The Kendall tau-c value is the similarity between the two variables based on ranking, and the Pear- son correlation coefficient is measure of linear correlation between two sets of data.
For Table 3, M-CapEval k is used ation set for measuring the performance of image captioning metric, and it demonstrates excellent characteristics of PR-MCS for various languages within the dataset. As introduced in Section 3.2, our created M-CapEivallk serves as a benchmark for measuring the correlation of given metrics with human judgment across various languages. PR- MCS exhibits perturbation robustness while also showing similar performance to CLIPScore and MCS with Kendall correlation in English and even higher performance in Pearson correlation.
evalu-
The experimental results for the few-shot setting are shown Table and Figure 6 (the yellow line in rightmost graph). When the distribution for the fine -re-grained caption is given, the overall perfor- mance in perturbation detection, in terms of the av-
In Table 4, it is evident that PR-MCS remains

---

valuable metric for existing English benchmark datasets such Composite (Aditya et al., 2015) and Flickr8k Expert (Hodosh et al., 2013). The correlation of PR-MCS with human judgment for English sentences is notably strong compared to conventional reference-bused metrics and is sim- ilar to CLIPScore. Furthermore, as asserted in Table 3, PR-MCS operates in multilingual setting and demonstrates perturbation robustness, distin- guishing it from CLIPScore. PR-MCS's adaptabil- ity multilingual context allows for showeasing its effectiveness by translating these datasets into languages besides English using machine transla- tion. PR-MCS consistently demonstrates strong human correlation across all other languages, and the comparative results with reference-basee image captioning metries can be found in Appendix I. In light of the cumulative findings discussed pre- viously, for sentences that potentially include per- turbations that an image captioning model may output, existing metrics such as CLIPScore and MCS are vulnerable. PR-MCS is a useful metric that can evaluate good sentences positively and bad sentences negatively for potentially perturbed sen- tences; thus, it can be used instead of CLIPScore.
are translated into other languages by utilizing the MT model, so we have no choice but depend on the performance of the MT model and avoid model bias.
Recently, Reimers and Gurevych (2019) released the MultiLingual CLIP based the ViT-B/32 CLIP model". We constructed MCS with this model as the backbone and measured human cor- relation using M-CapBvvllkk. The results showed slightly lower correlation compared to the MCS with our implemented Multilingual Text Encoder, but were still similar (average Kendall score for 5 languages: Sentenceformer based MCS 0.249, our MCS 0.255). However, this model is large-scale model, it has the disadvantage of being slow in inference speed as an automatic metric. We conducted finetuning with our more lightweight model with high human correlation. In future work, if experiments are conducted with this large-scale model, additional analysis on our proposed method- ology and new metrics can be done through va- riety of experimental results and interpretations. Furthermore, trends in large-scale models can also be confirmed.
Ethics Statement
6 Conclusion
The annotators for the two newly created datasets (M-FineCap3k and M-CapEval1 were hired through a data annotation service. The remuner- ation was calculated according to the annotators' country of residence, as determined by the com- pany. Annotators were asked not to write any toxic content (1. offensive, sexist, or racist comments; 2. toxic words; 3. sexual behavior). All other datasets and models used the experiments are from the publicly available website or Github.
In this paper, we propose PR-MCS, perturbation- robust metric for multilingual image captioning using language-agnostic fine-tuning. PR-MCS, de- veloped by fine-tuning the text encoder of CLIP, can distinguish lexically perturbed text from orig- inal text. We also propose a new fine-grained multilingual image captioning evaluation set, M- FineCap3k, for use in perturbation robustness eval- uation. Experimental results for existing datasets and new dataset show that PR-MCS detects perturbation well and robust to perturbation in multiple languages. Furthermore, we verify that PR-MCS is a useful metric with strong correlation with human judgment, using our M-CapEvallkk,
Acknowledgements
We thank anonymous reviewers for their construc- tive and insightful comments. K. Jung is with ASRI, Seoul National University, Korea. This work was supported by Samsung Electronics and the BK21 FOUR program of the Education and Re- search Program for Future ICT Pioneers, Seoul National University in 2023. This work was partly supported by Institute of Information & communi- cations Technology Planning & Evaluation (IITP) grant funded by the Korea governmen((MSIT)
Limitations
Model bias of machine translation in training In our study, evaluation set is created by directly annotating languages other than English to remove the bias of the machine translation (MT) model in the evaluation phase. However, in the training phase, the dataset size is too large annotate di- rectly in multiple languages other than English. Therefore, the pre-training set and the fine-tuning
1:2--mulilinguallvv

---

[NO.2021-0-01343, Artificial Intelligence Grad- uate School Program (Scoul National University) & NO.2021 NO.2021-0-02068, -0-020688, Artificial Intelligence Inno- vation Hub (Artificial Intelligence Institute, Seoul National University)).
Zihang Dai. Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoe V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language mod- els beyond fixed-length context. arXiv preprint 1rXiv:1901.02860
Jacob Devlin, Ming-War Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. 8. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
References
Somak Aditya, Yezhou Yang. Chitta Baral, Cornelia Fermuller, and Yiannis Aloimonos, 2015. From im- ages to sentences through scene description graphs using commonsense reasoning and knowledge. arXiv preprint arXiv: 151 1.03292.
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine trans- lation and multilingual image description. arXiv preprint arxivv1710.0710
Akiko Aizawa. 2003. An cifoommatiionthhoretttt perspec- tive of tf-idf measures. Information Processing & Management, 39(1):55-55.
Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. arXiv preprint 100504990
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pages 382-398. Springer.
Manuel Romero Fredrik Carlsson, Ariel Ekgren. 2022.
K. Anitha Kumari, C Mouneeshwari, RB Udhaya, and Jasmitha. 2019. Automated image captioning for flickr8k dataset. In International Conference on Arti- ficial Intelligence, Smart Grid and Smart City Appli- cations, pages 679-687. Springer.
Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artifi- cial Intelligence Research, 61:65-170.
Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. 2020. Captioning images taken by people who are blind. In European Conference on Computer Vision, pages 417-434 Springer.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint 1eXiv:1409.0444
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXh::2004.007
Shuang Bai and Shan An. 2018. survey on automatic image caption generation. Neurocomputing, 31 1:2911 304.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the aci workshop intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853-899.
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, -5119-35..
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Is- rael Cohen. 2009. Pearson correlation coefficient. Noise reduction in speech processing, pages 1-4. Springer.
Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multi- lingual alignment of contextual word representations. arXiv preprint arXiv:2002.00 03518.
Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30( 1/23:81-933
Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. Eumpean conference on computer vision, pages 104-120. Springer.
Grzegorz Kondrak. 2005. N N-gram similarity and dis- tance. In International symposium on string pro- cessing and information retrieval, pages 1 115-126. Springer.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations.
Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Demoncourt, Trung Bui, and Mohit Bansal. 2022. Fine-grained image captioning with clip reward. arXiv preprint arXiv:2205.131 13.11.

---

International journal of computer vision, 123(1):32- 73.
et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Wein- berger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957-966. PMLR.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Ir- wan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alon self-attention in vision models. Ad- vances in Neural Information Processing Systems. 32.
Hwanhee Lee. Seunghyun Yoon, Franck Deronncour, Trung Bui, and Kyomin Jung. 2021. Umic: An unref- erenced metric for image captioning via contrastive learning. arXiv preprint arXiv:2106. 14019.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bett-networks. arXiv preprint arXiv:1908 10084.
Hwanhee Lee, Seunghyun Yoon, Franck Demonccurt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2020. Vilhertscore: Evaluating image caption using vision- and languag bert. In Proceedings of the First Work- shop on Evaluation and Comparison of NLP Systems, pages 34-39.
Ananya B Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas
Mohan, and Mitesh M Khapra. 2021. Perturba- tion checklists for evaluating nlg evaluation metrics. arXiv preprint arXiv:2109.05771
Chin-Yew Lin. 2004. Rouge:
A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022. A survey of evaluation met- ries used for nig systems. ACM Computing Surveys (CSUR), 55(2):1-39
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays. Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European confer- ence on computer vision. pages 740-755. Springer.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 191.01900000108.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. arXiv preprint arXiv:2109. 13..:...09.13388
Yuging Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2020. Multilingual translation with exten- sible multilingual pretraining and finetuning. arXiv preprint (rXxx:2008.044:
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li. Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoye. 2020. Multilinuual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- fics, 8:726-742.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4566 4575.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE con- ference on computer vision and pattern recognition. pages 3156-3164
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint 1rX
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via visual sentinel for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375-383.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. 2016. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions paltern analysis and machine intelligence, 39(4)652 6633.
Pranava Madhyastha, Josiah Wang, and Lucia Specia. 2019. Vifidel: descriptions. arXiv preprint 1uXiv:1907.094.
Evaluating the visual fidelity of image
Qi Wang, Junyu Gao, Wei Lin, and Yuan Yuan. 2019. Learning from synthetic data for crowd counting in the wild. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pages 8198 8207.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computa- tional Linguistics, pages 31 111-18
Kan Wu, Houwen Peng. Minghao Chen, Jianlong Fu. and Hongyang Chao. 2021. Rethinking and improv- ing relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 10033-10041.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark,

---

Kelvin Xu. Jimmy Ba, Ryan Kiros, Kyunghyun Cho. Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual atten- tion. In International conference on machine learn- ing, pages 2048-2055 PMLR.
Yanzhi Yi, Hangyu Deng, and Jinglu Hu, 2020. Im- proving image captioning evaluation by considering inter references variance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 985-994.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. arXiv preprint arXiv:1906.00 08069.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675.

---

A Pre-training Details
while maintaining high human correlation, and finally, the best-performing À values of the objective function for fine-tuning are as follows: A - 0.1, A2 - 0.05, Å - 0.05. the main text, we reported single-run scores after finding the best performing parameters.
Like Fredrik Carlsson (2022)'s method, mul- tilingual CLIP is trained by pre-training through teacher learning using MSE loss as shown on the left of the Figure 2. The datasets used for pre- training are GCC (Wang et al., 2019), VizWiz, and MSCOCO with total 2.2M sentences. Each En- glish sentence is translated into German, Spanish, French, and Japanese using MBART-50. The pre- trained CLIP model used as the teacher model is the RN50X4 model, and the Distill-Multilnguall BERT (Sanh et al., 2019) is used as the student text encoder. The model trained in total of epochs, and takes about 20 hours per epoch with our computing power.
Hy pper-parameters for optimizer We use A (Loshchilov and Hutter, 2017) optimizer with 0.9, f2 0.999, e 1e 8. The initial learning rate 5e 5.
C M-CapEvallk examples
The examples of the M-CapEvallk can be seen in Figure 8 and Figure 9. Native speakers of each lan- guage translated into each language while maintain- ing the score based on the image, original caption, and score assigned the pair. The word transcreate is used because was translated while maintain- ing the score, not simple translation. The API for collecting the M-CapEval Ik dataset is shown in Figure 10, and instructions for collecting the dataset can be show in Figure 11.
B Experimental Details
B.1 Reproductabilty checklists
Dataset and Source code We provide our pre- training, fine-tuning, and evaluation source code along with configuration code for perturbations as supplementary materials. We will publicly release our dataset M-FineCap3k, and the full codes with weight parameters.
D M-FineCap3k statisties
Computing Resources AMD Ryzen Threadripper 2950X (3.50 GHz) with GeForce GTX 2080 Ti is used for the experiments. All codes implemented on Python 3.6.15 and PyTorch 1.7.1. The fine-tuning of each model trains epochs, and takes about hours per epoch.
Language
DE
JA
Datasize
1,000
3,000
3000
3,000
3,000
Sentence length
23.42
19.42
23.21
22.09
48.39
critical objects
2.87
3.56
4.17
3.11
4.42
backgrounds
1.25
1.27
1.39
1.25
1.56
relationships
1.57
2.05
1.78
1.81
2.46
Number of Parameters of our multilingual CLIP is about 66M as like as Distil-Muuttlliggen BERT.
The number of parameter
Table 5: M-FineCap3k statistics
Table provides detailed statistics for M- FineCap3k, including the dataset size for each lan- guage, the average sentence length, and the aver- age numbers of critical objects, backgrounds, and relationships. M-FineCap3k consists of lengthy sentences of approximately 20 word tokens av- erage. In the case of Japanese, since there is spacing sentence, sentence length is calculated using tokenizer based on word extractor, and the sentence length is almost the same as the character level. In addition, there are three to four critical objects in all languages, so each sentence describes the visual content of an image in great detail.
Train-Valid- Test split MSCOCO used for fine- tuning consists of 414k training set and 25k valida- tion set. We split the training set by 9:1 and used it for fine-tuning and validation. We also randomly extracted 3k samples from the existing validation set and used it test set.
B.2 Hyper-parameters
Hyper-parameters for fine-tuning In order to find the best-performing model, we conducted an experiment on 16 hyper-parameter combinations 0.1).
(A1 0 0.5, The parereprr was manually tuned based on the effective detection of lexical noise
0.1,

---

The results for all perturbation of E M-FineCap3k eval set examples
all languages for Flickr8k eval can be found in Table 8.
The examples of the M-FineCap3k eval set for lan- guages other than English can be seen in Figure 7.
VizWiz The results for all perturbation
all languages for Vizwiz eval set can be found Table 9.
French
Caption: Une femme attend en ligne à côté d'une femme avec sac marron dans une piáca bondéa.
Multi30k The results for all perturbation of all languages for Multi30k eval set can be found in Table 10.
Critical Objects femme
Relationships: attend
Backgrounds piáca bondée
German
M-FineCap3k The results for all perturba- tion of all languages for M-FineCap3k eval set can be found in Table 11.
Caption Vor
tragt goldene
Halskette und hat grüngraue Augen.
Critical Objects eine lächelnde Frau, goldene Halskette, grüngraue Augen
Relationships:: befindet trägt
grauen Hintergrund
I Correlations with human judgement
Spanish
Captions: grupo de personas sonriende y haclendo gesto con las manos, fondo panel diferentes logos
DE
FR
ES
Composite
Flickrtik
Compusile
Flickrtkk
Compuedte
Flickrit
Critical Objects: grupo de personas, gesto, manos, logos
BI.R--1
0.301
0.313
0:300
0.314
0,308
0.313
Relationshipa: sonriendo,
BLE1-4
0.299
0.301
0.303
0.302
0,300
0.303
Backgrounds panel blance
ROUGE-L
Japanese
MCS
Captions: Bist è S
PR-MUS
0.8%
E72
R 7 9) 11
Pova
Table 6: Kendall Tau (T.) correlations with human judg- ment for existing datasets other than English.
Critical Objects: Bist $867=v2
2. 2 2.99 12994722 7289, 729. 3
The results for diverse languages on the Com- posite (Aditya et al., 2015) and Flickr8k. Expert (Hodosh al., 2013) datasets shown in the Table 6. Given that these benchmark datasets are initially in English, we employed the MBART-50 machine translation model (Tang et al., 2020) to translate them into various languages. When con- trasted with reference- image captioning met- rics, PR-MCS showcases significantly strong cor- relation across all languages for both datasets, and its performance closely resembles that of MCS in comparative analysis.
Relationships:: n 2o
Backgrounds A. Wi. ib. 74 170
Figure 7: M-FineCap3k eval set examples for each lan- guages.
F Perturbed caption examples
The examples of the perturbed captions for lan- guages other than English can be seen in Figure 12- Figure 15. The critical objects shuffledd for in- sentence substitution perturbation are displayed using each color.
Implementation Details
In Alg. 1, show the Python implementation of each perturbation criterion: "Repetition", "Re- moval", "Masking". "Jumble", and "Substitution".
All results tables
MSCOCO The results for all perturbation of all languages for MSCOCO 3k eval can be found in Table 7.
Flickr8k

---

Algorithm Python implementation of perturbation
def rep_rem_mask caption_list). # Repetition, Removal, and Masking
caption_rp []
caption_ rm []
caption mask
for i in range (caption_lis) )):
words caption [rption_list[]]________________ substitued_rp []
substitued_rm
substitued_rm_mask
substitued_rmrp []
for in range (words]
substitued_rp append a words [j]
substitued_rm mask. append words
if random.randon () threshold:
substitued_rpp -appeeddddppppeddddeeeee [words[]]
substitued_rm append (words]
substitued_rm__malt (astm__mass[11_____ [[MASK]
elif random random) () threshold:
substitued_rmrp. caaptuuddrrrrpaappeee.e (words[j])
caption wtp.ppp.p... a"ppeddd "j...on"....join (substitued (bsstitued_rp))
caption_res rm append join (substitued_rm)
caption rm mask. append jooin (substitued ("u"tooinsuubstiued rm mask) )
return caption_rpe _pp caption_rm rm. caption_mes rm mask
def jumble ( (caption_list))
Jumble
caption_jumble = for i in range(len) (caaa(tttooniis
words caption_liia list(ii. random.. shuffle (words)
caption caption_jumble appeedd ("_"."""""""" join (words))
return caption_jumble
def sub_in sub_in_sent (captionnnalii
pritical_obj_listy # In-sentence substitution
caption_sub_il = for i in range (Ice(((totoon_llsss)) current_caption = caption_listli]
cetrent_critticl_ooo_ooo shuffled current_ciitiaal_oo__iis.
critical_obb lis_tisttti
words caption spli
if len (etrreent_errticcl_00// caption_suh_ine "aah__in_appedd""""__ (words))
2:
else
while current waapalt_rriilllloooo obj_lisi_ list shuffled:
random. shuffle
target current caption for in range len (shufflededd
target shuffled J. join caption_sub_ append (appedd(aaa) (target)
currentteriiticall [ist[
return caption_sub_ii

---

Examples
Iie cwirek Tresscratinn witton giob crekntercing
Wourrs reight
Aellaa ca
Tacck uk uiiig sppuute
Aataa prate sut bastel
Good Trancreation:
Trunsr wertas
bled attting on top of
insorrcc
caption correctedd
Original Captions
Searish rargtt
Aullowr Coution
woman sitting at a table with a cake
insurppeoeeeee wiitl teve
Aale loking collutrr with butuu be kiilg
Score (0-1 range) 0.6
son commona www.aaaiiieennn in froec lastep
Good Transcreon
Trrneraatioos
person holding mondl phoee in thvir hard
gili folding cel' plone
Transcreated Captions
(Resson: incoerect unknoww information (Py pird addidd which is in includedinn in or original Caption
(FR) une femme assise à une table avec un gâteau
(DE) eine Frau sitzt an einem Tisch mit einem Kuchen
AArea in place aenool mecuae
(ES) una mujer sentada en mesa con un pastel
Geagle
-nnall se tout with lealtn food
enne olece of ares.
(JA) 5 0 f 7 Iwō1
Goos nnt
Bad Transceationn
of fncd heers with bunch zanameee
troy of food indluiig nonona.
Figure 8: M-CapEvallk example (1).
(Rezson: Incomnect information the ginaa prion (o busich of bomonesd revised connectis!
(hoblly Biy aid faniou toot tock
Thasok vicccigg sucharann moutt. A bere der desi the le tounte on mouue.
Trarocreat en:
brushing
Figure 10: M-CapEvallk collection API.
Original Captions
bunch of knives cutting board with knife
[tspo]
Score (0-1 range)
muredee promided melll
0.2
dewn mnneee oaher (it trankire the conker intorusls imiaseee
the mage may bec photo, Ratirttor oure beckgroand
axptirn genersted the
chey dountis imrage
Transcreated Captions
atfentin to trterallte the with the ong Dii cartoo..
the similar cvaluatioos
(FR) un tas de couteaux sur une planche découper avec un couteau
you typa miiallaned castions, beep these during conatation.
selvence castinnd
retennes pros ol caption rown of is wit not soviddinch wechy mamberiio
(DE) einige Messar auf einem Schneidebrett mit einem Messer
our gool in ceste oodioss offeree asguggeshhr thei soores
[ES) un conjunto de cuchillos sobre una tabla de cortar con un cuchillo
(M) a Lots Aod 6160747
Figure 11: M-CapEvallk collecting instructions.
Figure 9: M-CapEvallk example (2).

---

French
Japanese
Critical Objects
Critical Objects Mo b. 4OXT
blanc, matelas pour animaux de compagnie rose
Original: Un chat noir et un dhat blanc allongés ensemble sur un matelas
Original:
S B BA &.C. MI 39 Fitic
pour animaux de compagnie blanc rtrose posé sur un tapis blanc.
X TTNSSATT
Jumble: pour Un compagnie blanc chat blanc rose matelas de allongés un blanc. posé animaux et noir tapis ensemble chat
Jumble: i 2 9 WEAN La Mo Stt.HIiHN.t7LLL SAA
Removal: Un chat blanc allongés un pour animaux compagnie posé blanc.
Removal: to NeW
Un noir un un chat blanc blanc allongés allongés ensemble compagnie compagnie blanc et rose rose posé posé sur un tapis blanc
Repetition:
Repetition:
BUNT LOAT
matelas pour pour animaux animaux de
Removal_mask: XASKK]S445111K (MASK] / JALASK [AAKK AAS M0 (MASK GIMASII
Removal_ mask: @emoval_mask: [MASK]
[MASK] [MASK] [MASK] [MASK]
ensemble sur [MASK] matelas [MASK] [MASK] [MASK] blanc
[MART] (MASK// 2 (AAASI[[AASK]]AAAAAAAA (MASK], SIMASK) b
[MASK] [MASK] sur tapis [MASK]
Sub_in_ MBORBEE R2X
Sub_in_sent ichat blanc un matelas pour animaux compagnie blanc rose allongés ensemble chat noir tapis blanc.
Figure 12: Eval set perturbed captions example (FR).
Figure 14: Eval set perturbed captions example (JA).
German
Spanish
Critical Objects
Packungen,
Critical
Flaschen, Pralinen
uniforme deportive blance, casce con protector facial, quantes
Original: ver einem weißlen Hintergrund stehen mehrere Packunger und Flaschen, in den Packungen befinden sich Pralineni
Hombre vistiendo un unilorme departive blance con casco can protector facially compuantes blancos, sosteniendo mano derecha palo cricket Tondo claro diusso
Original:
Jumble: einem und Hintergrund stehen weilen Packungen Flaschen, sich Packungen mehrere Pralinen befinden
Jumble: protector casco Hombre blanco con pala un vistiendo de crickel un mano la facial deportive fondo
sosteniendo
guantes
Removal:
vor weiflen Packungen Pralinen
Removal: vistiendo uniforme con casco facial blancos, fondo claro difuso
vor vor einem welllen weißen Hintergrund stehen mehrere Packungen Flaschen, den Packungen Packungen befinden Pralinen Pralinen
Repetition:
Hombre vistiendo vistiendo un uniforme uniforme deportiva blanco protector facial con guantes sosteniendo en en la mano derecha un palo de cricket con con fondo fondo daro difuso difuso
Repetition:
con
Removal mask: [MASK] einem [MAS]] Hintergrund stehen mehrere Packungen und Flaschen, in den [MASK] [MASK] [MASK] befinden sich [MASK]
Removal, mask: Hombre [MASK] un [MASK] deportivo blance [MASK] [MASK] con protector [MASK] y con guantes [MASK] sostaniendo [MASK] mano derecha un pala de cricket [MASK] [MASK] [MASK] [MASK]
Sub job_in__entt vor einem welßenn Hintergrund stehen mehrere Packungen und Pralinen, den Packungen befinden sich Flaschen
Sub in_sent: pala de cricket vistiands un casco con protector facial con y guantes uniforme deportiva blanes, sosteniendo mane derecha Hombre blancos fondo claro difuso
Figure 13: Eval set perturbed captions example (DE).
Figure 15: Eval set perturbed captions example (ES).

---

Language
Metric
Original
Repetition
Removal
Masking
Jumble
Substitution
Average
0.7871
0.7241
0.7137
0.7573
0.7817
0.7442
1MCS
0.7944
(-0.92%)
(-8.84%)
(-10.15%)
(-4.67%)
(-1.59%)
(-5.23%)
1,0963
1.0177
0.987
1.0966
1.0928
1,05808
English
MMCS
L.0939
(0.21%)
(-6,96%)
(-9.77%)
(-0.24%)
(-0.10%)
(-3.27%)
0.8800
0.3103
0.0112
0.0433
0.2534
0.29964
PR-MCS
1,4177
(-37.93%)
(-78.11%)
(-99.21%)
(-96.95%)
(-82.13%)
(-78,86%)
1.0764
0.9989
0.9501
1.0792
1.0735
1,03562
iMCS
1.0743
(0.19%)
(-7.01%)
(-11.56%)
(0.45%)
(-0.07%)
(-3.60%)
1,0851
1,0119
1,0044
1,0887
1,0855
1,05512
German
MMCS
1.0839
(-0.11%%
(-6.64%)
(-7.33%)
(0.44%)
(0.15%)
(-2.66%)
0.9082
0.686
0.0175
0.166
0.800
0.51562
PR-MCS
1.3153
(-30.95%)
(-47.84%)
(-98.67%)
(-87.38%)
(-39.15%)
(-60.80%)
1,0753
0.9983
0,9453
1,0768
1.0692
1,03298
MMCS
1.0719
(0.32%)
(-6.87%)
(-1 (118%%)
(0.46%)
(-0.25%%
(-3.63%)
1,0845
1.0152
0.9918
1.0878
1.0857
1.053
French
MMC
L.0837
(0.07%)
(-6.32%)
(-8.48%)
(0.37%)
(0.18%)
(-2.83%)
1.0021
0,9086
0.0091
0,2873
1.1167
0.66434
PR-MCS
1,4127
(-29.21%)
(-35.68%)
(-99.36%)
(-76.66%)
(-20.95%)
(-52.97%% %
1,0591
0.9865
0.9331
1.0624
1.0528
1,01878
1MCS
1.0564
(0.26%)
(-6.61%) (6)
(-1 (-11.67%)
(0.57%)
(-0.34%%
(-3.56%%
1,0807
1,012
0.9909
1.0846
1.0812
1,04988
Spanish
MMCS
1.0799
(0.07%)
(-6.29%)
(-8.24%)
(0.44%)
(0.12%)
(-2.77%)
1,0316
0.8841
0.0101
0.2882
1.193
0.6814
PR-MCS
1.4086
(-26.76%)
(-37.24%)
(-99.28%)
(-79.54%)
(-15.31%)
(-51.63%)
1.0769
1,0001
0.9247
1.0819
1.0726
1,03122
MMCS
1.0727
(0.39%)
(-6.68%)
(-13.80%)
(0.86%)
(0.01%)
(-3,85%)
1,0348
0.9669
0.9105
1.0294
1.0271
0.9813
Japanese
MCC
1.0286
(0.60%)
(-6.00%)
(-11.48%)
(0.08%)
(-0.15%)
(-3.39%)
1.3114
1.0539
0.0116
0.9114
1.3128
0.92022
PR-MCS
1.3948
(-5.97%)
(-24.44%)
(-99.16%%
(-34.66%)
(-5.88%)
(-34.02%)
Fredrik Carlsson (2022) based
*: Our Multilingual CLIP based
Table 7: MSCOCO 3k results table.

---

Language
Metric
Original
Repetition
Removal
Masking
Jumble
Substitution
Average
1.0562
0.9882
0.8991
1.0524
1.0611
1.0114
1MCS
1.0659
(-0.91%)
(-7.29%)
(-15.65%)
(-1.27%)
(-0.45%)
(-5.11%)
1,0919
1.0102
0.9839
1.0952
1.0903
1,0543
English
$ MMCS
1.0915
(0.04%)
(-7.45%)
(-9.86%)
(0.34%)
(-0.11%)
(-3,41%)
0.7679
0.2721
0.0065
0.0163
0.138
0.24016
PR-MCS
1.1484
(-33.13%)
(-76.31%)
(-99.43%)
(8.58%%)
(-87.98%)
(-79.09%)
1.067
0.9852
0.9059
1.0738
1.0664
1,01966
fMCS
L.0688
(-0.17%)
(-7.82%)
(-15.24%)
(0.47%)
(-0.22%%
(-4.60%)
1,0763
0,9958
0,9921
1,0839
1.0796
1,04554
German
MMCS
1.0779
(-0.15%)
(-7.62%)
(-7.96%)
(0.56%)
(0.16%)
(-3.00%)
0.7511
0.3756
0.0203
0.0346
0.1556
0.26744
PR-MCS
1.1621
(-35.37%)
(-67.68%)
(-98.25%)
(9..02%)
(-86.61%)
(-76.99%)
1,0636
0.9815
0.8863
1,0625
1.0594
1,01066
MMCS
1.0653
(-0.16%)
(-7 -7.87%)
(-16.80%)
(-0.26%)
(-0.55%)
(-5.13%)
1,0745
0.9986
0.9855
1.0824
1.0778
1,04376
French
MMC
L.0758
(-0.12%)
(-7.18%)
(-8.39%)
(0.61%)
(0.19%)
(-2.98%)
1.0369
0.6367
0,018
0.0545
0.3915
0.42752
PR-MCS
1.6681
(-37.84%)
(-61.83%)
(-98.92%)
(-96.73%)
(-76.53%)
(-74.37%% %
1,0517
0.9707
0.8773
1.0445
1.0451
0.99786
MMCS
1.0527
(-0.09%)
(-7.79%)
(-16.66%)
(-0.78%)
(-0.72%%
(-5.21%%
1,0732
0.999
0.9829
1,0802
1.0756
1,04218
Spanish
MMCS
1.0747
(-0.14%)
(-7.04%%
(-8.54%)
(0.51%)
(0.08%)
(-3,03%)
1,0358
0.6189
0.0111
0.0587
0.618
0.4685
PR-MCS
1.6534
(-37.35%)
(-62.57%)
(-99.33%)
(-96.45%)
(-62.62%%
(-71.66%
1,0722
0.9875
0.8573
1.0796
1.067
1.01272
MMCS
1.0703
(0.18%)
(-7.74%)
(-19.90%)
(0.87%)
(-0.31%)
(-5.38%)
1,0275
0.9655
0,8932
1.0278
1.0255
0.9879
Japanese
MCS
1.0293
(-0.17%)
(-6.20%)
(-13.22%)
(-0.15%)
(-0.37%)
(-4.02%)
0.7931
0.6235
0.0106
0.0894
0.715
0.44632
PR-MCS
1.6279
(-51.28%)
(-61.70%)
(-99.35%)
(-94.51%)
(-56.08%)
(-72.58%)
Fredrik Carlsson (2022) based
*: Our Multilingual CLIP based
Table 8: Flickr8k results table.

---

Language
Metric
Original
Repetition
Removal
Masking
Jumble
Substitution
Average
0.7134
0.6763
0.625
0.7007
0.7114
0,68536
1MCS
0.7217
(-1.15%)
(-6.29%)
(-13.4%)
(-2.91%)
(-1.43%)
(-5.04%)
1,0606
1.0063
0.9756
1,0651
1.0614
1,0338
English
$ MMCS
1.0626
(-0.19%)
(-5.3%)
(-8.19%)
(0.24%)
(-0.11%)
(-2.71%)
0.6281
0.2754
0.0069
0.0892
0.4726
0.29444
PR-MCS
0.9769
(-35.7%%
(-71.81%)
(-99.29%)
(-90.87%)
(-51.62%)
(-69.86%)
1.0213
0.9757
0.9176
1.0352
1.0254
0.99504
fMCS
L.0244
(-0.69%)
(-5.12%)
(-10.77%)
(0.66%)
(-0.29%%
(-3.24%)
1,0433
0,9952
0,9848
1,0515
0.9754
1,01004
German
MMCS
1.0491
(-0.55%)
(-5.14%)
(-6.13%)
(0.23%)
(-7.03%)
(-3.72%)
0.6857
0.4573
0.0154
0.1558
0.7607
0,41498
PR-MCS
0.9895
(-30,7%)
(-53.78%)
(-98.44%)
(-84.25%)
(-23.12%)
(-58.06%)
1,0206
0.9788
0.9034
1,0369
1.0228
0.9925
MMCS
1.0266
(-0.58%)
(44666%)
(-12%)
(1%)
(-0.37%%
(-3,32%)
1,0456
1.0001
0.9764
1.0534
1.0499
1,02508
French
MCC
L.0509
(-0.5%)
(-4.83%)
(-7.09%)
(0.24%)
(-0.1%)
(-2.46%)
0,9655
0.7088
0.0097
0,2314
1.166
0.61628
PR-MCS
1,442
(-33.04%)
(-50.85%)
(-99.33%)
(-83.95%)
(-19.14%%
(-57.26%)
1,0189
0.9794
0.8924
1.0377
1.0207
0.98982
1MCS
1.0226
(-0.36%)
(-4.22%)
(-12.73%)
(1.48%)
(-0.19%%
(-3.21%%
1,0463
1,002
0.973
1.0537
0.9982
1.01464
Spanish
MMCS
1.0505
(-0.4%)
(-4.62%)
(-7.38%)
(0.3%)
(-4.98%)
(-3,41%%
0,9893
0.6767
0.003
0.209
1.2505
0.6257
PR-MCS
1.4526
(-31.89%)
(-53.41%)
(-99.79%)
(-85.61%)
(-13.91%)
(-56.93%)
1,0279
0.9855
0.8733
1.05
1.0259
0,99252
MMCS
1.0297
(-0.17%%
(-4.29%)
(-15.19%)
(1.97%)
(-0.37%)
(-3.61%)
1,0242
0.9963
0.9275
1.0406
1.0403
1,00578
Japanese
MCS
1.0414
(-1.65%)
(-4.33%)
(-10.94%)
(-0.08%)
(-0.11%)
(-3.42%)
0.8612
0.7769
0.0035
0.3241
1.1465
0.62244
PR-MCS
1.4113
(-38.98%)
(-44.95%)
(-99.75%)
(77.04%%
(-18.76%)
(-55.9%
Fredrik Carlsson (2022) based
*: Our Multilingual CLIP based
Table 9: VizWiz results table.

---

Language
Metric
Original
Repetition
Removal
Masking
Jumble
Substitution
Average
0.8201
0.7613
0.7896
0.8221
0.7968
0.7978
fMCS
0.8234
(-0.40%)
(-7.54%)
(-4.10%)
(-1.58%)
(-3.23%)
(-3,09%)
1,0568
0.9965
0,9801
1,058
0.9827
1,01482
English
MCC
1.0611
(-0.41%)
(-6.09%)
(-7.63%)
(-0.29%)
(-7.39%
(-4.36%)
0.7963
0.2309
0.0095
0.032
0.4516
0.30406
PR-MCS
1.1008
(-27.66%)
(-79.019))
(-99.13%)
(97.09%)
(-58.97%)
(-72.37 %
1,0298
0.9863
0.9691
1,0308
0.976
0.9984
MMCS
L.0325
(-0.26%)
(-4.47%)
(-6.14%)
(0.16%)
(-5.47%)
(-3,30%)
1,0581
0,9809
0.9737
1.0606
1.0586
1,02638
German
MMC
1.0625
(-0.41%%
(-7.68%
(-8.36%)
(0.18%)
(-0.36%)
(-3,40%)
0.7419
0,3583
0.0123
0.0544
0.6396
0.3613
PR-MCS
1.096
(-32.3%)
(-67.31%)
(-98.88%)
(-95.04%)
(-41.64%)
(-67.03%)
1.031
1.0098
0.9782
1.0329
0.9853
1.00744
MMCS
L.0353
(-0.42%)
(-2.46%)
(-5.51%%
(-0.23%)
(-4.82%%
(-2.69%)
1,0581
0.9848
0.9651
1.058
1.0578
1,02476
French
MMCS
1.0624
(-0.40%)
(-7.30%)
(-9.15%)
(-0.41%)
(-0.43%)
(-3,54%)
0.9374
0.590
0.0192
0.052
0.9585
0,51142
PR-MCS
1,5767
(-40.54%)
(-62.58%)
(-98.78%)
(-96.70%)
(-39.21%
(-67.56% %
: Fredrik Carlsson (2022) based
Our Multilingual CLIP based
Table 10: Multi30K results table.

---

Language
Metric
Original
Repetition
Removal
Masking
Jumble
Substitution
Average
0.7524
0.6577
0.6265
0.7275
0.7515
0.70312
1MCS
0.7593
(-0.91%)
(-13.38%)
(-17.49%)
(-4.19%)
(-1.03%)
(-7.40%)
1.0439
0.9699
1.0484
1.0628
1.061
1,0372
*MCS
1.0657
(-2.05%)
(-8.99%)
(-1.62%)
(-0.27%)
(-0.44%)
(-2.67%)
English
0.5984
0.2382
0.4844
0.7262
0.7421
0.55786
PR-MCS
1.0429
(-42.62%)
(-77.16%)
(-53.55%)
(-30.37%)
(-28.84%)
(-46.51% %)
PR-MCS
0.3183
0.0653
0.0012
0.0027
0,4045
0.1584
0.7125
(Few-shot)
(-55.33%)
(-90.84%)
(-99.83%)
(-99.62%)
(-43.23%)
(-77.77%)
1.0576
0.9516
1.0011
1.0546
1.0643
1.02584
|MCS
1.0708
(-1.23%)
(-11.13%)
(-6.51%)
(-1.51%)
(-0.61%) (6)
(-4.200%)
1.063
0,9638
1.0484
1.0815
1,0781
1.04696
*MCS
1.0803
(-1.60%)
(-10.78%)
(-2.95%)
(0.11%)
(-0.20%)
(-3. (3.09%)
German
0,5437
0,2521
0.4996
0.5151
0.495
0.4611
PR-MCS
0.7261
(-25.12%)
(-65.28%)
(-31.19%)
(-29.06%)
(-31.83%)
(-36.5%)
PR-MCS
0.396
0.0617
0,0011
0.1059
0,3062
0.17418
0.589
(Few-shot)
(-32.77%)
(-89.52%)
(-99.81%)
(-82.02%)
(-48.01%)
(-70.43%
1.045
0.9462
0.991
1,048
1.0463
1.0153
1MCS
1.053
(-0.76%)
(-10.14%)
(-5.89%)
(-0.47%)
(-0.64%)
(-3.58%)
1.0472
0.957
1.0322
1.0574
1.0536
1 1,02948
*MCS
1.0575
(-0.97%)
(-9.50%)
(-2.39%%
(-0.01%)
(-0.37%%
(-2.65%)
French
0.96
0.4739
1.0976
0.7335
0.9791
0,84882
PR-MCS
1.3634
(-29.59%)
(-65.24%)
(-19.50%)
(-46.2%)
(-28.19%)
(-37.74%)
PR-MCS
1.025
0.1559
0.0082
0.1034
0.9032
0.43914
1.4862
(Few-shot)
(331.03%)
(-89.51%)
(-99.45%)
(-93.04%)
(-39.23%)
(-70.45%%
1,048
0.9548
0.9996
1.0464
1.0536
1.02048
|MCS
1.0599
(-1.12%)
(-9.92%)
(-5.69%)
(-1.27%)
(-0.59%)
(-3.72%%
1,0385
0.9629
1.0393
1.0542
1.0537
1. 1.02972
*MCS
1.0577
(-1.82%)
(-8.96%)
(-1.74%)
(-0.33%)
(-0.38%)
(-2.65%%
Spanish
1.206
0,5811
1.1648
0.9128
0.8481
0.94256
PR-MCS
1.3821
(-12.74%)
(-57.96%)
(-15.72%)
(-33.96%)
(-38.64%)
(-31.80%)
PR-MCS
1.187
0.1798
0.0019
0.1405
0.473
0,39644
1.4292
(Few-shot)
(-16.95%)
(-87.42%)
(-99.87%)
(-90.17%)
(-66.90%)
(-72.26%)
1.0546
0.9768
0,9976
1.0475
1.0559
1.02648
1MCS
1.06
(-0.51%)
(-7.85%)
(-5.89%)
(-1.18%)
(-0.39%)
(-3.16%)
0.9692
0.982
1.0323
0.9813
0.963
0,98556
*MCS
0.963
(0.64%)
(1.97%)
(7.20%)
(1.90%)
(0.00%)
(2.34%)
Japanese
0.2719
0.3357
0.7354
0.6373
1.0862
0.6133
PR-MCS
1.1551
(-76.46%)
(-70.94%)
(-36.33%)
(-44.83%)
(-5.96%)
(-46.91 %)
PR-MCS
0.1385
0.1285
0.0012
0.0048
1.2969
0.31398
1.4136
(Few-shot)
(-90.2%)
(-90.91%)
(-99.92%)
(.99.66%)
(-8.26%)
(-77.79%)
f Fredrik Carlsson (2022) based
Our Multilingual CLIP based
Table 11: M-FineCapEval results table.


Reviews:
[{"review_id": "q3oJBmLcXi", "reviewer": "Reviewer_j2CY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for the thorough feedback on our study.\nWe will offer explanations, individually addressing the queries and questions you hold.\n___\n### [Response to ‘Reasons to Reject’]\n**Comment 1**. It has not been verified whether the perturbed captions really contain errors in content. It is also unclear whether the artificially created perturbations in this study are similar to those output by the model.\n\n**Answer 1**. We consider your question about the practical utility of perturbation robustness, and we aim to provide responses to this question from two perspectives: rationale and practicality.\n\nFirstly, from a **rationale standpoint**, we want to explain why image captioning metrics should possess perturbation robustness. A good metric should be able to measure good sentences as good and bad sentences as bad. In the foundational work of our study by Sai et al. [1], various criteria in both image captioning and NLG metrics are shown to lack robustness against simple perturbations. These perturbed bad sentences should be evaluated as bad according to these multiple criteria. The perturbations we employed were not directly determined by us empirically, but rather, they were leveraged from the perturbation types outlined by Sai et al. We present methodologies to address these perturbation types.\n\nMoving on to the **practicality perspective**, we have identified errors in the model output for various cases in image captioning. Particularly, issues of \"Repetition\", where certain tokens are repeated, and \"Substitution,\" where attribute placement is incorrect, occur more frequently in the model's output than other perturbations. These sentences are difficult for reference-free metrics to detect. Although such sentences are not very common, they must be filtered out as bad sentences. Furthermore, the proposed PR-MCS, while maintaining strong human correlation for normal sentences, is a comprehensive metric that maintains perturbation robustness. Thus, it can be practically utilized as it evaluates good sentences as good and has the ability to evaluate bad sentences as bad, unlike existing metrics like CLIPScore.\n___\n**Comment 2**. N-gram based metrics, e.g., CIDEr and METEOR, are considered relatively robust to perturbations, but this study has not made a comparison.\n\n**Answer 2**. N-gram-based image captioning metrics like BLEU, CIDEr, and METEOR have certain limitations: (1) they require references for sentence evaluation, and (2) their correlation with human judgment is low. The correlation with human judgment for the Flickr8k-expert dataset, as measured using Kendall's Tau, is as follows:\n\n| Metric    | τ_c |\n|-----------|--------------|\n| BLEU-1    | 0.323         |\n| BLEU-4    | 0.308         |\n| ROUGE-L   | 0.323         |\n| METEOR    | 0.418         |\n| CIDEr     | 0.439         |\n| SPICE     | 0.449         |\n| CLIPScore | **0.530**         |\n(With the exception of CLIPScore, all the others are reference-based metrics.)\n\nAs a result, recently, reference-free metrics like CLIPScore have been proposed. These metrics do not require references yet exhibit a high correlation score with human evaluations. When it comes to evaluating image captioning models in practical terms, reference-free metrics are more favorable, and among them, CLIPScore is widely adopted. \n\nAs per your suggestion, we measured the perturbation robustness of N-gram based metrics for further analysis. Below are the results of the perturbation experiments for N-gram based metrics. Due to time constraints, we did not conduct experiments in a multilingual context. Instead, we examined how well N-gram based metrics capture the presence of perturbations in English sentences. As anticipated, N-gram based metrics demonstrate robustness against perturbations.\n\n|Dataset| Jumble | Removal  | Repetition | Masking | Substitution |\n|-------|-------|-------|-------|-------|-------|\n| MSCOCO | B1:0.99 B4:0.03 R:0.44 C:2.78 S:0.39| B1:0.51 B4:0.27 R:0.69 C:3.30 S:0.42| B1:0.71 B4:0.45 R:0.85 C:4.38 S:0.74| B1:0.60 B4:0.27 R:0.59 C:2.28 S:0.23| B1:0.99 B4:0.29 R:0.67 C:3.25 S:0.45| \n| Flickr8k | B1:0.99 B4:0.03 R:0.40 C:2.77 S:0.37| B1:0.51 B4:0.27 R:0.64 C:3.12 S:0.42| B1:0.71 B4:0.46 R:0.77 C:4.12 S:0.72| B1:0.59 B4:0.27 R:0.53 C:2.11 S:0.21| B1:0.99 B4:0.36 R:0.73 C:4.27 S:0.46| \n| VizWiz| B1:0.99 B4:0.03 R:0.42 C:2.73 S:0.35| B1:0.51 B4:0.26 R:0.70 C:3.02 S:0.41| B1:0.71 B4:0.46 R:0.86 C:4.00 S:0.76| B1:0.59 B4:0.27 R:0.59 C:2.11 S:0.23| B1:0.99 B4:0.39 R:0.71 C:3.75 S:0.39| \n\n(B1, B4, R, C, and S stand for BLEU-1, BLEU-4, ROUGE-L, CIDEr, and SPICE metrics, respectively. The original scores are as follows: B1: 1.0, B4: 1.0, R: 1.0, C: 10.0, S: 1.0.)\n\nIndeed, while N-gram-based metrics demonstrate robustness to perturbations, as mentioned earlier, their value as metrics is considered low due to the limited correlation with human judgment within the current research field. Therefore, we highlight that we have achieved perturbation robustness in reference-free metrics. **We consider high human correlation to be a practical priority when metrics are actually utilized**. Therefore, we have proposed methodologies to further enhance perturbation robustness for CLIPScore.\n\nIn summary, the advantages of PR-MCS can be summarized and visualized in the table below:\n\n| Metric | Perturbation Robustness | Correlation with Human Judgment|\n|-------|-------|-------|\n| Reference-based| **Robust(√)** | Low(↓)|\n| CLIPScore | Not robust(x) | **High(↑)**|\n| PR-MCS |  **Robust(√)**  |**High(↑)**|\n\nCertainly, adding this information to the appendix would indeed be helpful in explaining our motivation.\n___\n### In Conclusion,\n\nOnce again, We’re thankful for your in-depth review and insights. We have provided responses to your concerns and questions. If your concerns are sufficiently addressed, we appreciate it if you would consider accepting our paper. Otherwise, please let us know about your remaining concerns.\n\n[1] Perturbation CheckLists for Evaluating NLG Evaluation Metrics (https://aclanthology.org/2021.emnlp-main.575) (Sai et al., EMNLP 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The susceptibility of evaluation indicators to perturbations is a severe problem; therefore, it is an important contribution to propose a method that could address this.\n2. The perturbation evaluation dataset collected in this study will be useful and significantly impact future evaluation metrics studies.\n3. This study collected data in English and four other languages and evaluated the evaluation metrics in those languages. This contributes to the construction of more general evaluation metrics.", "reasons_to_reject": "1. It has not been verified whether the perturbed captions really contain errors in content. It is also unclear whether the artificially created perturbations in this study are similar to those output by the model.\n2. N-gram based metrics, e.g., CIDEr and METEOR, are considered relatively robust to perturbations, but this study has not made a comparison.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study aims to improve evaluation metrics for image caption generation. Recent CLIP-based evaluation metrics are incapable of correctly evaluating captions with perturbations. To learn a robust evaluation metric model, this study performs contrastive learning between images and captions with and without perturbations. As test data, this study collected multilingual captions annotated with details and important elements. Evaluation on this data shows that the proposed metric can robustly evaluate perturbed captions in five languages while maintaining a high correlation with human evaluations.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jfl2ucDXM1", "reviewer": "Reviewer_J7jR", "summary": "", "questions": "Q1: line 199: why the translation process can ensure that the good and bad scores are maintained?\n\nPlease refer to Reasons to Reject for more questions.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to extend our gratitude for the exhaustive insights you have offered in relation to our research. With your thoughtful review in mind, we'd now like to address a few points to provide further context and clarification.\n___\n### [Response to ‘Reasons to Reject’]\n**Comment 1**. If the model is already aware of which perturbation would be performed, is there any comparison between using the current learned model and using rule-based systems to remove the perturbation? Conducting further analysis over the selected perturbation type may be good.\n\n**Response 1**. We agree with your viewpoint that assuming the metric's backbone model is aware of the type of perturbation, employing a rule-based approach to remove perturbations can be considered as a method to impart perturbation robustness. However, the process of the model recognizing the perturbation type might also be dependent on parameters. In this study, we opted for a parameter-based approach where the backbone model's parameters inherently recognize perturbations and assign lower scores accordingly. We conducted comparative experiments against the baseline using this method in the current research. \n\nWe also find your opinion of \"Conducting further analysis over the selected perturbation type may be good\" to be very agreeable and insightful. **Figure 1** shows a case study showcasing our fine-tuning technique's ability to achieve perturbation robustness presented for both English and German. We will add more examples, including additional case studies, to the appendix.\n___\n**Comment 2**. Lack of comparison with other baseline methods. Different backbone metrics can also be used to further validate the extensibility of the current method. For the dataset side, authors only compare Multi30K after line 318. What about the Comparison between M-FineCap3k and MSCOCO or VizWiz? Why translation with MBART-50 is not enough? \n\n**Response 2**. Expanding the experiments to include various backbone models, including UMIC, would indeed add more strength to the analysis in the study. However, CLIPScore demonstrates a higher correlation with human judgment and is more widely used than other reference-free image captioning metrics, including UMIC (Kendall tau 0.530 for CLIPScore, 0.431 for UMIC, and 0.336 for VIFIDEL[1] in case of Flickr8k-Expert benchmark). Considering that our proposed PR-MCS also performs well on good sentences similar to CLIPScore (as shown in Table 3), we believe that conducting experiments using CLIPScore as the backbone provides sufficient support for our methodology analysis.\n\nFor the dataset side, the reason for comparing the advantages of the M-FineCap3K dataset with Multi30K lies in the fact that both datasets are multilingual. We constructed the M-FineCap3K dataset to facilitate perturbation across multiple languages, as Multi30K comprises only short sentences and is composed exclusively of languages other than English (only French and German). MSCOCO and VizWiz, being English-only datasets, were not included in the comparison for multilingual image captioning datasets. Since they consist solely of English, they require machine translation models for translation into other languages, which cannot be considered human-annotated datasets. Moreover, like Multi30K, these datasets also share the limitation of containing only short sentences.\n___\n**Comment 3**. The correlation between humans and MCS is slight in Table 1: 0.236 on FR does not seem really usable. It would be good to provide some results from reference-based metrics as a comparison. Also, in Table 2, what is the expected scale of the scores? Can the scores be compared across rows?\n\n**Response 3**. In Table 1, we have provided numerical values in terms of baseline for reference-free metrics for languages other than English. A comparison between reference-based metrics for M-CapEval1K is presented below.\n\n| Metric    | M-CapEval1K (FR) | \n|-----------|--------------|\n| BLEU-1    | 0.223        |\n| BLEU-4    | 0.224        |\n| ROUGE-L   | 0.210         |\n| CIDEr     | 0.239         |\n| MCS  | 0.236\t\t|\n| PR-MCS | 0.236\t|\n\nThe value of 0.236 might not seem very high, but considering that the CLIPScore for CapEval1k in English is 0.270, we believe it can serve its purpose as a baseline for French. The CapEval1K demonstrates a higher probability distribution of moderately difficult sentences, ranging from 0.6 to 0.8 on a scale of 1 point. Arranging the sequence of these sentences is more challenging than arranging sequences of obviously good and bad sentences. Therefore, compared to benchmarks with a simple binary distribution of low and high scores, CapEval1K exhibits significantly lower human correlation. This contributes to the fact that CapEval1K itself is a challenging benchmark with a lower correlation to human judgments, partly due to its inherent nature of having a distribution that is not strongly aligned with metrics.\n\nFurthermore, the results of measuring human correlation after translating **Composite** and **Flickr8k** into French using a machine translation model are as follows:\n\n| Metric    | Composite (FR) | Flickr8k (FR) |\n|-----------|--------------|--------------|\n| BLEU-1    | 0.302        | 0.314\t|\n| BLEU-4    | 0.303        | 0.302\t|\n| ROUGE-L   | 0.317         | 0.311\t|\n| CIDEr     | 0.376         | 0.435\t|\n| MCS  | **0.497**\t\t|  **0.530**\t|\n| PR-MCS  | **0.496**\t|  **0.527**\t|\n\nIn the case of benchmarks like Composite and Flickr8k, MCS and PR-MCS achieved high human correlations (close to 0.5 for Composite, over 0.5 for Flickr8k.). Through this, we believe that the results of MCS and PR-MCS for French are quite usable as image captioning metrics.\n\nRegarding the questions in Table 2, the scales for MCS and PR-MCS range from 0 to 2.5 points. In metric evaluation, we think what matters more than the scale is the correlation with human judgment. In Table 2, as the score drop becomes more pronounced between the original output and perturbed output, it can be interpreted as a better detection of perturbations.\n___\n**Comment 4**. More verification of the dataset is necessary.\n\n**Response 4**. We have constructed the M-FineCap3k dataset through a professional data collection company, and we have translated the M-CapEval1k Benchmark using a specialized translation agency. In line with your opinion, we believe that detailed information about the dataset construction is crucial for the future utilization of the dataset. The **API and instructions** for collecting the dataset can be found in **Figure 10** and **Figure 11** in the Appendix. We will provide more detailed information about the composition of each dataset, including verification aspects such as annotation details, annotator agreement, and quality assurance methods, in the upcoming version.\n___\n**Comment 5**. There is slight relation between the multilingual part and the perturbation-prevention part of the paper. I expect more coherence in the proposed contribution.\n\n**Response 5**. We believe that leveraging multilinguality in the Perturbation-prevention part is a promising approach that can greatly contribute to the improvement of the methodology and the analysis of the research. Based on the research directions you have suggested for future studies, we think that developing methods to leverage multilinguality will further advance PR-MCS. We appreciate your insightful and valuable suggestions.\n___\n### [Response to ‘Questions’]\nQ1: line 199: why the translation process can ensure that the good and bad scores are maintained?\n\nA1. Across benchmarks for image captioning metrics, including CapEval1k, Flickr8k-expert, and Composite, intentional inclusion of poor sentences alongside reference sentences is employed to assign low scores deliberately. This practice is designed to measure human correlation by incorporating poorly constructed sentences. When utilizing machine translation models, deliberately manipulated poor sentences can be corrected to appear normal through the model's translation. For instance, an English sentence deliberately altered for grammar mistakes, receiving a score of 0.4 out of 1 for quality, might be translated into German with correct grammar. To address this concern, we employed native speakers. In the APIs and instructions detailed in Appendix Figures 10 and 11, annotators were directly instructed to translate the poorly constructed sentences while retaining their low quality.\n___\n### In Conclusion,\n\nWe greatly appreciate your insightful input. Your valuable feedback is highly valued by us. We have taken the necessary steps to respond to your inquiries and address the matters you raised. Should these solutions effectively address your apprehensions, we would sincerely appreciate your consideration in accepting our paper. However, if there are still areas of concern, we kindly request that you inform us so that we can address them appropriately.\n\n[1] Pranava Swaroop Madhyastha, Josiah Wang, and Lucia Specia. 2019. Vifidel: Evaluating the visual fidelity of image descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6539–6550.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe authors bring image captioning metric to a multilingual perspective with fair performance. In addition, M-FineCap3K is a useful resource for the community.\n\n2.\tThe motivation of the work is clear, the reference-free metrics are indeed not robust to lexical perturbations.", "reasons_to_reject": "1.\tThe current method seems to have limited contribution: multilinguality of CLIP is from the original multilingual CLIP model itself. The perturbation signals are also inherited from previous work. Conducting further analysis over the selected perturbation type may be good, e.g., showing examples of which the fine-tuning helps, developing a pattern-agnostic method, or if there are further patterns identified. If the model is already aware of which perturbation would be performed, is there any comparison between using the current learned model and using rule-based systems to remove the perturbation?\n\n2.\tLack of comparison with other baseline methods. MCS and PR-MCS are the only metrics compared in the main results table (Table 2). However, other methods such as UMIC (Lee et. al., 2022) can also be included here with minimum adaptation. These different backbone metrics can also be used to further validate the extensibility of the current method.  \nFor the dataset side, authors only compare Multi30K after line 318. What about the Comparison between M-FineCap3k and MSCOCO or VizWiz? Why translation with MBART-50 is not enough? More quantitative or qualitative results needed to be shown.\n\n3.\tThe results are not clear and sometimes confusing. For example, the correlation between humans and MCS is slight in Table 1: 0.236 on FR does not seem really usable. It would be good to provide some results from reference-based metrics as a comparison. Also, in Table 2, what is the expected scale of the scores? Can the scores be compared across rows?\n\n4.\tThe dataset is a good resource for the community. However, more verification of the dataset is necessary (e.g., annotation details, annotator agreement, quality assurance methods).\n\n5.\tThere is slight relation between the multilingual part and the perturbation-prevention part of the paper. I expect more coherence in the proposed contribution. In the area of WSD, various methods have been proposed to leverage multilinguality to improve word sense understanding. Similar approaches can be applied here. For example, Kang et. al., 2023. Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models.", "questions_to_authors": "Q1: line 199: why the translation process can ensure that the good and bad scores are maintained?\n\nPlease refer to Reasons to Reject for more questions.", "paper_topic_and_main_contributions": "This paper presents PR-MCS, a multilingual metric evaluating image captioning with images and outputs. PR-MCS is built upon multilingual CLIP and further finetuned with three losses with pre-defined perturbation signals. To evaluate the proposed method, the authors in addition propose a multilingual image captioning dataset M-FineCap3k. Experimental results on five languages show that PR-MCS has good performance across dataset sets and languages, and can assign lower scores for the perturbated cases.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None as far as I know", "Typos_Grammar_Style_and_Presentation_Improvements": "None as far as I know"}, "extra_text": {}}, {"review_id": "Nf2GBXL90y", "reviewer": "Reviewer_ccYd", "summary": "", "questions": "- From Table 1, the scores of the MCS(dagger) and MCS(star) are almost same. What is the advantage of doing the pre-training? The MCS(dagger) already provides pre-trained models with different languages.\n- For the loss of PR-MCS, the idea is to make the distance between the image embedding and the original caption embedding close and the distance between the perturbed caption embedding and other embeddings further. I wonder if the Triplet loss can be used instead of pair-wise loss. Have you tried or thought about the triplet loss when you design the model?\n- In Table 2, the few shot results are based on the different test sets on the regular M-FineCap3k dataset. It can mislead the performance results by reporting under the same dataset. Also, how many few shots are used for the evaluation? It would be more informative if the statistics of the sequence length on the datasets is provided for L510-L516.\n- In Table 3, PR-MCS results underperform in general except for the Pearson correlation in English. Also, can you provide the significant test results for the correlations? It does not provide convincing evidence to claim that PR-MCS is a metric with a strong correlation with human judgment.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, We'd like to express our appreciation for the comprehensive input regarding our research. We'll proceed to provide clarifications, addressing each of your inquiries and concerns respectively.\n___\n### [Response to ‘Questions for the authors’]\n\n**Q1**. From Table 1, the scores of the MCS(dagger) and MCS(star) are almost same. What is the advantage of doing the pre-training? The MCS(dagger) already provides pre-trained models with different languages.\n\n**A1**. The reason we conducted pre-training on the text encoder was to develop our own implementation of the backbone text encoder for use in our experiments. We would like to note that the pre-training methodology itself is not a contribution to our research. Our primary contribution lies in proposing the multilingual CLIPScore (MCS) based on the multilingual CLIP framework, which can be applied to various languages. Therefore, the choice of which multilingual CLIP model to use as a backbone is not a critical issue, as our focus is on introducing MCS that is applicable to multiple languages using the multilingual CLIP foundation. Multilingual CLIP models have been presented not only by Fredrik et al. [1] but also by Others [2]. One reason we chose to utilize our pre-trained text encoder as the backbone for the MCS metric across multiple languages was its superior performance. Subsequently, we conducted experiments on perturbation robustness by further fine-tuning this backbone.\n___\n**Q2**. For the loss of PR-MCS, the idea is to make the distance between the image embedding and the original caption embedding close and the distance between the perturbed caption embedding and other embeddings further. I wonder if the Triplet loss can be used instead of pair-wise loss. Have you tried or thought about the triplet loss when you design the model?\n\n**A2**. The concept of constructing the loss function through Triplet loss using image embeddings as anchors seems highly insightful. Although we didn't initially contemplate incorporating training through Triplet loss during the model design phase, we've devised a notion for a distinct loss that quantifies distances. This is intended to allow us to assess and fine-tune the contribution of individual loss components using Lambda values. This approach will enable us to measure the influence of each loss and make adjustments accordingly, enhancing the overall effectiveness of our model.\n___\n**Q3**. In Table 2, the few shot results are based on the different test sets on the regular M-FineCap3k dataset. It can mislead the performance results by reporting under the same dataset. Also, how many few shots are used for the evaluation? It would be more informative if the statistics of the sequence length on the datasets is provided for L510-L516.\n\n**A3**. I agree with your perspective. Since the results in the few-shot setting were obtained using a different test set, it's possible that they could lead to misunderstandings within the same sector of the table. I believe that modifying the table by adding horizontal lines would be helpful to address this issue.\n\nWe split M-FineCap3k into subsets proportioned to 1:9 in size and use only 300 perturbed captions as the few-shot input. (L520-523). The statistics for M-FineCap3k, including sequence lengths, can be found in Appendix D and Table 4. The sentences consist of around 20 words each, making them quite lengthy. Following your suggestion, providing this information along with the lengths of MSCOCO, VizWiz, Flickr8k, and Multi30k in lines L510-516 would indeed make the content more informative.\n___\n**Q4**. In Table 3, PR-MCS results underperform in general except for the Pearson correlation in English. Also, can you provide the significant test results for the correlations? It does not provide convincing evidence to claim that PR-MCS is a metric with a strong correlation with human judgment.\n\n**A4**. In Table 3, a Kendall score of PR-MCS (0.267), similar to CLIPScore's 0.27, is a quite strong indicator of a favorable correlation with human judgment compared to other metrics. Furthermore, the correlation with human judgment measured using Kendall's Tau for other English datasets like Composite and Flickr8k is as follows:\n\n| Metric    | Composite | Flickr8k |\n|-----------|--------------|--------------|\n| BLEU-1    | 0.313        | 0.323\t|\n| BLEU-4    | 0.306        | 0.308\t|\n| ROUGE-L   | 0.324         | 0.323\t|\n| METEOR    | 0.389        | 0.418\t|\n| CIDEr     | 0.377         | 0.439\t|\n| SPICE     | 0.403        | 0.449\t|\n| PR-MCS  | **0.493**\t|  **0.527**\t|\n| CLIPScore | **0.497**        | **0.530**\t |\n\nThe correlation of PR-MCS with human judgment for English sentences is notably strong compared to conventional reference-based metrics and is similar to CLIPScore. Furthermore, as we assert in Table 3, PR-MCS operates in a multilingual setting and demonstrates perturbation robustness, distinguishing it from CLIPScore.\n___\n### In conclusion,\nOnce again, we appreciate your detailed reviews and insightful comments. We greatly appreciate your insightful input. We've addressed your questions and issues accordingly. Especially considering that you have requested multiple questions regarding the **[Reasons To Reject]**, if our rebuttal has adequately and appropriately addressed those questions, it is our belief that your level of interest and preference towards this study could potentially increase. If these resolutions adequately meet your concerns, we would be grateful if you could consider accepting our paper. In case there are any unresolved matters, please keep us informed.\n\n[1] Manuel Romero Fredrik Carlsson, Ariel Ekgren. 2022. https://github.com/freddefrallan/multilingual-clip. \n[2] Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper shows the proper motivation for the proposed datasets and reasonable method and experimental setup.\n- Though contrastive loss is not a novel method for multi-lingual settings, pair-wise contrastive loss for different languages can have benefits.\n- The experiments are performed on different datasets.", "reasons_to_reject": "Please see the questions below.", "questions_to_authors": "- From Table 1, the scores of the MCS(dagger) and MCS(star) are almost same. What is the advantage of doing the pre-training? The MCS(dagger) already provides pre-trained models with different languages.\n- For the loss of PR-MCS, the idea is to make the distance between the image embedding and the original caption embedding close and the distance between the perturbed caption embedding and other embeddings further. I wonder if the Triplet loss can be used instead of pair-wise loss. Have you tried or thought about the triplet loss when you design the model?\n- In Table 2, the few shot results are based on the different test sets on the regular M-FineCap3k dataset. It can mislead the performance results by reporting under the same dataset. Also, how many few shots are used for the evaluation? It would be more informative if the statistics of the sequence length on the datasets is provided for L510-L516.\n- In Table 3, PR-MCS results underperform in general except for the Pearson correlation in English. Also, can you provide the significant test results for the correlations? It does not provide convincing evidence to claim that PR-MCS is a metric with a strong correlation with human judgment.", "paper_topic_and_main_contributions": "- They finetune the text encoder of CLIP to distinguish the perturbed text from the original caption.\n- They introduce a new evaluation dataset, M-FineCap3k, containing 5 translated languages from English captions.\n- The proposed PR-MCS can distinguish lexically perturbed text in different datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new metric that's robust to perturbations and that can be used for multilingual image captioning. Alongside the metric, a data set is introduced that is used to verify the robustness of this metric.

Pros:
- Tackling an important problem; many other evaluation metrics are highly susceptible to perturbations.
- Multilingual approach, much better than just focusing on one language.
- Introduces both a new metric and a new data set that can both be used by the community.
- Maintaining a strong correlation with human judgment while improving robustness to perturbations.

Cons:
- The comparison to other metrics -- especially as it relates to human judgment correlation -- in the current review copy could be clarified further, e.g. with a table as shown at the bottom of https://openreview.net/forum?id=EpBNf4Arod&noteId=SIuJnYRUqN (it would make sense for this to be in the main body, not relegated to an appendix).
- There's not a lot of detail around how the data set was constructed.
Decision: Accept-Findings