Mutual Information Divergence: A Unified Metric for Multimodal Generative Models
Authors: Jin-Hwa Kim
Yunji Kim
Jiyoung Lee
Kang Min Yoo
Sang-Woo Lee
Venue: NeurIPS 2022 Accept (2022)
Keywords: text-to-image generation
image captioning
evaluation metric
mutual information
vision and language

Abstract: Text-to-image generation and image captioning are recently emerged as a new experimental paradigm to assess machine intelligence. They predict continuous quantity accompanied by their sampling techniques in the generation, making evaluation complicated and intractable to get marginal distributions. Based on a recent trend that multimodal generative evaluations exploit a vison-and-language pre-trained model, we propose the negative Gaussian cross-mutual information using the CLIP features as a unified metric, coined by Mutual Information Divergence (MID). To validate, we extensively compare it with competing metrics using carefully-generated or human-annotated judgments in text-to-image generation and image captioning tasks. The proposed MID significantly outperforms the competitive methods by having consistency across benchmarks, sample parsimony, and robustness toward the exploited CLIP model. We look forward to seeing the underrepresented implications of the Gaussian cross-mutual information in multimodal representation learning and future works based on this novel proposition. 

Mutual Information Divergence: A Unified Metric for Multimodal Generative Models
Jin-Hwa Kim NAVER AI Lab, SNU AIIS
Yunji Kim
Jiyoung Lee NAVER AI Lab
Republic of Korea
Republic of Korea
1nhwa. wwwmnavercor.ppppp.....oo com
fyunji. .iim lee. @@navercorp.com
Kang Min Yoo NAVER Lab, CLOVA, SNU AIIS Republic of Korea
Sang- Woo Lee NAVER CLOVA, Lab, KAIST Republic of Korea
kangmin.yo@@haverooop..... com
sang woo www..eeeeeeerrrr....oo com
Abstract
Text-to-image generation and image captioning are recently emerged as a new experimental paradigm assess machine intelligence. They predict continuous quantity accompanied by their sampling techniques in the generation, making evaluation complicated and intractable get marginal distributions. Based recent trend that multimodal generative evaluations exploit a vison-and-languaa, pre-trained model, propose the negative Gaussian cross-mutual information using the CLIP features as unified metric, coined by Mutual Information Diver- gence (MID). To validate, extensively compare it with competing metrics using carefully-generatte or human-annotated judgments in text-to-image generation and image captioning tasks. The proposed MID significantly outperforms the competitive methods by having consistency across benchmarks, sample parsimony, and robustness toward the exploited CLIP model. We look forward seeing the underrepresented implications of the Gaussian cross-mutual information in multi- modal representation learning and future works based on this novel proposition. The code is available at https: htt///gtttuu giggtuu.b....... www/noaeo/oeeeeeeeeeew ai/mid metric.
Introduction
multimodal generative model, including text-to-image generation [1] and image captioning [2] models, is emerging research topic showing interpretative multimodal understanding, text image retrieval, machine creativity, etc. The gist of learning multimodal generative models is to understand how to connect one modality to the other and generate the corresponding representations following the desired data distribution. However, measuring the distance or divergence between the model and data distributions is generally intractable due the finite data and generation cost. Therefore, the proposed metrics attempt to approximate it with polynomial-sin sized data and generated samples [3].
For text-to-image generation, the widely-used metrics Inception Score (IS) [4] and Fréchet Inception Distance (FID) [5]. These metrics are originally proposed for non-conditional generative models, which are repurposed to measure the distance between the data and model conditional distributions using a validation split. This idea was supported by the effectiveness of deep features perceptual metric [6], although these metrics use the Inception V3 [7]. Not surprisingly, there are attempts to develop more robust metrics using multimodal pre-trained models, such as object
"Correpooding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1.0
60
Eincoder (shared)
CLIP
MOTUAL INFORMATION
CUB
COCO
LN-COCO
Real Images
Z
CUPPS
0.6
40
MMF-CELLSAAHIO
Autlbckk tiee wit.
MI Divergence (MID)
GT Captions
0.4
20
COCO
CLIP Encoder (shared)
0.2
LN-COCO
10
MM Carnn HQ
MID SCORE
Kiwin aes witicio
0.2
0.4
0.8
0.2
0.4
0.6
0.8
Evaluating sample(s)
TXT-IMG SHOFFLE RATIO
TXT-IMG SHIVPBLE
Figure I: Left. The schematic diagram of our proposed method. Right. The consistent property of Gaussian MI is in stark contrast to CLIP-S [19]. We vary the ratio of text-image shuffling where the counterparts in the selected pairs are deliberately shuffled, depicting misalignment. For CLIP-S, we observe varying slopes across the datasets depending on the data domain while MI shows relatively consistent tendency. The CUB and MM-CelebA describe birds and human faces, respectively. having narrow domains compared to COCO and LN-COCO with various objects.
detectors [8], image captioning models [9], and Iision-and-languue pre-trained models [10, 11]. Significantly, the image captioning method transforms texi-to-image measurement into image-to-text measurement, providing a different viewpoint with cyclic consistency.
For image-to-text generation, image captioning, the COCO Caption Evaluation toolkit seems to be the standard to measure the divergence from ground-truth captions having BLEU [12], METEOR [13], ROUGE [14], CIDEr [15]. and SPICE [16]. Similar the IS and FID of text-to- totttto-mmage generation metrics, these metrics are merely n-gram-based statistical methods neglecting conditional images. The breakthrough in improving the correlation with human judgment comes from the utilization of
the pre-trained vision-and- lnnna-aan models, egg., TIGEr [17], ViLBERTScore-- [18], and RefCLIP-
S [19]. We speculate that these two directional metrics getting closer measuring the generative divergence of text-image alignment.
This paper proposes unified metric for multimodal generative models. In probability theory and information theory, mutual information (MI) measures how much one random variable tells us about the other. In multimodal generation, the MI of two modalities quantitatively measures how much the generated is well-aligned with the condition. From this motive, we propose to use the Gaussian mutual information where the probability distributions of visual and textual features and borrow the idea of cross-mutual information [20] to measure the MI divergence from the real data distribution, which is the expectation of point-wise mutual information with respect to evaluating samples. Surprisingly, the proposed method outperforms previous works with significant margins on the assorted benchmarks of text-to-image generation and image captioning, including standard human judgment correlation benchmarks.
defined by the means and covariances
Section 2 introduces the previous works text-to text-to-image generation and image captioning metries, and the work on cross-mutual information. Section describes the proposed method defining the continuous mutual information using multivariate Gaussian distributions and the negative cross- mutual information, which is the Mutual Information Divergence (MID) that we term. Section consists of two parts, evaluation rext-to-image generation and image captioning evaluation, including related discussions. Section 6 concludes the work with remarks.
We summarize our contributions follows:
To the best of our knowledge, we firstly propose the negative cross-mutur information under the Gaussian assumption unified metric for multimodal generative models.
We provide three theoretical analyses the proposed method MID, out-of-distribution detecting by the squared Mahalanobis distances, bias and variance decomposition, and its relation to the Kullback-Leibler divergence.
We achieve the state-of-the-art in text-to-image generation and image captioning benchmarks including the generated and human Likert-scale judgment correlations, visual reasoning accuracy, Flickr8K- -Eppert, Flickr8K-CF, Pascal-50S, and FOIL hallucination detection.

---

2 Related work
2.1 Metrics for assessing text-to-image generation
Traditional metrics. One of the widely used metrics is the Fréchet Inception Distance (FID) [5] that measures the distributional difference between synthetic (fake) and real-world images (real). Training and validation distributions are independent, making model that fails to match the conditional distributions it does not reflect the given textual information. Although it measures fidelity along with Inception Score [4], it cannot directly measure the alignment of text and image. The alternative metrics [21-23] are proposed to evaluate the fidelity and diversity. Text-to-image metries. Dedicated to assessing text-to-image generation, the R-Precision exploits the Deep Attentional Multimodal Similarity Model (DAMSMM) [24] to calculate the top-1 retrieval accuracy from one hundred text candidates for the generated image query. Besides, the CLIP R-Precision [11] exploits the CLIP [10] showing a better retrieval performance and human judgment correlation. However, false- negative candidates (accidentally correlated) or strong negative candidates (totally unrelated) may interfere with the accurate assessment [25]. To evaluate the quality of individual objects, the SOA [8] attempts to measure the object detection accuracy using the YOLOv3 [26] based on the object classes that appeared in the text but cannot consider other factors. Caption generation [9] is another approach using the vision-and and-language pre-trained model. The motivation is the cyclic consistency that the generated caption from the generated image should match with the text for image generation. However, the model bias including object hallucination [27] and the accumulated errors from metrics are drawbacks. Diagnostic datasets. Park et al. [11] provide the curated splits of the CUB [28, 29] and Flowers [30] to assess unseen color and shape compositions in the narrow domains. DALL- Eval [31] proposed a diagnostic dataset PaintSkills to evaluate visual reasoning skills to assess models based on this dataset. Since the dataset is generated from a 3D simulator using limited configurations, the data distributions deviate from other real-world datasets.
2.2 Metrics for assessing image captioning
Reference-only metrics. Borrowing from machine translation literature, image captioning models are evaluated using reference-based textual metrics where five references are usually used. BLEU-4 [12], ROUGE-L [14], and METEOR [13] are n-gram precision or recall-based metrics, CIDEr [15] uses if-idf weighting and stemming, while SPICE [16] uses semantic parsing and scene graph analysis. Notably, BERT-S++ [32] considers inter-reference variance using the fine-tuned BERTScore [33] for image captioning. Weeterene-witth-image metrics. The recently proposed metries are considering the images used for generating captions. TIGEr [17] uses a pre- trained SCAN [34] while ViLBERTScore- F[18] uses pre-trained ViLBERT [35] exploiting the vision-and -nd-o-and-- aligmments from large- scale data and multiple tasks. Similarly, CLIP-S and RefCLIP-S [19] use the pre-trained CLIP, a more powerful vision-and- lan-uuua model, outperforming the previous methods. Implications. CLIP R-Precision for text-to-image generation and RefCLIP-S for image captioning share the same motivation exploiting the same vision-and-lan lad-langgge pre-trained model. Here, we remark on the unifying metrics in two different tasks (e,g., CLIP-S), text-to-image generation and image captioning. and propose a new unified metric for the multimodal generative models based on the continuous mutual information considering the covariances of two modality groups
2.3 Metrics for assessing other tasks
One of the other applications worth mentioning is text- based motion generation tasks [36]. Since traditional metrics, e.g., average position error or average variance error, merely rely on numerical differences instead of high-level semantics, they proposed to use a motion-text alignment pre-trained model (mCLIP), enabling to measure CLIP-S and our proposed MID. It validates our speculation that the unified metric can effectively apply beyond vision and language.
2.4 Cross-mutual information in machine translation
Bugliarello al. [20] proposed the cross-mutual information (XMI) as a metric of machine translation exploiting a probabilistic view in neural machine translation models. XMI is an analogue of mutual
information for cross-entropy defined as XMI(S - T) = Has (T)- H H (T) Hag!! (T) denotes
the cross-entropy of the target sentence T under a language model qLM and H yur (T) S) is the cross-
3

---

conditional entropy under gross-lingual model qMr. In practice, they exploit two model distributions (tm(t) and gwr(t([s) to approximate the XMI as follows:
ALM (t(9)
XMI(S +)) ~
Σ log
QMT (t)) ((())
where N denotes the number of held-out evaluating samples. Since language models consider finite size of vocabulary, the cross-entropy can be efficiently approximated using the target sentences; however, it limited readily applying other generative models.
Method
3.1
Continuous mutual information
We introduce a unified metric for conditional generative models not depending on the modalities of condition and generation. To measure the alignment of condition and generation, we first consider the continuous mutual information of the condition x and generation y as follows:
T(X: Y) (xp())) log p(x,y) y) p(x)p(y)
where the probability and joint probability distributions are multivariate Gaussian, which is the maximum entropy distribution for the given mean μ and covariance Σ [37]. The first two moments are used for practical reason [5]. The multivariate Gaussian distribution is defined as:
p(x) (2 det(E)
1
exp | (x- - - (x [4]
where D the dimension of x. The mutual information with the Gaussian distributions is reduced to:
T(X: Y) = log (del(EEx) det(E))
1 2
det(Ez)
where Σx and Σy E R DxD are the covariances of the condition and generation and Σg E R2Dx2D is the covariance matrix of the concatenation z of x and y representing the joint distribution. The proof can be found in Appendix A.1. Note that we use log det(E) = >, log X; for numerical stability, where \; is the eigenvalue Σ.
For high-dimensional features, we consider two encoders fx and fy to get & and y, respectively, maximizing the mutual information of the encoded representations, I(X; Y). Specifically, the image and text encoders of the CLIP [10] are used since these are pre-trained on 400M image-text pairs using the InfoNCE loss, maximizing lower bound on mutual information [38]. Without loss of generality, we use x, y instead of x, y to denote the feature vectors to calculate the moments.
3.2 Point-wise mutual information for pair-wise evaluation
Based on the previous continuous mutual information, derive the point-wise mutual information (PMI) for pair-wise evaluation. Please see Appendix A. for the detail. The PMI is defined as:
PMI(x:y) = I 1 (DM + (x) -
(4)
(x where D3, denotes the squared Mahalanobis distance (SMD), (x -x
Tx px)
where Px and Σx are the mean and covariance of x, and similarly for D (y) and (z). z denotes [x; y. The MI is from the normalization of the Gaussians and the SMDs are from the exponential of
the Gaussians. The second term measures the difference between the distances, D2, + D2, (y)
and D2 (z), assessing the deviation from the MI. Notice that the expectation of the second term with respect to the sample distribution is zero (see Appendix A.

---

Table I: Generated Likert-scale judgment correlation using LAFITE, uses the i..dd samples having
at least one detected object, which was 88.3% of samples, to calculate the SOA accuracy per image.
Method
Backbone
Kendall Te
Kendall T
SOAT [8]
YOLO-V3
47.3
51.8
CLIP-S [19]
CLIP (ViT-B/32)
40.8
35.3
InfoNCE [38]
CLIP (VIT-B/32)
44.1
38.2
CLIP-R-Precision [11]
CLIP (VIT-B/32)
66.0
56.1
OFA-Captioning+CL.R [19, 39]
OFA-Large CLIP (ViT-B/32)
72.0
62.3
CLIP-S [19]
CLIP (ViT-L/14)
52.2
45,2
InfoNCE [38]
CLIP (ViT-L/14)
64.8
56.1
CLIP-R-Precision [11]
CLIP (ViT-L/14)
69.6
58.1
CPA-CCppioning-CCIPPSS [19, 39]
OFA-Large + CLIP (ViT-1J14)
73.7
63.8
MID (ours)
CLIP (VIT-B/32)
74.6
64.6
MID (ours)
CLIP (ViT-L/14)
87.3
75.6
3.3
Mutual Information Divergence: the expectation of PMI w.r. evaluating samples
We propose to use the expectation of PMI with respect to the evaluating sample (X. ÿ), measuring the divergence from the ground-true truth or reference samples (X, Y). The metric is defined as follows:
E ((x.))
Y) B D2 + D3, (9) - D3 (2)
(5)
where D and (x, y) denote the of evaluating samples and pair of evaluating sample, respectively, z denotes y]. Notice that the expectation of D (x) can be decomposed to the bias and variance terms as follows (Appendix A.2 for the proof):
Ex Dr (x)] (μxx Mx) TEx (1x px) + tr (Ex Ex))
(6)
considering the mean and covariance deviations from the reference, . N( Ex). Eg (3) is D when y y generative condition since the moments equal to the counterparts.
By the way, we can show that follows (The proof can be found in Appendix A.3):
ý) related to the Kullback-Leibler divergence as
E (iax)-PPMI(S ÿ) (X: Y) + DKL (p(x) |l p(x)) D (p(2) || p(z))
For simplicity, we denote our proposed method ((s,)))DPMII ý) as Mutual Information Diver- gence (MID), comparable of the FID. In practice, we use Equation using the double-precision CLIP features. For piint-wise evaluation, we use the PMI(X, y) without the expectation.
Experiment
4.1 Evaluation on text-to-image generation
Implementation details. Without an explicit mention, we use the CLIP (ViT-L/14) to extract image and text embedding vectors. Note that it is crucial use double-precision for numerical stability.
Generated Likert-scale judgments. To carefully assess the text-image alignment, we consider the four-scale alignment using the real and fake images from the COCO dataset [2]. We regard the real images as a four-point set, the fake images generated by the ground tround-truth captions as three- three-point two-point set, and the randomly sampled (misaligned) fake images as one-point set. We assume that the fake images generated by the foiled captions should be inferior compared with the fake images generated by the ground-truth captions because the model cannot exploit the critical information to generate key objects. The current state-of-the-ar LAFITE [41] pre-trained on the COCO is used for our text-to-image generation
set, the fake images generated by the foiled captions 2 [40]
For the details, please refer to object hallucination section Section 4.2 Figure 11 Appendix. Phttps: // thub. com/ /rbooo//afitte
5

---

Table 2: AMT human Likert-scale judgment correlation. Using the 2K fake images from LAFITE and VQ-Diffusion, we collected fine-grained text-image alignment judgments and measure the judgment correlation of competing metrics. i 90.2% of samples having at least one detected object.
Method
Backbone
Kendall Te
Kendall T
SOA! [8]
YOLO-V3
5.6
7.2
CLIP-S [19]
CLIP (VIT-B/32)
10.6
10.6
InfoNCE [38]
CLIP (VIT-B/32)
10.9
10.9
FLIP-R-Precision [11]
CLIP (ViT-B/32)
5.9
6.8
OFA-CapiionnggCCL.PP [19, 39]
OFA-Large + CLIP (ViT-B/32)
8.1
8.1
CLIP-S [19]
CLIP (ViT-L/14)
11.1
11.1
InfoNCE [38]
CLIP (VIT-L/14)
11.1
11.1
[11]
CLIP (ViT-L/14)
9.4
9.7
OFA CrAACCattoninnCLLI [19, 39]
OFA-Large + CLIP (ViT-L/144
7.8
7.8
MID (ours)
CLIP (VIT-B/32)
11.9
11.8
MID (ours)
CLIP (VIT-L/14)
12.4
12.4
FREQUENCY
3K
REAL
FoILzD FAKz
MISALINEE FAKE
FaRz
3K
2K
2K
1K
1K
-150
775
50
POINT-WISE INFORMATION (OURS)
OFA-CAPIINING-CLIISS
KK
FArQueNC
3K
2K
2K
1K
1K
-10.5
10045
-10.4
-10.55
+10.3
-10.25
-10.2
-10.15 -10,1 -10.05
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
INFONCE
CLIP-S
Figure 2: The histogram shows the frequencies of the competing methods using the CLIP of ViT-L/14 for the four-scale judgments from the COCO dataset. For the detail of method, please see the text.
model. Notice that when evaluate the metrics, the ground-truth captions used to measure the text-image alignment of the generated images. We believe this generated benchmark can proxy to human judgments with careful manipulation of the fake images using the foiled captions.
randomly sample 30K captions from the FOIL dataset [40] to build 120K judgments. We report the Kendall's coefficient [42] to measure the rank correlation, variant of or accounting for ties. Table 1, our method consistently outperforms competing methods. InfoNCE denotes negative InfoNCE loss [38] calculating the softmax function over the 30K captions. Remind that the InfoNCE maximizes a lower bound on mutual information [38]. Since this is estimated using smaller batch size when optimizing, it shows limited capability alignment. Please refer to Table 8 and Figure 8 in Appendix for the VQ-Diffusion [43] benchmark.
metric for the text-image
Figure 2 (top left) shows the histogram of the frequencies of PMI showing the four categories. Fake samples have generally lower values than Real's. Foiled Fake has lower values than Real and Fake having long tail. We observe that foiled caption broadly impacts the text-image alignment as we expected. The negative PMI often observed for the fake images, which are deviated from the distribution of real images. Whereas, the histogram of CLIP-S (bottom right) and InfoNCE (bottom left) show Fake samples where its scores are higher than Real's, ss while the overlapping areas greater
of Fake and Foiled Fake (62.2% and 61.6% for CLIP-S and InfoNCE, respectively) than PMI's (55.2%). making it difficult to differentiate the degree text-image alignment. For the caption generation method [9), exploit the current state-of-the-art image captioning model of the OFA-Lerge [39] pre-trained on huge mixture of publicly available datasets to generate captions for
'The CLIP-S [19] is defined as 2.5 cos(x, y) which is the scaled cosine similarity of the CLIP features.

---

Table 3: Visual reasoning accuracy using the foiled caption trick.
Object
Count
Color
Spatial
Object
Count
Color
Spatial
Metric
CLIP ViT-B/32
CLIP ViT-L/14
CLIP-S [19]
0.318
0.026
0.068
0.025
0.585
0.157
0.209
0.169
CLIP-R-Precision [11]
0.168
0.016
0.031
0.019
0.238
0.046
0.058
0.041
SOA [8]
0.367
0.028
0.039
0.035
0.365
0.030
0.035
0.030
InfoNCE [38]
0.416
0.042
0.094
0.049
0.675
0.230
0.284
0.243
MID (ours)
0.792
0.290
0.332
0.280
0.843
0.443
0.481
0.457
the generated images. Then, CLIP-S [19] is used to assess the quality of image captioning. Notice that CLIP-S outperforms traditional image captioning metrics as shown in Section 4.2.
Visual reasoning accuracy using the foiled caption trick. Inspired by the foiled caption trick, we extend four visual reasoning skills for object, count, color, and spatial relationship. For each category, we define a set of tokens, and we foiled those tokens in the caption by randomly swapping to the other token. We build three sets images, the real images, the fake images, and the foiled fake images. We measure the accuracy that getting one point for the foiled fake images having the lowest score, or zero for the other cases. Table? 3 shows that our method achieved the best performance across all categories. The runner-up was InfoNCE, while SOA was ineffective differentiate among the two fake images and the real image. Although DALL-Eval [31] proposed to use a detector and its dedicated heads for the count, color, and spatial relationship tasks, this was limited to the 3D-generated images with their near pear-pe detection ability. Remind that the accuracy of random guessing is 33.3%, where MID requires powerful feature extractor of the CLIP ViT-L/14 to get meaningful performances on the count, color, and spatial relationship tasks. For the detail, please refer the text in Appendix C.
Human Likert-scale judgment. We collect 10K one-to-four Likert-scale human judgments for 2K fake images using the LAFITE and VQ-Diffusion from the Amazon Mechanical Turk (AMT) for the fine-grained comparison with the competing metrics. For each image, we collect five annotations from unique workers taking its median for reliable correlation measurement. Table 2 shows the consistent results with our generated judgment correlation benchmark. Since this benchmark aims for the fineigrained judgment among only fake images, overall scores are relatively lower than the generated benchmark. All correlation results are significant having the p-value < 0.001. For the details of the collection procedure and data statistics, please refer to Appendix D, where Figure 10 shows the visualization of some examples comparing with human judgment scores. We also report the comp-t2i benchmark [11] results for the compositional evaluation of the CUB and Flower datasets in aspects of color and shape in Appendix F.
4.1.1 Discussions
Consistent metric across datasets. A distinct property of our method is the consistency across datasets. As shown in Figure 1, the cosine similarity- bbssdd method, CLIP-S suffers the inconsistent results. For example, the CUB [28, 29] and MM-CelebA-HQ [44] have narrow domains, birds and human faces, respectively, which is prone to get similar score for all samples in the datasets by cosine similarity. To validate our hypothesis, we vary the ratio of text-image shuffling using the real datasets where the counterparts in the selected pairs are deliberately shuffled, depicting misalignment. For CLIP-S, we observe the inconsistency depending on datasets, while MI shows consistent tendency. Notice that the expectation of PMI is reduced to MI for the real images (Appendix 1.).
Inspecting possible over-fitting with the CLIP features. Figure shows the normalized scores of metrics across feature extractors. To get the normalized score, we subtract the score of RN101 and divide by the standard deviation of four scores across feature extractors reminiscent of z-score. We expect the footprint of metrics should be consistent across different generative models if the model is not over-fitted to the metrics. LAFITE used both encoders of CLIP ViT-B/32 and VQ-Diffusion used the text encoder of CLIP ViT-B/32, while DM-GAN did none of them [45]. InfoNCE and HIIP-R-Preciiin are related to the contrastive training losses (LAFITE, and DM-GAN for the DAMSM loss [24]), which may lead to drastic change of the normalized scoring signature across the feature extractors. While the proposed MID was relatively stable across the generative models.
7

---

LAFITE
VQ-DarrssiOn
DM-GAN
RN101 WITBB/X WIT-B/16 WIT-/11
RN101 WI-B/2 VIT-B/16
WT-L11
RN101
VIT-B/22 WT-BB166 WI-LL/1
MID
CLIPS
CLIP-X PRECIOON
SUA
IxFONCE
CAPTION
Figure 3: The footprints of metrics across feature extractors using the normalized scores.
Table 4: Flickr8K-Expen -xpeert human judgment correlation.
Method
Kendall
Table 5: Flickr8K-CE human judgment correlation.
BLEU-1 [12]
32.3
BLEU-4 [12]
30.8
Method
Kendall 7b
ROUGE-L [14]
32.3
BERT-S (RoBERTa-F)
39.2
BLEU-4 [12]
16.9
METEOR [13]
41.8
CIDEr [15]
24,6
CIDEr [15]]
43.9
METEOR [13]
22.2
SPICE [16]
44.9
ROUGE-L [14]
19.9
LEIC (n.) [46]
46.6
SPICE [16]
24.4
BERT-S++ [32]
46.7
BERT-S (RoBERTa-F)
22.8
TIGEr [17]
49.3
LEIC [46]
29.5
NUBIA [47]
49.5
CLIP-S [19]
34.4
ViLBERTSSore- [18]
50.1
RefCLIP-S [19]
36.4
CLIP-S [19]
$1.2
MID (ours)
37.3
RefCLIP-S [19]
53.0
MID (ours)
54.9
Comparison with the state-of-the-art models using MID. Table 9 in Appendix E exhibits the MID performance of the recent text-to-image generative models along with the other major metrics.
4.2 Evaluation on image captioning
Implementation details. For fair comparison with the current tfte--ff-thearr evaluation metric (RefCLIP-S), we use the same pre-trainee trained CLIP (ViT-B/32) used in the prior work to extract image and caption embedding vectors. We the images and the corresponding reference captions to build the covariance matrices Ex, Ey and the joint covariance matrix Σz, For the numerical stability of the inverse of covariance matrix, we replace Ex1 with Σ- (Ex + +II- which handles the near-zert zero eigenvalues of covariance. We found that of 5e-4 generally works across all benchmark evaluations, except for the FOIL benchmark where we used of 1e-15, which was slightly better. Note that use an identical prompt "A photo depicts" for all caption embeddings as employed RefCLIP-S [19].
Flickr8K-Expert and Flickr8k-CF, We measure the correlation of the proposed method with the Likert-scale judgments, which indicate the relative correctness of given captions. Fliker8K- Expert [48] provides 17K human expert judgments for 5,664 images with a four-scale where the higher better. Following prior works, we flatten all human judgments a list of 16,992 (5,664 x3) samples, and we exclude 158 pairs where their captions appear in the reference set. Flickr8K-CE [48] has 145K binary judgments from the CrowdFlower for 48K image-caption pairs. Each pair receives at least three judgments, and we take the proportion of positive as corresponding score. Kendall's T coefficient [42] measures rank correlation, and т. and т, are used for KeaeerKK-xxper and Flickr&K-CTT respectively. Although is more suitable when the underlying scales differ in two variables, we follow the previous works for fair comparison. Tables and show the evaluation results. For both cases, our MID significantly improves the correlation of human judgments with

---

Table 6: Pascal-50S accuracy. Please see the text for the definition of subsets, HC, HI, HM, and MM.
Table 7: FOIL. hallucination pair- wise detection accuracy results.
The methods utilize either one or four references.
Method
HC
HM
MM
Mean
length
51.7
52.3
63.6
49.6
54.3
BLEU-4 [12]
60.4
90.6
84.9
54.7
72.6
Method
1-ref
4-ref
SPICE [16]
63.6
96.3
86.7
68.3
78.7
length
50.2
50.2
METEOR [13]
63.8
97.7
93.7
65.4
80.1
BLEU-4 [12]
66.5
82.6
ROUGE-L [14]
63.7
95.3
92.3
61.2
78.1
METEOR [13]
78.8
85,4
CIDEr [15]
65.1
98.1
90.5
64.8
79.6
BERT-S (RoBERTa-F)
65.4
96.2
93.3
61.4
79.1
ROUGE-L [14]
71.7
79.3
TIGEr [17]
56.0
99.8
92.8
74.2
80.7
CIDEr [15]
82.5
90.6
SPICE [16]
75.5
86.1
ViLBERTScore-F [18]
49.9
99.6
93.1
75.8
79.6
BERT-S++ [32]
65.4
98.1
96.4
60.3
80.1
BERT-S
88.6
92.1
CLIPSS [19]
87.2
87.2
CLIP-S [19]
56.5
99.3
96.4
70.4
80.7
RefCLIP-S [19]
64.5
99.6
95.4
72.8
83.1
ReICLIP-S [19]
91.0
92.6
MID (ours)
90.5
90.5
MID (ours)
67.0
99.7
97.4
76.8
85.2
54.9 and 37.3, respectively. Notably, MID improves further than RefCLIP-S, which uses the same vision-and-land vision-and-language pre-trained CLIP.
Flickr8K reference parsimony. our method does not directly rely on the corresponding references, but through the mean and covariance. Therefore, we could exploit the sample statistics with limited number of references. Figure 4 shows the Kendall's t correlation utilizing a subset of the available references. Interestingly, even though 30-40% of images are avail- able, it retains the majority of performance. The dashed lines indicate the correlations of RefCLIP-S, which exploits all references. Notice that our method can be positioned between the reference-with-inng and reference-free metrics. Because sufficient amount of samples are required to assess image captioning models, the sample statistics can be reliable from the sufficient samples.
Compared with the other methods,
CORRELATION RARELLATION
0.52
Fricsaak ExPInt
0.45
0.38
0.30
0.3
0.9
REFERENCE USAGE RATIO
Figure 4: Flickr8K parsimony.
Pascal-50S. a different evaluation setting of accuracy, the Pascal-50S [15] offers 4K pair-wise preference judgments between two captions, evenly splitting four categories, two human correct captions (HC), both human written, but one incorrect (HI), one is from human, the other is by model (HM), and both are generated by machine (MM). For each pair, there are 48 human judgments and the majority of votes decides which caption is preferred, where ties are broken randomly. As in the previous work [19], randomly sample 5 references among 48 candidates and average over five evaluations. Table shows the consistent results outperforming competitive methods. Except for HI, achieve the oftte-of-the-ar while the margin of HI is 0.1 having pear-perfect score of 99.7.
Object hallucination sensitivity. Rohrbach al. [27] argue that image captioning models prone to generate the objects not presented in the image due to learned bias. To assess this aspect, the FOIL-COCO [40] builds the carefully modified captions from the COCO captions [2] by swapping a single noun-phran e.g.,
0.92
ACCURACY
0.90
substituting "cat" for "dog". To measure the accuracy whether assigning a higher score to the ground-truth caption over the FOIL caption, we evaluate 32K test images with exclusive four reference captions competitive scores of our method. Since RefCLIP-S [19] directly accesses the reference captions of the evaluating image, they can exploit the original words (before foiling) in the references, which roughly 87% four references and 67% in randomly selected reference. The RefCLIP-S is defined as follows:
0.88
PMI (4-azE)
(1-amp)
REFCLIP-S (1-a88)
the COCO dataset. Table shows the
0.7
REFERENCE USAGE RATIO
Figure 5: FOIL parsimony.
RefCLIP-S(x. y. R) H (CLIP-S(xx, y), max(max cos(r, y),r),)),)) 0),
(8)
reR

---

where H is harmonic mean, x and y denotes image and caption embeddings, and R is a set of the references. Figure supports this. Our method robust via covariance estimation in both 1-ref (green line) and 4-ref (blue line), while RefCLIP-SS $ is reducing to its reference- free version of CLIP-S degrading performance. We interpolate the scores of RefCLIP-S and CLIP-S as reference (yellow line). If we similarly utilize the term of max cos(r, and 93.7, outperforming RefCLIP-S (Appendix for the detail, Figure for the visualization).
(rrrr y), 0) as in Equation 8, MID gets 92.4
General discussion
Reliance the pre-trained CLIP models. Do the CLIP-based models take advantage of overfitting to the metric comparing other models? Section 4.1.1, we partly answer that. Notice that multiple versions of the pre-trained CLIP vary the architectural type of visual backbone networks. We found that the normalized scores are generally consistent across generative models and the backbones, especially for our metric (Figure 3). However, it is worth noting that encourage to of the feature extractor (e.g,, CLIP) trained by the loss maximizing mutual information of two modalities (ref. the last paragraph of Section 3.1). Whether two independently-trainn trained encoders perform well is under-explored however, since we define the joint distribution in the proposed metric using the multivariate Gaussian distribution of concatenated features, we expect reasonably works.
Relations to the FID and CLIP-score. The Fréchet inception distance (FID) assesses the perceptual quality of fake images from generative model [5]. It is symmetric distance where two differences between two means and two variances are independently considered and summed. While our method an asymmetric divergence (Equation 7), the distances of means and variances are re-scaled have unit variance as in the Mahalanobis distance (Equation 6), which is the source of domain robustness (Figure Right). Note that the CLIP-S is the cosine similarity using CLIP features. The CLIP features are trained to minimize the InfoNCE loss by increasing mutual information approximately: however, our experiment shows that their approximation significantly degrades the evaluating performance.
Measurement on diversity. Our method does not entirely ignore the diversity by considering the covariance matrices from real and fake samples, similarly to the FID. However, few unique fake samples (e.g., mode collapse) may have small errors in the estimated statistics.
Ethical considerations. Multiple reports show that the CLIP and the generative models based on this pre-trained CLIP have problematic social biases toward racial or ethnic groups [31, 49). Unless appropriate measures toward these biases in the CLIP, our metric is potentially subject to the risk of fairness, accountability, and transparency.
Conclusion
We, to our best knowledge, firstly argue that the negative cross- mutual information with multivariate Gaussian distributions can be used as a unified metric for multimodal generative models. We provide the theoretical analyses of the proposed metric by out-of-distribution detecting by the squared Mahalanobis distances, bias and variance decomposition, and the relation to the Kullback-Leebler divergence, along with the empirical experiments. We achieve the state-of-the-art performances on text-to-image generation and image captioning benchmarks, the generated and human Likert-scale judgment correlations, visual reasoning accuracy, Flickr8K-Expert, Flickr8K-CF, Pascal-5OS, and FOIL hallucination detection. We look forward to seeing future works on the Gaussian cross- moss-mutual information multimodal representation learning based on this novel proposition.
Acknowledgments
We sincerely thank Dongyoon Han for reviewing our manuscript and providing helpful comments. Also, we give thanks to Jung-We Woo Ha for early discussions and suggestions throughout the project. And, we thank anonymous reviewers for their constructive feedback to solidify the manuscript. The NAVER Smart Machine Learning (NSML) platform [50] has been used in the experiments.
10

---

References
[1] Stanislav Frolov, Tobias Hinz, Federico Raue, Jörn Hees, and Andreas Dengel. Adversarial text-to-image synthesis: A review. Neural Networks, 144:187-209, 2021.
[2] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: /504. 00325, 2015.
[3] Hoang Thanh-Tung and Truyen Tran. Toward generalization metrie for deep generative models. In Proceedings of the NeurlPS Workshop on "I Can't 't Believe It's Not Better!", 2020.
[4] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
[5] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pages 6627-6638, 2017.
[6] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 586-595 2018.
[7] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2818-2826, 2016.
[8] Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to-image synthesis. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020.
[9] Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 7986-7994, 2018
[10] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference Machine Learning, pages 8748-8763, 2021.
[11] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[12] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[13] Satanjeev Banerjee and Alon Lavic. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings the ACL workshop intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72,
2005.
[14] Chin-Yeu Lin. Rouge: A package for automatic evaluation of summaries. In Proceedings of the ACL Workshop on Text Summarization Branches Out, pages 74-81, 2004.
[15] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-bard based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566-4575, 2015.
[16) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In Proceedings of the European Conference on Computer Vision, pages 382-398. Springer, 2016.

---

[17] Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang. Zhe Gan, Jana Diesner, and Jianfeng Gao. Tiger: Text-to-image grounding for image caption evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 2141- 2152, 2019.
[18] Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. Vilbertscore: Evaluating image caption using aaiionann-langgage bert. In Proceedings of the 1st Workshop on Evaluation and Comparison of NLP Systems, pages 34-39, 2020.
[19] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: Reference-free Evaluation Metric for Image Captioning. Proceedings of the Empirical Methods in Natural Language Processing, 2021.
[20] Emanuele Bugliarello, Sabrina J Mielke, Antonios Anastasopoulos, Ryan Cotterell, and Naoaki Okazaki. It's easier to translate out of english than into it: Measuring neural translation difficulty by cross-mutual information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1640-1649, 2020.
[21] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assess- ing generative models via precision and recall. Advances in Neural Information Processing Systems, 31, 2018.
[22] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
[23] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. In International Conference Machine Learning, pages 7176-7185. PMLR, 2020.
[24] Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine- aineggrined text image generation with attentional generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1316-13244 2018.
[25] Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang, and Seong Joon Oh. Ecev caption: Correcting false negatives by collecting for ms-coco. In Proceedings of the European Conference on Computer Vision, 2022.
and-human-vvrrifeed image-caption associations
[26] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.0220 027777 2018
[27] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035-4045, 2018.
[28] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 20rds-200-2011 dataset, 2011.
(29] Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 49-58, 2016.
[30] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
[31] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.044053 2022.
[32] Yanzhi Yi, Hangyu Deng, and Jinglu Hu. Improving image captioning evaluation by considering inter references variance. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 985-994, 2020.
12

---

[33] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In Proceedings of the 7th International Conference on Learning Representations, 2019.
[34] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision, pages 201-216, 2018.
[35] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-lane language tasks. Advances in Neural Information Processing Systems, 32, 2019.
[36] Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language-based motion synthesis & editing. arXiv preprint arXiv:2209.00349, 2022.
[37] D Dowson and A Wragg. Maximum-entroop distributions having prescribed first and second moments. IEEE Transactions on Information Theory, 19(5):689-69)) 1973.
[38] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03748 2018
[39] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to-seqqeeee learning framework. arXiv preprint arXiv:2202..0 03052,, 2022.
[40] Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. Foil it! find one mismatch between image and language caption. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 255 2655-2652220 2017.
[41] Yufan Zhou, Ruiyi Zhang. Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxi- ang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. arXiv preprint arXiv:2111.13799 2021.
[42] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
[43] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. arXiv preprint arXiv:2111.14822 2021.
[44] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2265,,26, 2021.
[45] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-imaae synthesis. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5802-5810, 10, 2019.
[46] Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. Learning to evaluate image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5804-5812, 2018.
[47] Hassan Kane, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, and Mohamed Coulibali. Nubia: Neural based interchangeability assessor for text generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pages 28-37, 2020.
[48] Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47: 853-899, 2013.
[49] Robert Wolfe, Mahzarin R. Banaji, and Aylin Caliskan. Evidence for hypodescent in visual semantic In 2022 ACM Conference on Fairness, Accountability, and Transparency, page 1293-1304, 2022.
13

---

[50] Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyun- woo Jo. KyungHyun Kim, Youngil Yang. Youngkwan Kim, al. Nsml: Meet the mlaas platform with real-world case study. arXiv preprint arXiv: 1810.0995,, 2018.
[51] Jin-Hwa Kim, Do-Hyeong Kim, Saehoon Yi, and Taehoon Lee. Semi-orthogonal embedding for efficient unsupervised anomaly segmentation. arXiv preprint arXiv:210S.14377, 2021.
[52] John Duchi. Derivations for linear algebra and optimization. Berkeley, California, 3(1): 2325-5870, 2007.
[53] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Iya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arX1v:2112.10711, 2021.
[54] Ming Tao, Hao Tang, Songsong Wu, Nicu Sehe, Xiao Yuan Jing, Fei Wu, and Bingkun Bao. Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008. 05865, 2020.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Possible numerical issues in Section 3.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Appendix A
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes)
(b) Did you mention the license of the assets? [Yes] See the reference. All assets are available to public.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] The abstract includes the URL the implementation of the proposed metric and the collected human judgments.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] Appendix D
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] Appendix
15


Reviews:
[{"review_id": "xvsw8z5WRyD", "reviewer": "Reviewer_W8Mn", "summary": "This paper presents an automated metric based on mutual information divergence for multimodal generative models. The method exploits gaussian mutual information framework and cross-mutual information. It uses image and text encoders of CLIP to compute the mutual information divergence. The method is theoretically well motivated and presents promising empirical results.  ", "questions": "In section 4.1: there is an assumption being made where it says that fake images are generated by 'foiled' captions are inferior compared to fake images created using regular captions. How true is this assumption? FOILed captions are not necessarily \"wrong\" captions. They can indeed be plausible and hence the generated images can be of better quality. Some validation for this would be useful. ", "limitations": "The paper does not present critical limitations of the metric (such as reliance of CLIP/lack of validity over languages) or issues relating potential dual use of the metric. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The metric is theoretically well motivated and is consistent across datasets and domains. \n2. The proposal in the paper correlates well with human judgement across several datasets. \n3. Thorough comparison to previous work. \n\n\nPotential concerns: \nThe results currently do not support if this metric is absolute - it is not clear how reliant the empirical results are with CLIP. What if there was no CLIP? \nMissing reference: VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions by Madhyastha et al (ACL 2019) \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}, {"review_id": "aJ_QBYMN52C", "reviewer": "Reviewer_WuSm", "summary": "This paper introduces the Mutual Information Divergence (MID) metric for multimodal generative models, where the metric attempts to measure the mutual information between conditions and generations under the Gaussian assumption. Various experiments on text-to-image generation / image captioning, and theoretical analysis are performed to demonstrate the effectiveness and rationality of the proposed metric. It is interesting that Human Likert-scale judgment is employed to validate the superiority of MID.", "questions": "(1)   Are there any analysis on the relation between MID and existing measuring metrics, such as FID, or image-text matching score? \n(2)   Intuitively, MID impose higher alignment between image and text, does this hurt the diversity？", "limitations": "Please see questions and weaknesses.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n(1) This paper introduce the MID metric for multimodal generative models, which is somewhat interesting and brings more reasonable results.\n(2) The paper conducts solid experiments to validate the effectiveness of MID on both text-to-image generation and image captioning tasks, such as the generated and human Liker-scale judgment correlations, visual reasoning accuracy, Flickr8K-Expert, Flickr8K-CF, Pascal-50S, and FOIL hallucination detection. \n\n- Weaknesses\n(1) It lacks necessary analysis on relations between MID and existing metrics listed in Related Work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_oJvIlmFgNt", "reviewer": "Reviewer_Ezue", "summary": "This paper introduced Mutual Information Divergence (MID), a new metric for measuring the quality of image-to-text and text-to-image generation. Similar to model-based metrics like FID, MID relied on a pre-trained backbone (CLIP) to extract features which are used for computing scores. The score is computed by measuring the point-wise MI between to two feature spaces (with Gaussian assumption). Results show that MID can better measure the quality of image-text pairs by comparing against existing metics on augmented dataset.", "questions": "The feature from the transformer-based CLIP model (layer-normed) seems to be contradicting the assumption that both X and Y are Gaussians, am I missing something or MID simply works regardless of the normalization of the input feature.\n", "limitations": "Besides numerical limitation, there are other limitations not discussed in the paper. For example, as a trade-off for leveraging pre-trained backbone model, this metric is might not be applicable to image/text that mismatches the pre-training scenario of the backbone model. (e.g. different language, different shape of image) In other word, this metric is only feasible for high-resource problems.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n+ The metric is simple and straightforward (in a good way) by combining powerful multi-modal backbone that allows cross-modal relation to be considered and MI to be computed rather easily. (my only concern is the Gaussian assumption, see Questions)\n+ The most important contribution is perhaps the fact that this metric can be applied regardless of the input/output modality and the choice of backbone can potentially be flexible as well.\n+ Experiments showed that MID is better than existing metrics and aligns nicely to human evaluation.\n+ The authors considered many different aspects (such as consistency, overfitting backbone, hallucination sensitivity) of designing a new metric and conducted experiment respectively to show that MID is robust.\n\nCons\n- Only few models are considered for evaluation. It would be nice to see the results when applying MID to evaluate more models since metrics are designed to be used to compare models.\n- While the advantage over simple metrics like BLEU is clear, the disadvantage is not discussed throughout in the paper. (see limitations)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QrI68_w7YN", "reviewer": "Reviewer_adUF", "summary": "This submission proposes a new evaluation metric for visual-language generation tasks. Compared to previous metrics, the authors propose to leverage negative cross-mutual information with multivariate Gaussian distributions to calculate mutual information. The authors conduct experiments on both text-to-image generation and image captioning datasets and show the effectiveness of the proposed metrics.  ", "questions": "Overall I am satisfied with the submission. There is one problem with the metric:\n\nThe title focuses on \"Multimodal Generative Models\". However this metric is limited only within visual and textual modalities. As shown in Figure 1 left, to calculate the metrics, it also needs to use CLIP model to extract embedding first. This may limit generalization of the metrics. In the experiments, the authors also focus on text-to-image and image captioning tasks. It would be better to show the usage of the metric on more \"Multimodal Generation\" tasks (e.g., Video, sound, point cloud, etc.) to increase its scope.", "limitations": "Please check the \"Questions\" section above. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The motivation is clear and proposed framework is easy to understand. Besides, the authors also released the source code to reproduce the metric calculation.\n\n+ The technical details are clearly explained with valid experiments' results support.\n\n+ The authors also conduct ablation study to discuss the metrics usage on both text-to-image and image captioning tasks on various datasets, which is much appreciated.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NkN4kZD9ZdF", "reviewer": "Reviewer_wkz2", "summary": "This paper proposes a new metric for evaluating text-to-image and image-to-text models. The metric is mutual information divergence, which uses CLIP features as a source of ground truth. The authors provide theoretical analysis of their metric and demonstrate that it outperforms other metrics on numerous evaluations of real vs. fake images and captions.", "questions": "To what extent does CLIP’s training distribution play a role in the strong performance observed? If the model were scaled to 4 billion images, for example, would MID improve further?", "limitations": "The paper does not have a limitations section. It could use a discussion of the biases in CLIP. See, for example, Wolfe et al. 2022 ACM FAccT; Birhane et al. 2021 arXiv.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\nStrengths: This is a novel area of inquiry and the metric employed achieves better results than any other proposed method.\nWeaknesses: There have been methods that employed CLIP features previously, and the improvement over these is relatively small.\n\nQuality\nStrengths: Outperformance of other methods. Strong theoretical evaluation of the proposed method.\nWeaknesses: No discussion of the potential limitations and biases of CLIP.\n\nClarity\nStrengths: Visualizations are easy to understand. Intro and mathematical descriptions are well presented.\nWeaknesses: I found the paper hard to follow at times, especially as it moves into evaluation and discussions. \n\nSignificance\nStrengths: Outperformance of other methods. Wide variety of environments in which the method might be useful for making sense of the relationship between text and images.\nWeaknesses: Many of the best synthetic generators and image captioners now directly employ the CLIP embedding space to achieve strong results (DALL-E models; GLIDE; VQGAN-CLIP; Antarctic Captions, CLIP prefix-LM and its cousins; etc.). Won’t using the CLIP embedding space as a means of evaluating multimodal representations be a confound whenever these systems that already employ a CLIP model need to be evaluated? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept