Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an automated metric based on mutual information divergence for multimodal generative models. The method exploits gaussian mutual information framework and cross-mutual information. It uses image and text encoders of CLIP to compute the mutual information divergence. The method is theoretically well motivated and presents promising empirical results.  
strengths_and_weaknesses: Strengths: 
1. The metric is theoretically well motivated and is consistent across datasets and domains. 
2. The proposal in the paper correlates well with human judgement across several datasets. 
3. Thorough comparison to previous work. 


Potential concerns: 
The results currently do not support if this metric is absolute - it is not clear how reliant the empirical results are with CLIP. What if there was no CLIP? 
Missing reference: VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions by Madhyastha et al (ACL 2019) 

questions: In section 4.1: there is an assumption being made where it says that fake images are generated by 'foiled' captions are inferior compared to fake images created using regular captions. How true is this assumption? FOILed captions are not necessarily "wrong" captions. They can indeed be plausible and hence the generated images can be of better quality. Some validation for this would be useful. 
limitations: The paper does not present critical limitations of the metric (such as reliance of CLIP/lack of validity over languages) or issues relating potential dual use of the metric.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduced Mutual Information Divergence (MID), a new metric for measuring the quality of image-to-text and text-to-image generation. Similar to model-based metrics like FID, MID relied on a pre-trained backbone (CLIP) to extract features which are used for computing scores. The score is computed by measuring the point-wise MI between to two feature spaces (with Gaussian assumption). Results show that MID can better measure the quality of image-text pairs by comparing against existing metics on augmented dataset.
strengths_and_weaknesses: Pros
+ The metric is simple and straightforward (in a good way) by combining powerful multi-modal backbone that allows cross-modal relation to be considered and MI to be computed rather easily. (my only concern is the Gaussian assumption, see Questions)
+ The most important contribution is perhaps the fact that this metric can be applied regardless of the input/output modality and the choice of backbone can potentially be flexible as well.
+ Experiments showed that MID is better than existing metrics and aligns nicely to human evaluation.
+ The authors considered many different aspects (such as consistency, overfitting backbone, hallucination sensitivity) of designing a new metric and conducted experiment respectively to show that MID is robust.

Cons
- Only few models are considered for evaluation. It would be nice to see the results when applying MID to evaluate more models since metrics are designed to be used to compare models.
- While the advantage over simple metrics like BLEU is clear, the disadvantage is not discussed throughout in the paper. (see limitations)
questions: The feature from the transformer-based CLIP model (layer-normed) seems to be contradicting the assumption that both X and Y are Gaussians, am I missing something or MID simply works regardless of the normalization of the input feature.

limitations: Besides numerical limitation, there are other limitations not discussed in the paper. For example, as a trade-off for leveraging pre-trained backbone model, this metric is might not be applicable to image/text that mismatches the pre-training scenario of the backbone model. (e.g. different language, different shape of image) In other word, this metric is only feasible for high-resource problems.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This submission proposes a new evaluation metric for visual-language generation tasks. Compared to previous metrics, the authors propose to leverage negative cross-mutual information with multivariate Gaussian distributions to calculate mutual information. The authors conduct experiments on both text-to-image generation and image captioning datasets and show the effectiveness of the proposed metrics.  
strengths_and_weaknesses: + The motivation is clear and proposed framework is easy to understand. Besides, the authors also released the source code to reproduce the metric calculation.

+ The technical details are clearly explained with valid experiments' results support.

+ The authors also conduct ablation study to discuss the metrics usage on both text-to-image and image captioning tasks on various datasets, which is much appreciated.  
questions: Overall I am satisfied with the submission. There is one problem with the metric:

The title focuses on "Multimodal Generative Models". However this metric is limited only within visual and textual modalities. As shown in Figure 1 left, to calculate the metrics, it also needs to use CLIP model to extract embedding first. This may limit generalization of the metrics. In the experiments, the authors also focus on text-to-image and image captioning tasks. It would be better to show the usage of the metric on more "Multimodal Generation" tasks (e.g., Video, sound, point cloud, etc.) to increase its scope.
limitations: Please check the "Questions" section above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces the Mutual Information Divergence (MID) metric for multimodal generative models, where the metric attempts to measure the mutual information between conditions and generations under the Gaussian assumption. Various experiments on text-to-image generation / image captioning, and theoretical analysis are performed to demonstrate the effectiveness and rationality of the proposed metric. It is interesting that Human Likert-scale judgment is employed to validate the superiority of MID.
strengths_and_weaknesses: - Strengths
(1) This paper introduce the MID metric for multimodal generative models, which is somewhat interesting and brings more reasonable results.
(2) The paper conducts solid experiments to validate the effectiveness of MID on both text-to-image generation and image captioning tasks, such as the generated and human Liker-scale judgment correlations, visual reasoning accuracy, Flickr8K-Expert, Flickr8K-CF, Pascal-50S, and FOIL hallucination detection. 

- Weaknesses
(1) It lacks necessary analysis on relations between MID and existing metrics listed in Related Work.
questions: (1)   Are there any analysis on the relation between MID and existing measuring metrics, such as FID, or image-text matching score? 
(2)   Intuitively, MID impose higher alignment between image and text, does this hurt the diversity？
limitations: Please see questions and weaknesses.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new metric for evaluating text-to-image and image-to-text models. The metric is mutual information divergence, which uses CLIP features as a source of ground truth. The authors provide theoretical analysis of their metric and demonstrate that it outperforms other metrics on numerous evaluations of real vs. fake images and captions.
strengths_and_weaknesses: Originality
Strengths: This is a novel area of inquiry and the metric employed achieves better results than any other proposed method.
Weaknesses: There have been methods that employed CLIP features previously, and the improvement over these is relatively small.

Quality
Strengths: Outperformance of other methods. Strong theoretical evaluation of the proposed method.
Weaknesses: No discussion of the potential limitations and biases of CLIP.

Clarity
Strengths: Visualizations are easy to understand. Intro and mathematical descriptions are well presented.
Weaknesses: I found the paper hard to follow at times, especially as it moves into evaluation and discussions. 

Significance
Strengths: Outperformance of other methods. Wide variety of environments in which the method might be useful for making sense of the relationship between text and images.
Weaknesses: Many of the best synthetic generators and image captioners now directly employ the CLIP embedding space to achieve strong results (DALL-E models; GLIDE; VQGAN-CLIP; Antarctic Captions, CLIP prefix-LM and its cousins; etc.). Won’t using the CLIP embedding space as a means of evaluating multimodal representations be a confound whenever these systems that already employ a CLIP model need to be evaluated? 
questions: To what extent does CLIP’s training distribution play a role in the strong performance observed? If the model were scaled to 4 billion images, for example, would MID improve further?
limitations: The paper does not have a limitations section. It could use a discussion of the biases in CLIP. See, for example, Wolfe et al. 2022 ACM FAccT; Birhane et al. 2021 arXiv.
review_title: null

Meta-review: metareview: The paper studies the evaluation metric for multimodal generation models. The authors propose a method MID based on estimating mutual information of visual and text embeddings at sample and distribution level. From experiments, the MID correlates with human evaluation on multiple tasks (text-to-image and image captioning). The authors provide theoretical intuition and analysis of MID and relation to other divergence scores. Experiments are solid and convincing. The reliance on CLIP is discussed though other multimodal encoders than CLIP are not evaluated in experiments. Author discussion with reviewers are helpful to better understand the paper. 

Overall, it is a solid paper with a clearly described, simple, and effective method. 
confidence: Certain
Decision: Accept