Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The author proposes a new metric for evaluating the multi-mode image synthesis method, and has tested new metrics. However, method details are limited. In addition, the author's contribution point 1,3 is difficult to capture in the introduction. Besides, the papers investigated are old.
strength_and_weaknesses: +The author proposes a new metric for evaluating multi-mode image synthesis methods
+The author used a large number of experiments to verify

-Recently, there are many meaningful works for multi-mode image synthesis. I don't know why the author has not conducted more detailed research in the Introduction section. Instead, follow the work of 2020 and 2021.

Dalmaz O, Yurt M, Çukur T. ResViT: residual vision transformers for multimodal medical image synthesis[J]. IEEE Transactions on Medical Imaging, 2022, 41(10): 2598-2614.
Isaac-Medina B K S, Bhowmik N, Willcocks C G, et al. Cross-Modal Image Synthesis Within Dual-Energy X-Ray Security Imagery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 333-341.
Wu G, Chen X, Shi Z, et al. Convolutional neural network with coarse-to-fine resolution fusion and residual learning structures for cross-modality image synthesis[J]. Biomedical Signal Processing and Control, 2022, 71: 103199.
Lv Z, Li X, Niu Z, et al. Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 11214-11223.
Tan Z, Chu Q, Chai M, et al. Semantic Probability Distribution Modeling for Diverse Semantic Image Synthesis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

-In denying that Measure is an important metric, as Section 7 of [1], I hope that the author can further compare the advantages of the metric and have more proof of the metrics.

[1] Zhou R, Jiang C, Xu Q. A survey on generative adversarial network-based text-to-image synthesis[J]. Neurocomputing, 2021, 451: 316-336.

-In the introduce section, the author introduces MIS-Score, but it is difficult capture the first contribution point and the third contribution point by reading this section.

-The introduction of the method is limited, and it is difficult to capture the details of the method. The author's description of the implementation framework is incomplete and it is difficult to understand the proposed framework.

clarity,_quality,_novelty_and_reproducibility: The introduction of the method is limited, and it is difficult to capture the details of the method.
summary_of_the_review: The research topic is worth recommending, but the author's description of the method needs to be strengthened.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new metric to evaluate the image quality for multi-modal image synthesis task,  which relies on a parser to find different semantic concepts and a detector to find the corresponding objects and attributes in the image. Besides, a robustness testing technique is proposed to evaluate the robustness of a generative model.
strength_and_weaknesses: Strength:

1. The paper is well written and easy to follow.

2. Authors further consider the fine-grained semantic consistency between the given text and synthetic image to completely evaluate the correlation between them.

Weaknesses:
1. Basically, the proposed metric mainly focuses on measuring the semantic consistency between the given text and the generated image, which is different from IS and FID. So some statements shown in introduction, e.g., "These metrics work well for the generation from simple prompts, e.g., description of a single object. However, for prompts with multiple objects and additional context information, simply adopting these metrics is insufficient and may lead to inaccurate or inconsistent results. " might not be completely accurate. Also, the R-precision and CLIP score can also be adopted to measure the correlation between multiple object by adopting a similar detector used in the propose method.

2. Although the proposed MIS-Score further considers the fine-grained correlation between the text and image, it mainly depends on a parser to find semantic concepts and a detector, so its performance might be affected by both components. 

3. For equation 2, could authors give more details to explain why it can be used to measure the correlation?
clarity,_quality,_novelty_and_reproducibility: The novelty of proposed metric might be limited, but authors provide sufficient information to reproduce the proposed method.
summary_of_the_review: Please see above weaknesses, and I am happy to change my rating based on authors' responses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 1, Present a new method to evaluate the cross-modal consistency by inspecting the decomposed semantic concepts.
2, Introduce a new metric, called MIS-Score, which is designed to measure the fine-grained semantic alignment between a prompt and its generation quantitatively.
3, Developed an automated robustness testing technique with referential transforms to test and measure the robustness of multi-modal synthesis models.
strength_and_weaknesses: Strength:
1, The first time to propose a new approach for measuring the fine-grained semantic consistency for the multi-modal image synthesis tasks.
2, A robustness testing technique is designed and implemented to evaluate the robustness of text-to-image generation models.
3, Comparisions between different methods are interesting.
Weaknesses:
1, How to calculate weight wi in eq3?
2, Can you calculate your metric directly on the benchmark dataset to check the annotation quality of the caption?
3, Why Stable Diffusion is not the best model here?  It seems that they use more data.
4, Do you still need a pretrained CLIP to calculate your score?
5, How can you locate the object region?
clarity,_quality,_novelty_and_reproducibility: Clarity, Quality, Novelty, and Reproducibility are OK.
summary_of_the_review: A new metric is designed for measuring the fine-grained semantic consistency for the multi-modal image synthesis tasks. The measurement for multi-modal image synthesis tasks are important.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new metric to evaluate image quality in multi-modal image synthesis (i.e. text-to-image synthesis). It is specifically interested in assessing how well the generated imagery respects the input text prompt, and it claims to do so at a finer level of granularity than previously-available metrics.

Strengths: reviewers found the paper well-written and easy to follow. They also appreciated the extensive experimental evaluation.

Weaknesses: all reviewers expressed a concern about a lack of novelty. There already exist methods for assessing consistency between text prompts and generated images (e.g. CLIP score, R-precision), and the reviewers were not convinced that the method proposed here offered anything useful beyond those.

Overall, reviewers were not in favor of accepting the paper, given this weakness.

The authors wrote very thorough responses to each of the reviewers, but none of the reviewers responded during the OpenReview discussion period. Thus, I decided that an in-person meeting was warranted.
justification_for_why_not_higher_score: Reviewers raised a number of concerns related to novelty, evaluation, and lack of implementation details. These concerns reasonably persist even after accounting for the authors' response.
Decision: Reject