Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an approach to automatically discover the failure cases of vision models under real-world settings. Off-the-shelf image-to-text and text-to-image models are leveraged to find such failures. Firstly, a conditional text-to-image synthesis model generates synthetic data based on the ground-truth label. Then, a captioning model is used to describe misclassified inputs. Next, the descriptions generated by the captioning model are used to synthesize more images to test whether the specific description causes a higher failure rate. This pipeline can be used to find the failure modes with language explanations for the classification models.
strength_and_weaknesses: Strengths:
1. Incorporating large-scale text-to-image and image-to-text models for diagnosing classification models is an interesting topic. The authors provide some insightful explorations.
2. The proposed method can be used to diagnose image classification models and provide language explanations for the failure modes.

Weaknesses:
The major limitation is that the proposed approach assumes that the text-to-image and image-to-text models are perfect. However, if error happens in text-to-image synthesis or image captioning synthesis, then the results will not be reliable. For example, what if the text-to-image synthesis model generates an image of a bee with the caption "a realistic photograph of a fly"? The authors only discussed this issue in one paragraph in limitations, but I think this is an important issue that should be investigated deeper. How frequently do the text-to-image synthesis and image captioning models make mistakes, and how these mistakes will affect the results? How robust is the proposed approach?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The topic of incorporating large-scale text-to-image and image-to-text models for finding the failure cases of image classification models is novel. The authors provide details of the method for reproducing the results.

summary_of_the_review: My major concern is the robustness of the approach. If the text-to-image models and the image captioning models make mistakes, the results will be misleading. This approach assumes that the text-to-image and image captioning models are perfect, or at least the error rate is much smaller than the image classification model. But I suspect this assumption is not always true.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes a pipeline that can be used to identify failures of image classifiers. Proposed approach utilizes SOTA generative models to enable interpretable failure discovery of a model by providing targeted realistic images. Experiments are well conducted to demonstrate the benefit of the approach. Discussions on its limitations and potential concerns are well considered in the paper as well.
strength_and_weaknesses: Strength

* Formulated the problem of finding failures of image classifier in a probabilistic framework and showed how one can use off-the-shelf image-to-text captioning model and text-to-image generative model to create an interpretable and easy-to-use debugging tool.
* Proposed approach is more interpretable and naturalistic than existing approaches and shows promise in generalizability.


Weakness / other comments

* “In particular, we use three shots, with each shot being an image and caption pair. Images are publicly available online and are manually described using between four to seven short sentences that describe the subject of the photograph and its physical position with respect to the camera, as well as the background or context in which the subject appears” → This indicates that the method requires human input with some expert knowledge which raises a question on scalability and potential bias for this process.
* How can identified failures help improving the actual model? Further discussion / analysis on how the discovered failures could be used would be useful in order to give a better idea to the practitioners who would want to use this system to improve their model.
* “For each label y, we manually select target labels  ŷ (snow leopard, bee and chainlink fence respectively)” → Why was this set of ŷ selected? What is the impact of the choice ŷ on failure discovery?


clarity,_quality,_novelty_and_reproducibility: It is well written and clear enough for someone unfamiliar with the topic to easily follow. The method is technically sound for the most part. This work uses existing SOTA models but the way they combined them as a new debugging tool is fairly novel. 
summary_of_the_review: This paper clearly describes their proposed method which is well designed and backed up with proper analysis and experiments. Although I have some doubts of the claimed scalability and how practical it would be if we were to use them for model improvement, as a proof-of-concept I think it falls above the acceptance bar.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes utilizing off-the-shelf text-to-image and image-to-text models to discover vision models' failures automatically. The experimental results are delighting and convincing to some extent. This paper is also inspiring and potentially useful to improve the generalizability of vision models.
strength_and_weaknesses: Overall, the idea of exploiting off-the-shelf generative and captioning models to discover failures of image classifiers trained in ImageNet is interesting and novel. I am also impressed by the extensive experimental results (including visualization analysis and variants effect analysis). 

Nevertheless, I have several small issues:
- Although adequate experiments are conducted on ResNet-50 on ImageNet, I am curious to see experiments on a stronger network backbone such as ResNet-152. This lies in the concern that a stronger network may not be easily found failures using the proposed method. As a matter of fact, I’m mostly convinced by the provided results. But I think this experiments will make the conclusions more solid.
- Do the discovered failures can further improve the object classification model’s performance? 

clarity,_quality,_novelty_and_reproducibility: The paper is well organized, technically sound and novel.
summary_of_the_review: Currently, I lean to positive for its novel idea.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper contributes a method to automatically find failure cases of image classifiers in an  oepn-ended manner by leverageing off-the-shelf, large-scale generative models. 
strength_and_weaknesses: Strength: Automatically discovering failures is an interesting problem; The idea of applying image-to-text generative models to discover failures and obtain human-interpretable explanation of failure is novel.

Weaknesses:
The failure cases in this paper are all synthetic and there is still a large margin gap remaining between real-world data and data generated by generative models; Though a lot of well-trained generative models can be applied in the method, the costs of training them should not be ignored; The limitations mentioned in the paper are difficult to solve and will affect the effectiveness of the work. Due to the randomness in the process of selecting cases via generative models, it is difficult to prove that this framework can be reproduced and effective.
clarity,_quality,_novelty_and_reproducibility: Clarity Quality Novelty and Reproducibility
Clarity and quality are good while novelty and reproducibility are not good enough.
summary_of_the_review: This paper proposes a method to automatically find failure cases of image classifiers. The applying of generative models is novel, but the limitations mentioned in the paper are difficult to solve and will affect the effectiveness of the work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an approach to automatically discover the failure cases of vision models via using off-the-shelf image-to-text and text-to-image models. 

Initially, this paper received scores of 3566; after rebuttal, the scores have changed to 3366. On one hand, all the reviewers agree that the studied problem is interesting, and the proposed method is novel. On the other hand, the major concern shared by the two reviewers who gave a score of 3 is that the proposed approach kind of assumes that the text-to-image and image-to-text models are perfect (or, performs very well), leaving the robustness of the approach unclear and the results might be misleading. For this concern, the authors have provided detailed response; however, it is unfortunately not convincing enough for reviewers to increase the scores. 

On balance, the AC decides to recommend rejection based on the current draft, and encourages the authors to further refine the paper and submit the paper to a future conference. 
justification_for_why_not_higher_score: Two of the reviewers had major concerns about the robustness of the approach, as the proposed approach kind of assumes that the text-to-image and image-to-text models are perfect (or, performs very well). After rebuttal, one reviewer even downgraded the score to 3.
Decision: Reject