Zero-Shot Natural Language Explanations

Authors: Fawaz Sammani, Nikos Deligiannis
Primary area: interpretability and explainable AI
Keywords: Natural Language Explanations, interpretability, explainability

Abstract:
Natural Language Explanations (NLEs) interpret the decision-making process of a given model through textual sentences. Current NLEs suffer from a severe limitation; they are unfaithful to the model’s actual reasoning process, as a separate textual decoder is explicitly trained to generate those explanations using annotated datasets for a specific task, leading them to reflect what annotators desire. In this work, we take the first step towards generating faithful NLEs for any visual classification model without any training data. Our approach models the relationship between class embeddings from the classifier of the vision model and their corresponding class names via a simple MLP which trains in seconds. After training, we can map any new text to the classifier space and measure its association with the visual features. We conduct experiments on 38 vision models, including both CNNs and Transformers. In addition to NLEs, our method offers other advantages such as zero-shot image classification and fine-grained concept discovery.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729449132824

thread_type:
official_review

<thread_object>
creator: reviewer_n3tx

<content>
summary:
The paper proposes a method to use natural language to explain the decisions made by an image classification model. To achieve this, they train a small MLP to map the text features to the feature space of the image classifier. Then, they use a pretrained language model and apply prefix-tuning, fine-tuning a small set of trainable parameters (added to the key-value pairs in the attention module). This enables the language model to generate explanations for a given image sample, using the image and the image classifier to extract its features.

strengths:
- Provides a method for generating explanations for image classifier decisions using natural language.
- Does not require large-scale training data.
- Is inexpensive to train (only a small MLP and small prefixes in the attention mechanism).
- Includes good visualizations.
- Has strong paper motivation.
- Shows good performance on CNNs compared to supervised methods.

weaknesses:
- Some parts of the paper are vague. For example:
  - The section discussing learnable prefixes set as key-value pairs in the attention modules. Could you please add one sentence to this part explaining how this works?
  - The last paragraph discussing zero-shot NLE generation (If you attach each of the $K$ tokens to the previously sampled top-K tokens, wouldn’t there be $K^2$ possible sentences instead of $K$?).
- It requires optimization for each image sample to produce an NLE.
- It requires loss calculation and backpropagation for each image sample to produce an NLE.
- Although the model performs well on unseen classes, it deteriorates performance on seen classes.
- LPIPS is not a suitable evaluation metric for this application because the NLE is not intended to generate captions for images (as T2I models require). Therefore, passing NLEs generated by this model to a T2I model to generate images does not make sense.
- No limitations are mentioned in the main text of the paper.

questions:
- Regarding the last paragraph of section 3.2: If you attach each of the $K$ tokens to the previously sampled top-K tokens, how would there be $K$ possible sentences? Wouldn't it be $K^2$?
- In the first paragraph of section 4.1:
  - What is the number of classes in $\hat{W}$? (What is the dimension of $\hat{W}$?)
  - When training the MLP, which classes did you use? Only the 900 classes, or all of them?
- As a metric to evaluate your method, the cosine similarity between the text embeddings of the predicted explanation and the ground truth explanation (from ImageNet-X) using the BERT model is interesting and missing. Have you tried this?
- Do you have an intuition about why your model performs worse than other supervised models on ViT models (even though it performs better on CNNs)? Do you have any experimentations to support your intuition?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729511288256

thread_type:
official_review

<thread_object>
creator: reviewer_y6z6

<content>
summary:
This paper presents a zero-shot natural language explanation approach. 
The proposed method involves two sequential steps. 
For the first step, the authors propose to align the text encoder with that of the classifier from a visual encoder.
Subsequently, an LLM is then equipped with a prefix that is trained by the pseudo label learned from the former stage.
The experiments are primarily focused on the NLE task. 
When compared with several baselines, the proposed method achieves improved performance on diverse vision backbones.

strengths:
- The studied problem, zero-shot natural language explanation, is quite practical and useful for many researchers.
- The writing of this paper is good, and most parts of it are easy to follow.
- The authors utilize a variety of vision encoders to validate the generalization capability of the proposed method.

weaknesses:
- My biggest concern is why we simply remove the first stage and use the CLIP model for the alignment.
To me, the authors try to align the output from a text encoder with that of a vision encoder classifier. 
This process is utilized to offer a pseudo label for the subsequent LLM prefix-tuning.
I feel like we can directly use a pre-trained CLIP as a replacement which can generalize well to open settings.
- Following the first concern, how did the authors address out-of-distribution classes? 
It seems the classifiers are trained on 1,000 ImageNet classes.
What if there are more fine-grained classes?
- It is controversial to call this method a ``classifier-agnostic`` method. 
  - The authors did use a classifier in the first training stage.
  - The MLE is evaluated on the ImageNet domain.
- There is only 1 baseline compared in Table 1.
Additionally, the improvement over this method is not significant.

questions:
One more question other than the aforementioned weaknesses: 

How about the model performance using a tiny vision large language model (VLLM)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729898602080

thread_type:
official_review

<thread_object>
creator: reviewer_9tnt

<content>
summary:
The paper proposes a new method for zero-shot natural language explanation (NLE) for classification. The method trains an MLP that maps the encoding space of an LLM text encoder to the space of class weights of arbitrary classifiers. Consequently, the method works on arbitrary classifiers. It produces NLE by iteratively adapting a prefix to the image classifier’s feature representation, and the prefix serves as a condition for generating text descriptions. In this way, the generated text is faithful, in contrast to the case of models trained on human description data specifically. It achieves the state-of-the-art performance on zero-shot NLE, and is also on par to the SOTA methods in zero-shot classification and concept discovery.

strengths:
1. The method proposed by the paper is novel, taking a completely different perspective from that of the previous works in the field of NLE. 
2. The method is powerful, in that it achieves the state-of-the-art performance on zero-shot NLE, and is also on par to the SOTA methods in zero-shot classification and concept discovery. 
3. The method also demonstrates and utilises the possibility of matching the representation space of LLM to arbitrary vision classifier (or any models whose outputs are texts), which gives deep insights into this mechanism and can be inspiring to other fields as well.
4. Overall, the presentation of the paper is very clear.

weaknesses:
1. Section 3.2 (as well as Section F) is somewhat confusing to me. From my understanding, the generation of NLE is the following process: 1. there are K sentences starting with some initial prompts like “this is a”; 2. a random prefix is present to generate top-K tokens; 3. the top-K tokens are concatenated to the K sentences to get K continued sentences; 4. the representation of these sentences are compared to the image representation, and the prefix is updated to reflect the similarities, so that next step it can generate better tokens that make the sentences aligned to the image; 5. the whole process continues so that the sentences become longer. I still have the following doubts (I will update my score and confidence once I understand them better; I also think Figure 4 should be more clear on this whole process):
    1. At the first time step, the sentences will become “this is a [object]”. How to make sure the top-K tokens actually contain the correct object, so that the continuation does not follow a completely wrong object?
    2. How is it possible for the generated sentence to automatically have “because” in it, so that the NLE is generated to be “this is a [object], because…”
    3. In Section F, I am a bit confused between “iterations” and “time steps”
    4. In Section F, I am not sure why the CLIP-score is used to choose the sentences. Shouldn’t it be that the sentence is chosen based on its similarity to the image representation? And is it fair to use CLIP-score here when comparing to other methods?
2. In the NLE comparison results, it seems that with the “cosine” metric the proposed model seems to be systematically inferior than the best models. The author might want to analyse the cause and give one explanation.

questions:
1. The author might consider improving the presentation in section 3.2 by making the process more clear.
2. The author might want to analyse the cause and give one explanation for the inferior "cosine" metric in NLE results.
3. I wonder if this method can be applied to directly improve the robustness of image classifiers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730095553377

thread_type:
official_review

<thread_object>
creator: reviewer_zjsf

<content>
summary:
The paper primarily explores generative natural language explanations for vision models. Alternatively, the paper considers learning a simple MLP to retrieve text most similar to the visual embedding of a given vision model. They claim to be the first work to this in a zero-shot manner. The authors conduct extensive ablations across various models and demonstrate strong results.

strengths:
1. The proposed method is simple and intuitive, I liked the motivation behind the work. 
2. The paper is well written and easy to follow, this makes it easy to understand what the authors have done and the current State of The Art.
3. The authors experiment on a wide variety of models. 
4. The authors are able to extend "Visual Classification via Description from Large Language Models", in a practical manner to a wide variety of models.

weaknesses:
1. Based on Table 1, the simple method seems to work less well for bigger models such as ViT and DEIT in the generalized zero-shot setting. There is a lack of discussion of the results, a little discussion about Table 1 would be helpful.
2. Since this is a zero-shot method, it would have been cool to see how does it work on other popular yet specialized datasets like Flowers / Aircraft / CUB etc. Although ImageNet results are good having more datasets would have given more insights.

questions:
1. Did the authors ablate the prompt template? CLIP can be sensitive to prompt templates, was a similar trend seen for the learnt MLP?
2. Were there any biases in the generated explanations? Like bias towards color or specific features?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes an approach for Natural Language Explanations (NLEs) for vision models. The paper received positive reviews from all 4 reviewers. The reviewers like the new perspective on faithfulness, the motivation behind this work, the simplicity and soundness of the proposed approach, and the comprehensiveness of the experiments. Thus, I recommend acceptance. The authors are strongly encouraged to incorporate reviewers' suggestions, especially on expanding the discussion of limitations (fine-grained/OOD classes) and design decisions of the proposed model.