Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales
Authors: Tang Li
Mengmeng Ma
Xi Peng
Venue: NeurIPS 2024 poster (2024)
Keywords: Explainable Machine Learning
Vision-Language Models
Primary area: interpretability_and_explainability

Abstract: Large pretrained foundation models demonstrate exceptional performance and, in some high-stakes applications, even surpass human experts. However, most of these models are currently evaluated primarily on prediction accuracy, overlooking the validity of the rationales behind their accurate predictions. For the safe deployment of foundation models, there is a pressing need to ensure *double-correct predictions*, *i.e.*, correct prediction backed by correct rationales. To achieve this, we propose a two-phase scheme: First, we curate a new dataset that offers structured rationales for visual recognition tasks. Second, we propose a rationale-informed optimization method to guide the model in disentangling and localizing visual evidence for each rationale, without requiring manual annotations. Extensive experiments and ablation studies demonstrate that our model outperforms state-of-the-art models by up to 10.1\% in prediction accuracy across a wide range of tasks. Furthermore, our method significantly improves the model's rationale correctness, improving localization by 7.5\% and disentanglement by 36.5\%. Our dataset, source code, and pretrained weights: https://github.com/deep-real/DCP

Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales
Tang Li
Mengmeng Ma
Xi Peng
DeepREAL Lab: https://deep-reat aetaa..gghub...... io Department of Computer & Information Science, University of Delaware {tangli, mengma, xipeng) (oonnngdudeeeeee
Abstract
Large pretrained foundation models demonstrate exceptional performance and, in some high-stakes applications, even surpass human experts. However, most of these models currently evaluated primarily on prediction accuracy, overlooking the validity of the rationales behind their accurate predictions. For the safe deployment of foundation models. there pressing need to ensure double-correct predictions, i.e. correct prediction backed by correct rationales. To achieve this, we propose a two-phase scheme: First, we curate new dataset that offers structured rationales for visual recognition tasks. Second, we propose a rationale-infomme method to guide the model disentangling and localizing visual evidence for each rationale, without requiring manual annotations. Extensive experiments and ablation studies demonstrate that our model outperforms state-of-the-ar models by up to 10.1% in prediction accuracy across wide range of tasks. Furthermore, our method significantly improves the model's rationale correctness, improving localization by 7.5% and disentanglement by 36.5%. Our dataset, source code, and pretrained weights:
optimization
Introduction
Large foundation models, such as CLIP and GPT- 4V (2), exhibit exceptional performance pass human experts in some high-stakes applications, such medical diagnosis 3 and autonomous driv- ing [4] (3 However, most of these models are cur- rently evaluated primarily prediction accuracy, overlooking critical aspect for ensuring safety, i.e., the validity of the reasons behind their accurate pre- dictions. Understanding the rationales the "how" and "why" behind model predictions is crucial for developing safe predictions. Fig. [ shows typical examples of unsafe predictions: CLIP might predict accurately yet based on wrong rationales, whereas GPT-4V might make wrong predictions based on ra- tionales that are plausible to humans. To build trust in real-world deployment, natural question arises: Can models make predictions backed by correct rationales?
troffic image?
Prediction
Explain
CUP
Rationale
Inout Image
User the door the truck cab open?
GPT-4V
Prediction
User: Explain why..
GPT-4V: The line of the Boor allons
eaB showing to gans
open. that would
Image
Rationale
Figure Unsafe prediction examples. Cor- rect prediction, incorrect rationale: CLIP identifies a red light, but wrongly based on red balloons. Incorrect prediction, correct tionale: GPT-4V incorrectly predicts a closed door, yet based on plausible visual evidence.
correct predictions, i.e., correct
Correct rationales generally align with how humans would reason about the same decision and based
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

There are existing attempts to provide rationales for machine learning models' predictions. They either explicitly force the models to make decisions based on human-understandable concepts by introducing bottleneck layers 9 I0], or implicitly inject commonsense knowledge into models by contrastive learning between similar yet distinet textual concepts [ 12]. However, none of them ensures double-correct predictions. Observations from our previous research [3] and recent studies in the field [14] reveal that these models might provide incorrect rationales, as they fail to base the rationales on valid visual evidence.
on valid visual evidence 7 8)
To this end, we develop double-corren predictions by focusing on two foundational aspects:
i) "What" are the correct rationales? Structured rationale acquisition. Existing vision datasets typically provide ground truth labels of predictions, whereas missing the rationales behind these decisions [16 17 To fill this gap, we curate a new dataset that offers over 4,000 unique textual rationales designed for predicting the 1,000 categories in ImageNet [18), structured in tree format. This design differs from existing knowledge graphs [19] 20 21,, which either provide irrelevant knowledge for the vision task or are too coarse-grained, providing insufficient information. Our rationale dataset is tailored to capture the detailed reasoning processes for visual recognition.
ii) "Where" are the correct rationales? Rationale-informed optimization. The other challenge in developing double-correct predictions is the absence of pixel-wise annotations for rationales' visual evidence. Although some datasets provide segmentation masks of object parts [17] (2), they lack sufficient rationale coverage and are limited to small-scale use cases [23]. To address this issue, we propose a rationale-informel optimization method to guide the model in disentangling and localizing the visual evidence of rationales, without requiring manual annotations. Our method can be integrated into the existing model training process without architectural changes and extra parameters.
We evaluate the proposed method on a wide range of benchmark datasets and tasks. For prediction correctness, our model outperforms state-of-the-ar models in zero-shot, linear probe, and fine-tuning settings by 2.6%, 2.0%, and 10.1%. %. For rationale correctness, the empirical results exhibit that our model significantly improves ground truth rationale localization and rationale disentanglability by 7.5% and 36.5%. Furthermore, the extensive qualitative results and ablation studies demonstrate the effectiveness of the proposed method.
Our contribution includes: 1) We curate a new structured rationale dataset. 2) faithful explanation method tailored for explaining CLIP-ViT predictions. 3) A principled optimization method that seamlessly integrates structured rationale information to develop double-correct predictions. 4) Empirical results in a wide range of benchmark datasets and tasks including image classification and retrieval demonstrate the superior prediction and rationale correctness our model.
2 Problem Formulation
In this section, we first formally define rationales, then provide the mathematical formulation of the double-correct prediction problem.
Definition (Rationales) Given a category y. rationales are a set of K underlying abstract notions {rk)== and relations that capture the reasoning process leading to the recognition of
For example, when recognizing a specific breed of dog in an image, the rationales could be a set of concepts such as the shape of the ears, the color of the fur, and the size of the dog. Mathematically. given textual rationale r, we assume the existence of a ground truth labeling function V(x, r) that can provide the pixel-wise annotations of visual evidence corresponding
In the real world, rationales be represented through textual descriptions 24 25
on an input x.
Definition (Double-Corre Correct Predictions) A correct prediction is double-correct when backed by correct rationales that are based on valid visual evidence.
Denote (z, y) ~ P(X, Y) as a data point sampled from the training distribution YY.,Y). g() as an explanation method that attributes the prediction of text r to group of pixels in input x depending on model f. f(-) as the task-specific loss function, and F as function class that model- for the prediction task. To ensure the model f makes double-corred prediction, we propose to solve the following constrained optimization problem:
min R(f)= == ((cp)))rxx))(((())))) fe
s.t. g(I,r:f) = V(z,r), E (r
(I)
2

---

The problem in Eq. is challenging to solve, since we neither have access to the rationales {rk) K nor to the ground truth labeling functions V()). There are existing attempts that employ domain experts to manually collect textual descriptions of rationales [22] 26], or pixel-wise annotations of
object parts on the image [[7]. However, these approaches are often limited to small-scale datasets, and impractical in large-scale settings due to the high cost of fine-grained annotations [27 23].
Double-Correct Predictions
To bridge the gaps, in Sec. 3.1 we present how acquire rationales 1- in Sec. 3.2 we propose a new explanation method g-)), and in Sec. 3.3 we develop double-correct predictions without (-)).
3.1 Structured Rationale Dataset
In this section, we curate a new rationale dataset to offer {r-)== in Eq. [ According to Def. ratio- nales are structured human knowledge. Therefore, ontologies that encapsulate complex, interconnected information while maintaining semantic relationships between entities [28] 29], present a proper tool to rep- resent rationales. The benefits are bi-directional: i) in the human-to-machhe to-machine direction, it offers a standard- ized, machine-readabl format; ii) in the machine-to- human direction, ontology structure mirroring how humans organize and retrieve information explain the model's s decision-mmking process.
American Robin
Arriner
Wombat
Ceres on Bilr))
Winzee For
(Braa)) Hors
Window
T
Buly
(ang
PE (Lane
Shorl
(ars)
Figure 2: Our structured rationales capture the major attributes and their sub-attributes that lead to the recognition of objects. Our dataset offers over 4,000 unique rationales covering all 1, 1,000 categories from ImageNet [18]
Different from existing works that are limited to small-scale manual annotation, we generate our rationale dataset in a scalable manner. Specifically, we utilize Large Language Models (LLMs) like GPT-4 2
Acquire structured rationales:
to extract the structured rationales. Existing studies prove that GPT-4 has expert-level expertise in commonsense 30 and domain knowledge [31]. However, find that directly querying LLMs would yield inconsistent tree structures that hardly be used by machine learning models. To address this issue, we provide series of exemplary structured rationales before the query, employing in-context learning [32) to extract standardized rationales in JSON format. See Appendix A for our full prompt and rationale examples.
Our dataset covers all 1,000 categories in the ImageNet 18] For each category, we generate an ontology tree with a maximum height of two. As illustrated in Fig. 2 the root node is the category, the children of the root are the attributes, and the leaves are the sub-attributes. The edges represent the relationships between nodes. Combining attributes and sub-attributes, our dataset contains over 4,000 unique rationales. Our rationale ontology trees capture the reasoning processes leading to the recognition of the corresponding root categories.
Rationale dataset statistics:
Can we trust the rationales extracted from GPT-4? Although there are plenty of works showing GPT-4's remarkable capabilities 30 31 it still could suffer from hallucinations 33 34 However, evaluations on the generation quality are largely missing from existing works that generate data from L.LSs comprehensive human and machine evaluations. As detailed in Sec. 4.1 on a 5-point Likert scale across three metrics, 964 out of 1,000 categories are scored as having high-quality rationales (24.0).
35 36. To fill this gap and ensure the quality of our rationale data, we conduct.
In contrast to existing Knowledge Graphs [19 20 21 that either offer knowledge unrelated to the visual prediction task, or are too coarse- coarse-grained that provide insufficient information, our structured rationales are tailored for visual recognition tasks in a fine- grained attribute level. Furthermore, our dataset can expand accommodate new rationales, providing flexibility to dealing with evolving datasets where more data becomes available. For example, our rationale ontologies can be seamlessly integrated following the ImageNer [18 category ontology derived from WordNet 20
fine-grained
3.2 Faithful Explanation Method
In this section, we develop new explanation method to implement g(-) in Eq. 1 To incorporate both image and text inputs, we instantiate the model f using the CLIP-ViT architectures [37] because of
3

---

ImageNet, Accuracy
75
Table I: Weakly-supervised segmentation aceu- racy ImageNer-See [45] We threshold explana- tion heatmaps from CLIP-ViT-L-14 as segmentation masks. Our method outperforms existing explana- tion methods in segmentation accuracy, demonstrat- ing the high faithfulness of our explanations.
VIT-8/32
30
VIT-L/14
15
VIT-H/14
8 10
12
14
16
18
20
26 30 52
Acoumulated mean-ablated layers
Exp. Methods
Pixel Acc. t
mloU
mAP f
Figure 3: Multi-head Self Attention (MSA)
LRP 46
52.81
33.57
54.37
accumulated mean-ablation study. Based on 2. we replace the direct effects of MSAs up to specific layer with their mean values calculated across the ImageNer [88] valida- tion set. Most of the performance gains can be attributed to the final layers of the VIT.
rollout 30
60.63
40.64
74.47
row attention
65.67
43.83
76.05
GradCAM
70.27
44.50
70.30
Chefer et al. 40
69.21
47.47
78.29
TextSpan 44
75.21
54.50
81.61
Ours
76.27
58.04
82.17
their proven capability [ Existing methods for explaining the ViT model either directly use the attention maps as explanations 39], or weigh them using gradients 440 41]. However, these methods might be unfaithful to the ViT predictions. This is because the computation of each ViT prediction involves queries, keys, and values, whereas the attention maps only capture the inner products of
queries and keys, ignoring information in values that also affect predictions 44 40 Therefore, explanations based on attention maps might not fully reflect the reasons behind ViT predictions.
Decompose ViT outputs: Recent works 43 44 prove that, for ViT models, the image embeddings can decomposed into the contributions of each token within each attention head. Let Φ and parameterize the image- and text-encoder of the CLIP-ViT model, P is the projection matrix, L,, M, N are the numbers of layers, heads, and image tokens, a; I.m is the output of the m-th attention head in layer for the i-th image token, then the embedding of image I can be decomposed as:
er (T) PV)T(T) Σ E M
E E Pat
(2)
By contracting along layers and heads, [44 calculates the contribution of the i-th image token the final image embedding using E/=, EM
Faithful explanations weighted by mean-ablation results: As indicated by our mean-ablation results in Fig. the final layers contribute the most the predictions, whereas the earlier layers have minimal impact. Thus, noise from early layers could obscure key information by a naive summation across all layers in [44. To address this issue, we weigh each layer's contribution based its importance, measured by the corresponding performance drop in the mean-ablation study. Denote the performance drop of layer as A,, we calculate the contribution of the i-th image token by:
A
e El_ D Pa Pal where w
Σイ=1
Note that e; is projected onto the image-text embedding space by P. Thus, we can use g(l,))
(le fe(r)) her to calculate the explanations of rationale on an image 1, i.e., visual evidence.
Our method significantly improves the explanation accuracy, as shown in Tab. I In contrast to satention-bbasd explanations [39], our method fully utilizes the information from queries, keys, and values that are used for ViT predictions. Compared to sundient-weighter attention maps [40] 411, our method cuts down the computational complexity from Onn²) to (n)) over n image tokens.
3.3 Rationale-informedd Optimization
In this section, we develop double-correct predictions by disentangling and localizing rationales without pixel-wise human annotations V(-) in E
Disentanglement via reconstruction: Drawing insights from our previous research 47 13 propose to contrast between explanation heatmaps of rationales to guide the model training in a self- supervised manner. Specifically, we enforce the following two constraints: i) the image embeddings for different rationales within the same category are disentangled, and ii) the aggregated image embedding of all rationales within the same category aligns with the text embedding of the category.
we

---

Mathematically, the backbone objective is to learn a mapping function f E F such that for each image-text pair (I PII, T), the embeddings (I) and fo are aligned in a shared space if they are correct match, where 7 is text description of category Let E(.) be the InfoNCE loss [8]. h(g(T,r)) = e 1
(g) r), > 7) extracts the image embedding of a given rationale. D(: is a distance metric such as I.2 distance. e, and ő are thresholding hyperparameters. For all [ we propose to develop double-correct predictions by optimizing:
r,
min
R()) == s.t. ((h(o(T,))) (olTTrr)),,((T,. ((((Tr 2 (Q(,))) S 6.
< Correct Predictions
FEF
Correct Rationales
(4)
Disentanglement
Reconstruction
Intuitively, the reconstruction term prevents the disentanglement from collapsing into trivial solutions,
thereby ensuring localization. Solving Eq. 4 Jotten leads to a non-convex problem, wherein methods
such as stochastic gradient descent (SGD) cannot guarantee constraint satisfaction [49 50]. To
address this issue, we leverage Karush-Kuhin Tucker (KKT) conditions [5]
52 and introduce
Lagrange multipliers À and y to convert the constrained problem into its unconstrained counterpart: min JEF {R()) E...r-PLL))
+ XDD((o(L,,)) (g(t,r)) + YDIE, (((,))))
Our method has the following merits: i) In contrast to existing works that rely on expensive pixel-wise annotations to localize objects [53] [54), the proposed informeed optimization achieves
more fine-graine grained. attribute-level localization without manual annotations. ii) Our method can be integrated into vision vision-language model training without architectural changes and extra parameters.
4 Experiments
In this section, we first evaluate the quality of our curated rationale dataset in Sec. 4.1 To best validate double-correct predictions, we then conduct a series of experiments to compare the proposed method The experimental results prove that our model achieves superior prediction and rationale correctness on wide range of benchmark datasets and tasks.
with existing methods in Secs 4.2 4.7
4.1 Evaluation of Rationale Quality
Metrics: We focus three essential aspects of the rationale quality. (1) Factual Consistency: whether the rationales are consistent with facts. (2) Comprehensivennss: whether the rationales provide sufficient information necessary predict the category. (3) Visual Disentanglement: whether the rationales are visually disentanglable or non-overlap. We rate them on a 5-point Likert scale scoring system, where higher scores indicate better performance. For example, in Factual Consistency, score 5 means 100% of the generated rationales are consistent with facts, score 4 means 75%, score 3 means 50%, score means 25%, and score means completely wrong.
Evaluators: (1) Human Evaluators: We recruited four human evaluators, who are mostly graduate students. They asked to conduct assessments based on commonsense knowledge and perform Internet searches for validation. On average, it takes them around one minute per sample. (2) Machine Evaluators: The latest GPT-4o and GPT-4v models (date accessed: Aug. 6th, 2024). For each evaluation, we perform three independent runs and calculate the average scores. Note that expanding human evaluations to the entire dataset is not scalable. To this end, we first prove the reliability of machine evaluations, then use to automatically evaluation the entire dataset.
We sample three independent groups of data from our rationale dataset, each consisting of 50 categories and their corresponding rationales. Specifically, categories were randomly selected from their superclasses: Animals (20), Objects & Artifacts (15), Natural Scenes (5), Plants (5), and Human Activities (5). This ensures that not only each superclass represented but also that results are robust (55). As shown in Tab. 2 The dataset consistently achieves scores of 4.61 or higher on the average of evaluators for each metric, indicating that over 90.3% of the rationales for each category are highly factual, comprehensive, and visually disentanglable.
Human evaluations:
Machine evaluations: Note that the scores of three metrics are almost identical between machines and humans. The Pearson Correlation coefficient of 0.82 reveals the strong positive correlation
5

---

Table 2: Evaluation results of rationale quality. Both machine and human evaluators receive the same instructions about the metrics. The scores for all three metrics are nearly identical between machine and human evaluators, indicating that over 90.3% of our rationales are of high quality.
Evaluators
Factual Consistency Comprehensiveness
Visual Disentanglement
GPT-4o
4.89=0.05
4.55=0.06
4.66+0.06
GPT-4v
4.92-00.3
4.67:10.05
4.70.10.02
Machine Avg.
4.91
4.61
4.68
Human-A
4.85=0.11
4.64-10.19 19
4.42:10.15
Human-B
4.97=0.02
4.77:00.2
4.20110.11
Human-C
4.78=0.04
4.60+0.11
4.78+0.10
Human-D
4.81=0.08
4.65++0.05
4.77+0.07
Human Avg.
4.85
4.66
4.54
between machine and human evaluators. Based on this observation, we further conduct machine evaluations on the entire dataset efficiently. Our results indicate that 964 out of 1,000 categories have high-quality rationales (24.0). See detailed results for the entire dataset in Appendix,
4.2 Benchmark Datasets and Implementation Details
Due to the computational cost of training large vision-language models (VLMs) from scratch, we focus on fine-tuning experiments. Specifically, we fine-tune the ViT-B/32 variant of CLIP on the ImageNNr [18] dataset combined with our curated rationale dataset. To maintain simple and interpretable rationales, the ontology graph for each category is limited to a maximum depth of two, allowing for the extraction of five to six independent concepts on average.
Backbone model:
We compare our model with sftteoof-the-art VL that use ViT-B/32 as their vision encoders, including large-scale pretrained models (CLIP [1 DeCLIP (56)). knowledge-augmeented augmeeted model (NegCLIP IT), and fine-grai grained alignment models (FILIP (57. PyramidCLIP (53).. For
Baseline models:
fair comparisons, also compare our model with ImageNet [18] fine-tuned models using the same CLIP initialization and augmented text descriptions as our model, including full model fine-tuning (-ft) and rision-encoder-olle fine-tuning (-ft-vision).
Evaluation datasets: We validate the prediction correctness of the models on image classification and image-text retrieval tasks. For image classification (zero-shot, linear probe), experiments are carried out on nine benchmark datasets, including CUB [17), Caltech101 (58), OxfordPets (59) Food/0/ (60)
SUN397 [6], StunfordCars 622 DTD (3) CIFAR-IO [64]. and CIFAR-100 (64] For retrieval, we
conduct experiments on Flickr30K [65] and MSCOCO [66]. To evaluate the correctness of rationales, evaluate the models' rationale localizability
CUB-Part [67] and PartImageNet [68] that provide
ground truth segmentation masks of object parts, e.g., "head" and "body". Furthermore, we evaluate the rationale disentanglability on the aforementioned nine benchmark datasets. More details can be found in Appendix D
We follow the same architecture design as CLIP [ for ViT-B/32. The aneciii
Implementation details:
input resolution image encoder is 224 x 224 and the maximum context length of text encoder is 77.
We train our model using an Adam W [69 optimizer and the cosine learning rate scheduler with
linear warmup. Specifically, the learning rate linearly increases from the peak value within 10% of the total steps, and then decreases with cosine anneal strategy. Our learning rate is set to 5e-7 and train the model for eight epochs. More details can found in Appendix
4.3 Evaluation Metrics
Prediction correctness: We use standard category prediction accuracy to evaluate the prediction correctness for zero-shot, linear probe, and fine-tuned settings.
Rationale correctness: We define two new metrics to measure rationale correctness.
i) Rationale localizability. We evaluate the correctness of rationales using ground truth segmentation
masks of object parts 67 68). Following the standard evaluation protocol [70], we threshold the
rationale explanation heatmaps to segmentation masks and caleulate a mean Intersection over Union

---

Table Comparison of prediction accuracy (%) on nine benchmark datasets. Our results are on the average of three trials of experiments using different random seeds. We highlight the best results and the second best results. Surprisingly, different from most interpretability methods that compromise benchmark performance, our method also enhances prediction accuracy.
Metries
Models
C10
C100
CUB
CAL
PETS
F101
SUN
CARS
DTD
AVG
CLIP
91.3
65.1
51.5
87.9
87.0
84.4
63.2
59.4
44.5
70,5
DeCLIP
91.2
66.4
51.2
89.5
79.5
74.6
63.4
50.6
42.7
67.7
NegCLIP
85.7
60.9
37.4
81.0
79.7
71.1
57.0
45,4
37.5
61.7
Zero-shot
FILIP
86.9
65.5
37.5
91.9
88.1
82.8
69.1
55,4
49.3
69.6
Accuracy (%)
PyramidCI.I
81.5
53.7
52.7
81.7
83.7
67.8
65.8
65.0
47.2
66.6
CLIP-ft
83.6
83.6
81.6
78.7
45.3
33.9
CLIP-ft-vision
86.1
56.0
42.2
42.2
Ours
90.8
89.3
88.5
84.3
62.3
73.1
CLIP
95.1
88.8
81.1
83.7
DeCLIP
96.5
84.7
65.0
94.8
85.0
75.0
81.6
78.5
83.4
NegCI.IP
94.3
71.8
98.7
85.6
78.6
75.0
81.3
83.8
Linear Prohe
FILIPP
95.1
82.4
99.1
83.4
76.8
85.5
Accuracy (%)
PyramidCLIP
96.0
82.5
72.3
96.4
87.8
83.3
77.5
82.6
77.3
84.0
CLIP-ft
93.1
76.5
70.7
98.1
88.1
81.7
75.8
58.6
76.3
79.9
CI.I--f-vision
93.7
77.9
71.7
98.3
88.6
84.3
76.4
73.9
75.4
82.2
Ours
95.6
82.7
77.2
99.3
88..1
79.8
83.0
87.5
(mloU 1) score with the ground truth masks across different object parts. Specifically, the dynamic threshold r = e + a, where I and a are the mean and standard deviation of importance values of all pixels in a heatmap. The pixel with an importance value larger than r is set to otherwise 0.
ii) Rationale disentanglabiliry. As shown in Fig. for the CLIP model [1, the visual evidence of different rationales is entangled. Specifically, treat the disentanglemer between the visual evidence of different rationales as an important metric to evaluate whether the model can distinguish rationales. Specifically, treat rationale explanation heatmaps m and m' vectors and calculate 1- (m, m')l as an intuitive measure of disentanglability. the higher metric value the better.
4.4 Evaluation on Prediction Correctness
We compare model against other state-of -ot-oo--thhtttt and fine-tuned VLMs on zero-shot image classification tasks. The results are shown in Tab. 3 On the average of nine datasets, model outperforms the second-best result by 2.6%. The results indicate the strong transferability of our model to other vision datasets.
Zero-shot image classification:
7 we conduct linear probe experiments on the nine image classification datasets. shown in Tab. 3 our model outperforms the second-best result by 2.0%. These results demonstrate the superior vision representations learned by model.
Linear probe: Following the common practice II
Fair comparison with fine-tuned models: As shown Tab, 3 our model outperforms the best fine-tuned model by 10. % and 5.3% on zero-shot and linear probe results. This suggests that the proposed Rationale-informn Optimization is essential in improving the model's performance.
4.5 Evaluation on Rationale Correctness
Rationale localizability: We compare our model with state-of-the-art and fine-tuned VL.Ms. As shown in Tab. our model significantly improves the localization accuracy of rationales by 7.5% and 6.0% on CUB-Part [67] and PartlmageNer [68). This suggests that even without using explicit region annotations, our method significantly enhances the model's localizability of rationales.
Rationale disentanglability: We compare the rationale disentanglement performance of our model with state-of-the-as and fine-tuned models. As shown in Tab. [ the average of nine image classification datasets, our model outperforms the second-best best result by 36.5%. This significant improvement reveals that our model can distinguish between different rationales.
Fair comparison with fine-tuned models: To evaluate whether our model's 's performance gain can be obtained by solely introducing information from our rationale dataset, we conduct fair comparison

---

Original
Near deea
Calegory
Rateenale Explonation
Wethwar
Originl
Canggry
Toeis
Nwatowe
Ratirnie Exparations
"oguave bur
CLPP
Our
CLIP
Oue
Figure 4: Qualitative results of rationale disentanglement and localization. The rationales' visual evidence of the CLIP model [ typically highlights the entire object, lacking precise localization. In contrast, our model correctly localize rationales, thereby enhancing trust in predictions.
Table 4: Comparison of rationale localizability CUB-Pari [67 and PartImageNel (68 As detailed Sec. 4..3 we threshold rationales' explanation heatmaps as segmentation masks and calculate their mloU (1) with ground truth masks of corresponding object parts. Our model significantly improves the localization accuracy fine-graines object parts. Full table in Appendix C
Training
CUB-Pat
Models
ParlIageeNet
Size
Head
Beak
Tail
Wings
Eyes
Torso
Avg.
CLIP
400M
9.9
3.3
28.0
14.4
5.2
DeCLIP
5.1
1.5
18.4
8.3
3.7
NegCLIP
400M+COCO
7.5
2.5
29.4
13.9
FILIP
10.3
7.3
20.5
3.4
23.7
11.2
4.0
PyramidCI.I
143M
10.7
2.9
6.0
17.0
1.8
20.5
9.8
3.9
CLIP-fi
400M+IN
13.5
3.3
5.8
22,9
2.1
25.5
12.1
4.5
CLIP-ft-vision
400M+IN
7.4
2.5
7.9
26.4
1.6
22.0
11.3
4.4
Ours
400M-IN
25.3
10.1
12.7
32.6
15.7
35.2
21.9
11.2
experiments with fine-tuned CLIP models. We compare our model with baseline (CLIP-zs), full model fine-tuning (CLIP-ft), and vision-only fine-tuning (CLIP-ft-vision). All fine-tuned models the same CLIP initialization and receive the same language supervision as our model. As shown model outperforms the best fine-tuned model by 9.8% and 41.1% 1% rationale
in Tabs. 4 localizability and disentanglability, This indicates that naive fine-tuning using augmented information without constraints would deteriorate the rationale correctness of the model.
In Fig. we show visualizations our visual evidence different rationales. As shown, the rationales' visual evidence of the CLIP model [T] are entangled and mislocalized. In contrast, the rationales' visual evidence of our model are visually distinct and correctly localized.
Qualitative results:
4.6 Ablation Study
Ablation on rationale disentanglement: The "w/o disen." refers to a variant of our method without rationale disentanglement constraint. As shown in Tab. [ the rationale localizability decreased by 10.4%, indicating the model might not learn to distinguish between rationales without constraints.
Ablation reconstruction: The "w/o recon." refers to variant of our method without reconstruc- tion constraint. As shown in Tab. 7 the rationale localizability and prediction accuracy drastically decreased by 13.3% and 30.2%. This reveals that recklessly optimizing the disentanglement between rationale can easily fall into trivial solutions.
using the text embeddings of con- cepts as bottleneck layer force the CLIP model ( predict based them can improve prediction accuracy and interpretability. Specifically, the final prediction will be made by the average embedding similarity between the image and all concepts, namely argmax, RR Ik=(ff()) fo fo())). We
Generalize to different rationale sets: According to DCLIP [.

---

Table Comparison of rationale disentanglement. We conduct experiments on nine image classifica- tion datasets. Our results are on the average of three trials using different random seeds.
Models
C10
C100
CUB
CAL
PETS
F101
SUN
CARS
DTD
AVG
CLIP
0.249
0.445
0.475
0.442
0.540
0,481
0.519
0.353
0.287
0.420
DeCLIP
0.303
0.297
0.359
0.395
0.388
0.392
0.360
0.332
0.258
0.343
NegCLIP
0.386
0.319
0.456
0.401
0.440
0.491
0.495
0.389
0.261
0.404
FILIP
0.367
0.359
0.267
0.260
0.305
0.384
0.427
0.371
0.378
0.346
PyramidCIIP
0.299
0.300
0.428
0.418
0.391
0.318
0.397
0.359
0.283
0.355
CLIP--tt
0.378
0.,36
0.469
0,389
0.434
0.374
0.383
0.339
0.251
0.374
Cl.s--ftvisson
0.327
0.3993
0.433
0.431
0.491
0.475
0.423
0.357
0.274
0.400
Ours
0.697
0.714
0.831
0.821
0.920
0.823
0.749
0.776
0.734
0.785
Table 6: Comparison of zero-shot image-text re- trieval accuracy (%). Double-correct prediction enhances the model's visual understanding. (Note that NegCLIP is trained MSCOCO [66])
Table 7: Ablation study on proposed con-
straints using CUB-Part [67] and CUB [17].
Models
mloU (t)
Acc. (%)
Ours w/o disen.
11.5 + 1.3
43.3 + 0.8
MSCOCO
Flickr30K
Models
Ours w/o recon.
8.6 + 0.8
25.8 + 0.7
12T
T21
12T
T21
Ours (full)
21,9 + 1.6
56.0 + 0.5
CLIP
32.5
28.6
64.0
60.9
DeCLIP
32.6
22.1
59.8
46.2
Table 8: Comparison of rationale- based pre-
NegCLIP
69.3
68.1
diction accuracy (%) on ImageNer (18).
FILIP
33.6
36.4
52.9
53.3
PyramidCLIP
37.1
37.6
69.0
69.6
Model
CLIP
CLIP-Rt
Ours
CLIP-ft
24.3
25.1
42.5
41.6
concepts
63.1
67.5
70.5
CLIP-ft-visinn
25.9
27.2
49.1
55.5
rand. str.
63.3
68.6
68.3
Ours
38.4
37.1
69.5
68.9
A
+0.2
+1.1
-2.2
the concept set provided in [9] rather than our training rationale dataset. As shown in Tab. 8 our model can generalize to an unseen concept set with improved prediction accuracy.
Ablation using random string: WaftleCLIP 722 shows that random concept strings bottlenecks conduct an ablation study using the random
can achieve similar performance gains in DCLIP 9.
strings provided by [22. As shown in Tab. since model can distinguish between different rationales, the random strings deteriorate the prediction accuracy of our model.
4.7 Evaluation on Retrieval Tasks
Zero-shot image-text retrieval: We evaluate our model zero-shot image-text retrieval tasks. As shown in Tab, 6 the improved rationale correctness also benefits retrieval tasks.
Rationale- based text-to-image retrieval: To better evaluate the rationale correctness of our model, we conduct novel retrieval task: rationale-based text-to-image retrieval. The model should retrieve the image with a specified rationale presented. As shown in Fig. [ in contrast to the CLIP model [ that entangles rationales with specific categories, our model precisely understands the semantic meaning of rationales independent to categories.
5 Related Works
Vision model explainability. A widely adopted branch of explainability methods post hoc generates heatmaps identify the image regions most crucial the model's predictions, e,g., GradCAM [T), LIME (7), and SHAP [73]. Although useful for revealing the correlations between inputs and outputs, such explanations might be ambiguous, and fail to correspond to high-level concepts that humans easily understand [8]. Methods like TCAV [74] curate attribute datasets to explain vision models using concepts familiar to humans. However, such methods can fail when the models do not learn these concepts [75). Another branch of methods attempts to design specific architecture to intrinsically interpret model predictions, e.g., CBM (76 and ProtoPNet [77]. However, they cannot guarantee the model learns the semantic meanings of the concepts correctly [1] and yield compromised prediction

---

Query: photo of long neck."
(b) Query: 'a photo of wings.
CLIP
CUP
Ours
Ours
Figure 5: Qualitative results of zero-shot text-to-image retrieval MSCOCO 666. retrieve the 1op-5 images with a given rationale presented. The CLIP results reveal a significant entangle of rationales with specific category, such as "long neck" with giraffes and "wings" with airliners. In contrast, our model treats rationales independently from categories, thus offering diverse retrieval results. For example, the "long neck" found in birds, giraffes, dears, and bottles.
The task is to
accuracy [76]. Different from existing works, method incorporates explanations to guide the model training, achieving accurate predictions backed by correct rationales.
Knowledge augmentation for vision-language models. Visual models often learn spurious corre- lations that stem from data biases unrelated the causal explanation of interest 78
79 whereas external knowledge allows models to learn the right features [77 80). Existing attempts for injecting
knowledge into the models are often from the language modality. K-LITE [8]] enrich the image caption using knowledge from WordNet (20] and Wikitionary [82). NegCLIP [1]] and DANCE [12] improve the commonsense understanding CLIP by generating hard negative captions, the latter uses knowledge from ConceptNet [19]. StructureCLIP [83 leverages acene-graphs [1] to incorporate knowledge into text embeddings. However, our results (Tabs. 4 7 reveal that solely augmenting information in the language cannot guarantee the model learning correct features. In contrast to these works, our method offers supervision signals from both modalities to ensure couble--orrectness.
Contrastive vision-language alignment. Different from conventional multimodal learning that fuses different modalities 84 85], large-scale vison-language pretrained models, such as CL.IP and ALIGN (11. exhibit promising zero-shot transferability to downstream tasks. However, their global alignment objective coarse, which only learns the existence of objects like bag-of-wort while ignoring their localizations (II. Recent attempts like PyramidCLIP (53) and X-VLM [54] leverage object region annotations to align word phrases with image regions. DeCLIP (56) and FILIP (57) align text with image regions through self-supervised learning. However, their supervision is limited to a coarse, object-level granularity. Different from these works, our method offers fine-gre onn-grai concept-level supervisory signals of rationales without expensive manual annotations.
6 Limitation
While our study advances the double-correetness of predictions, it is not without limitations. First, the absence of explicit ground truth for rationale localization in large-scale datasets remains significant challenge. We mitigated this by leveraging self-supervised rationalc disentanglement and localization method, but this approach depends heavily on the quality of the structured rationale ontologies. Second, our methods, though effective, are computationally intensive, which may limit their applicability in resource-constrained scenarios.
Conclusion
We introduce new concept of double-correct predictions aimed at training vision-language founda- tion models to make accurate predictions backed by correct rationales, thereby enhancing their safety for real-world deployment. To support this, establish solid foundation for the development of double-correct predictions. Specifically, we develop a unique dataset with structured rationales that clearly outline the reasoning processes necessary for visual recognition tasks. Furthermore, propose a principled mational--ifformer optimization method tailored for double-correct prediction. Our comprehensive empirical evaluations demonstrate that method significantly enhances the double correctness of lisiun-language model predictions.
10

---

Acknowledgments
This work is supported by the NSF CAREER Award No. 2340074, the NSF SAFE Award No. 2416937, the NSF III CORE Award No. 2412675, and the DoD DEPSCoR Award AFOSR FA9550- 23-1-0494, Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the supporting entities.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 7748-8763. PMLR, 2021.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal. Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report. arXiv preprint arXivv2303.0877 088774 2023.
[3] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capa- bilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13337. 2023.
[4] Xiwen Liang, Yangxin Wu, Jianhua Han, Hang Xu, Chunjing Xu, and Xiaodan Liang. Effective adaptation in multi-task co-training for unified autonomous driving. In Advances in Neural Information Processing Systems, 2022.
[5] Xiwen Liang, Minzhe Niu, Jianhua Han, Hang Xu, Chunjing Xu, and Xiaodan Liang. Visual exemplar driven task prommpppg for unified perception in autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9611-9621, 2023.
[6] Randy L Teach and Edward H Shortliffe. An analysis of physician attitudes regarding computer- based clinical consultation systems. Computers and Biomedical Research, 14(6):542-5588 1981.
[7] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1 1135-1144, 2016.
[8] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
[9] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2022.
[10] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2022.
[11] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of of-wordss and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
[12] Shuquan Ye, Yujia Xie, Dongdong Chen, Yichong Xu, Lu Yuan, Chenguang Zhu, and Jing Liao. Improving commonsense in vision-langual models via knowledge graph riddles. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2634-2645, 2023.
[13] Tang Li, Mengmeng Ma, and Xi Peng. Deal: Disentangle and localize concept-level explanations for vims. In European Conference on Computer Vision, pages 383-401. Springer, 2025.
[14] Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? ICLR Workshop, 2021.
11

---

[15] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9568 95788, 2024.
[16] Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Iransactions on pattern analysis and machine intelligence, 28(4):594-6) 61 2006.
[17] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[19] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence. volume 31, 2017.
An open multilingual
[20] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-4) 41, 1995.
[21] Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73 2017.
[22] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 9(1):1-9,200 2018.
[23] Ribana Roscher, Bastian Bohn, Marco F Duarte, and Jochen Garcke. Explainable machine learning for scientific insights and discoveries. leee Access, 8:42200 42216, 2020.
[24] Lawrence W Barsalou. Grounded cognition. Annu. Rev. Psychol., 59:617-645, 2008
[25] Jeffrey Mark Siskind. Grounding language in perception. Artificial Intelligence Review, 8:371
391, 1994.
[26] Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Zou. Skincon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. Advances in Neural Information Processing Systems, 35:18157-18167, 2022.
[27] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12275-12284, 2020.
[28] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE transactions on knowledge and data engineering, 29(12):2724-277 2017.
[29] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494-514, 2021.
[30] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arxiv. arXiv preprint arXiv:2303, 12712, 2023.
[31] Zhengliang Liu, Hanqi Jiang, Tianyang Zhong, Zihao Wu, Chong Ma, Yiwei Li, Xiaowei Yu, Yutong Zhang, Yi Pan, Peng Shu, et Holistic evaluation of gpt-4v for biomedical imaging. arXiv preprint arXiv:2312.052550 2023.
12

---

[32] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[33] Timothy R McIntosh, Tong Liu, Teo Susnjak, Paul Watters, Alex Ng, and Malka N Halgamuge, culturally sensitive test to evaluate nuanced gpt hallucination. IEEE Transactions on Artificial Intelligence, 2023.
[34] Mikaël Chelli, Jules Descamps, Vincent Lavoué, Christophe Trojani, Michel Azar, Marcel Deckert, Jec Raynier, Gilles Clowez, Pascal Boileau, and Caroline tuetsch-Chelli. Hallu- cination rates and reference accuracy of chatgpt and bard for systematic reviews: Comparative analysis. Journal of Medical Internet Research, 26xe53164) 164, 2024.
[35] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- Buriih,n-Burh,,, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19187-19197,202
[36] Ziyuan Qin, Hua Hui Yi, Qicheng Lao, and Kang Li. Medical image understanding with pretrained vision language models: comprehensive study. In The Eleventh International Conference on Learning Representations, 2022.
[37] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[38] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816 (6816-16825, 2022.
[39] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.0093 000988 2020.
[40] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 782-791, 2021.
[41] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618 626, 2017.
[42] Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, and Shiqi Wang. Rethinking attention-mode explainability through faithfulness violation test. In International Conference on Machine Learning, pages 13807-13824. PMLR, 2022.
[43] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Hemighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(I):12, 2021.
[44] Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image represen- tation via base- decomposition. In The Twelfth International Conference on Learning Representations, 2023.
[45] Matthieu Guillaumin, Daniel Küttel, and Vittorio Ferrari. Imagenet auto-annotation with segmentation propagation. International Journal of Computer Vision, 1 110:228-348, 2014.
[46] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Woj- ciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In Artificial Neural Networks and Machine Learning- 2016: 25th International Conference on Artificial Neural Networks, Barcelona, Spain, September 6 2016, Proceedings. Part II 25, pages 63-71. Springer, 2016.
13

---

[47] Tang Li, Fengchun Qiao, Mengmeng Ma, and Xi Peng. Are data-driven explanations robust against out-of- of-dissribuution data? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3821-3831, 2023.
[48] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807. 0.3744, 2018.
[49] Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Advances in Neural Information Processing Systems, 34:20210-20229, 2021.
[50] Fengchun Qiao and Xi Peng. Topology-aware robust optimization for out-of-distribution generalization. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[51] Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[52] Mengmeng Ma, Tang Li, and Xi Peng. Beyond the federation: Topology aware federated learning for generalization to unseen clients. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[53] Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramidelip: Hierarchical feature alignment for vision-language model pretraining. Advances in neural information processing systems, 55:35959-359700 2022.
[54] Yan Zeng, Xinsong Zhang, and Hang Li. Multi Multi-grained vision language pre-training: Aligning texts with visual concepts. In International Conference on Machine Leurning, pages 25994- 26009, PMLR, 2022.
[55] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 20/1, pages 1521-1528. IEEE, 2011.
[56] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations, 2022.
[57] Y. L. et al. Filip: Fine-grained interactive language-image pre-training. ICIR, 2022.
[58] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[59] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and patlern recognition, pages 3498-3505. IEEE, 2012.
[60] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCO 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 2014.
Springer,
[61] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[62] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554 561, 2013.
[63] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-361 2014.
[64] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
14

---

[65] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[66] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th Europeun Conference, Zurich, Switzelland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[67] Oindrila Saha, Zezhou Cheng, and Subhransu Maji. Improving few-shot part segmentation using coarse supervision. In European Conference Computer Vision, pages 283-299 Springer, 2022.
[68] Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille. Partimagenet: A large, high-quality dataset of parts. In European Conference on Computer Vision, pages 128-145. Springer, 2022.
[69] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[70] R. et al. Grad-cam: Visual explanations from deep networks via gradient localization. ICCV, 2017.
[71] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916 PMLR, 2021.
[72] Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15746- 15757, 2023.
[73] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[74] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668 2677. PMLR, 2018.
[75] Reduan Achtibat, Maximilian Dreyer, Ilona Eisenbraun, Sebastian Bosse, Thomas Wie- gand, Wojeiech Samek, and Sebastian Lapuschkin. From" where" to" what": Towards numan-understandable explanations through concept relevance propagation. arXiv preprint xrXiv:2206.03208, 2022.
[76] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference machine learning, pages 5338-5348. PMLR, 2020.
[77] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
[78] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprini arXiv: 000000007999990 2019.
[79] Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12556 12565, 2020.
[80] Tang Li, Jing Gao, and Xi Peng. Deep learning for spatiotemporal modeling of urbanization. Advances in Neural Information Processing Systems Workshops (Best Paper Award), 2021.
[81] S. S. et al. K-lite: Learning transferable visual models with external knowledge. NeurIPS, 2022.
15

---

[82] Christian M Meyer and Iryna Gurevych. Wiktionary: A new rival for expert-buite lexicons? Exploring the possibilities of collaborative lexicography. na, 2012.
[83] Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, et al. Structure-clip: Towards scene graph knowledge to enhance multi- modal structured representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2417-2422 2024.
[84] Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2302-2310, 2021.
[85] Mengmeng Ma. Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18177-18186 2022.
16

---

Appendix
A
Full Prompts
Our Prompt to Obtain Structured Rationales
American Robin = {
"nodes":
("id": "American Robin", "label": "American Robin"),
{"id": "Breast", "label": "Breast"),
{"id": "Tail", "label": "Tail"},
{"id": "Beak", "label": "Beak"),
{"id": "Eyes", "label": "Eyes"},
("id": "Red", "label": "Red"},
["id": "Gray", "label": "Gray"),
("id": "Yellow", "label": "Yellow"),
("id": "Round", "label": "Round"), ("id": "Long", "label": "Long"}
1,
"edges": [
{"source": "American Robin", "target": "Breast", "relation": "has"),
("source": "American Robin", "target": "Tail", "relation": "has"},
{"source": "American Robin", "target": "Beak", "relation": "has"},
("source": "American Robin", "target": "Eyes", "relation": "has"},
{"source": "Breast", "target": "Red", "relation": "is"},
{"source": "Tail", "target": "Gray", "relation": "is"},
("source": "Beak", "target": "Yellov", "relation"; "is"},
("source": "Eyes", "target": "Round", "relation"; "are"),
("source"; "Tail", "target"; "Long", "relation": "is"}
Airliner
"nodes": [
("id": "Airliner", "label": "Airliner"),
{"id": "Wings", "label": "Wings"),
("id": "Tail", "label": "Tail"},
("id": "Fuselage", "label": "Fuselage"),
("id": "Engines", "label": "Engines"), ["id": "Windows", "label": "Windows"),
["id": "Logo", "label": "Logo"},
["id": "Large", "label": "Large"),
["id": "Horizontal stabilizer", "label": "Horizontal stabilizer"),
["id": "Gylindrical", "label": "Gylindrical"),
("id": "Under wings", "label": "Under wings"},
("id": "Rowed", "label": "Rowed"),
("id": "Tail fin", "label": "Tail fin"}
1.
"edges": [
{"source": "Airliner", "target": "Wings", "relation": "has"},
{"source": "Airliner", "target": "Tail", "relation": "has"),
{"source": "Airliner", "target": "Fuselage", "relation": "has"),
{"source": "Airliner", "target": "Engines", "relation": "has"},
{"source": "Airliner", "target": "Windows", "relation": "has"},
{"source": "Airliner", "target": "Logo", "relation": "has"),
{"source": "Wings", "target": "Large", "relation"; "are"),
("source": "Tail", "target": "Horiz. stabilizer", "relation"; "has"),
17

---

("source": "Fuselage", "target": "Cylindrical", "relation"; "is"},
("source": "Engines", "target": "Under wings", "relation": "are"),
("source": "Windows", "target": "Rowed", "relation": "are"),
("source": "Tail", "target": "Tail fin", "relation": "has")
}
What are useful visual concepts for distinguishing (category_name _amm) in photo? These features should visually distinctable and have limited overlap with each other. These features should include attributes and their relations. For each item, you should be concise and precise, and use more than five words. No ambiguous answers. Show your answer using a tree structure
JSON format strictly following the examples shown above. Only contains two depths of nodes (depth 1: attributes, depth 2: subattributes). No connections between node with the same depth. Do not contain node without edge connected it. other explanations, only provide the graph.
B Machine Evaluation on Full Dataset
Table 9: The machine evaluation results on the quality of the full rationale dataset.
Evaluators
Factual Consistency Comprehensiveness
Visual Disentanglement
GPT-4v GPT-4o
4.74
4.39
4.52
4.89
4.59
4.61
C Full Table
Table 10: Evaluation on PartlmageNet [68] with ground truth region of parts using ViT-B/32 vision encoder. We summarize the annotated parts for different categories into 13 common parts. We apply thresholds to the explanation heatmaps and calculate their mloU with ground truth masks. Our model improves the localization accuracy of each part, even though they appear significantly different across categories, such as "wings" for birds and airliners.
Model
Head
Body
Foots
Tail
Hands
Fin
Wings
Tiers
Mirror
Seat
Seal
Engine
Mouth
Avg.
4.2
7.6
DeCLIP
5.9
6.5
3.2
2.4
3.7
T.8
5.2
3.0
1.5
1,7
5.4
2,7
5.9
3.7
NegCLIP
8.3
8.1
4.7
2.1
5.2
3.7
4.1
0.7
2.0
6,9
5.2
FII.IP
5.4
6.9
3.2
2.6
3.9
3,0
5.8
3.4
0.5
6.9
3,1
4,9
4.0
PyramiCCLIP
4.9
7.4
3.3
1.7
5.7
2.8
5.2
3.8
0.8
1.6
7.7
1.8
4.0
3,9
CLIP-ft
6.9
7.3
3.5
1.9
4.9
2.6
5.5
4.2
0.5
1.9
9.7
2.6
6.5
4,5
CLIP-ft-vision
6.4
6.9
3.4
2.1
4.4
2.5
5.4
4.1
0.7
2.1
9.3
2.9
6.4
4.4
Ours
16.7
21.1
8.9
7.2
10.4
10.1
26.3
12.3
11.2
D Implementation Details
18

---

Table 11: Datasets for classification task.
Dataset
Abbreviation
Classes
Train Size
Test Size
CIFAR-10
C10
10
50,000
10,000
CIFAR-100
C100
100
50,000
10,000
Describable Textures
DTD
47
3,760
1,880
Stanford Cars
CARS
196
8.144
8,041
Food-100 101
F101
101
75,750
25,250
Oxford exford-IIIT Pets
PETS
37
3,680
3,669
SUN397
SUN
397
19,850
19,850
Caltech- 101
CAL
102
3,060
6,085
CUB-200-201
CUB
200
5,994
5,794
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the main claims of the paper, delincating both the theoretical and experimental contributions, which are supported by the results presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper comprehensively discusses the limitations of the proposed methods,
including robustness against violations of underlying assumptions and scalability concern..
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
19

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides a detailed presentation of full assumptions and definitions, as presented Secs. 2 Each equation and its definitions are clearly numbered and
cross-referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper fully discloses all necessary details for reproducing the main experi- mental results, including comprehensive descriptions of the methodologies, experimental setups, and parameter settings. addition, the code and specific datasets are provided as well.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
20

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provides an Anonymous GitHub link with open access to both the data and code used in the experiments, complete with detailed instructions in the supplemental material that enable faithful reproduction of the main experimental results. This includes exact commands, necessary environment details, and scripts for preprocessing data, ensuring that other researchers can replicate the study's 's findings without ambiguity. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce
Soaaddess/CoeeSSuuniiiiiiiiiiii sossssubbissiooooccccoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
guw.////i/e//eeee/ /oooeeebbiissionnllll soooooooic for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper details all aspects of the experimental settings, including data splits, hyperparameter selection processes, and the types of optimizers used.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper provides statistical measures such as error bars and confidence intervals for all major experimental results, such as Tab. 7 These measures are correctly defined, and the paper details the variability factors they capture, including train/test splits and initialization randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper adequately details the computational resources required for each experiment, including the types of compute workers (CPU or GPU), memory specifications, and execution times.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
22

---

Question:
Does the research conducted in the paper conform, in every respect, with the
NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research presented in the paper adheres fully to the NeurlPS Code of Ethics, ensuring ethical considerations are addressed and complied with throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper effectively discusses both the potential positive and negative societal impacts of the research conducted. It acknowledges the benefits of the proposed technology in enhaneing data-driven decision- makinn processes while also addressing possible negative implications, such unsafe predictions.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For cxample, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: The paper outlines comprehensive safeguards for the responsible release of data and models, particularly those with potential for high misuse.
Guidelines:
The answer NA means that the paper poses such risks.
23

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The paper properly credits the creators and original owners of all assets used, including datasets, models, and code. Each asset is clearly cited, with references to the original sources and explicit mention of licenses and terms of use.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi paperswithcode thcode. com /atasets
has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced in the paper, including datasets and models, are well documented with comprehensive details provided in structured templates.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?

---

Answer: [NA]
Justification: The paper does not involve any experiments or research activities that include crowdsoureing direct interactions with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve any experiments research activities that include crowdsourcing or direct interactions with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Q58Wc1acAd", "reviewer": "Reviewer_VWuj", "summary": "The goal of the paper is to train Vision Transformer models in a way that they make the \"right predictions for the right reasons\". The paper follows up on prior work in ML explainability that extracts \"rationales\" for model predictions. The idea of the paper (which is also considered in some prior works) is to align the rationale with the model predictions. The key issue that the paper faces is that rationales are quite difficult to label at a large scale. The paper gets around this problem by using GPT-4 based annotations where given a ImageNet class, 6 rationales in the form of a two level deep tree are generated. The paper then fine-tunes some Vision Transformer models to that maximize accuracy while disentangling rationales from each other. Experiments show an increase in accuracy and explainability performance.", "questions": "1. Please see the questions in points 1-4 in the \"Weaknesses\" section.\n\n2. Minor nit: The example in Figure 1 seems a bit problematic. From a distance, the balloons could be taken as red lights in the shape of globes. Wouldn’t the correct question be: Is there a red *traffic* light in the image?\n\n3. Eq. 3: Shouldn’t $w_l$ be inside the summation?\n\n4. Table 2 caption: The last part of the caption is difficult to parse and looks like there is a typo here “our method also hence the prediction performance”. Is this phrasing intended?\n\n5. Line 28: Looks like the base model predictions and the rationales (which often themselves are kinds of predictions) are two often independent things. Theoretically, is it possible to have models that always make the *right prediction* (100% generalization) but always with the *wrong rationale*? Could one train such models, e.g., via using some adversarial training? Would such models be trustworthy, even when one knows they are 100% accurate?", "limitations": "The reviewer sees no societal issues. The paper does discuss some limitations, though there are many (see the Weaknesses section) that need discussion.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. Explainablity of ML models is an important topic as it is critical for building trust in models.\n2. The idea of disentangling rationales is interesting and looks like it should indeed lead to better explanations.\n3. Experiments cover a wide range of datasets.", "weaknesses": "1. The biggest issue with the paper is that it makes some choices without assessing their merit, which leads to a number of open questions. For instance, the paper uses GPT-4 to build the graph of rationales. How were the resulting rationales from GPT-4 evaluated? How were the examples that were passed to the model in Appendix A constructed? Did the paper study how well GPT-4 followed the instructions? For instance, some of the instructions are quite intricate. Consider for instance: “These features should be visually distinctable and have limited overlap with each other”. It is not clear how the model interprets “visually distinctable” and “limited overlap”. Did the model actually follow these instructions?\n\n2. Continuing from the previous point, in line 180: How was the value of 6 chosen? What would be the effect of changing this value? Should all classes have the same number of rationales associated with them? Are some classes inherently \"more complex\" than others? Similarly, should all rationales be disentangled? One would assume that some rationales might be overlapping. For instance, a traffic light and a traffic light pole are overlapping whereas a traffic light and the road aren’t. \n\n3. The paper also adds no discussion on the presence of rationales in the image. Are all the rationales (e.g., beak, tail) always visually present in every image of that class? Does it ever happen that the rationale is not visually present but the explainer still predicts it presence? Similarly, how does one ensure that the model “looks at” the rationales in the input when making the prediction? Could we actually blur that part and see if the model predictions / rationale still stay the same?\n\n4. The writing of the paper is also quite rushed. Important details missing which make it hard to properly judge the merit of the paper. For instance, in line 158: What does the function $h$ actually look like? How does it connect to the vision and the language components mentioned in the previous paragraph? How is $c_i$ computed in Eq. 3 used later on? Similarly, in line 205: The rationales were extracted from GPT-4, so how is the localization ground truth (in terms of pixels in the image) obtained? In Figure 4: What exact procedure was used to get the heatmaps for CLIP explanations? How does the conversion of Eq. 4 to Eq. 5 via Lagrange multipliers solve the non-convexity issue? Isn’t the problem still non-convex? How exactly are the KKT conditions used here?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and instructive reviews. Below we provide point-to-point responses.\n\n**Q1. Why can we trust the rationales generated by GPT-4?** \\\n**1) Sufficient knowledge.** Existing studies prove that GPT-4 has expert-level expertise in commonsense [1] and domain knowledge [2]. **2) Prompt engineering.** Recent studies show that LLMs can follow structured instructions with examples [3-4]. Our examples in the prompt are manually refined across several runs to ensure GPT-4 follows our instructions. **3) Rationale quality evaluations.** As suggested, we conduct automatic evaluations on the rationale quality in a scalable manner, thereby eliminating expensive and subjective human assessment. Inspired by [5], we employ the latest GPT-4o and GPT-4v to assess rationale quality from three aspects rated on a Likert scale from 0 to 5. Summary results are presented below, with detailed score distributions in Fig.1 of our rebuttal's one-page PDF (metric definitions are in Fig.1 caption). Overall, 964 out of 1,000 categories have high-quality rationales (≥4.0).\n|Evaluator|Factual Consistency|Comprehensiveness|Visual Disentanglement|\n|:-:|:-:|:-:|:-:|\n|GPT-4o|4.74|4.39|4.52|\n|GPT-4v|4.89|4.59|4.61|\n\n[1] Bubeck et al. \"Sparks of artificial general intelligence: Early experiments with gpt-4.\" arXiv 2023. \\\n[2] Liu et al. \"Holistic evaluation of gpt-4v for biomedical imaging.\" arXiv 2023. \\\n[3] Menon et al. \"Visual classification via description from large language models.\" ICLR 2023. \\\n[4] Qin et al. \"Medical image understanding with pretrained vision language models: A comprehensive study.\" ICLR 2023. \\\n[5] Bills et al. \"Language models can explain neurons in language models.\" OpenAI 2023.\n\n**Q2. Do all categories have the same number of rationales?** \\\nNo. Each category has \"approximately five to six independent concepts\" (Line 180). We visualize the distribution of rationales, which vary from 3 to 7 with an average of 5.25 per category, in Fig.2 of the PDF.\n\nAs suggested, we conduct additional studies to evaluate the effect of changing the number of rationales. We force each category to have a certain number of rationales and train the models on ImageNet, then evaluate their zero-shot accuracy:\n|# rationales|0|2|4|6|8|10|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|CIFAR-10|83.6|86.7|88.7|90.8|87.2|86.4|\n|CIFAR-100|59.5|62.7|64.4|68.1|56.6|56.0|\n|CUB|46.3|46.5|48.9|56.0|14.4|17.1|\n\nBest performance is consistently observed when the number is close to six. When the number is small, rationales have limited predictiveness; when the number is large, GPT-4 might generate low-quality rationales.\n\n**Should all rationales be disentangled?** Yes. **1)** In Appendix.A, we instruct GPT-4 to generate visually disentanglable rationales. **2)** Our evaluations in Q1 further prove the disentanglability of the rationales. For example, the traffic light, as you mentioned, our rationales are [\"Red, yellow, green lights\", \"Vertical pole structure\", \"Reflective surfaces\", \"White background color\", \"Hanging wires\",...], which are mostly non-overlap and should be disentangled.\n\n**Q3. Are all the rationales always visually present?** \\\n**For training data (ImageNet):** Not always. However, ImageNet images are mostly single object front view in which most of the rationales are visually presented. Rationales could be shared across categories, which significantly increases their overall presence. **For testing data (e.g., CUB-Part):** Always. These small datasets offer ground truth segmentation mask rationales for each image. The existence of a mask indicates the presence of the rationale.\n\n**Will the model predict rationale that is not visually present?** No. Based on Table 7 of the main paper, using not-presented rationales (random strings) will deteriorate performance.\n\n**Does the model \"look at\" rationales when making predictions?** Thanks for the constructive suggestions, we conducted a fidelity evaluation to assess this, and the answer is **Yes**. We measured the area under the insertion curve (iAUC $\\uparrow$), which tracks the incremental prediction probability increase as features from the original input are sequentially added to a blurry reference based on the rationale heatmaps' order. The results indicate that our model's predictions are more sensitive to the perturbation of rationale regions:\n|Models|ImageNet|CIFAR-10|CIFAR-100|CUB|\n|-|:-:|:-:|:-:|:-:|\n|CLIP|0.590|0.457|0.462|0.548|\n|CLIP-ft|0.533|0.420|0.405|0.494|\n|Ours|0.612|0.498|0.509|0.636|\n\n**Q4. Additional details for equations.** \\\nThanks for pointing out the typos, $w_l$ should be inside the summation in Eq.3. The $c_\\mathrm{token}^i$ is then used for calculating $\\langle c_\\mathrm{token}^i, f_\\theta(t) \\rangle$ that measures the importance score of the image token to the prediction of $t$ (Line 145). The function $h(R) = \\frac{1}{|R|}  {\\sum_{i\\in R}^{}} c_\\mathrm{token}^i$, where $R$ denotes the set of token indices within the region. Converting Eq.4 to Eq.5 via Lagrange multipliers does NOT solve the non-convexity issue. This is the standard solution to the constrained optimization problem [6]. It allows for the application of optimization techniques (e.g., SGD) that can handle non-convex problems more efficiently.\n\n[6] Boyd et al. Convex optimization. Cambridge university press, 2004.\n\n**Q5. Minors and other questions.**\n- In the PDF, we visualize the results for the suggested \"red traffic light\" in Fig.3, it still highlights incorrect visual evidence.\n\n- Theoretically, it is possible to train a Clever Hans predictor that always relies on wrong rationales (i.e., spurious correlations) to make correct predictions. For example, in [7] the model is intentionally trained to use artificial hospital tags for classifying MRI images. Even if we know these models might achieve 100% accuracy, they should NOT be considered trustworthy or safe.\n\n[7] Adebayo et al. \"Post hoc explanations may be ineffective for detecting unknown spurious correlation.\" ICLR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TqM80fCMmG", "reviewer": "Reviewer_L5Ee", "summary": "Machine learning models are mostly commonly evaluated based on accuracy, yet highly accurate models can conceal issues with the underlying reasoning. For example, models can predict the right label while predicting it for the wrong reason. To improve the ability of foundation models to make good predictions for the right reasons, the authors contribute two items: 1) they develop a dataset where prediction labels are connected with a series of rationales, and 2) they develop a fine-tuning strategy to incorporate this into the model.", "questions": "1. Why is a tree structure used when designing rationales (line 46)?\n2. Why does incorporating rationales also improve accuracy? \n3. What is the difference between localizability and disentaglability?", "limitations": "The authors acknowledge the limitations of their paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Formulation as rationales matches intuition** - The core idea of rationales is interesting as a building block for developing explanations. Such an idea seems to be present in different settings/scenarios, and matching up explanations with the underlying rationales seems to be an intuitive way to ensure that predictions are made for the right reason. \n2. **The method is pluggable/usable across different models** - As pointed out in Section 3.3, the method proposed in the paper is usable across different architectures. Essentially, the proposed method is accomplished through optimization techniques rather than additional parameters or architecture changes. Because of this, the model proposed in the paper can easily be used with different models. \n3. **Evaluation is across many datasets** - The impact of their fine-tuning method is evaluated against nine different datasets, which makes it clear what the results + impact is. In Figure 4, different images are shown comparing a CLIP baseline with the method proposed by the authors. Such a figure demonstrates the utility provided by their method, as they can improve heatmaps, which implies that they improve a model's ability to make predictions for the \"right reason.\"", "weaknesses": "1. **Unclear if the Dual-correct Prediction Problem is the same as the Right for the Right Reason** - Equations 4 and 5 sketch out the optimization problem that defines the dual-correct predictions, and how to optimize for right for the right reason. However, it's unclear if the optimization problem listed here is the correct one to pursue, and the reasons behind such a problem are unclear. It seems to be that far-apart rationales should still have dissimilar embeddings, and close-together rationales have similar embeddings, but such intuitions are not explained in the paper. More generally, the reason and structure of this optimization problem should be explained better, as it is unclear how or why such a problem connects with the overall issue of producing correct rationales. \n2. **Generalizability of the dataset generation method is unclear** - The dataset generation method is focused on the ImageNet dataset, where different hierarchies are created based on the underlying ontologies. It is unclear whether such ontologies can be generated in non-ImageNet datasets, and how the location of such rationales can be determined in generality. Because of this, it's unclear whether the method to generate the structured rationale dataset can be widely replicated across domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging comments. Below we provide point-to-point responses.\n\n**Q1. Whether dual-correct prediction problem is the same as the Right for the Right Reason?** \\\nNo, our dual-correct prediction problem is different from the Right for the Right Reason in [1]. **1) Explanation granularity.** [1] is limited to providing category-level explanations, i.e., a single explanation heatmap for one prediction. In contrast, our dual-correct prediction can provide concept-level explanations, i.e., multiple explanation heatmaps for one prediction. **2) Comprehensibility.** For image data, [1] is limited to heatmap explanations that are ambiguous in their interpretation. For example, given a heatmap highlighting the object, it remains unclear whether the model is focusing on the object’s color, texture, or shape. Differently, our dual-correct prediction provides explanations in user-understandable natural language based on valid visual evidence. **3) Supervision.** [1] requires ground truth location of explanations during training, which is prohibitively expensive for fine-grained rationales. In contrast, our dual-correct prediction method trains the model to learn correct rationales in a self-supervised manner without human annotations. **4) Real-world applications.** [1] is limited to experiments on small-scale toy datasets, it is unclear whether the method can be applied to real-world data. On the contrary, our experiments demonstrate the effectiveness of our dual-correct prediction large-scale datasets.\n\n[1] Ross et al. \"Right for the right reasons: training differentiable models by constraining their explanations.\" IJCAI 2017.\n\n**Why solving Eq.4 and Eq.5 can achieve dual-correct predictions?** **To achieve correct predictions**, our backbone objective ensures the correct alignment of image and text embeddings in a shared space (Line 157-159), the same as the CLIP model. **To achieve correct rationales**, the biggest challenge here is the absence of human annotations of rationale explanations (Line 85-88). To address this, we propose two constraints to learn correct rationales in a self-supervised manner. One is the disentanglement constraint, which ensures the vision embeddings of different rationales from the same category are disentangled (Line 154-155). However, only disentangling rationale embedding can easily fall into trivial solutions (as shown in Table 6 of our ablation study). Therefore, we propose the other reconstruction constraint, which ensures the aggregation of vision embeddings for all rationales from the same category aligns with the language embedding of the category (Line 155-156). This constraint regularizes the model to achieve semantically meaningful disentanglement of rationale embeddings, i.e., correct rationales. Our experiments demonstrate the effectiveness of the proposed optimization in achieving dual-correct predictions.\n\n**Q2. Whether the ontology generation method can be generated to non-ImageNet datasets?**\nYes, our method can be generalized to arbitrary image recognition datasets. Our method for dataset generation does NOT rely on the underlying hierarchies for creating ImageNet, i.e., WordNet. Specifically, we utilize GPT-4 to extract the desired rationales ontologies (Line 112-113) and our detailed prompt provided to the LLM is detailed in Appendix.A. As demonstrated, our method requires only the category name to generate the rationale ontology, making it applicable to any conventional image recognition dataset, such as CIFAR, CUB, and Caltech101. \n\n**How are the locations of rationales determined?** As detailed in Sec.3.3 and **Q1**, our method can learn the locations of rationales without human annotations of location.\n\n**Q3. Why is a tree structure used when designing rationales?**\nThe reasons are two-fold: **1)** In the context of ensuring rationale correctness, a tree structure helps in disentangling the contributions of different rationales to the final decision. Each path from the root to a leaf can represent a logical chain of reasoning, isolating the impact of individual rationales and their interactions. This disentanglement is essential for training our model that not only performs well but also aligns their decision-making process with human-understandable reasoning. **2)** Trees are scalable structures that can adapt to varying levels of complexity in data. They can expand to accommodate new rationales or condense by merging similar nodes, providing flexibility in how data is organized. For example, our rationale ontologies can be seamlessly integrated following the ImageNet category ontology. This flexibility is crucial when dealing with complex or evolving datasets where more data becomes available.\n\n**Q4. Why does incorporating rationales also improve accuracy?**\nThis is achieved by alleviating the model's reliance on spurious correlations, which are unrelated to the causal correlations of interest [2]. Our method forces the model to make correct predictions based on correct rationales, which makes the learned representations of the model more transferable [3]. Our model is more robust, thereby achieving improved prediction accuracy on *unseen data*, as demonstrated by zero-shot and linear probe accuracy in Table 2.\n\n[2] Arjovsky et al. \"Invariant risk minimization.\" arXiv 2019. \\\n[3] Radford et al. \"Learning transferable visual models from natural language supervision.\" ICML 2021.\n\n**Q5. What is the difference between localizability and disentaglability?**\nOur disentaglability measures the overlap between rationale explanation heatmaps within an image (Line 211-215). Our localizability measures the mean Intersection over Union (mIoU) between rationale explanation heatmaps and their corresponding ground truth segmentation masks (Line 205-210). This ground truth is only available for a few small datasets, such as CUB-Part and PartImageNet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P6yaZhXriH", "reviewer": "Reviewer_WT2A", "summary": "This paper proposes a method for incorporating rationales into the predictions of foundation models. The goal is to build models that make \"dual-correct predictions\": predictions that are correct because the model \"reasons\" using the correct rationale. To do this, the authors collect a new rationale dataset that captures additional information for each category in ImageNet. They then develop a method that faithfully attributes predictions to rationales for CLIP-VIT models, and propose an optimization approach that uses the rationale to guide the model during training. \n\nThis method is then evaluated through a series of experiments. The authors find that their proposed method outperforms state-of-the-art models in both prediction accuracy and rationale correctness across image classification and retrieval tasks. As a result, the paper concludes that the proposed dataset, explanation method, and rationale-informed optimization approach provide a way to develop models with dual-correct predictions.", "questions": "See questions above relating to limited applications, trusting the rationales of GPT-4, and missing related work.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I believe the strengths are as follows:\n\n- Problem goal: With the rise of foundation models being applied in increasingly impactful settings, it is imperative for them to not only make correct predictions but for them to do so for the right reasons; otherwise, they run the risk of failing in unpredictable ways. This paper does a nice job of discussing this problem and the goal of \"dual-correct\" prediction is intuitive and informative. I thought Figure 1 was a really nice figure.\n\n- Rationale dataset: The collected rationale dataset could be useful for future research that builds on rationale methods for ImageNet. \n\n- Experimental results: The paper conducts extensive experiments across a wide range of tasks, including image classification, retrieval, rationale localization, and disentanglement. The proposed method consistently outperforms other methods and ablated versions.", "weaknesses": "Although the experimental results are strong, I think there are weaknesses that take away from the overall impact of the work:\n\n- Limited applications: The paper is framed in the title and intro as providing general methods for forming models with dual-correct predictions. However, the results are only for vision transformers: the faithful explanation method used is based on CLIP-ViT models, and the collected dataset is only relevant for object classification on ImageNet. Since the method and problem are framed generally, there should be more experiments across other domains (e.g. text) and models (e.g. diffusion models). \n\n- Using GPT-4 to generate gold-label rationales: The premise of the paper is that foundation models may not be using the right rationales when making predictions. However, their dataset of rationales is constructed using GPT-4, a foundation model. If we believe the premise (which I do), why should we trust GPT-4's generations as gold-label rationales?\n\n- Missing related work: There is a large literature in NLP that seek to develop methods that form sensible rationales (e.g. [1, 2, 3]). These should be included in the related work and there should be a discussion about how the method developed in the paper differs from them.\n\n- Clarity: Parts of the paper are confusing. For example I thought the rationale-informed optimization section was difficult to understand. There were also typos throughout the paper (e.g. \"[33] averaging across layers and heads to get contribution of image token\" in line 137 isn't a complete sentence). \n\n[1] T. Lei, R. Barzilay, and T. Jaakkola. 2016. Rationalizing neural predictions.   \n[2] J. Bastings, W. Aziz, and I. Titov. 2019. Interpretable neural predictions with differentiable binary variables.   \n[3] S. Gurrapu, A. Kulkarni, L. Huang, I. Lourentzou, L. Freeman, and F. A. Batarseh. 2023. Rationalization for Explainable NLP: A Survey.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your encouraging comments and recognition of our contributions. Below we provide point-to-point responses.\n\n**Q1. Since the method and problem are framed generally, there should be more experiments across other domains (e.g. text) and models (e.g. diffusion models).** \nIndeed, in this paper, we aim to introduce a general problem, dual-correct prediction, for existing models. We then instantiate the research problem using Vision-Language Models (VLMs) and develop a method tailored for the CLIP-ViT model to achieve dual-correct predictions. This is because the CLIP-ViT model is the most representative model that serves as the foundation of other state-of-the-art models. For example, most generative Multimodal Large Language Models (MLLMs), such as LLaVA [1], use a fixed CLIP-ViT as their vision encoder. Recent works reveal that the capability of the CLIP model is the bottleneck of improving these MLLMs [2]. Moreover, the CLIP-ViT model is widely used to guide the generation of diffusion models, e.g., Stable Diffusion. Therefore, ensuring the CLIP-ViT model to make dual-correct predictions has a crucial impact across different domains and tasks.\n\nIn our experiments, we further demonstrate the superiority of our model in text-to-image retrieval tasks on MSCOCO and Flickr30K datasets. This indicates that the enhanced dual-correctness of the vision encoder will improve the performance of its jointly trained text encoder. Ensuring dual-correct predictions for language and diffusion models will require adapting current methods or developing new ones. We will leave this for future work.\n\n[1] Liu et al. \"Visual instruction tuning.\" NeurIPS 2023. \\\n[2] Tong et al. \"Eyes wide shut? exploring the visual shortcomings of multimodal llms.\" CVPR 2024.\n\n**Q2. Why should we trust GPT-4's generations as gold-label rationales?**\n**1) Sufficient knowledge.** Existing studies prove that GPT-4 has expert-level expertise in commonsense [3] and domain knowledge [4]. **2) Prompt engineering.** Recent studies show that LLMs can follow structured instructions with examples [5-6]. Our examples in the prompt are manually refined across several runs to ensure GPT-4 follows our instructions. **3) Rationale quality evaluations.** We further conduct automatic evaluations on the rationale quality in a scalable manner, thereby eliminating expensive and subjective human assessment. Inspired by [7], we employ the latest GPT-4o and GPT-4v to assess rationale quality from three aspects rated on a Likert scale from 0 to 5. Summary results are presented below, with detailed score distributions in Fig.1 of our rebuttal's one-page PDF (metric definitions are in Fig.1 caption). Overall, 964 out of 1,000 categories have high-quality rationales (≥4.0).\n|Evaluator|Factual Consistency|Comprehensiveness|Visual Disentanglement|\n|:-:|:-:|:-:|:-:|\n|GPT-4o|4.74|4.39|4.52|\n|GPT-4v|4.89|4.59|4.61|\n\n[3] Bubeck et al. \"Sparks of artificial general intelligence: Early experiments with gpt-4.\" arXiv 2023. \\\n[4] Liu et al. \"Holistic evaluation of gpt-4v for biomedical imaging.\" arXiv 2023. \\\n[5] Menon et al. \"Visual classification via description from large language models.\" ICLR 2023. \\\n[6] Qin et al. \"Medical image understanding with pretrained vision language models: A comprehensive study.\" ICLR 2023. \\\n[7] Bills et al. \"Language models can explain neurons in language models.\" OpenAI 2023.\n\n**Q3. Discussion of rationale-related works in NLP.**\nIn NLP, the rationale is exchangeable to language explanation [8], however, our rationale refers to visual evidence. In [9], the authors develop a greedy method to generate rationales, which selects the minimal subset of input tokens that produces the same output as the full sequence. [10] and [11] enhance the interpretability of language models in a two-model manner, one latent model selects rationales from the input text, and the other model makes predictions based on the rationales alone. In [12], the authors annotate a commonsense explanations dataset to train a language model that can provide rationales for commonsense reasoning tasks. **1)** Different from these works that are focused on language models, our method can not only provide language explanation for the vision predicitons, but also be based on correct visual evidence. **2)** In contrast to existing works that require human annotations, our method learns correct rationales in a self-supervised manner. We will add more discussion in the final version.\n\n[8] Gurrapu et al. \"Rationalization for explainable NLP: a survey.\" Frontiers in Artificial Intelligence 2023. \\\n[9] Vafa et al. \"Rationales for Sequential Predictions.\" EMNLP 2021. \\\n[10] Lei et al. \"Rationalizing Neural Predictions.\" EMNLP 2016. \\\n[11] Bastings et al. \"Interpretable Neural Predictions with Differentiable Binary Variables.\" ACL 2019. \\\n[12] Rajani et al. \"Explain Yourself! Leveraging Language Models for Commonsense Reasoning.\" ACL 2019.\n\n**Q4. Details of Rationale-informed optimization section.**\nThank you for pointing out the typos. For the rationale-informed optimization section, our key idea is to leverage the two proposed constraints to force the model to learn correct visual evidence for prediction. Specifically, the disentanglement constraint guides the model to learn distinct visual embeddings for different rationales (Line 154-155). However, without human annotations of ground truth regions of rationales, the model can easily fall into trivial solutions that randomly disentangle rationale embeddings without correct localization (as shown in Table 6 of our ablation studies). Therefore, our reconstruction constraint forces the embeddings of rationales can reconstruct the embedding of the category (Line 155-156). By regularizing the search space, our method achieves semantically meaningful disentanglement solutions without relying on costly human annotations. We will correct the typos and make the section more clear in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iKjEOneGUr", "reviewer": "Reviewer_dtoB", "summary": "This paper introduces a novel method called dual-correct predictions, designed to train models to make accurate predictions based on correct rationales, thereby improving their safety for deployment. Additionally, the authors develop a unique dataset containing structured rationales that explicitly outline the reasoning processes required for visual recognition tasks.", "questions": "1. What is the difference between Rationale ontologies proposed by the authors and prototypes in prototype learning? Is it considered to add prototype learning as baseline?\n2. How does the authors' approach scale to a data without ontology labeling?\n3. How does the model predict when the authors are reasoning?\n4. Authors present the background of the problem in Figure 1, using GPT4V as an example. Why is it not in the experiment, comparing GPT4V?", "limitations": "See weaknesses and questions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Authors constructed a new fine-grained rationale ontology dataset, which provides new solution ideas for generating faithful explanations\n2. Authors propose a dual-correct method to improve the effectiveness of the models.", "weaknesses": "1. Equation 4 lacks sufficient explanation. Especially the correct rationale.\n2. Lack of algorithm flowchart to show the training and inference process of the model.\n3. Authors do not have a more up-to-date explainable baselines.\n\nOther:\n\nEquation 4 and 5, missing in the last part ')'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and instructive reviews. Below we provide point-to-point responses.\n\n**Q1. Detailed explanation of Equation 4.**\nIn Eq.4, **to achieve correct predictions**, our backbone objective ensures the correct alignment of image and text embeddings in a shared space (Line 157-159), the same as the CLIP model. **To achieve correct rationales**, the biggest challenge here is the absence of human annotations of rationale explanations (Line 85-88). To address this, we propose two constraints to learn correct rationales in a self-supervised manner. One is the disentanglement constraint $\\mathcal{D}(h(g(I,r)), h(g(I,r')))\\ge\\epsilon$, which ensures the vision embeddings of different rationales from the same category are disentangled (Line 154-155). However, only disentangling rationale embedding can easily fall into trivial solutions (as shown in Table 6 of our ablation study). Therefore, we propose the other reconstruction constraint $\\mathcal{D}(\\sum _ {r\\in\\{r _ k\\} _ y}h(g(I,r)),f _ \\theta(y))\\le\\delta$, which ensures the aggregation of vision embeddings for all rationales from the same category aligns with the language embedding of the category (Line 155-156). This constraint regularizes the model to achieve semantically meaningful disentanglement of rationale embeddings, i.e., correct rationales. Our experiments demonstrate the effectiveness of the proposed optimization in achieving dual-correct predictions. Thank you for pointing out the typo, we will make this part more clear in the final version.\n\n**Q2. Authors do not have more up-to-date explainable baselines.**\nWe respectfully disagree. The explanation method we proposed is developed based on the recent works that decompose the CLIP-ViT predictions into the contributions of attention heads [1][2]. As shown in Table 1, we compare our explanation method with both classic methods (e.g., LRP, rollout, GradCAM) and state-of-the-art methods (e.g., Chefer et al., CLIP-PRS). The results reveal that the explanation accuracy of our method outperforms up-to-date state-of-the-art methods. \n\n[1] Gandelsman et al. \"Interpreting CLIP's Image Representation via Text-Based Decomposition.\" ICLR 2024. \\\n[2] Chen et al. \"Interpreting and Controlling Vision Foundation Models via Text Explanations.\" ICLR 2024.\n\n\n**Q3. What is the difference between the proposed Rationale ontologies and prototypes in prototype learning?**\nRationales and prototypes are conceptually related but fundamentally different. Through the lens of prototype learning works, such as [3] and [4], our rationales can be viewed as a set of prototype vectors for a certain category. However, the differences are significant. **1)** Our rationales are more comprehensible to users. Prototypes are \"visual words\" (image patches), while our rationales are free text. Using image patches and their activations as explanations is ambiguous; their interpretation heavily relies on subjective user judgment. For example, while a prototype may highlight the wing of a bird, it remains unclear whether the model's prediction is based on the wing’s shape, color, or texture. In contrast, our rationales expressed in natural language deliver unambiguous explanations and can precisely localize the regions on the image. **2)** Our rationale aligned with human knowledge and reasoning process for making predictions. Since prototypes are learned in a data-driven manner, these approaches inevitably introduce spurious correlations and biases inherent in the data, i.e., correct prediction with wrong rationales. In contrast, our rationales are generated from human knowledge extracted from pre-trained LLMs, ensuring the model makes correct predictions with correct rationales, i.e., dual-correct predictions.\n\n[3] Snell et al. \"Prototypical networks for few-shot learning.\" NeurIPS 2017. \\\n[4] Chen et al. \"This looks like that: deep learning for interpretable image recognition.\" NeurIPS 2019.\n\n\n**Q4. How does the authors' approach scale to data without ontology labeling?**\nOur rationale generation method can be generalized to arbitrary image recognition datasets, and does NOT rely on ontology labeling of the original dataset, such as the WordNet for ImageNet. Specifically, we query the GPT-4 with example responses and instructions (as shown in Appendix. A) to automatically generate rationale ontologies. Our method requires only the category names, making it applicable to any image classification dataset.\n\n**Q5. How does the model predict and reasoning?**\n(Although we are not sure whether this is the correct question you are asking, we will try to provide an explanation.) **For prediction**, the model will compare the similarity of the image embedding with the text embedding of all category names in the shared space [5]. The category prediction will be the one that has the highest similarity score. **For reasoning**, in Table 7 of our ablation studies, we demonstrate that the model can first compare the similarity between image embeddings and all rationale embeddings, then average the scores of rationales within each category. The category prediction will be the one that has the highest average similarity score. This approach not only enhances prediction accuracy but also exposes the model's reasoning process by revealing intermediate predictions of rationales.\n\n**Q6. Why not compare with GPT-4V in the experiments?**\nThis is because GPT-4V is a closed-source model,  we cannot access its internal features or produce heatmaps through the current API. In Figure 1, we can only tell from the language responses of the GPT-4V that it will also produce incorrect predictions with correct rationales. We tried to prompt GPT-4V to highlight corresponding regions for rationales. However, it always utilizes other classic pretrained smaller models for detection or segmentation, rather than performing these tasks itself. We plan to include this evaluation once more APIs for GPT-4V become available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, Area Chairs, and Program Chairs,

Thank you for your time and effort in reviewing our paper. We appreciate the reviewers find we `"does a nice job"` (WT2A) on studying an `"important topic"` (VWuj) which is `"critical"` (VWuj) for building trust in models, our research problem of dual-correct prediction is `"intuitive and informative"` (L5Ee, WT2A), our collected dataset is `"unique"` (dtoB) and `"useful for future research"` (WT2A), our idea of `"disentangling rationales is interesting"` (VWuj, L5Ee), `"new"` (dtoB), and `"can easily be used"` (L5Ee), our experiments `"cover a wide range of datasets"` (VWuj, L5Ee, WT2A), and our results are `"strong"` (WT2A). We have responded to the individual comments of each reviewer.

**Contribution of this work.** 
The main contribution of this paper is we introduce and formally define the dual-correct prediction problem, i.e., correct prediction with correct rationale. We define the rationales as the valid visual evidence for the language justifications of the prediction. This problem is critical since most of the machine learning models are currently evaluated primarily on prediction accuracy, overlooking a critical aspect for ensuring safety, i.e., the validity of the reasons behind their accurate predictions. To achieve dual-correct prediction, we propose a scalable pipeline to automatically augment existing datasets with language rationales. Based on our dataset, we develop a principled Rationale-informed Optimization method to disentangle and localize rationales on the images. We evaluate the proposed method on 14 benchmark datasets for prediction accuracy and rationale validity.

**Summary of reviews.** Reviewers L5Ee and dtoB vote for acceptance. Reviewers VWuj and WT2A vote for rejection. We summarize the main comments of the four reviewers below:

1. The major concern of Reviewers VWuj and WT2A is the trustworthiness of the rationales generated by GPT-4. On the one hand, the feasibility of this idea is supported by literature that GPT-4 has expert-level expertise in commonsense and domain knowledge, and can correctly follow user instruction via prompt engineering. On the other hand, we provide automatic evaluations on the quality of generated rationales in terms of three metrics. The results show that 96.4% of the rationales are assessed as high-quality. These additional results further enhance the trustworthiness of our generated rationales.

2. The common concern of Reviewers L5Ee and dtoB is the generalizability of our rationale generation pipeline to non-ImageNet datasets. We provide detailed explanations of our automatic rationales generation pipeline through GPT-4. We clarify that our method does not rely on the underlying hierarchies for creating the original dataset, such as WordNet for ImageNet, it only requires the category names. Therefore, our rationale generation pipeline can be applied to arbitrary conventional image classification datasets.

3. Reviewers L5Ee, WT2A, and dtoB request details on achieving dual-correct predictions by solving Eq.4 and Eq.5. We provide explanations on how our backbone objective ensures correct predictions. Moreover, we clarify why we need the proposed disentanglement and reconstruction constraints. We provide detailed explanations on how these constraints work together to achieve correct rationales, i.e., localize rationale using valid visual evidence. We will make this part more clear in the final version.

We believe that these clarifications and additional results have addressed the concerns of the reviewers, and kindly ask the reviewers to take this into account when considering score adjustments. We welcome any further discussion with the reviewers.

Meta-review: This paper proposes a method for enabling vision-based transformer models to more fully consider logical or conceptual components of a class when making predictions, based on a conceptual hierarchy for each class generated via an LLM, referred to in the paper as rationales.  The reviewers in general agreed that the work is addressing an important issue in explanability and furthermore improves robustness, and that the empirical evaluation is extensive and solid.

The lowest score reviewer's main concern was the validity of the GPT-4 generated hierarchy -- the reviewer rightly notes that there are a huge range of potential failure modes for this model in a context like this.  However, the authors in response have done detailed additional verification, including both automated and human evaluation of the hierarchy itself.  The reviewer did not respond to this new information, but I find it compelling.  Additionally, having read the paper and the appendix myself, I have done some additional experimentation to spot check this method for generating rationales and have found it to be highly convincing in practice.  I'm generally skeptical of methods that just ask us to "trust the LLM", but here I believe the authors have established strong evidence that things are behaving reasonably here.  Because this was the main concern of the lowest scoring reviewer, I am discounting that score here and recommending acceptance, as the remaining scores are 6, 6, and 7.

I very much encourage the authors to take the full reviews and the lengthy discussion into account when preparing the revision.
Decision: Accept (poster)