Bootstrapping Variational Information Pursuit with Large Language and Vision Models for Interpretable Image Classification
Authors: Aditya Chattopadhyay
Kwan Ho Ryan Chan
Rene Vidal
Venue: ICLR 2024 poster (2024)
Keywords: Interpretable ML
Explainable AI
Information Pursuit
Large Language Models
Large Multimodal Models
Vision Language Models
Primary area: societal considerations including fairness, safety, privacy
TL;DR: Extending the Variational Information Pursuit framework by annotating data with Large Language and Multimodal Models

Abstract: Variational Information Pursuit (V-IP) is an interpretable-by-design framework that makes predictions by sequentially selecting a short chain of user-defined, interpretable queries about the data that are most informative for the task. The prediction is based solely on the obtained query answers, which also serve as a faithful explanation for the prediction. Applying the framework to any task requires (i) specification of a query set, and (ii) densely annotated data with query answers to train classifiers to answer queries at test time. This limits V-IP's application to small-scale tasks where manual data annotation is feasible. In this work, we focus on image classification tasks and propose to relieve this bottleneck by leveraging pretrained language and vision models. Specifically, following recent work, we propose to use GPT, a Large Language Model, to propose semantic concepts as queries for a given classification task. To answer these queries, we propose a light-weight Concept Question-Answering network (Concept-QA) which learns to answer binary queries about semantic concepts in images. We design pseudo-labels to train our Concept-QA model using GPT and CLIP (a Vision-Language Model). Empirically, we find our Concept-QA model to be competitive with state-of-the-art VQA models in terms of answering accuracy but with an order of magnitude fewer parameters. This allows for seamless integration of Concept-QA into the V-IP framework as a fast-answering mechanism. We name this method Concept-QA+V-IP. Finally, we show on several datasets that Concept-QA+V-IP produces shorter, interpretable query chains which are more accurate than V-IP trained with CLIP-based answering systems. Code available at https://github.com/adityac94/conceptqa_vip.

Published as a conference paper at ICLR 2024
BOOTSTRAPPING VARIATIONAL INFORMATION PUR- SUIT WITH LARGE LANGUAGE AND VISION MODELS FOR INTERPRETABLE IMAGE CLASSIFICATION
Aditya Chattopadhyay" Kwan Ho Ryan Chan René Vidal=
I Johns Hopkins University, USA, achattol 0 (ejuu.ee University of Pennsylvania, USA, (zyanckh, vidalr]@seas upenn. edu
ABSTRACT
Variational Information Pursuit (V-IP) is interpretable-by-essii framework that makes predictions by sequentially selecting a short chain of user-defined, interpretable queries about the data that are most informative for the task. The prediction is based solely on the obtained query answers, which also serve as a faithful explanation for the prediction. Applying the framework to any task re- quires (i) specification of query set, and (ii) densely annotated data with query answers to train classifiers to answer queries at test time. This limits V-IP's appli- cation to small-scale tasks where manual data annotation is feasible. In this work, we focus on image classification tasks and propose to relieve this bottleneck by leveraging pretrained language and vision models. Specifically, following recent work, we propose to use GPT, a Large Language Model, to propose semantic con- cepts as queries for given classification task. To answer these queries, pro- pose a light-weight Concept Question-Answering network (Concept-QA) which learns to answer binary queries about semantic concepts in images. We design pseudo-labels to train our Concept-QA model using GPT and CLIP (a Vision- Language Model). Empirically, we find our Concept-Qu. model to be competi- tive with state-of-the-art VQA models in terms of answering accuracy but with order of magnitude fewer parameters. This allows for seamless integration of Concept-QA into the V-IP framework as fast-answering mechanism. We name this method Concept-QA+V-Ip Finally, we show on several datasets that Concept-QA+Vanl produces shorter, interpretable query chains which are more accurate than V-IP trained with based answering systems. Code available at httpo:///attaao.ccoo/ddaaa/////e httpt/yccceeeecocptpa....uo///
INTRODUCTION
With the increasing complexity of modern deep network architectures, there is a growing concern over the unintelligible nature of their decision-makee making process (Gunning & Aha 2019). Initial ap- proaches to explain deep networks were post-hoc, where algorithms were designed to output scores (attributions) that reflect how "important" a particular input feature is to the network's prediction Selvaraju et al. 2017 Lundberg & Lee 2017, Ribeiro et al. 2016). However, these methods have been widely criticized since their explanations are often not aligned with human expectations which more in terms of high-level semantic concepts (Koh et al. 2020 Chattopadhyay et al. 20022)
Moreover, the reliability of these methods faithfully representing the importance different fea-
tures for prediction has often been questioned (Adebayo et al. 2018 Yang & Kim 2019).
Consequently, there is a growing need for developing deep learning methods that are interpretable by design (Koh et al. 2020 Rudin al.) 2022. Chattopadhyay et al.) 2002). One line of research in this direction is based on Information Pursuit (IP) (Chattopadhyay et 2022 2023). In this framework, the user first defines set of task-specific image queries, each one having clear interpretation to the user as semantic question about the image. Given new image bbs IP then proceeds to select queries one at a time, until the obtained answers are sufficient to make prediction with high
"Equal contribution

---

Published as a conference paper at ICLR 2024
confidence. The query-answer sequence obtained upon termination serves faithful explanation for the prediction. This is because, IP never observes the image directly, but only through the obtained query answers, thereby guaranteeing that the prediction is made entirely on the explanation. Finally, to achieve short explanations, IP selects queries in order of information gain, that is, in each iteration the query selected has maximum mutual information with Y (the prediction variable) given the history of query-answers obtained so far. An illustration of the framework is shown in Figure|
A central challenge in implementing IP is its reliance on mutual informa- tion, which hard to estimate in high dimensions (Belghazi et al.) 2018). To address this, a variational char- acterization of IP, called Variational Information Pursuit (V-IP), was pro-
Inpor image gale
sequence querios about you
rsamenal?
Does it have Mauk and brown madiings?
Yes
Sic it sruil to meciam sord day?
No
ful il lave sail hango Jown?
Sur in tope ofa mocke??
Prediction:
Sei
sharry
Bixa with 90% coufidence
coller?
it aatialats?? Dost in have chorn troee lege?
305
Your Does i huve long haull with served horad Yew
posed (Chattopadhyay et al. 2023 Covert et al. (023). In this approach, deep network is trained to directly learn the most informative next query from data, bypassing the need to plicitly estimate mutual information. A major limitation of V-IP is that it still requires the user to specify the query set that is relevant to the task (e.g. image classification). Moreover, this also requires datasets manually annotated with answers to the queries allow for training classifiers which would be able infer the answers at test time. This limits the applicability of V-IP to datasets where such annotations are available.
Figure 1: Illustration of the IP framework for making interpretable predictions. The task is image classification. The user-speeified query set Q consists of queries about the presence or absence of different semantic attributes of objects. Given an image nnth, IP proceeds by selecting queries from Q in order of information gain until prediction can be made with high confidence.
In this paper, we focus on image classification tasks and seek to address this limitation by employing large language models (LLMs) to specify interpretable queries for task and vision-languee models (VI.Ms) to answer those queries. This would allow us to scale V-IP to any image classification task without the need to manually specify query sets and their corresponding answers. To achieve this, we draw inspiration from recent work in concept-based image classification (Oikarimen et al. 2023 Yang et al. 2023 Menon & Vondrick 2023), which shows the effectiveness of GPT (Brown et al, 2020 an LLM, at generating relevant semantic concepts that are discriminative for a given image classification task. While the semantic concepts proposed by GPT can be seamlessly integrated as interpretable queries into the V-IP framework, we still need a mechanism to answer these queries.
Prior work on concept-based image classification grounds concepts to images using CLIP (Radford) eet al., (021). a VLM,, which computes an lz-normalized dot product between any given image's embedding and any given concept's text embedding as measure of how strongly the concept is associated with the image contents, thereby answering the query whether the particular concept is present/absent in the image. Unfortunately, continuous-valued answers, as provided by CLIP, are not directly compatible with the V-IP framework since they disrupt the interpretability of the query- answer chains. As an example, consider the first query 91 in Figure [] CLIP's dot product between the image embedding of 8e and the text embedding for concept "mammal" is about 0.24. It is not clear from this value if the image is of mammal or not? This in turn makes it hard to interpret the rationale behind the selection of subsequent queries which depend on the query-answers obtained so far. This issue is not resolved by simply thresholding and binarizing the CLIP dot products since, as we show in Table this results in noisy inaccurate answers when compared with the ground truth.
Our main contribution, is to show that one can effectively train a Concept Question-Answering net- work Concept-QAA which produces binary answer to the question "Is the given concept present in the image?" by utilizing soft pseudo- labels derived from both GPT and CLIP. We show via exper- iments that GPT complements concept-image association scores generated by CLIP since GPT has learnt from large bodies of text what concepts are salient for recognizing a particular class. We find that on several image classification datasets (via self- snlf-annotations), our Concept- QA system is com- petitive with contemporary pretrained VLM models used for visual question-answerte being computationally much more efficient. Moreover, we empirically show that V-IP achieves much shorter query chains with higher accuracy when trained on answers provided by Concept-Qu compared to answers provided by CLIP dot products thereby enhancing interpretability.
tasks while
Paper Contributions. (1) We propose methodology to train concept question-amp answering system (called Concept-QA) which given concept and an image, gives a binary answers as to whether the
2

---

Published as a conference paper at ICLR 2024
concept is present in the image or not using pseudo-labels provided by GPT and CLIP. In particular, our method does not require any manually annotated training data grounding concepts to images. (2) We empirically validate, on multiple datasets, the performance of Concept-QA in faithfully rep- resenting the true concepts. (3) We show how Concept-Qu QA enables us to extend V-IP's applicability to image classification datasets where manually specified query sets and answer annotations not a priori available, including ImageNet (Deng et al.) 2009). (4) Finally, we demonstrate that v-IP trained on answers provided by Concept-QA leads to much shorter and more interpretable query- chains than using the continuous dot products provided by CLIP as answers.
2 RELATED WORK
Here we briefly discuss prior work on other contemprary interpretable-D sealle-----eeee besign methods and rele-
gate discussion of prior work on large language and vision models used in this work to Appendix $
snterpretable-b--deiit methods without language. response to the criticisms levied on post-
hoc interpretability methods (Adebayo et al.) 2018 Kindermans et al.] [019] Sllak et al., 2020
Shah et al. 2021; Rudin 2019) many paterpretable-b--eesie
methods have been proposed. Bohle
et al. (2021) propose a novel B-cos layers that incentivizes the network weights to align with task-
relevant features of the input. [lvaree Melis & Jaakkola (2018) regularizes the training of deep networks such that they are locally well-approximmaed by linear classifiers, eL al. (2018) proposes to learn abstract patterns from data, called prototypes, which are then linearly combined to make a prediction. These methods aim explain predictions via input features abstract prototypes. In sharp contrast, in this work, we seek to ground our explanations in language in terms of queries of semantic concepts. thus providing a colloquial description of the underlying models' decisions.
Concept Bottleneck Models (CBMs). This family of models, first introduced by Koh et al. (2020) seeks first map a given input a concept feature space, where every feature value corresponds to the presence or absence of the given concept. The final prediction is made by a linear classifier trained on these concept features. Our proposal in this paper to use concepts as queries for classi- fication tasks was inspired by work on CBMs. However, V-IP and CBMs fundamentally differ in the nature of the explanations they provide for their predictions. V-IP explains the prediction by sequentially asking queries about the input in order of information gain. At each step. the selection of the next query is determined by the history of step, the user can inspect how the model's 's posterior over the class labels changes more and more evidence is accumulated from each new query answer. This provides a progressive and transparent description of the model's 's decision-making process. CBMs the other hand provide static expla- nation by reporting the contribution (as the magnitude of the concept feature value times the weight assigned by the linear network to that feature) of every concept to the final prediction.
observed so far. Morcover, in each
METHODS
3.1 BACKGROUND: INFORMATION PURSUIT AND VARIATIONAL INFORMATION PURSUIT
We will use capital letters to denote random variables and lower-case letters for their realizations.
Information Pursuit (IP). IP as an interpretable-by-dees predictions for any given task was first introduced by Chattopadhyay et al. (2022). Let X : + X and Y Ω + y denote the random variables for input data and corresponding labels/outputs, and Ω be the underlying sample space were all random variables are defined. The user first defines a query set Q. consisting of task-specific and interpretable queries X' - A, where (2x) E A is the answer to the query q E Q evaluated at x E X. Following the specification of Q, the IP algorithm described as follows: Given data point ob X, the algorithm sequentially selects queries in order of information gain, until all remaining queries are nearly uninformative. Specifically,
framework for making interpretable
q1 arg max I (g(X): 4k-1 arg max I [g((X): Y (z=h)= ))))
(1)
gEQ
qEQ
The symbol I denotes mutual information. Here 9k+1 denotes the query selected at step A + 1 given the history of query-answer pairs observed so far, which is defined as
11-k (z = { E X (1.. (((()d))) (a..)((a))) 1:k
(2)
3

---

Published as a conference paper at ICLR 2024
In other words, the history is the set of all realizations of X that share the same answers to the first queries as xht.. The IP algorithm terminates if all the queries in Q have been asked or the remaining queries
uninformative about Y, conditioned on the history obtained so far. In other words, IP terminates after I. iterations (I. depends on x= if either = IQ or maxgeQ [((()) Y 91:L)) S 6, where is user-defined threshold. The final prediction after termination is given by arg yay P(Y = (1:t(z )).
predict posterior stopping criterion reached
Variational Information Pursuit (V-IP). V-IP was introduced by Chaat- topadhyay al. (2023) an effi- cient approach for carrying out IP. V- IP defines a querier network By (with weights 17) function which maps arbitrary finite length query-answer chains, denoted as s, to query g Q, and predictor network fo (with weights 0) that maps any given to distribution over the label Y. Now, given any randomly selected query-answer chain S and data (X, Y), ; let the querier output some query Q and subsequently the predictor output distribu- tion P(Y S, (()).. The authors show that by minimizing the KL divergence between the true distribution P(Y X) and the predicted distribution P(Y S. (X)) expectation over the data and random query-answe chains, one can learn to select the most informative next query given any history of query-answer pairs. More precisely, the optimal querier and predictor of the described optimization problem would be:
5.
Predictor
mag P(Y (zx(("")))
stoppiig crlterion not reached
stack sil?
2. Bright blue color?
3.
Redd with pattrr??
Querler
9--1 E Next
Ro-s Ansver
(o..o/=""))a query-answer chain
Figure 2: Overview of the V-IP algorithm
9-- (s) = arg max (( fo-(s) P(Y S s),
BEQ
where s is any given query-answer chain and P(Y| S= is the true posterior distribution over Y given At inference time, V-IP makes predictions by using the trained querier 9,, to recursively select most informative queries (starting from the empty set at iteration 1), obtain the answer and append to the current history of query-answer pairs observed so far. See Figure for a schematic diagram of this process. The authors proposed two choices for termination, (i) Posterior-based, which terminates if the posterior is sufficiently peaked, that is, max (,x 2 7, and (ii) -base, which terminates if the posterior is stable for some k number of steps, that is, for и consecutive steps, the difference between the entropies of the posterior over Y before and after asking question is less than (7, K & are user-defined), More details on V-IP in Appendix E
3.2 GENERATING QUERY SETS USING LANGUAGE MODELS
As discussed in $3. applying the V-IP framework to any task requires two ingredients, (i) specify- ing set of interpretable task-relevant queries, and (ii) defining mechanism to answer these queries for any given image at inference time. This presents a major bottleneck as curating user-defined query sets and their corresponding answers is a costly time-consuming process. In this paper, focus on image classification tasks and seek to liberate the V-IP framework from this bottleneck by:
Leveraging the zero-shot capabilities of GPT to propose appropriate queries for a given task. Training Concept Answering (Concept-QA) system using pscudo-labels provided by CLIP and GPT for answering these queries at inference time about given image.
3.2.
PROPOSING INTERPRETABLE QUERIES USING GPT
Prior work on concept- based classification methods has shown the effectiveness of GPT at gener- ating semantic concepts that are predictive for solving image classification tasks (Oikarinen et al.) 2023 Yang et al. 2023; Menon & Vondrick 2023). These methods involve the following steps:
1. Given an image classification dataset, extract all the class labels.
2. Iteratively, for each class label, prompt GPT to specify concepts useful for describing the class. For example, if the class label is "ship", Oikarinen et al. (2023) use the prompt "List the most important features for recognizing something as ship".
4

---

Published as a conference paper at ICLR 2024
3. The above process generates large number of candidate concepts relevant to the task. This set is then pruned to remove redundancies among the concepts. This can be done via handcrafted filtering rules (Oikarinen & Weng 2023 submodular optimization (Yang et al. 2023
The assumption of these methods is that since GPT has been trained on a large corpus, it understands which semantic concepts are salient for recognizing a class. These concepts can be used as queries in the V-IP framework by converting every concept C into a binary question: "Is concept C' present in the given image?". This ensures every query has clear interpretation to the user since they grounded in language. Thus, we propose to aggregate the final list of concepts (after the pruning step) and use them queries for our query the concepts extracted by Oikarinen et al. (2023) as our query set. Refer to Appendix Figure 7 for example extracted concepts. Having specified procedure for automatically generating task- relevant query set, we next discuss mechanisms to answer these queries for given image.
In this paper, for all datasets employed, we
3.2.2 ANSWERING QUERIES WITH CLIP ADVERSELY AFFECTS V-IP'S EXPLANATIONS
Prior work on concept-based classification methods proposed to CLIP for grounding the con- cepts to images. This is achieved by passing the image and the concept text through CLIPP' image and text encoders respectively get their embeddings. Then, the fz-normalized dot product be- tween the two embeddings measures how strongly a particular concept is correlated with the image contents. Unfortunately, we will show that using CLIP's dot product as the answer the query "Is concept C' present in the given image?" hampers the interpretability of V-IP's reasoning process.
Recall that V-IP's explanation of a predic- tion is the sequence of query-answer pairs. The choice of every query depends only on the prior history of query-answers. In- terpreting this reasoning process crucially depends on the interpretation of this his- tory, We claim that this history (equa- tion 2 is interpretable to the user only if given any image, the user can ascertain if the image belongs the history. We mo- tivate this definition using Figure [ The first question asked is whether the image is of a mammal, and the answer is "Yes". The user then knows that given any im- age sampled from the data distribution, if it contains mammal, will belong to the model's history. Continuing like this, we see the model ask several other questions about different characteristics of mammals like hair texture, tail shape. etc. before it concludes it is bison. Since, ev- ery step, the history is interpretable, V-IP's reasoning up to the prediction of "bison" is transparent. However, when the answers are continuous- valued dot products, as supplied by CLIP, the history is no longer interpretable (according to our definition). We elaborate on this next with an example.
P(Y | (aaa"")))
Imput imagee sbo
ingneddennts label: 1 113546
2. donsal fine 0.1014
088
reddssh-gray frr 0.4637
sausage 1
1.2015
0.6
Whitegray feathers 0.5587
7. feed dog- 0 0.6204
orange.peling scalee-
black
0.1236
02
11. hooves with two toes: 1 1.0953-
12. shrimp 2.6706
0.0
Bison
Arabial andd Brteer
American shark
Figure V-IP's explanation for an image from the Imagenet validation set using CLIP's dot products as answers. The in- put image, =d is shown on the left and the query-answer chain is shown the right along with the evolution of the model's posterior distribution. Every row corresponds to the posterior distribution over the classes (shown in the x-axis) given the history of answer pairs observed so far. The y-axis indicates the query asked at that iteration along with CLIP's dot product answer. For brevity, we only mention the concept, instead of the corresponding query "Is given concept present in the image?" (best viewed in colour).
We train V-IP on the Imagenet dataset using the query set described in $.2.1 and CLIP's dot prod- uets (standardized by subtracting and dividing by the mean and std computed over image-concept pair from the entire dataset) to answer the queries. In Figure we depict our trained V-IP's expla- nation for an image from the Imagenet validation set. Positive and negative dot products are colour coded as green and red respectively. The first query, "Is the concept ingredients label present in the image?", gets an answer of 1.33. According to our definition, the history after obtaining this answer, qı (x. is not interpretable to the user since we do not know what images would give the same dot product of -1.35 when asking the same question. The choice of the second query about dorsal fins as well the remaining queries, are hard to interpret. Moreover, the final query about "shrimp" makes no sense given the posterior after 11 queries was peaked on "bison", "bighom", and "Arabian camel". We argue this is because histories in this context are not interpretable.
5

---

Published as a conference paper at ICLR 2024
3.2.3 OUR PROPOSAL: CONCEPT QUESTION-ANSWERTE NETWORK
One possible solution to solving this interpretability conundrum is to make the query answers dis- crete, with every answer having a well-defined interpretation to the user. However, in the absence of data annotated with query answers, how can one learn classifier that would "accurately" predict the answer for a given image at test time? To answer this, we rely on the zero-shot capabilities of GPT and CLIP.
A first attempt at this might be to simply threshold the standardized CLIP dot products and interpret a positive value as the concept being present the image. However, as we will empirically show in $4 such a method results in inaccurate answers (compared with the ground truth) and makes them difficult to interpret. We thus propose to complement the knowledge CLIP has about image-concept association with GPT's knowledge about different concepts that are important for recognizing particular class. In particular, we train a Concept Question-Ansseringg network (Concept-Qu, -Aaceptt-AAAae using pseudo-labels (which our target answers for training Concept-QA) generated from GPT and CLIP. We first describe the intuition behind these pseudo-answers and then explicitly define them.
In image classification tasks, the class is often the focal point of the image. This means that priori before inspecting the image, if we know the class then we can be pretty confident that some concepts will not be present in the image. For example, if know the image is of class "Tiger", we know that it is unlikely the concept "a garment" will be present in the image. However, answering a query about a concept like "white" requires inspecting the image contents since some tigers can have white fur, while others are predominantly orange. We formalize this intuition using GPT and CLIP. Specifically. for given image classification dataset, we take every image-label pair in the corresponding train set and construct image-concept- pateeppt-psuu-labbeeee triplets for all concepts in our query set in the following manner:
1. For every image-label pair and concept in our query set, we ask GPT if the concept is salient for recognizing the class label.
2. If GPT responds "No", we take the pscudo-label for that concept-image pair as "No".
3. If GPT responds "Yes/Depends", we use CLIP's dot product between the concept's text embed- ding and the image's 's embedding disambiguate whether the concept is present in the image. Since CLIP's dot product is a measure of how strongly the concept is aligned with the image content, we use it (after suitable normalization which we will describe next) to model the proba- bility that the concept is present in the image.
Moving forward, we would use the value 1 (0) interchangeably with "Yes" ("No") to talk about the answer/prediction for a particular concept query. Now, given image-label pair (I, y) E X' x y and a binary concept C which corresponds to a query qc E Q, we define the following terms,
Part (C= 1 y) = 0 otherwise
if GPT responds "Yes" to the above mentioned prompt
(0 -
where I(x) and Tcc) correspond to the ez-normalized image and concept's text embedding respec- tively using CLIP's encoders. (-) is a per image min-max normalization function which ensures that the concept with the highest dot product with the x is given a probability score of 1, and the concept with the smallest dot product with æ is given a probability score of O. The two probabilities from equation 4 are then combined give the pseudo-label for the pair (Ir v) and concept C as,
P(C=1 [[C=1|X=0)) xIx Y
PG = Porr 1 x=x)
(5)
P(C=0| X=x,
Parr(C==0 Y= (C=)
Finally, we propose to train deep network, called Concept-QA. which takes CLIP's embeddings of the image and the concept's text and makes a binary yes/no prediction to the query "Is the given concept present in the given image?". We use the generated pseudo-labels, P. to train our Concept- QA network using the following loss function for given concept (which is reminiscent of the popular cross cross-entropy loss when the target labels are one-hot),
Loss(C)= = -Exy
Σ cE(0,1)
P(C X, log (C = c xx)
(6)

---

Published as a conference paper at ICLR 2024
where defines the Concept-QA model weights. Notice our model predicts whether the concept C is present in the image purely based on the image contents (no Y in- formation is provided), as embodied in the notation (C X). The optimization problem is defined as, min, Eceq Loss(C), where the sum- mation is over all concepts (or their cor- responding queries) in our given query giies
every (clem]
noner to WT:
Telekkng
follewiic coccapee aEe eRCinEs foe
L.N. peide
CUP 2,7
Conceett 04
bescripcice: bese JAEEE DANDAL poeeacess Se61 bUACe, MIEE SROGE seAep
Asswer Predicike
PefC [
ieepite DOGiG top
11x
SEEON: DO1 SRACE eSpieeicy acloeacy
Condmnct Panaketabel
Trakkng
ParyCC
P10 [0==|xx-2 Y Y
Figure 4: Overall pipeline for training and performing inference the Concept-QA system. Details about the GPT and CLIP prompts used in experiments can he found in Appendix $D
set Q. Figure 4 gives an overview of Concept-QA training. More training details in Appendix $D
After training, for given image td and any concept C, we take the corresponding query's answer (x
X 2 т and otherwise. The constant T is a user-defined parameter which determines how confident we want the Concept-QA to be in its prediction of before accepting it as the binary answer. We use T 0.4 for all datasets considered this paper except CIFAR-(10 100} for which consider T = 0.5.
(C
EXPERIMENTS
In this section, we empirically evaluate the efficacy of our proposed Concept- QA model in extending the application of V-IP image classification tasks where manually specified query sets and answer annotations are not available. We do so in two complementary directions.
1. Interpretability of query sets. In prior work (Chattopadhyay et al. 2022 2023) the query sets were manually specified and every query had a clear interpretation to the user. However, this work, train our Concept-QA model using pscudo-labels to answer different queries. Thus, the interpretability our queries depend on how faithfully Concept-QA reflects the ground truth. For example, if for some query "Is concept "stripes" present in the image?", Concept-QA erroneously predicts "1" (corresponding to a "Yes") when the image has the concept "checkerboard pattern" in it (but no stripes), then the answer no longer has clear interpretation to the user.
Description length and accuracy of V-IP explanations. Consequently, given an interpretable query set, V-IP makes predictions by asking sequence of query-answer pairs about given input. This sequence serves as an explanation for the prediction. accordance with Occam's razor, Chattopadhyay
al. (2022) advocate that shorter explanations are preferred over longer ones if both achieve the same prediction accuracy. Thus, it is important to have query sets that result in both short and accurate explanations.
Finally, we provide qualitative examples of explanations provided by V-IP using Concept-QA as an answering mechanism and contrast it with the uninterpretable query-answer chains obtained when using CL.IP dot products as the answering mechanism.
We carry out all our experiments on 5 popular image classification datasets; ImageNet (Deng et al., 2009) CUB-200 (Wah al. 2011), Places365 (Zhou et al.) 2017), CIFAR-(10,100) (Krizhevsky et al. 2009). In all our experiments involving CLIP, we used the ViT-B/11 backbone, and for GPT we used the gpt-3 5-turbo 0613 version, except for CUB-200 where we found GPT-3.5 ineffective due to the commain-peccffi nature of the dataset and opted to use gpt-4 instead. For every dataset, the query set employed the set of concepts extracted by Oikarinen al. (2023)
Interpretability of query sets. We evaluate how faithful Concept-QA in representing the true answers and compare its performance with the fol- lowing baselines: (i) CLIP-Binad. which uses the binarized CLIP dot products (standardized as explained in (3.2.2 as answers with a posi- tive/negative value taken as 1/0. (ii)
Table I: Results evaluating the faithfulness of different answer- ing mechanisms to the true answers. Acc. and Fl refer to accuracy and Fi-score metrics respectively.
Model
ImapeNet F
Places365 Acc.
Fj
Anc.
CUB-200
F,
Acc:
CIFAR-10
F,
CIFAR-I100
F2
Concept-QA (Ours)
0.56
0.90
0.62
0.38
CLIP Binuz
0.55
0.39
0.58
0.42
0.56
0.48
0.58
0.47
0.51
0.21
CLIP Ditaao.
0.50
0.27
0.59
0.26
0.56
0.45
0.56
0.53
0.54
0.25
BLIP2 ViT-g OPT's
0.55
0.31
0.76
0.18
0.53
0.35
0.73
0.13
0.86
0.07
BL.1P2 ViT-g FlanTSar
0.86
0.56
0.87
0.62
0.70
0.40
0.83
0.59
0.87
0.41

---

Published as a conference paper at ICLR 2024
gom which obtains answers by thresholding Pcur(C=1|X==)) (equation 4 at 0.5. (iii) BLIP2 ViT-g FlanT5xx & BLIP2 ViT-g OPT2.BB. which are state-of- oatte-o--hhearrr art are-trained VLMs that effective in zero-shot Visual Question-Answerinn datasets we consider do not come with annotated query answers we randomly sample 2.5 image- concept pairs from each dataset and self-annotate them with the true answers. Details of the sampling and annotation process can be found in Appendix CC)
(VQA) tasks (Li et al. 2023 Since the
For each dataset, we report the accuracy and
Fl-score and report the results in Table !
We observe that Concept-QA outperforms CLIP-Binga and CLIP-Binnom on all 5 datasets on both metries, supporting our claim that simply binarizing CLIP dot products based on threshold will lead to noisy answers and integrating GPT with CLIP to provide the pseudo-labele enhances the quality of the supervision signal for training the answering model. Furthermore, compared to the BLIP2 Flan'55 model, Concept-Qu is competitive on the CIFAR datasets and ImageNet. Our model considerably better on CUB-200 a fine-grained bird species identification dataset which is too specific for the general -uupros- VQA model to have good zero-shot performance. Finally, BLIP2 FlanT5 performs better on the Places365 dataset which is a scene classification dataset. We hypoth- esize that this is due to the inability of CLIP's image encoder at capturing essential image semantics for this dataset. Table in the Appendix corroborates this, which shows that a linear probe trained on CLIP's image embedding achieves a test accuracy of only 55% which is much lower than the accuracy achieved on all the other datasets.
While being competitive with BLIP2 most datasets, Concept- QA is much more computationally efficient than these large VLMs which have billions of parameters. Concretely, ConceptQA is a lightweight MLP with 70K parameters which operates on top of CLIP resulting in a total of 150M parameters. The competitive VLM, BLIP2 FlanT5, on the other hand has about 4.1B parameters. Consequently, our model takes about 0.04 seconds per image per query, while the VLM takes about 1.52 seconds for the same. This makes the latter prohibitive for utilization within the V-IP framework for training and inference. For example, carrying out one epoch of V-IP training on Imagenet using BLIP2 Flan-T5 as the answering mechanism would take about 4. 128 GPU hours (evaluated on NVIDIA RTX A5000 GPU by averaging the time taken to process 100 batches of image-query pairs and multiplying by the size of the dataset.) which is infeasible. Comparatively, V-IP with Concept- QA takes just about 0.5 GPU hours to finish one epoch! Similar computational challenges also plague the use of VLMs to directly provide the pseudo-labels for training the Concept- QA model in place of our proposed pseudo- labels (equation 5). Refer Appendix $ for a discussion on this.
Description length and accuracy of V-IP explanations. Next, we analyze the effec- tiveness of V-IP in obtaining short and ac- curate explanations using our query sets and compare different answering mech- anisms. Following are the models con- sidered: (i) V-IP trained with Concept- QA, Concept-QA+V-IP (ii) V-IP trained with CLIP continuous-valuee dot prod- uets (standardized), CLIP+V-IP. (iii) V-IP trained with CLIP- -Bin-ingdd. CLIP-Bingat+ +- IP; (iv) V-IP trained with CLIP-Bin CLIP-Bin V-IP; and (v) V-IP trained with BLIP2-FlanT5: HLII2-FlanT5+VAIp For training and architecture details for these models refer to Appendix $ We report our results in Figure 5 For each model, the curve was generated by selecting the stopping criterion that yielded the best results: the stability-based criterion (see Section $3.1) for Concept-QA+V.Ip and BLIP2- FlanT5+V-IP, and the posterior- based criterion for the others.
3.00
Cifar-10
Cifar-100
CUB-200
0.0
ga.r
0.0
0.6
0.4
R0.2
6.2
75
5B
7.5
16.8
25
20 Cueries
Image/Net
Places-365
0.50
Concepi-gA--VP
45
CUP -Vi
Bo35
CUP CUP-Binu, 1V-P
Fian--
0.0
Aug.
#
of Ouerre
Aog Suarten
Figure 5: Trade-off between test accuracy vs. avg.
# of
V-IIP
queries (explanation length). View in colour.
We see in Figure that Concept-QA+V-J outperforms CLIP+V-IP (along with its binarized CLIP- Bin variants) on all datasets in terms of the average number of queries needed to reach desired level of accuracy (except on CIFAR-10 for a small "low test accuracy, low avg. number of queries" regime). As articulated previously, state-of-the-aa VLMs are computationally intensive and thus,
Forr both these VLMs we the prompt "Question: Is concept present in the image (Yes/No)? An- swer:", where concept is replaced by the name of the concept we interested in asking the VL.M about.
8

---

Published as a conference paper at ICLR 2024
CLIP-VIP; V-I with CUI dot products as answers
Inout Image x
Concagt QANKIP: VPP with Concegt QA answers
3.0
Enpry
Varrma
sten, farcoverdd legs 6425
2. hending day 1 2621
0.
8 5.80 5k
3. larg, wrta bid
g vanamous life
sapport
other lemurs 2 0976
7. mocruto 5677
poserfall enning ahility
5. larg, anderwater graduue 2 4007
8.a manon-
0.4
pray aransl
10 ushagg mote:
15 rack eccnnding from the body
0.2
ersek beadyy ayes
colers
shark fit
0.0
Figure 6: Example run of the V-IP algorithm on an example image from the validation set ImageNet. Column shows CLIP+V-IP trajectories, while Column 3 Concept-QAs QA+V trajectories. Refer to the caption of Figure 3 for a description of the heatmaps. The x-axis of every heatmap shows the top- 10 classes according to the posterior after termination. For queries/concepts. green (red) colour denotes positive (negative) dot product in col and "Yes"("No") answer in col 3. View in colour. See Appendix şG more examples.
inefficient to use for answering queries in V-IP training for large datasets like Imagenet. As a result, we could only compare with BLIP2-FlanTS on the three relatively smaller datasets. Our result V-II indicates that Concept-QA++ V-IP outperforms BLIP2-FlanT5+v-IP on CIFAR-10 and CUB-200 (the latter by a huge margin). On CIFAR- 100, our model has a better trade-off for short explanation lengths but eventually the BLIP2- F222-FFlaT55 gets better results. Interestingly, these observations correlate well with the results in Table I where the answers supplied by Concept- -A are more faithful to ground truth than the VLM on CUB-200, competitive on Cifar- 10 and slightly worse on C0f Finally, since interpretability can be seen as a constraint on learning, we discuss in Appendix T the gap between V-IP's performance using query sets to that of black-box model.
Interpretability of Predictions by V-IP. Having quantitatively established the efficacy of our pro- posed Concept-QA with V-IP, we now present qualitative examples illustrating example runs of V-IP (See Figure 6 and more examples in Appendix $G Our examples show a sharp contrast in inter- pretable explanations when answers are supplied by Concept-QA vs. uninterpretable explanations generated when answers are supplied by CLIP continuous-valuee dot products.
starts by asking if the image is of a mammal. This first query is independent of the image contents (see equation []. Upon receiving "No" answer, it proceeds to ask if it is an animal and it has scaly skin. Since both these answers were "Yes", the model knows the image is of an animal which has scales. The next query asks whether the animal lives near water, down the
Observe, in Conneeeeeonnneeee-Aaaaa
for which the Concept-QA model incorrectly says "No". This leads Concept-QA+VV wrong branch where it asks questions to see if the animal is a snake (which does not necessarily live near water bodies). After repeated "No" responses, animal is marine animal, which gets a positive response from Concept- Finally, after query 13, Concept-QA+ V-IP knows the image is of a shark and we see the posterior distribution concentrate all its mass three different species of sharks. Subsequently, the class "hammerhead" is eliminated from consideration after asking whether the shark has a hard, spiny shell; a neck extending from the body and small, beady eyes all characteristies of a hammerhead shark. Finally, Concept-QA+ disambiguates between the great white and tiger shark by asking whether the shark has a "stocky body", which a characteristic feature great white sharks compared with tiger sharks which have relatively sleckier bodies. Such rich transparent elucidation of the docision-making process in V-IP is lost when the answers are taken as dot products. This clear by observing the explanation provided by CLIP+V-IP for the same image. Since the answers are now dot products, the queries selected by V-IP are no longer the same. For example, after 8 queries, CLIP+V-IP has all its posterior mass on great white and tiger sharks. However, the next two queries are about a mahout and a shaggy mane, neither of which are related to the two shark species in consideration. However, this information is used by CLIP+V-IP to conclude the image is of a great shark with more than 80% confidence. We conclude this section by directing the reader to {H (in the Appendix) for a discussion on limitations of our current approach and potential future directions.
Mancept-QA+-III circles back and asks if the
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was supported by the Army Research Office under the Multidisciplinary University Research Initiative contract W91 1991N--17-100004, the NSF grant 2031985 and by Simons Foundation Mathematical and Scientific Foundations of Deep Learning (MoDL) grant 135615. Moreover, the authors acknowledge support from the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE2139757. Any opinions, findings, and conclusions or recommenda- tions ex expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems, 31, 2018.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.), Proceedings the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 531-540. PMLR, 10-15 Jul 2018. URL mlr
htttps://proceedin www.:em.....prr....mmmmmmoe.eeee httpps/s//.//peeeaaaaaaaaaaaaaaa
Moritz Bohle, Mario Fritz, and Bernt Schiele. Convolutional dynamic alignment networks for in- terpretable classifications. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 10029-10038, 2021.
Michael Bommarito II and Daniel Martin Katz. 1:Xiv:2212.14402, 2022.
Gpt takes the bar exam. arXiv preprint
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint wwXiv:2108.07200 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2031v:2303.12712, 2023.
Aditya Chattopadhyay, Stewart Slocum, Benjamin D) Haeffele, Rene Vidal, and Donald Geman. In- terpretable by design: Learning predictors by composing interpretable queries. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Aditya Chattopadhyay, Kwan Ho Ryan Chan, Benjamin David Haeffele, Donald Geman, and Rene Vidal. Variational information pursuit for interpretable predictions. In The Eleventh International Conference on Learning Representations, 2023.
Ian Connick Covert, Wei Qiu, Mingyu Lu, Na Yoon Kim, Nathan White, and Su-In Lee. Learning to maximize mutual information for dynamic feature selection. In International Conference on Machine Learning, pp. 6424-6447, PMLR, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs. vision-language models with instruction tuning, 2023.
Dmitry Demidov, Muhammad Hamza Sharif, Aliakbar Abdurahimov, Hisham Cholakkal, and Fa- had Shahbaz Khan. Salient mask- gaiided vision transformer for fine- -iee-grannd classification. arXiv preprint 2rXiv:2305.07102. 2023.
10

---

Published as a conference paper at ICLR 2024
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scate scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Sedigheh Eslami, Gerard de Melo, and Christoph Meinel. Does clip benefit visual question an- swering in the medical domain as much as it does in the general domain? arXiv preprint arXiv:2112.13906, 2021.
David Gunning and David Aha. Darpa's explainable artificial intelligence (xai) program. Al maga- zine, 40(2))44-58, 2019.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
wlarxiv.org///aa / 1412.11110 6980
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. International Conference on Machine Learning, pp- 5338-5348. PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2023.
Tuomas Oikarinen and Tsui- Wei Weng. Clip-dissect: Automatic description of neuron representa- tions in deep vision networks. In The Eleventh International Conference on Learning Represen- lutions, 2023.
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottle- neck models. In The Eleventh International Conference on Learning Representations, 2023.
Maria Parelli, Alexandros Delitzas, Nikolas Hars, Georgios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Clip-guided vasion-language pre-training for question answer- ing in 3d scenes. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pp. 5606-5611, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2024
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206-2155 2019.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. In- terpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1-85, 2022.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference computer vision, pp- 618-626, 2017.
Harshay Shah, Prateck Jain, and Pranceth Netrapalli. Do input gradients highlight discriminative features? Advances in Neural Information Processing Systems, 34, 2021.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 804 814, 2022.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 180-186, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397, 2023.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Mengjiao Yang and Been Kim. Benchmarking attribution methods with relative feature importance. arXiv preprint 1oXiv:1907.090000 2019.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable im- age classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19187-19197, 2023.
Shuhong Ye, Weikai Kong, Chenglin Yao, Jianfeng Ren, and Xudong Jiang. Video question an- swering using clip-guided visual-text attention. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 81-85. IEEE, 2023.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A RELATED WORK ON LARGE LANGUAGE AND VISION MODELS
OpenAl's GPT-3 (Brown et al. 2020) is an LLM trained on large amounts of unlabeled text in self-supervised manner. Recent benchmarks have demonstrated that LLMs can perform complex human tasks, such as passing the Bar Exam (Bommarito II & Katz 2022) and show signs of general understanding (Bubeck et al.) 2023). Adding other data modalities, CLIP (Radford al. 2021) VLM pre-trained on millions of image-text pairs. CLIP has been building block for many recent VLMs that achieve state-of-the-art zero-shot performance image classification (Menon & Vondrick 2023), Visual Question Answering (Eslami et al. [021]] Parelli et al. 2023 al.
and Video Question Answering (Ye et al., (023). Also known as Foundational Models (Bommasani et al. 2021), L.Ms and VLMSs have demonstrated strong capabilities to learn complex concepts,
which motivates our proposal to use GPT and CLIP to propose query sets for V-IP.
B EXTENDED RESULTS
B.1 ACCURACY OF CONCEPT-QA+V-IT VS. BLACK-BOX MODELS
We compare the test accuracy obtained by of Concept-QA+V-IP vs. black-box non interpretable models in Table Recall that V-IP obtains query answer chains by sequentially selecting the most informative features until a stopping criteria is reached. For every dataset, we use the stability criterion with parameter " and 6: the stability criterion is reached when the difference between two consecutive posterior entropies is below for к consecutive iterations. In practice, the user chooses the parameters for the stopping criterion by balancing the trade-off between the number of queries used to explain the prediction and the accuracy of the prediction: longer query chain can potentially achieve a better accuracy but at the cost of interpretability due to longer query-answar chains. On the extreme case, evaluated the test accuracy of Senncept-Q++-JJ. with all available query answers QX)) are used, which corresponds to the upper limit to how well Concept-QA V-IP can perform with respect to using variable number of queries. Next, compare Concept- QA+ A-I-'S accuracy with vision-only baselines to ascertain how much accuracy is sacrificed due to the constraint of interpretability (through query sets). For a fair comparison, we compare with CLIP's ViT-B/16 image encoder since the Concept-Qu -Ancept-QAnne network takes as input the image embeddings produced by this encoder. We evaluate the test accuracy of using Linear Probe on CLIP's ViT-B/16 (which solves a logistic regression problem using image embeddings of images as input features) since this was the metric reported the original CLIP paper (Radford et al., 202Tb. Moreover, we also compare with the accuracy the ViT-B/16 vision transformer achieves on each of these datasets (supervised training baseline). Table 2 shows that for all datasets Doucept-QA++a accuracy that is close vision-only baselines, with the exception of CUB-200 where appreciable gap remains (about 20%).
is able to achieve a test
Test Performance of Concept-QAAV.IP The second column shows the sizes of the query set. Expla- nation Length is the number of queries needed average over all test samples to reach the stopping criterion. Accuracy of Concept- PerepttQA+V+ QA+V-IP's prediction after the stopping criteria is reached given in column five. In column six we report the accuracy of Concept-QA+ Sareppt-Q+V-III V-IP using available query answers Q(X). Accuracy on pretrained CLIP VIT-B/16 using linear probe is shown in column seven. Finally, we report the supervised classification accuracy of the ViT-B/16 vision transformer in the last column.
Table 2:
Dataset
IQI
Stopping Criterion (7.,)
Explamation Length
Given (7..)
V-IP Avc.
V-I V-IPAsc Ave. Given Q(X)
CLIP VIT- Probe
VST B/16
CUB-200
208
Stability (0.127, 4)
24.7
67.0
70.8
80.0
90.6
Demidov et al
2023
CIFAR-10
Stability (0. 127,
5.5
95.6
95.7
96.2
99.0
CIFAR- 100
824
Staility (0. 127, 4)
21.3
77.3
80.2
83.1
91.9
Doovissky el al
21
ImageNet
4523
Stability (0. 127. 10)
49.6
68.2
73.8
80.2
84.2
(Dsoviiskyy
2120
Places365
Subility 127, 10)
32.1
(ingh et
B.2 EXAMPLES OF QUERIES IC
Recall, for every concept C, the corresponding query 9G is "Is concept C present in the image?"
See Figure 7
13

---

Published as a conference paper at ICLR 2024
1. blue sheen on the wings
7. orange and black wings
CUB-200
2. all-dark plumage
8. pale gray or white color
3. thin, black bill
9. red legs
black cap and white "eyeline"
10. yellow tips the wings
5. bright golden-yellow plumage fast, erratic flight patterns
11. black and white plumage
12. a small, black bird
1. a barn
7. taillights
2. beetle
8. living thing
CIEAR-IN
3. a driver
9. A cargo
4. long neck
10. a wet nose
5. a seatbelt
11. Feathers
6. four legs
12. a mosquito
1. alarm clock
7. large number of trees
CIFAR- 100
a bandit
8. a large, rounded bottom
bathing suit
9. a lobster trap
4. deer
10. a long stem with thorms
kcypad
11. a mirror
jungle
12. a phone book
small, round disk at one end
7. a buffalo
ImageNet
2. hitch for towing
8. erect ears
slock, metallic exterior spacecraft curved or V-shaped bottom dress
9. protective gear
10. doctor
11. a large, rotund body
6. pump
12. long, snake-like shape
1. paved with asphalt or concrete
7. a brightly lit interior
Places365
large, cabin-like structure
8. pipes
3. whitc or light-colored walls
9. has crater at the top
Hair
10. a gas cap
band
11. a sign that says "no dumping"
tennis court
12. a family of penguins
Figure 7: Sampled queries from the query set generated by GPT. Query generation process is discussed in
Appendix D
C DETAILS OF USER-ANNOTATED QUERY ANSWERS
Survey Creation. Each dataset is given query set Q and has 1000 images randomly selected from the test set. For each VQA system, all queries 9c E Q are evaluated on the images sampled. Given the set of all query answers, we sample 250 positive image-query pairs where the VQA system answers "yes" (gc(x) 1) and 250 negative image-query pairs where the answer is "no" (x)(x) = 0). Hence, given dataset and VQA system should have 500 image-query pairs with equal number of positive and negative query answers pairs. For evaluation, each VQA system is evaluated on the 2,000 image-query pairs collectively gathered from the four different VQA systems. Since the performance of VQA systems may vary query query, sampling processing ensures that the evaluation set does not favor any the VQA system by oversampling with positive/negative query-answer pairs from any single system.
14

---

Published as a conference paper at ICLR 2024
Survey format. Each evaluation requires sampled image z and query qc- The evaluator (the person performing the evaluation) is presented a picture of the plotted image and question "Is the concept {concept) present in the image?" title, with {concept) replaced by the concept being evaluated. The evaluator asked to answer only in yes or no. Two examples for evaluating ImageNet presented in Figure
concept "a sandy-colored coat" present the image?
concept "a scaly skón" present the image?
100
300
700
400
300
300
200
200
300
400
100
202
300
400
Figure 8: Two examples of pictures presented to the evaluator to evaluate the presence of a concept in a given image
Providing Labels. Combining the surveys for each dataset and for each model, there are a total of 10,000 individual evaluations of to answer. Every evaluation is completed by the authors of this work over the span of two weeks. average, each person spends about 20 mintues to perform 500 image-concept evaluations.
D IMPLEMENTATION: CONCEPT-QA
Query Set Generation. To obtain task-relevant and interpretable query set, we follow the practice
Oikarinen et al. (2023) to prompt GPT. We briefly reiterate the process of prompting GPT and filtering concepts below:
For a given dataset, we first query GPT-3 with the following prompts:
List the most important features for recognizing something as a (class)
List the things most commonly seen around a {class}
Give superclasses for the word {class}
where {class} is the name of a class. We query GPT with this prompt layout for every class. Then, number of filters are applied the the set of concepts generated improve the interpretability of the concepts. For instance, concepts with more than 30 characters are removed, and concepts that are too similar to class names are also removed. For filtering details, please refer directly to Oikarinen et al. (023). The resulting sizes for the set of concepts used in our following experiments can be found in Table 2 (second column to the left).
pseudo-labels from GPT. To obtain pseudo-labele from GPT, we use OpenAl's API to prompt GPT with the following lines of text:
Answer des/nd/ for whether the following concepts are salient for recognizing a (class)': (conceptl), (concept2), .... (conceptN). Output format: <concept>: <answer>: <explanation>, Answer as list.
where (class) is the name of the class, [concept1), fconcept2), ..., (conceptN] corresponds to the list of concepts generated, separated by commas, assuming there total of N concepts. We repeat this prompting process for every possible class of given dataset. The number of concepts N varies for each dataset.
15

---

Published as a conference paper at ICLR 2024
Architecture. The network architecture of the Concept-QA is the same for every dataset and diagram of its architecture shown in Figure 9 Inputs to the network are CLIP image embedding I(x) of a given image and text embeddings TC') for given concept C. The assume that T(x) and I(C) are fz-normalized. Output of the Concept-QA is a scalar-valued probability, indicating whether the concept C' is present in the image
Z(s] C
T(()) a
Unwor
Lnnor (294. 128)
Lnoor (299, 128)
1)
x
Figure 9: Architecture of Concept-QA. ReLU nonlinearity followed by a BatchNorm are used between each Linear layer.
Optimization. For each dataset, Concept-QA is optimized using the Binary Cross Entropy ob- jective. For CIFAR-10, the network optimized using Stochastic Gradient Descent (SGD) with learning rate 0.01, momentum 0.9 and weight decay 5 10 For CIFAR-I00, the network is also optimized using Stochastic Gradient Descent (SGD), but with learning rate 0.I, momentum 0.9 with Nesterov=True, weight decay 4-10-a For CUB, the optimizer is the same as CIFAR- 10, except momentum 0.9 with Nesterov=Fale with learning rate 10-4 weight decay 0, and amsgrad=True, For learning rate schedulers, each dataset uses a Cosine Annealing Learning Rate scheduler with T_max=200.
For ImageNet and Places365, the network optimized
E IMPLEMENTATION: V-IP
We specify the details of the the query answers, architectures and optimization methods used for training Concept-QA+V-IP in Appendix E.4 training CLIP+V-IP in Appendix E.I Meanwhile, the V-IP objective and the way we represent histories are stated below.
5.
Predictor
(Y (t)))) Update hetory
2. Bright blue colar?
3.
potteen?? Mediumasian gallsiae??
Querieer &,
Yes
gh-i 6 Q Next query
We Wi (aet Answer
{gr- ( Jix query answer chain
Append anower history
Figure 10: Overview of the V-IP algorithm.
V-IP Objective. V-IP was introduced by Chattopadhyay et al. (2033 as variational approach to IP which is very efficient in practice. V-IP defines predictor f P(Y) and a querier g: - Q that map
chain S of any finite length to distribution over Y (P(Y) denotes the set all possible distributions on Y) and query q E Q, respectively. Parameterized by deep networks 0 and 1), and 9n trained by sampling random query-answer chains and optimizing
16

---

Published as a conference paper at ICLR 2024
the following V-IP objective:
min 8,91
Ex Ex,s(DRL(P(Y) X) (X),
where
On := (S) Po (x), So((an 9n (x) US)))
where P(Y y X =) is the true posterior over Y given input X. Throughout this paper, conditioning on event S = should be understood as the set of all datapoints in X' that share the same answers to queries as in s. Both Concept-QA++ QA+V and CLIP+V-IP are learned by solving for 0 and 1y in the optimization problem stated in equation 7
Representing and Updating the History S. In V-IP framework, the input to predictor fo and querier 9n are both histories S. Similar to Chattopadhyay et al. (2023), the history for a sample x, S(x), is represented as the product of [Q -dimensional feature vector for all query answers Q(x) and binary mask M, i.e. Q(x) @ M, where @ here represents point-wise multiplication, also known as the Hadamard product. The i-th dimension corresponds to the query answer qi (z). We set M, to if the (x) & S and 1 if q, (x) S. Hence, history with all query answers has M equal to a vector of all ones, and an empty history is represented by a mask of all zeros.
Suppose have history for sample x of size k. (x)x) Q(x) MR. To update the history Sx with an additional query from the output of the querier (Sa), we simply update the mask and the history by:
Ma+1 Mj (Sk) and St+1 + Sk + Q(x) @ (Sx)
In other words. the new query is mathematically represented as a one-hot vector with dimension [Q], hence updating the representation of history is equivalent to updating the hinary mask M by setting i-th position 1, indicating that the (a()) the updated history (x+1
E.I TRAINING CLIP+V-IP
Query Answers (X).. Query answers are computed using CLIP dot-products. Again, we denote I(-) and T()) as the image and text encoder from CLIP. For any task and dataset, obtain the image embeddings of all images I(x) Vx E X' and T(4) Aq E Q. Each embedding is assumed to be fznoormalized, then the query answer qxx) = I(x) T(g). We further Z-score standardize each query answer by subtracting the mean and dividing the standard deviation of query answers from the training set:
(xx) -- ((x) -
ā i
ΣΣ x
((x).
o=
ΣΣ ZEX QEQ
(g(x) - A)
(8)
X
X
which can be computed easily with np mean and np. std.
For fair comparisons, use CLIP with ViT-B/16 16 backbone. Each image is preprocessed with CLIP's default preprocess function (Radford et al. 2021).
Architectures. In this work, only have two design choices for the predictor sa and querier Go- which we will denote them as shallow and deep. shallow is a two-layer fully connected neural network, and deep is four-layer fully connected neural network. Diagrams of their architecture are shown in Figure TT. The shal low architecture is used for medium-scale datasets CIFAR- 10, C00, CUB-200. The deep architecture is used for large-scale datasets ImageNet and Places365. The number of parameters for each experiment is listed in Table 3 The architecture design is chosen empirically based on the performance of the model. The size of the query set only affects the final output dimension of the querier 9n, and the number of classes affects the final output dimension of the classifier fo. In all of our experiments, we do not share the weights between fo and 9r, We apply Softmax operator to class logits and query scores to obtain probability for each class and probability for each query.During training, a Straight-through softmax with temperature parameter is used for computing query probabilities. For every experiment, we linear decay from 1.0 to 0.2 for 20 epochs|
We find V-IP training is rather insensitive to how we anneal T.
2 1.0 is equivalent regular Softmax operator, whereas corresponds to argmax(-) operator.
17

---

Published as a conference paper at ICLR 2024
Q(x) .M
M
@(x) M
A
Linerr 2000
shared
Linear 2000
Linear 2000
Linear 2000
Uieer 2000
< -h--harr----- >>
Linear 2000
Linear 2000
Linear 2000
Lineer 2000
ofweed
Linear 2000
Linar 2003
Linwar 2000
Concalensted
Concatonated
Cless Logits
Quary Scores
(a) deep architecture for (left) predictor fa and (right) querier
Q(X) .M
M
Q(X) M
M
Lineer 2000
Linear 2000
Linarr 2000
oahaned
Linear 2000
Concatensted
Concabenated
Cless Logits
Query Soores
(b) sha 1 ow architecture for (left) predictor fo and (right) querier Iu.
Figure 1I: Architecture designs for shallow and deep. "shared" implies the weights shared between the linear layers. "Concatenated" implies the output from previous layers concatenated R b Rix ca, b) [a|b] e R Rix(=+=)) Every arrow + before the concatenation and after the input layer is LayerNorm of appropriate dimension, followed by ReLU.
18

---

Published as a conference paper at ICLR 2024
of Parameters
CUB-200
CIFAR-10
CIFAR-100
ImageNet
Places363
CLIP+V-IP
3,051,408
2,596,138
5,773,924
41,659,523
29,441.572
Table The total number of parameters for each dataset.
Optimization. Every CLIP+V-IP experiment for CIFAR-10, 10AA-110, CUB- follow the two- stage training procedure mentioned in the Chattopadhyay et al. (2023), where we optimize the V-IP objective for Jo and Sn with random histories, by first using the Random Sampling strategy for 4000 epochs with learning rate 0.0001, followed by the Subsequent Biased Sampling strategy for 1500
epochs with learning rate 0.00005. In both stages, train using Adam Optimizer (Kingma & Ba 2015) with no weight decay, along with Cosine Annealing learning rate scheduler and hyperparam- eter T mammmax-..... 100.
Similarly, for large-scale datasets ImageNet and Places365, we also follow the two-stage training procedure. We use the same optimizer and learning rate scheduler with the same optimizer hyper- parameter as mentioned above, but we train the Random Sampling stage for 400 epochs, and train the Subsequent Biased Sampling stage for 40 epochs. In both stages, also train using Adam Optimizer (Kingma & Ba (015)) with no weight decay, but with Cosine Annealing learning rate scheduler and hyperparameter mma
E.2 TRAINING CLIP-BIN V-IP
The architecture and optimization exactly the same as described in $E.1 with the distinction that the query answers are now binarized by thresholding the standardized dot products from equation by 0. A positive value indicates "Yes" answer while a negative value indicates a "No" answer.
E.3 TRAINING CLIP-BIN V-IP
The architecture and optimization exactly the same as described $E.1 with the distinction that the query answers now binarized by thresholding (C [C=1] X =x) (equation 4) by 0.5. A higher value indicates a "Yes" answer while a lower value indicates a "No" answer.
E.4 TRAINING CONCEPT-QA+V.I
Query Answers Q(X). Query answers are computed using trained Concept-QA- To obtain binary query answers, we threshold the pre-sigmoid output logit of the Concept- Qaacept-QA, where the query an- swer (corresponding to a "Yes"" if logit is above the threshold and 0 (corresponding to a "No") if logit is below. The threshold is -0.4 for CUB-200 and Places365 and 0 for others.
Architecture. The network architectures for V-IP with query answers from Concept-QA are shown in Figure [2] Similar to the querier architecture in Chattopadhyay et al. (2023). the querier is fol- lowed by a Straight-through Softmax, hence the output is IQ dimmensional one-hot vector.
Table 4: Number of training epochs during Random Sampling stage (RS) and Subscquent Biased Sampling stage (SBS). The maximum lengths of history allowed during SBS (denoted as Max. (S]) are also listed. [Q] means for that dataset we used the size of the query set as the maximum allowed length of history for SBS.
Dataset
RS
SBS
Max. |
CUB-200
200
200
Q
CIFAR-10
200
100
Q
CIFAR- 100
200
200
[Q]
Places365
1500
80
500
ImageNet
600
100
500
Optimization. For every dataset, training Concept-QA+V.I follows the two-stage procedure as mentioned the Chattopadhyay et al.) (2023), where we optimize the V-IP objective for fo and Iy with random histories. In the Random Sampling Stage, the classifier so and the querier In,
19

---

Published as a conference paper at ICLR 2024
Q(X) M
Q(X) M
Linear 2000
Linear 2000
Linear
Linear 500
Query Logits
Class Logits
(a) Classifier fo
(h) Querier 9n
Figure 12: Architecture for Concept-QA+V-II Between each linear layer is ReLU nonlinearity followed by LayerNorm.
optimized joining using Adam optimizer with learning rate 0.0001 and amdgrad=True, while learning rate is scheduled using Consine Annealing with T_max=200. In the Subsequent Biased Sampling stage, the classifier fe is optimized using SGD with learning rate 0.0001, Nesterov's mo- mentum 0.9 and weight decay 5- -4, while learning rate is multi-step scheduled to decay at epochs with multiples of 30 by factor of 0.2. On the other hand, querier 9, is optimized with Adam with learning rate 0.0001 and no weight decay, while learning rate scheduled with Cosine Annealing with T ma0x Moreover, during Subsequent Biased Sampling, of the history for computational purposes. The number of epochs for each dataset during the two training phases, as well as the number of maximum length of the history allowed during training listed in Table 4
limit the maximum length
E.5 TRAINING BLIP2-FLANT5+V-IP
The architecture and optimization exactly the same as described $E.4 with the distinction that the query answers are now supplied by the pre-trained Vision-Language Model BLIP2 ViT-g FlanT5xxL
F.6 ORTAINING PGPT FROM LLAMA2
To obtain pseudo-lahels from language model Llama2, provide the instruction prompt and user prompt.
Llama2. We requested and downloaded the open-source weights for Llama model from Meta. We use the chat 1lama-2 2-13b 13b--bbom chat model in a zero-shot fashion (without any finetuning). The instruction prompt is follows:
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, uncthical, racist, sexist, toxic, dangerous, illegal content. Please ensure that your respenses are socially unbiased and positive in nature.
whereas the user prompt the same as that for GPT from Appendix D For hyperparameters, we set temperature to 0.6 and top-p probability for nucleus sampling to 0.9.
F EVALUATING THE FAITHFULNESS OF OUR PSUEDO-LABELS
We evaluate the quality of our pseudo-labels by computing their accuracy and Fi-score of over the 5 image classification datasets considered this paper. We also compare with baseline methods which state-of-the-an VLMs that present alternative approach to directly get pseudo-labels for training the Concept-QA model. Specifically, we compare with InstructBLiP (FlanT5xx.) (Dai al. 2023) and the I.LaV LL- Va-1a11155 1.55g models (Liu et al. 2024). For InstructBIIIP (Flan'55x.) we the
20

---

Published as a conference paper at ICLR 2024
same prompt in footnote 1 of the main paper. For LLaVa-1.57g present in the image? Answer only Yes or No.", where concept replaced by the name of the concept interested in asking the VLM about. Furthermore, compare the quality of pseudo- labels generated when using a different L.M (LLaMa2 (Touvron et al. 2023)) instead of GPT in For more details refer Appendix E.6
the prompt "Is concept
equation 5
We evaluate all four mod- els on the same anno- tated data of image-coneept pairs as used Table [ The results of this experi- ments are reported in Ta- ble 5 where "GPT+CLIP" refers to the pseudo-labels used in training our Concept-QA method in this work (specifically equa- tion 5). Similarly. "LLaMa2+CLIP" refers to pseudo-labels generated using equation [S] when LLaMa2 is used as the L.M in place of GPT. From the results we make observations. Firstly, we note that using GPT as the LLM provides far superior pseudo-labels than ILaMa2 on all the datasets employed. Secondly, on all datasets our pseudo-labels (GPT+CLIP) are more faithful at representing the true ground truth answers than the LLaVa model while being only slightly behind when compared to InstrucIBLIP
Table 5: Results for evaluating the faithfulness of different pseudo-label gen- crating mechanisms to the true answers. Acc. and F1 refer to accuracy and F Fi-score metrics respectively.
Pscudo-labelling method
ImageNet
Places365
CUB-200
CIFAR-10 10
CIFAR-IPE 100
Fi
Acc.
F1
FL
Acc.
GPT+CLIP
0.86
0.55
0.82
0.42
0.79
0.53
0.81
0.62
0.79
0.37
LLaMa2+CLIP
0.46
0.47
0.45
0.84
0.35
InstruciBLIP (FlanT5aa
0.85
0.58
0.86
0.66
0.68
0.46
0.82
0.62
0.84
0.40
LLaVa-L.5yp
0.63
0.41
0.66
0.47
0.49
0.46
0.69
0.52
0.66
0.25
However, as discussed in + in the main paper, using these VLMs for training Concept-QA model infeasible since they are computationally very intensive. For context, Imagenet has about 1.2 million images in its training set and the corresponding query set (obtained from GPT) has about 4.5K queries. In order to generate pseudo-labels for training our Concept-QA model we need to answer all queries about all images in the training data. This would require about 37K GPU hours using the InstructBLIP (5xa) model (cvaluated on NVIDIA RTX A5000 GPU, by averaging the time taken for computing a batch of 100 image-query pairs and then multiplying by the size of the training set and the query set) which is simply infeasible. In comparison, our GPT CLIP methodology generate psuedolabels (see equation takes about 600 GPU hours. The bulk of this computation time comes from using GPT, which takes about 1-2 minutes to answer batches of 200 queries per class. Compared the VLM, this is massive speed-up! In Table 6 taken to generate pseudo-labels using our method vs. the time taken to annotate all image-query pairs using the InstructBLIP (FlanT5xL) model for all 5 datasets employed. We do not compare with LLa Va, since our pscudo-labels outperform LLaVa on both metrics on all 5 datasets. However, since both LLL and InstructBI.IP (Flan'S5xu) have the number of parameters of the same order (7B vs. 4B respectively), their evaluation times are of similar magnitudes.
we report the time
Table 6: Time taken annotated the enture training dataset with query answers using different pscudo-label generating mechanisms. Numbers in GPU hours.
Dataset
GPT+CLIP (Ours)
InstructBLIP (FlanT5xx.)
CIFAR-10
0.2
CIFAR-100
12
266
CUB-200
20
6
Places365
83
25,650
ImageNet
600
37,344
'with the exception of CUB-20X and Places365. On CUB-200 our pseudo-labels perform much better due to the domain-specinte nature of this dataset which is too specialized for these ceneral-purposs. VLMs to have good zero-shot performance. On Places365, InstructBLIP achieves much higher F Fi-score than our pseudo- labels. We hypothesize that this is due to CLIIP's image encoder not heing able capture essential image semantics for this dataset discussed ( (iiccusion of results for Table
21

---

Published as a conference paper at ICLR 2024
G ADDITIONAL EXAMPLES OF QUERY-ANSWER CHAINS
We illustrate more test examples of query-answer chains using VonC and C-IP+V-IP on CUB-200 13 CIFAR-10 CIFAR-100 ImageNet [4 Places365 15 For datasets with more than 10 classes, we show only the classes with the top 10 probability class where IP terminates. For examples using CLIP+V-IP, green/red query (y-axis of the plots) represents whether the query- answer is above/below O. For examples using Concept- Concept-QA+ViIl QA+V-IP the green/red query represents whether query-answer is yes/no. See caption of Figure for description of the heatmaps.
H LIMITATIONS & FUTURE DIRECTIONS
Following are the limitations of our work.
Prior work V-IP required the user annotate datasets with query-answers which could be used to supervise the training of classifiers to provide answers at inference time. In this work, to overcome this data annotation bottleneck propose use GPT + CLIP to provide psuedo-labels to train our answering mechanism (the Concept-QA model). This inevitably leads loss in accuracy since we rely on the zero-shot performance of these large language models and vision language models. Future work would aim to address this limitation by exploring hybrid approach where we manually annotate a small portion of the data and use it in tandem with the pseudo-labels to train the Concept-QA QA model.
The current approach is limited to the problem of image classification due to the assump- tions involved in deriving our pseudo-labels, that is, the class object is the focal point of the image (mentioned in Section 3.2.3). Future work would be aimed at exploring other ways of generating pseudo-labels to extend the application of V-IP to more general vision tasks such as image captioning, scene segmentation and visual reasoning to name a few.
22

---

Published as conference paper at ICLR 2024
CLIP+VIP: VP with CLIP dot products as answers
Input Image abr
Concept QA+V with Concept QJ answers
Enery
L poematyysiin body 1700
3-1e corpled
2.a
sambbrs 2449
Meck ineeecttionn wer 2233-
prawiia aturage
Cruck
lay
(a) True class: Pine Grosbeak
CUP+V-IP: VPР with CUP dot products answers
Input Imagex***
Concept QA+V IP with Concept QJ answers
1. greetshyytioo body
088
2.3 redd heac:
seutentt 1 /22
teak and leps Mirck
4. small spamoeibe bhe tody a 5030
slack and plamage: 4357
thrats
02
pelliu thoir
groerad pellow huih
Besterone Cridd
02
yolltutann
10. 1its biaccill
feeet
feprit stec
sheriee
(b) True class: Hooded Oriole
CLIP+VIIP ViP with CLIP dot products as answers
Input Image Ad
Concegt-QA+V-P with Concept-QA answers
1.00
1.a small songhird
2. pontee bill
3 seabld
beat and legs black
4. smrl aparne- body. 8.4236
025
undertal feathers
Mox sppurpets 0.443
that beailtge
Blown dliseen Piom
0.00
a thin, black
small,
Rasty Blee
dark blue gray usporparts
firged
white threat and breast
cap and
wingtip
a swift. acrobatic fight brcemien plumage
0,0
Now Witeeo
Oh Hha
Crrestru unted Warliee
cain Thrres Mecotttenck
to
vireo
(c) True class: Eastern Towhee
CUP+V-IP: V-P with CUP dot products answers
Input Image 282
Concept QA+V IP with Concopt QLA answers
Jan 2015
Mact
S wewaeeenaaaaaaa 43912
uane 2aste-
66
S. arti, Bicce
6. alttmee becy than crows
underaal featers
3. to0 beree bab A4De
6. yolllosth bally
thrt artliii
creatishyyel yelloo
march
tivkk
co
1t. grean Biliye ya los beck vinga
Crange cooeere Wertier
(d) Class: Kentucky Warbler
Figure 13: Additional Examples from CUB-200.
23

---

Published as a conference paper at ICLR 2024
CUP-VIP: CLIP
GA-V-IP
Nartmal
sng
Honday
hergs
kidg sore cuuooo
Nalme
short,
2.. rulh dyyy manwee
15. aroee body adfiddacc flot
(a) True class: Armadillo
Input 2000
Concept-QAiv-B Concept-QA
cee cocorrti 141.
veria cokes
appinal
cartect
0.7
23 fargi roudd corpp
1%
62
(b) True class: Pinwheel
CLIPV-IP:
answers
Input Image
Concegt Concept answers
e-tand moust
LACH
larg Imposing diat
cinttar
(e) True class: Studio Couch
CUP-A-IP: Vi with dot products
Irput Image die
Conceps-QArV.R with Concept-QA anowers
Mareral
ANt244 calor
sppand
Fung
20,
moon snc
0.4
12 herr. sher oxsilaation
10, Food sourc
(d) True class: Artichoke
Figure 14; Additional Examples from ImageNet.
24

---

Published as conference paper at ICLR 2024
CLIP-V ViP with CLIP dot products answers
Input Image x
Concept QA+V-IP with Concegt QA answers
Dnaty
perten of bing
lage lurge deens 31300
agg,
64
6.305
airgs eipasss of 1,2004
cosalinet 1,553
redy bim
11. outlat: 2.00
63
Dreeting
lawe
(a) True class: Baseball Field
CLIP+V-P: V-P with CLIP dot products answers
Input Image gale
Concept-QAAV-BB with Concept-QA answers
Drioo
1.0
1. canden in te bull
large hanger
mosseies
60
0.350
fightrai penter 200141
gite ouals ouulsssoetts suatte 6.778
5. casher- 1,4200
stroge ded
landing
canat sen 2.20
parch- 1.100
Sanden Ploce
man-maadee dtructure
17 canatuly maniunr lawn 4u200
Sack.
DA plogronnt pement 1,200
00
(b) True class: Landing Deck
CLIP+VII: V-P with CUP dot products as answers
Input Image 2
Concept- QA+VIP with Concept QA answers
Erony
10
1. genden top a batting- 1006
L. soprs
larg soom narger soore 2003
6 was ountoar twsthmm 3367
noom with maguties 2519
3. verue
5. Sacaaaalee 5455
luble 5554
foed and chad served
7. oventhe
playomard dods (860
6.4
2.4
0.2
vautery
zo. hebriaa
clour tramauuont watl
proceiing
02
nabica hasorr pour
loos kapphex
sartersiing jood
unntorattes
oifice
aoazs louding up pands
80
(c) True class: Aquarium
CUP+V-IP: V-P with CUP dot products as answers
Input Image goba
Concept-OA+V-BB with Concept-QA answers
mours
16
garden kp ting: 4.134
water
mmnge thet Go00
cot indiatee dosrt pires Curr
sags eooie antiars a.t0e
struttue
2 atage urched 20004
mamemade atrucaae
32
anderaain
angs arthind fomeatio
Mace seca
(d) True class; Roch Arch
Figure 15: Additional Examples from P6aces365..
25

---

Published as conference paper at ICLR 2024
CLP+AHP:: 14P with CLIP dot products as answers
Input Image x
Concept QA+VIIP IP with Concept QJ answers
1. rist
2: cocosieeeeee
1. siow mrvvement
truit bowl
05
5.
04
cryplic
much
symmetyy
an acom
02
long. cytind shups
long, slonder trunk
0.4
0. trannuaar shape
D. lange numbur of broo
portel Pappred
10, biomo
pointed and
generally
suunpe cole
labby prosase
(a) True class: Apple
CUP+V-IP: V4Р with CUP dot products answers
2bs
Concept-OAvy-P with Concept-QA answers
Init
1. 3km movarnntt
motorcyce
0.6
4. cauoer
7. muticll doos
Istenal symmetry
smmpi, unadarnes dea
a CHERTY spout
uneven sur
I1 simm cook-
15, but
d. large. rounced bottom
12.2
0,2
13. rosch mate
14,
11. coft surfiie
courte
bustre
Coain
comertter chaar
Sontle
(b) True class: Bed
CLIPA4PP: V-P with CLIP dot products as answers
Input Image git
Concept QA+V IP with Concopt QA answers
10
kone
pion movemunt
centarns
2. toness
surface
sky
litaral symmety
Dewars
0.6
AG
fumiture
unedomed dea
mitl
device
7.
0.4
clnt
B. alarm docc
rounded
child
boxt
0.2
rupped.
11. Simple, unadomed design
Artyoeets wither <ide
liee
choic)
feleviiion
bay
15. dinder
(c) True class: Chair
CUP+V-IP: products
Input Image o
Concept-QANV-IP Concept-OA
Rone
Init
J. hul bow
2. cuntains
3.
3. large. surface
5. stem
6.4
atmole unademand design
anow capped paaks
6. large number trees
10. koniou
engne
C2
c. usually crypti coloration
1. milltry ven cle
6.0
622
11. aiome
stom
12. traanguuu shap
lanoe
de eeeernpee
(d) True Class: Skyscraper
Figure 16: Additional Examples from CIFAR- 1-00
26

---

Published as a conference paper at ICLR 2024
CLIPAVIP: V-P with CLIP dot products as answers
Input Image Ahs
Concept-QA+V-IP with Concept-QA answers
100
1.00
In
small, lithe body quqqaduuedd
engines wings: 2.923
3. a.amm 0.8817
watertruft
4. bed head igtte: 0.9299
025
0.25
4, ruddor
5. bralen 1.4005
mop
budk
0.00
0.00
simptones
cats
deaf
dogs
mrgg
harses
5tips trucka
5. trailer
True class: Trucks
CUP+W-IP: products
Input Image
Concept-QAvy.in Concept- answers
1.0
quadruped
linding yeir. L.045-
0.00
arplenes
beds
deer
days
teeg
mute
(b) True class: Airplanes
Figure 17: Additional Examples from CIFAR-10.
27


Reviews:
[{"review_id": "mZsQGulbci", "reviewer": "Reviewer_8xTQ", "summary": "This paper introduces a novel methodology for training a Concept Question-Answering system (Concept-QA) that determines the presence of a concept in an image using binary answers. This approach eliminates the need for manually annotated training data, relying instead on pseudo-labels generated by GPT and CLIP. The authors empirically validate Concept-QA on various datasets, demonstrating its accuracy in representing true concepts.", "questions": "While the paper highlights the advantage of not requiring manually annotated training data, it could delve deeper into the data efficiency aspect. Does Concept-QA require a large amount of unlabeled data to perform well, and how does its data efficiency compare to alternative approaches?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper introduces a methodology for training Concept-QA that doesn't require manually annotated training data, reducing the burden of data annotation.\n- Experiments on multiple datasets demonstrate the proposed method's effectiveness.\n- This paper is well-written and easy to follow.", "weaknesses": "- It would be better to try more LLMs to show the generality of this idea.\n- Limitations could be discussed. Is there anything about **the proposed method itself** that fails in certain scenarios or falls short compared to prior work? \n- While the paper mentions the use of pseudo-labels from GPT and CLIP, it could discuss the interpretability and potential biases associated with these labels. Are there instances where the pseudo-labels might lead to incorrect or biased answers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZEGegJRgpA", "reviewer": "Reviewer_8mpd", "summary": "The focus of this paper is to revisit \"Variational Information Pursuit (VIP) for image classification. VIP is an interpretable framework that makes predictions by sequentially selecting Question-Answer chains to reach the prediction. The paper states that VIP has limitations as it requires specification of a query set and densely annotated sequential QA data.  To relieve this limitation, the paper proposes that language models could be leveraged to generate queries for image classification, and a QA network is proposed (ConceptQA) to answer binary questions about the image. \n\nFindings: Experiments are conducted on 5 image classification datasets and compared to 3 vision-language models (VLMs). The model outperforms the baselines on 2/5 datasets and is second best on the others.", "questions": "1. Given that the method works by using QA -- could it also be useful for improving VQA performance (eg. for VQA-v2, VizWiz, GQA, CLEVR etc datasets).\n2. Do humans prefer shorter or longer explanations when it comes to trusting a model? While the tradeoff in figure 5 is a good finding, I'm not sure how it translates to helping humans understand decisions made by the model.\n3. Similarly, when humans know that the prediction has failed -- do they prefer shorter or longer explanations to debug why it failed? This study could be added.\n\nComment:\n1. I'm going to be \"that\" person and say that we should really avoid the marketing lingo of \"Foundation models\" in scientific papers -- who gets to decide which model is a foundation model and which isn't? Why is CLIP a foundation model but ResNet / RNN isn't?  The term \"Large\" language model (like \"deep\" learning) is also dicey, but at least we could define \"large\"/\"deep\" in terms of some threshold on number of trainable parameters/layers in the model. Is there really a definition for foundation models -- or is it just a marketing trick that everyone is accepting? I would recommend replacing \"Foundation Models\" in the title with \"Language Models\" -- it is more informative (because the paper relies on prompting GPT with natural language)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This is an interesting take to use VIP / sequential QA for image classification and the experiments demonstrate the effectiveness of this method for interpretability/explanations.\n2. In general the idea could be useful beyond image classification for other tasks with data-scarce settings, especially unlabeled datasets, where language models can help to create pseudo labels.\n3. The paper put together (main + appendix) is well written and explains the preliminaries, proposed method, implementation, and experiments in good detail.", "weaknesses": "1. Comparisons to vision-only models are missing: it would be useful to know how much of a gap there is between supervised as well as self-supervised vision models trained on these datasets. This comparison would tell us how much performance is being sacrificed for interpretability. Comparisons in terms of #parameters are also important in this regard.\n2. The method only uses OpenAI's GPT models (which are not open-source but only accessible via API calls -- i.e. need an internet connection and OpenAI account for inference) -- it would have been better to also implement the method with a local language model -- it is understandable that the performance could potentially be lower than GPT.\n3. A human study could have helped to supplement the study on explanation length. See Q2 and Q3 below.\n4. Useful details and useful visualizations are relegated to the appendix -- for instance Fig 9 could be moved to the main paper, some of the details on query set generation (App C) and training process (App D) could be briefly added in the main paper to improve the flow/readability of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dmvezA5oxj", "reviewer": "Reviewer_ZeKa", "summary": "Background: Variational Information Pursuit is an approach to generate interpretable explanations that leverages densely annotated data. \n\nKey idea: Overcome the bottleneck of dense-human-labeling by automatic labeling using an LLM. \n\nApproach: Start from an existing list of “semantic concepts” (queries) generated by GPT. Get pseudo-labels generated from CLIP + GPT. QA model provides the answers. The QA model is trained on the GPT queries and  Together the query-answer pairs serve as explanations.   \n\nContributions:\n\n- Lightweight “Concept-QA” model\n- Generated explanations (query-chains) are shorter, more accurate than with baseline QA model (CLIP-similarity score).", "questions": "1. The overall approach is inspired by work on label-free CBMs and the important modules (query generation, answer-generation and concept-qa) are reasonable. However, it is a bit difficult for me to articulate precisely what are the novel scientific ideas or findings from this paper. It might be nice if the authors could reiterate this.\n\n2. The main baseline for the proposed Concept QA model is binarized CLIP after normalization. There’s an assumption here that the value of “0” is a good threshold. Were there experiments performed to identify the optimal similarity threshold for accurate classification? For instance, the min-max normalization described in 3.2.3 might be a good starting point.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Performance: Concept-QA appears to perform well when evaluated along multiple axes. The results are highly correlated with concepts that are actually present in the images; i.e., there seems to be very little confabulation (often called “hallucination”). In addition, the explanations (query-answer chains) are shorter and more human-interpretable.\n\n2. The key contribution of the paper appears to be the formulation of the Concept-QA model based on query information and answers from GPT + CLIP. The paper demonstrates that naively using CLIP-score between (query-concept, image) does not work well out-of-the-box, and proposes learning a new light-weight network based on pseudo-labels.\n\n3. Impact: The labeling requirement was a huge bottleneck. With this approach, that requirement doesn’t exist anymore. It paves the way for more widespread application of VIP in scenarios where interpretable-by-design approaches are critical. However, we should credit the core idea and (part of the implementation) to the earlier work on label-free CBMs.", "weaknesses": "1. The framework of Variational Information Pursuit is quite similar to the Concept Bottleneck Models. The main difference is the sequential selection of informative queries (concepts) in VIP. The key contribution of the paper is the approach to overcome the limitation of annotating  query sets and labels. However, this was already proposed and implemented by an earlier paper (Oikarinen et al.).\n\n2. Sec. 3.2.2 presents the observation that choosing a set of queries from the dataset a-priori (agnostic to the image), does not result in either an optimal or interpretable query set. This is understandable, since two main information sources are ignored — the image content, and taking advantage of the answers to the queries. Considering the actual experimental setting is significantly different from the VIP setting, the title “Answering Queries with CLIP Adversely affects VIPs explanations” and the conclusions seem a bit misleading. While floating-point-dot-product-scores from CLIP might also be a problem (as discussed in Sec. 4.1), the more obvious problem in this setting appears to be the query selection strategy. Please let me know if I have misunderstood something.\n\n3. The proposed ConceptQA architecture is intuitive and quite straightforward. While this is not a downside by itself, it is probably something that one implement as a baseline. It is a bit hard to identify the interestingness or novelty in the approach. The closest baseline is simply a comparison against a CLIP(image, concept) similarity score.\n\nReferences:\n- Oikarinen et al.: Label-Free Concept Bottleneck Models; ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4zbVe48Iji", "reviewer": "Reviewer_3dNV", "summary": "This paper focuses on the problem of variational information pursuit (V-IP). The focus of V-IP is to generate the final answer by answering a series of interpretable queries/questions, facilitating the interpretability of model prediction. In this paper, the authors focus on a challenging problem of how to train a model to answer the interpretable queries as those queries are typically auto-generated and have no associated ground-truth labels. The authors thus proposed to generate pseudo-labels with the aid of GPT and CLIP.\n\nThe authors empirically show that the proposed Concept-QA model leads to shorter and more interpretable query-chains than the CLIP baseline.", "questions": "Please see the questions in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to follow. The authors explain their proposed method and give background knowledge about V-IP in an easy-to-follow way. More technical details are also explained and provided in the appendix.\n\n- The designed experiments indeed show that the proposed Concept-QA helps the whole V-IP improve interpretability and results in a shorter query chain compared to the CLIP baseline.\n\n- Fig 4 illustrates how using the proposed Concept-QA improves the interpretability of query chain.", "weaknesses": "- The major (if not the only) competitor of the proposed Concept-QA is CLIP baseline. It would be good to include other baseline methods for comparison.\n\n- What the quality of the pseudo-label when evaluated on the Table 1 experiment?\n\n- Since there exist many vision language models (commercial or open source), there might be other easier and more effective ways to generate pseudo-labels for training the Concept-QA model. For example, prompt a LLaVA or InstructBLIP model. The authors should compare to other pseudo-label generation methods as this is part of the major contribution of this paper.\n\n- What's the correlation between the performance of the QA model and the final V-IP system? In the paper, the authors $\\textbf{imply}$ that a better QA model leads to better overall V-IP performance, without supporting experiments and evidence. This weakens the connection between the results in Table 1 and the overall V-IP problem setting. For example, the author may show that using BLIP2 on Places 365 or CIFAR-100 brings better V-IP performance (improved interpretability of query chain, shorter query chain, etc). The reviewer is aware that the computation of BLIP2 is much more expensive than that of Concept-QA. The point here is to build the connection between QA model performance and V-IP performance.\n\n- It would be good to come up with a way to quantify the interpretability and the length of query chain so that the reader can be more confident that the proposed Concept-QA indeed improves over CLIP baseline in these two aspects. For example, a few qualitative results in Fig 4 and the appendix might not be representative enough and might be cherry-picked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)