SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} [ \mathrm {S N R} ] = 1 0 \log_ {1 0} \mathbb {E} \left[ \frac {\| D _ {o} \beta_ {o} \| _ {2} ^ {2}}{\| e \| _ {2} ^ {2}} \right] \mathrm {d B}.
$$
Expected SNR values were swept from 5 dB to 20 dB in 5 dB intervals. Thus, for each  $(m,n,s,\mathbb{E}[\mathrm{SNR}])$  setting, we generated 1000  $(D_o,\beta_o,x_o,e)$  tuples and ran OMP and IP-OMP
Recall that the NMSE was defined in equation 9.
Figure 7: Further numerical results on sparse recovery where the signal is corrupted by noise. These result add to those in Figure 3. On the  $y$ -axis we report  $\mathrm{NMSE}(\hat{\beta}_{\mathrm{Alg.}}, \beta)$  for OMP (lines) and IP-OMP (overlaid circles). On the  $x$ -axis, we report the number of measurements (signal dimension)  $m$ . Each value of the sparsity  $s$  corresponds to a different color.
for  $s$  iterations. Finally, for each trial, we measured the quality of the estimates  $\hat{\beta}_{\mathrm{Alg}}$ , by their NMSE as defined in equation 9. The results for  $\mathbb{E}[\mathrm{SNR}] \in \{5\mathrm{dB}, 20\mathrm{dB}\}$  are shown in Figure 3 in the main paper and the results for  $\mathbb{E}[\mathrm{SNR}] \in \{10\mathrm{dB}, 15\mathrm{dB}\}$  are reported in Figure 7.
Now we present a proposition that gives a formula for  $\mathbb{E}[\mathrm{SNR}]$  in terms of  $m$  and  $s$  when  $D_{o}$ ,  $\beta_{o}$ , and  $e$  are distributed according to our above described sampling process. For technical consistency, we denote  $\mathcal{D},\mathcal{B},E$  as the random variables and  $D_0,\beta_0$  and  $e$  as their respective realizations.
Proposition 1. Let  $\mathcal{D}$  be a  $m \times n$ -dimensional random matrix with columns distributed independently and uniformly on the unit sphere. Let  $\mathcal{S}$  be a random support of size  $s$ , chosen uniformly from all size- $s$  subsets of  $\{1,2,\ldots,n\}$ . Let  $\mathcal{B}$  be a random  $s$ -sparse vector with support on  $\mathcal{S}$  and with coefficients zero-mean, unit-variance Gaussian as described in § B. Let  $E \sim \mathcal{N}(0,\sigma^2 I)$ . Then for the noisy measurement model  $\widetilde{X} = \mathcal{D}\mathcal{B} + E$ , we have
$$
\mathbb {E} [ \text {S N R} ] := \mathbb {E} _ {\mathcal {D}, \mathcal {B}, E} \left[ \frac {\| \mathcal {D B} \| _ {2} ^ {2}}{\| E \| _ {2} ^ {2}} \right] = \frac {s}{\sigma^ {2} (m - 2)}, \tag {23}
$$
or on  $dB$  scale,
$$
\mathbb {E} [ \mathrm {S N R} ] = 1 0 \log_ {1 0} \left(\frac {s}{\sigma^ {2} (m - 2)}\right) \mathrm {d B}.
$$
Proof. Since  $E$  is independent of  $\mathcal{D}$  and  $\mathcal{B}$ , we can split the expectation in the definition of the expected SNR into
$$
\mathbb {E} [ \mathrm {S N R} ] = \mathbb {E} \left[ \| \mathcal {D B} \| _ {2} ^ {2} \right] \mathbb {E} \left[ \frac {1}{\| E \| _ {2} ^ {2}} \right].
$$
We will first compute  $\mathbb{E}[\| \mathcal{DB}\| _2^2 ]$
$$
\begin{array}{l} \mathbb {E} \| \mathcal {D B} \| _ {2} ^ {2} = \mathbb {E} \left[ \mathcal {B} ^ {T} \mathcal {D} ^ {T} \mathcal {D B} \right] \\ = \mathbb {E} \operatorname {t r} \left(\mathcal {B} ^ {T} \mathcal {D} ^ {T} \mathcal {D} \mathcal {B}\right) \\ = \mathbb {E} \operatorname {t r} \left(\mathcal {B} \mathcal {B} ^ {T} \mathcal {D} ^ {T} \mathcal {D}\right) \\ = \operatorname {t r} \left(\mathbb {E} \left[ \boldsymbol {\mathcal {B}} \boldsymbol {\mathcal {B}} ^ {T} \right] \mathbb {E} \left[ \boldsymbol {\mathcal {D}} ^ {T} \boldsymbol {\mathcal {D}} \right]\right). \tag {24} \\ \end{array}
$$
Rewriting as an inner product, we have
$$
\mathbb {E} [ \mathcal {D} ^ {T} \mathcal {D} ] _ {i j} = \mathbb {E} \left[ \langle \mathcal {D} ^ {i}, \mathcal {D} ^ {j} \rangle \right] = \delta_ {i j},
$$
where  $\mathcal{D}^i$  is the  $i^{\mathrm{th}}$  column of random matrix  $\mathcal{D}$  and  $\delta_{ij}$  is the Kronecker delta. The last equality if obtained by observing that since  $\| \mathcal{D}^j\| _2^2$  is a constant one and  $\mathcal{D}^i,\mathcal{D}^j$  are independent for  $i\neq j$ . We thus conclude that  $\mathbb{E}[\mathcal{D}^T\mathcal{D}] = I$ .
Notice that  $\mathcal{B}$  depends on the random support  $S$ . Furthermore, since  $\mathbb{E}[\mathcal{D}^T\mathcal{D}] = I$  and
$$
\operatorname {t r} \left(\mathbb {E} \left[ \boldsymbol {\mathcal {B}} \boldsymbol {\mathcal {B}} ^ {T} \right] \mathbb {E} \left[ \boldsymbol {\mathcal {D}} ^ {T} \boldsymbol {\mathcal {D}} \right]\right) = \operatorname {t r} \left(\mathbb {E} \left[ \boldsymbol {\mathcal {B}} \boldsymbol {\mathcal {B}} ^ {T} \right]\right),
$$
it suffices to consider the diagonal entries of  $\mathbb{E}[\mathcal{B}\mathcal{B}^T]$ :
$\mathbf{\sigma} = \mathbb{E}[\mathbf{1}_{\mathcal{S}}(i)]$  1s(i) is the set indicator function for random set S
$$
\begin{array}{l} \mathbb {E} \left[ \boldsymbol {\mathcal {B}} \boldsymbol {\mathcal {B}} ^ {T} \right] _ {i i} = \mathbb {E} \left[ \mathbb {E} \left[ \boldsymbol {\mathcal {B}} \boldsymbol {\mathcal {B}} ^ {T} \mid \boldsymbol {\mathcal {S}} \right] \right] _ {i i} \\ = \mathbb {E} \left[ \mathbb {E} \left[ \mathcal {B} _ {i} ^ {2} \mid \mathcal {S} \right] \right] \\ = \mathbb {P} [ i \in \mathcal {S} ], \\ \end{array}
$$
where the third equality results from the fact that coefficients  $\mathcal{B}_i$  on the support (i.e.,  $i\in S$ ) are Gaussian with zero mean and unit variance. In the fourth equality  $\mathbb{P}[i\in S]$  is the probability that a randomly drawn subset  $S$  would include element  $i$  in it. To compute  $\mathbb{P}[i\in S]$ , we count the number of cardinality- $s$  subsets of  $\{1,2,\dots,n\}$  containing  $i$  and divide by the total number of cardinality- $s$  subsets. This is described in detail in the following equations where we use  $S$  to denote the different values  $S$  takes on the sample space.
$$
\begin{array}{l} \mathbb {P} [ i \in \mathcal {S} ] = \frac {\left| \left\{S \in 2 ^ {[ n ]}: | S | = s , i \in S \right\} \right|}{\left| \left\{S \in 2 ^ {[ n ]}: | S | = s \right\} \right|} \\ = \frac {\binom {n - 1} {s - 1}}{\binom {n} {s}} \\ = \frac {(n - 1) !}{(s - 1) ! (n - 1 - s + 1) !} \frac {s ! (n - s) !}{n !} \\ = \frac {(n - 1) !}{(s - 1) ! (n - s) !} \frac {s (s - 1) ! (n - s) !}{n (n - 1) !} \\ = \frac {s}{n}. \\ \end{array}
$$
$$
\mathbb {E} \left[ \boldsymbol {\mathcal {B}} \boldsymbol {\mathcal {B}} ^ {T} \right] _ {i i} = \frac {s}{n}, \quad i = 1, \dots , n.
$$
Conclude that
$$
\begin{array}{l} \operatorname {t r} \left(\mathbb {E} \left[ \boldsymbol {\mathcal {B}} \boldsymbol {\mathcal {B}} ^ {T} \right] \mathbb {E} \left[ \boldsymbol {\mathcal {D}} ^ {T} \boldsymbol {\mathcal {D}} \right]\right) = \operatorname {t r} \left(\mathbb {E} \left[ \boldsymbol {\mathcal {B}} \boldsymbol {\mathcal {B}} ^ {T} \right]\right) \\ = \sum_ {i = 1} ^ {n} \frac {s}{n} \\ = s. \tag {25} \\ \end{array}
$$
Rewriting  $\| E\| _2^2$  as
$$
\| E \| _ {2} ^ {2} = \sum_ {i = 1} ^ {m} E _ {i} ^ {2},
$$
and recalling that  $E_{i} \stackrel{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, \sigma^{2})$ , we see that  $(1 / \sigma^2) \| E \|_2^2$  has a  $\chi^2$  distribution with  $m$  degrees of freedom. It follows that  $\sigma^2 / \| E \|_2^2$  has an inverse-  $\chi^2$  distribution with  $m$  degrees of freedom, which has an expected value of  $1 / (m - 2)$  [52, pp. 578-579, 583]. Thus
$$
\mathbb {E} \left[ \frac {1}{\| E \| _ {2} ^ {2}} \right] = \frac {1}{\sigma^ {2} (m - 2)}.
$$
Finally, we plug the value for  $\mathrm{tr}(\mathbb{E}[\mathcal{BB}^T]\mathbb{E}[\mathcal{D}^T\mathcal{D}])$  from equation 25 into equation 24 and multiply by  $\mathbb{E}[1 / \| E\| _2^2 ]$ :
$$
\begin{array}{l} \mathbb {E} [ \mathrm {S N R} ] = \operatorname {t r} \left(\mathbb {E} \left[ \boldsymbol {\mathcal {B}} \boldsymbol {\mathcal {B}} ^ {T} \right] \mathbb {E} \left[ \boldsymbol {\mathcal {D}} ^ {T} \boldsymbol {\mathcal {D}} \right]\right) \mathbb {E} \left[ \frac {1}{\| E \| _ {2} ^ {2}} \right] \\ = \frac {s}{\sigma^ {2} (m - 2)}. \\ \end{array}
$$
C Prior Work on Explainable AI
In this section we review prior work on explainable AI. For the references herein please refer to the References section at the end of this Appendix.
Modern deep networks are often criticized for their opaque nature resulting in a lack of transparency in their decision-making process [53]. As a result, they are often perceived as "black-boxes". Initial attempts to explain deep networks relied on post-hoc techniques like sensitivity-based or gradient-based analyses of the inputs and the outputs of a trained network [54, 55, 56, 57, 58, 59]. These methods attempt to explain a deep network's decision by generating a saliency map reflecting which parts of the input were most important for a particular prediction. The veracity of these methods at explaining the true underlying decision-making process has been questioned in several recent works [60, 61, 62, 63, 64, 65]. Moreover, the explanations obtained, in terms of importance of raw input features (like pixels) to the predicted output, are often not meaningful explanations to stakeholders [5, 41]. As a result, recent works have advocated that machine learning (ML) models should be explainable by design [66, 5, 67].
Over the past few years, there has been a flurry of papers that propose ML algorithms that are explainable by design; however, they vary in their definitions of what it means to be explainable. These can be broadly classified into three categories:
1. Regularization to make deep networks more interpretable. These methods propose to regularize the training of a deep network such that their decisions can be well-approximated locally by an interpretable ML model like linear classifiers [68, 69] or decision trees [70]. However, as in post-hoc techniques to explainability, the effect of these approximations on the fidelity of the generated explanations to the deep networks decision-making process is unknown. Alternatively, Wong et al. [71] show that learning a sparse final layer of the network improves interpretability, but requires post-hoc methods like LIME to explain the deep features used by the sparse layer.
2. Explanations based on learnt concepts. These methods propose to jointly learn prototypes or latent vectors from data and use these latent vectors to make a prediction [72, 73, 74, 75, 76]. The final prediction is then explained in terms of these learnt concepts, whose interpretation is based on exemplar-based reasoning. For example, Sarkar et al. [72] look at images that maximally activate a particular feature of the latent vector to decide what that feature represents. Similarly, Li et al. [75] replace their learnt prototype by the embedding of the nearest-neighbour training image patch. This selected patch then serves as an interpretation of the prototype. A limitation of these methods is that it is not necessary for these learnt prototypes/latents to be interpretable to the end-user [5].
3. Explanations based on user-defined concepts. In these methods, the user first defines a set of concepts that are relevant for the prediction task, along with annotated datasets indicating the presence or absence of concepts in each input. The final prediction is the made based on these concepts, which serve as an explanation for the prediction [5, 15, 67]. In contrast to the previous set of methods, joint training of concepts and label classifiers in these methods is done in a completely supervised manner (the concepts are not latent vectors). In another direction, Chen et al. [77] propose to make layers of a deep network more interpretable by learning a rotation matrix that rotates any selected deep network's layer activations such that every activation (after rotation) corresponds to one of the user-defined concepts. Unlike the previous two categories, the methods in this category have the flexibility of allowing the user to decide on the nature of explanations required for the prediction task by selecting the set of concepts as input to the learning algorithm. Our proposed algorithm CLIP-IP-OMP also belongs to this class of methods.
Despite its advantage, a major limitation of the third category of methods is the requirement for data annotated with these user-defined concepts. This bottleneck was addressed (for image classification tasks) very recently in [41, 78, 79] who showed that CLIP, a large vision-language pre-trained model can be used to provide zero-shot annotations for user-defined concepts. In particular, text embeddings of the concepts are obtained using CLIP's text encoder. The image annotations (whether or not a concept is present or not in the image) is then provided via the dot product between the text embedding of the concept and the image embedding (obtained by passing the image through CLIP's image encoder). This motivated our algorithm CLIP-IP-OMP, which uses CLIP embeddings to construct
the interpretable dictionary. As argued in the main paper, our proposed algorithm is computationally much simpler than existing state-of-the-art methods of explainable-AI (comparing with methods from the third category, which include Variational-IP [15] and Label-free CBM [41]). CLIP has also been applied for semantic interpretation of neurons in a Convolutional Neural Network [80].
D CLIP-IP-OMP Training Details
D.1 Semantic Dictionaries
In principle, a user can define a set of semantic concepts that are relevant for the classification task. A dictionary can then be constructed by obtained CLIP's text embedding for each one of those concepts. In this work, to illustrate the usefulness of our framework, we used the set of concepts used in [41] for each dataset. For completeness, we describe the process in which these concepts were obtained in more detail.
The authors of [41] employed GPT-3 [42], a large language model, to extract an initial set of semantic concpets. They used three prompts for this purpose.
- List the most important features for recognizing something as a "class".
- List the things most commonly seen around a "class".
- Give superclass for the word "class".
In the above prompts "class" is a placeholder that is replaced by the class names present in the dataset. This initial list of concepts was then further pruned to remove redundancies. The authors employed the following set of rules for pruning.
- Any concepts with textual description longer than 30 characters was removed.
- Any concept too similar to the target class was removed. This was done to prevent explanations of a class to be the class itself. To do this, the authors measured the cosine similarity between the concepts and the target classes in the text embedding space and pruned any concept with similarity higher than 0.85. The cosine similarity was measured by a weighted combination of CLIP's ViT-B/16 text encoder and the all-mpnet-base-v2 sentence encoder.
- Concepts similar to each other in CLIP's text embedding space were pruned to remove duplicate concepts. Specifically, any concept with more than 0.9 cosine similarity (same weighted combination as above) with any concept already present in their list.
- Concepts not present in the training data were pruned. This was done by removing any concept whose average top-5 CLIP activations (measured as similarity between the concept's CLIP text embedding and the image's CLIP image embedding) was below a pre-defined threshold.
- Finally, the authors employ a projection layer to learn to estimate CLIP's similarity between a text concept and an image given the image's deep features obtained from some Backbone network like Resnet-50 [81] (so at test time CLIP would not be needed). Any concept that this projection layer is not able to estimate well (using some measure of accuracy) is removed.
The number of concepts finally used for each dataset are as follows: 128 for CIFAR-10, 824 for CIFAR-100, 208 for CUB-200, 4523 for ImageNet, and 2207 for Places365.
D.2 More Training Details for the CLIP-IP-OMP Algorithm
For all our experiments we use the CLIP Vit-B/16 model. In our experimental setup in § 4.2, for a given dataset, we first set the desired number of iterations<sup>9</sup> to run the IP-OMP algorithm for sparse-coding the image embedding in terms of dictionary atoms. This is the desired explanation length. We then obtain sparse-codes for every image in the training set using the IP-OMP algorithm. This creates
9While we employed this termination criterion in our experiments, one can also alternatively use a variable number of iterations per image by using the entropy-based stopping criterion where one terminates once the norm of the residual is sufficiently small.
a new training dataset  $\mathcal{D}'$  where every (image, label) pair is transformed into a (sparse-code, label) pair. A linear network is then trained using the cross-entropy loss using  $\mathcal{D}'$ . For all datasets we used the same training configuration. We employed SGD with momentum optimizer with learning rate 1.0 and momentum 0.9. To prevent overfitting, we employed a drop-out layer before the linear layer with parameter 0.5. Finally, the linear classifier was trained for 1000 epochs. The input and output neurons for the linear layer for every dataset was taken as the number of concepts in the dictionary and the number of output classes respectively. The prediction probability of the trained classifier is obtained via a softmax on the output.
To obtain the curves in Figure 5 we repeated the above described process for different values of  $\tau$ . Since, our classifier is linear, training via SGD is relatively fast. The longest time being on the Imagenet and Places365 small datasets (since they have 1.28M and 1.8M images respectively) where every epoch takes about 17 and 50 seconds respectively on a single NVIDIA RTX A5000 GPU with a batch size of 1024.
E Example Runs for the CLIP-IP-OMP Algorithm
We show example runs of the CLIP-IP-OMP algorithm on 5 images selected from the test/validation set<sup>10</sup> of the different datasets on which we tested our algorithm. For each dataset, we run the IP-OMP algorithm for a fixed number of iterations  $\tau$  and make predictions using a linear classifier. In practice, a user can look at the accuracy vs. explanation length trade-off curves (Figure 5) and decide on a value for  $\tau$  based on their requirement (more accuracy vs. ease of interpretation of the results). In this appendix, for every dataset, we select some value for  $\tau$  to illustrate the efficacy of the proposed framework at making explainable predictions.
E.1 ImageNet
For this dataset, we set  $\tau = 9$ , that is, the sparse code  $\hat{\beta}_{\mathrm{IP-OMP}}$  for every image has support<sup>11</sup> of size 9. A linear classifier trained on these sparse codes gets about  $63\%$  accuracy, which is within 9 points of what can be achieved if we run the IP-OMP algorithm long enough. This can be seen from Figure 5, where the test accuracy vs. explanation length curve for ImageNet saturates at about  $70\%$  accuracy for relatively large  $\tau$  (say  $\tau = 50$ ). Results in Figure 9.
1. In Figure 9a, our model correctly predicts "Great White Shark". The sign of the weights of the linear classifier seem to correlate with the sign of the atom coefficients in the sparse code. The prediction can be attributed to positive concepts like "shark", "a large, white, rounded head", "powerful jaws", and "marine animal", all of which are salient features of a great white shark. Similarly, negative concepts like "swan", "planets", "streetcar", and "a toad" are concepts whose presence would indicate that the image is definitely not of a great white shark.
2. In Figure 9b, our model correctly predicts "Snow Leopard". The sign of the weights of the linear classifier seem to correlate with the sign of the atom coefficients in the sparse code. The prediction can be attributed to positive concepts like "spotted or striped fur", "a mottled gray and white coat", and "a cub"; which are also the concepts to which the classifier assigns the highest weights. Note other concepts like "a cell" (probably referencing a zoo cell) and "a fence" are also present in the image, and are attributed as positive concepts in the sparse code. However, since these are not salient for the class "Snow Leopard", these are attributed smaller weights by the linear classifier. Thus, our method shows that the model is making the right prediction for the right reasons, something that is very difficult to ascertain in black-box deep networks. Similarly, absence of concepts like "a striped coat" and "a white throat" probably help the classifier to distinguish a snow leopard from other big cats.
3. In Figure 9c, our model correctly predicts "Goldfish". The sign of the weights of the linear classifier seem to largely correlate with the sign of the atom coefficients in the sparse code. The prediction can be attributed to positive concepts like "a fish tank", "a [sic] aquarium",
<sup>10</sup>We use validation sets only for the ImageNet and the Places365 datasets since for these datasets we only have access to the validation set and not the test set.
11The support of a vector is the set of indices of all non-zero elements in the vector.
"orange and white stripes" (which is very salient to a goldfish); which are also the concepts on which the classifier assigns the highest weights. Note that "often has a decorative design" is also positively attributed in the sparse code by IP-OMP, most likely because goldfish is often seen as a decorative item in one's household. Indeed, inspecting the GPT3 concepts recovered for the "GoldFish" class in [43], we see that "decoration" is one of the concepts that is supplied by GPT3 for this class. Similarly, negative concepts like "trash", "gym clothes" and "spider" identified by IP-OMP are concepts whose presence would strongly indicate that the object in the image is not a goldfish. Accordingly the linear classifier assigns negative weights to these concepts.
4. In Figure 9d, our model incorrectly predicts "Electric Guitar". The true class is "Acoustic Guitar". Inspecting the selected atoms and the weights of the classifier, it is clear that from the positive concepts—"a guitar case", "a small, handheld instrument", "music", and "equipment"; along with the negative concepts, one cannot distinguish between an electric and an acoustic guitar. Indeed, the second most probable class in this case is "Acoustic Guitar". Note that although the image does not explicitly show "a guitar case", it is a concept positively correlated with the object in the image and CLIP's latent space is such that text embeddings that are related to image content have a high correlation with the image's embedding. Indeed, "a guitar case" is one of the concepts supplied by GPT3 for the class "Acoustic Guitar".
5. In Figure 9e, our model incorrectly predicts "Volcano". We see that the two most salient concepts according to the classifier (atoms to which the classifier assigns highest weight) are "eruption" and "a cone-shaped mountain", which are concepts a human would use to describe a volcano. Other positive concepts recovered by IP-OMP include "aflame", "a white or light-colored body" (probably looking at the bright flame) and "made of lava and ash" (which is the concept attributed the third highest weight by the linear classifier).
E.2 Places365
For this dataset, we set  $\tau = 11$ , that is, the sparse code  $\hat{\beta}_{\mathrm{IP-OMP}}$  for every image has support of size 11. A linear classifier trained on these sparse codes gets about  $45.5\%$  accuracy, which is within 3 points of what can be achieved if we run the IP-OMP algorithm long enough. This can be seen from Figure 5, where the test accuracy vs. explanation length curve for Places365 saturates at about  $48\%$  accuracy for relatively large  $\tau$  (say  $\tau = 30$ ). Results in Figure 10.