SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aditya Chattopadhyay†* Ryan Pilgrim†* René Vidal‡
†Johns Hopkins University, USA, {achatto1, rpilgri1}@jhu.edu
†University of Pennsylvania, USA, vidalr@upenn.edu
Information Pursuit (IP) is a classical active testing algorithm for predicting an output by sequentially and greedily querying the input in order of information gain. However, IP is computationally intensive since it involves estimating mutual information in high-dimensional spaces. This paper explores Orthogonal Matching Pursuit (OMP) as an alternative to IP for greedily selecting the queries. OMP is a classical signal processing algorithm for sequentially encoding a signal in terms of dictionary atoms chosen in order of correlation gain. In each iteration, OMP selects the atom that is most correlated with the signal residual (the signal minus its reconstruction thus far). Our first contribution is to establish a fundamental connection between IP and OMP, where we prove that IP with random projections of dictionary atoms as queries "almost" reduces to OMP, with the difference being that IP selects atoms in order of normalized correlation gain. We call this version IP-OMP and present simulations indicating that this difference does not have any appreciable effect on the sparse code recovery rate of IP-OMP compared to that of OMP for random Gaussian dictionaries. Inspired by this connection, our second contribution is to explore the utility of IP-OMP for generating explainable predictions, an area in which IP has recently gained traction. More specifically, we propose a simple explainable AI algorithm which encodes an image as a sparse combination of semantically meaningful dictionary atoms that are defined as text embeddings of interpretable concepts. The final prediction is made using the weights of this sparse combination, which serve as an explanation. Empirically, our proposed algorithm is not only competitive with existing explainability methods but also computationally less expensive.
Information Pursuit (IP), first proposed by Geman and Jedynak [1], is a classical algorithm for active testing: Given a set of queries whose answers are informative about some target variable, IP proceeds by adaptively selecting queries in order of information gain. Specifically, in each iteration, IP selects the query whose answer has maximum mutual information about the target variable given the history of query-answers observed so far. IP has found numerous applications in machine learning such as tracking roads in satellite images [1], face detection & localization [2], detecting & tracking surgical instruments [3], and scene interpretation [4]. More recently, IP has been touted as a framework for explainable AI [5]. Despite its utility, a major limitation for IP is its reliance on mutual information, which is challenging to compute in high dimensions [6]. In light of this, we ask the question, are there alternative sequential algorithms that could replace IP by employing a simpler objective in their selection step? One such algorithm is Orthogonal Matching Pursuit (OMP), which we describe next.
OMP [7, 8] is a classical algorithm for sparse coding: given a dictionary of atoms and an observed signal, OMP proceeds by adaptively selecting atoms in order of correlation gain. Specifically, in each iteration, OMP selects the atom that has maximum correlation with the current residual (the unexplained part of the observed signal given the previously selected atoms). OMP has been successfully applied to many signal processing applications such as image denoising using overcomplete dictionaries [9], image super-resolution [10, 11], and matrix completion [12].
Similarities between IP and OMP. Although the two algorithms were discovered independently in two different communities, IP in the active testing and OMP in the signal processing community, they share many similarities. Both algorithms are illustrated in Figure 1.
Figure 1: IP and OMP. Both are greedy sequential algorithms. IP selects the next query using conditional mutual information  $I(\cdot ;\cdot)$  while OMP selects the next dictionary atom based on dot products  $\langle \cdot ,\cdot \rangle$ . Previous choices are incorporated via the history  $\mathcal{H}_{1:k}(x)$  of previous query answers in IP, and the projection matrix  $\Pi_{D_k}^{\perp}$ , which we define in §2, in OMP.
- Both seek a parsimonious representation. IP tries to make a prediction by selecting the minimum number of queries on average. OMP tries to represent the observed signal by selecting the minimum number of atoms.
- Both are greedy algorithms used as efficient approximations to the true optimal solution. In each iteration, IP uses conditional mutual information to decide the next most informative query about the target variable  $Y$ . Similarly, in each iteration, and the residual (the observed signal's project all the atoms selected so far) to select the next signal. The conditioning (in IP) and residual decisions made by each algorithm.
- IP terminates when the posterior  $P(Y \mid \text{History})$  is sufficiently peaked, indicating  $Y$  can be predicted with low error. Similarly, OMP terminates once the mean squared error between the observed signal and its reconstruction is sufficiently low, indicating that the observed signal can be reconstructed from the selected atoms with high fidelity.
Differences between IP and OMP. Given these similarities, one may be tempted to conjecture that one of the algorithms is a particular case of the other. However, establishing such a connection is not trivial at all. The reason is that there are fundamental differences in the manner in which these two algorithms operate, despite their apparent similarities. In IP, the target variable and query answers are all random variables. An illustrative example of this is given by the popular parlor game "twenty questions", where a player asks queries about attributes of some entity  $Y$  (that another player has thought of) and would like to identify  $Y$  by asking the minimum number of questions. In this example,  $Y$  is a random variable, and as a result, the corresponding query answer which depends on  $Y$  is also a random variable. In OMP, on the other hand, one typically observes some fixed signal and then proceeds to encode it using the dictionary atoms. Consequently, IP's objective to select the next query requires estimating mutual information, which, as already stated, is difficult in high dimensions. OMP, on the other hand, only relies on dot products and least-squares involving the fixed signal and small numbers of dictionary atoms, which is computationally a far more tractable objective. Therefore, it is indeed surprising that one can derive a connection between the two algorithms.
Is OMP a particular case of IP? Formally establishing a connection between IP and OMP requires answering two key questions: (i) "what would the queries and their corresponding answers be?" and (ii) "what would the target variable be?" As IP operates on random variables and OMP on fixed observed signals, we need to inject the observed signal into the definition of the queries and/or target variable. We achieve this by taking queries as dot products of the dictionary atoms with a standard normal variable  $Z$ , and the target variable as the dot product between the observed signal  $x$  and  $Z$ . The intuition is that since  $Z$  has a radially symmetric distribution, in order to accurately
2In the case of IP, this is the solution that needs the minimal number of queries on average to predict the target variable with a desired level of confidence; in the case of OMP, this is the sparsest code that reconstructs the observed signal up to a desired level of accuracy.
reconstruct the target dot product, IP would need to encode the observed signal into the selected queries (scalar projections of the dictionary atoms). We formally show this in §3. In particular, we show that IP selects the atom in each iteration whose projection (onto the orthogonal complement of the span of the atoms selected so far) has the maximum normalized dot product with the residual. This contrasts with OMP which uses the unnormalized dot product as its selection criterion. We call this IP version of sparse coding  $IP-OMP$ . We then show via simulations over random Gaussian dictionaries and different sparsity levels that despite this algorithmic difference, IP-OMP and OMP have almost identical sparse recovery success rates. This empirical observation is further complemented by theoretical work by Soussen et al. [13] who prove that if for a given dictionary, exact recovery for all  $s$ -sparse signals is possible after  $s$  iterations of OMP, then the set of atoms selected after termination ( $s$  iterations) would be the same for both IP-OMP and OMP. $^3$
Implications of this connection. As alluded to before, computing mutual information is often intractable in high dimensions. Previous approaches tackle this issue by either explicitly learning a generative model for data [5, 14] or by learning the most informative next query from data using deep networks and stochastic objectives [15, 16]. However, IP-OMP presents a much simpler alternative which does not involve training any deep networks or learning data distributions using generative models. This begs the question of whether IP-OMP can be used as a cheap surrogate to IP in certain applications. In this paper, we show one such application to explainable AI, where IP has been recently proposed as a method for making explainable predictions by design [15]. More specifically, [15] proposes a framework called Variational Information Pursuit (V-IP) where the query set  $\mathcal{Q}$  consists of interpretable queries about the data. Using  $\mathcal{Q}$ , V-IP explains its prediction in terms of the selected interpretable queries. Similarly, we construct a dictionary comprised of interpretable atoms from text embeddings. We then propose to use IP-OMP to represent the image in terms of a sparse combination of dictionary atoms (the sparse code). The final prediction is made by training a linear classifier on top of the sparse code. In this way, we can make predictions explainable—in the sense that humans can understand predictions in terms of the sparse interpretable inputs to the classifier and their corresponding weights—without computing mutual information. This, however, presents a challenge: images will typically not be represented well as linear combination of text embeddings since they are very different modalities. To remedy this, we propose to use CLIP [17], a recently introduced large Vision-Language Model, to encode both images and text into a shared latent space.
Paper contributions. To summarize, our main contributions are:
- We formally establish a connection between IP and OMP. By choosing the parameters of IP appropriately, we obtain an algorithm, IP-OMP, which is equivalent to OMP up to a normalization in the objective.
- Inspired by this connection, we propose a simple algorithm using IP-OMP and CLIP for making explainable predictions in visual classification tasks. We show empirically on multiple image classification datasets that the performance of our algorithm is competitive with, if not better than, state-of-the-art methods for explainable AI.
2.1 Information Pursuit
In this paper, random variables are defined over a common sample space  $\Omega$  and are denoted by capital letters. Their realizations are denoted by the corresponding lowercase letter. Let  $Y$  be a target variable of interest. Let  $\mathcal{Q}$  be a set of queries, where every query  $Q \in \mathcal{Q}$  is a random variable. We call the realization of  $Q$ , written  $q$ , its answer. Given  $\mathcal{Q}$ , one is often interested in predicting  $Y$  by sequentially asking queries from  $\mathcal{Q}$  such that the average number of queries needed is minimized [18, 19, 20, 14, 5, 16]. Information pursuit (IP) [1] greedily approximates the solution to this problem. In particular, a single IP run proceeds as follows,
$$
Q _ {1} = \underset {Q \in \mathcal {Q}} {\arg \max } I (Q; Y); \quad Q _ {k + 1} = \underset {Q \in \mathcal {Q}} {\arg \max } I (Q; Y \mid \mathcal {H} _ {1: k}), \tag {1}
$$
where  $Q_{k + 1}$  is the query selected in iteration  $k + 1$ ,  $I(\cdot ;\cdot)$  denotes mutual information and  $\mathcal{H}_{1:k}$  is the history of query-answer pairs observed after the first  $k$  iterations. More precisely,  $\mathcal{H}_{1:k}$  is defined
<sup>3</sup>The authors of [13] refer to IP-OMP as the Orthogonal Least Squares algorithm since it can also be derived from a least squares optimization perspective. We expand upon this point in §3.
as the event  $\{\omega \in \Omega : Q_1(\omega) = q_1, \dots, Q_k(\omega) = q_k\}$ . The algorithm terminates after  $\tau$  iterations if the entropy (or differential entropy if  $Y$  is a continuous random variable) of the posterior  $P(Y \mid \mathcal{H}_{1:\tau})$  is below a user-defined threshold [1]. Alternatively, one could employ a fixed budget of  $\tau$  iterations and terminate [21]. After termination, IP's prediction for  $Y$  is given by  $\arg \max_{Y} P(Y \mid H_{1:\tau})$  [5].
2.2 Orthogonal Matching Pursuit
Given a matrix  $D \in \mathbb{R}^{m \times n}$  with  $m \ll n$  and an observed signal  $x \in \mathbb{R}^m$ , the goal of sparse coding is to find a sparse vector  $\beta \in \mathbb{R}^n$  that reconstructs the signal  $x$ . More precisely, the goal is to solve
$$
\min  _ {\beta \in \mathbb {R} ^ {n}} \| \beta \| _ {0} \quad \text {s . t .} \quad x = D \beta , \tag {2}
$$
where  $\| \cdot \| _0$  refers to the  $\ell_0$  pseudo-norm. Equation 2 has been of significant interest over the past few decades due to its applications to compressed sensing [22, Chapter 1] and sparse representation theory [23]. We will call  $D$  the dictionary,  $x$  the observed signal and  $\beta$  the sparse code. We will refer to the columns of  $D$  as atoms, where  $d^j$  refers to the  $j^{\mathrm{th}}$  atom.
Since the problem in equation 2 is NP-Hard [22, §2.3], approximations are employed. Orthogonal Matching Pursuit (OMP) [7, 8] is a popular greedy algorithm known for its computational efficiency [24] compared to alternatives like Basis Pursuit [25]. Given  $x$  and  $D$ , the OMP algorithm first  $\ell_2$ -normalizes all the atoms in  $D$ . Subsequently, it initializes its estimate of the sparse code as  $\hat{\beta}_0 = 0$  and its estimate of the support of  $\beta$  (set of non-zero indices in  $\beta$ ) as the empty set  $S_0 = \emptyset$ . OMP then proceeds iteratively. Its  $(k + 1)^{\mathrm{th}}$  iteration  $(k \geq 0)$  is defined as,
$$
j_{k + 1} = \operatorname *{arg  max}_{j\in \{1,\ldots ,n\}}|\langle d^{j},x - D\hat{\beta}_{k}\rangle |; \quad S_{k + 1} = S_{k}\cup \{j_{k + 1}\} ;
$$
$$
\hat {\beta} _ {k + 1} = \underset {u \in \mathbb {R} ^ {n}} {\arg \min } \left\{\| x - D u \| _ {2}: \operatorname {s u p p o r t} (u) \subseteq S _ {k + 1} \right\},
$$
where  $x - D\hat{\beta}_k$  is called the residual at iteration  $k$  [22, §3.2]. There are multiple choices for the termination criteria, including (i) to run some fixed number of iterations  $\tau$ , or (ii) to terminate once the norm of the residual is below some user-defined threshold  $\epsilon \geq 0$ , that is,  $\| x - D\hat{\beta}_{\tau}\|_{2} \leq \epsilon$  [26].
We now rewrite the atom selection step in equation 3 into a form that is more amenable to comparison with the IP version of OMP called IP-OMP that we will derive next. For a particular run of the OMP algorithm, let  $D_{k}$  be a sub-matrix of  $D$  composed of the first  $k$  atoms selected by the OMP algorithm and let  $\Pi_{D_k}$  be the projection to the range of  $D_{k}$ . From the update of  $\hat{\beta}_k$ , it is clear that  $D\hat{\beta}_k$  is a projection of  $x$  onto the range of  $D_{k}$ . Similarly, the residual  $x - D\hat{\beta}_k$  is the projection of  $x$  onto the orthogonal complement [27]. Using this observation, we rephrase the atom selection in equation 3 as
$$
j _ {k + 1} = \underset {j \in \{1, \dots , n \}} {\arg \max } | \langle d ^ {j}, \Pi_ {D _ {k}} ^ {\perp} x \rangle | = \underset {j \in \{1, \dots , n \}} {\arg \max } | \langle \Pi_ {D _ {k}} ^ {\perp} d ^ {j}, \Pi_ {D _ {k}} ^ {\perp} x \rangle |. \tag {4}
$$
3 IP-OMP: Information Pursuit for Sparse Coding
Recall from §1 that to establish a connection between IP and OMP, we need to specify a set of queries and a target variable, all of which are random variables. Moreover, since OMP operates on deterministic inputs (the observed signal), we have to inject the observed signal into the definition of the queries and/or the target variable. Inspired from the Bayesian perspective of compressed sensing [28, 29] one natural attempt could be to consider a sparsity-inducing prior over the sparse code, like the Laplace distribution [30]. This induces a distribution over the observed signal  $X = D\beta +$  measurement noise (capitalized to emphasize that  $X$  is now a random variable). Since the goal is to sparse-code the signal, a natural choice for the target variable could be  $X$  itself. Finally, since we want to recover the atom selection step in OMP (equation 4) using IP, which is obtained by maximizing the absolute dot product between the atom and the residual, a reasonable choice for the query set  $Q$  could be atoms of the dictionary where the answer to a query is the corresponding atom's dot product with  $X$ . Given this setup, the first query selected by IP is<sup>4</sup>
$$
Q _ {1} = \underset {Q ^ {j} \in \mathcal {Q}} {\arg \max } I \left(Q ^ {j}; X\right) = \underset {d ^ {j} \in D} {\arg \max } I \left(\langle d ^ {j}, X \rangle ; X\right). \tag {5}
$$
Note that we use subscript  $Q_{i}$  to denote the query selected by IP at iteration  $i$ , and superscript  $Q^{i}$  to denote the query that corresponds to the  $i^{\mathrm{th}}$  atom,  $d^{i}$ , of  $D$ .
It is immediately clear from equation 5 that the first query would be independent of the observed signal  $X = x$ . This is because IP decides what is the most informative query solely based on mutual information, which is a property of the distribution, and does not involve observing the realization  $x$ . This is in sharp contrast to OMP, where the selection of the first atom is driven by the observed signal (see equation 3). Thus, assuming a generative model for the signal  $X$  would not result in the OMP algorithm and calls for a more "non-standard" way of defining the queries and/or the target variable.
What is the right query set and target variable? How can we make the queries and/or the target variable depend on a realization of  $X = x$ ? Inspired by a measure called sliced mutual information, which defines the dependence between two random variables as the average mutual information between their one-dimensional projections onto random directions [31], we propose the following query set and target variable.
- Define  $Z$  to be a standard normal variable in  $\mathbb{R}^m$ . Let  $z^0$  be a sample from  $P_Z = \mathcal{N}(0, I_m)$ .
- Take  $\mathcal{Q}$  to contain all the atoms in  $D$  as queries, such that query  $Q^j \in \mathcal{Q}$  corresponds to the random projection  $\langle d^j, Z \rangle$  of atom  $d^j \in D$ , and the corresponding answer is  $q^j \coloneqq \langle d^j, z^0 \rangle$ .
- Take the target random variable of interest as  $\langle x, Z \rangle$ . Thus, given  $z^0$ , the value of the target variable that needs to be predicted is  $\langle x, Z \rangle = \langle x, z^0 \rangle$ .
The rationale behind these choices is as follows. Since  $Z$  is a random vector with a radially symmetric density, to accurately predict the target variable  $\langle x,Z\rangle$  for all realizations of  $Z$ , IP needs to query atoms that would in a way code for  $x$  itself. This intuition is formalized by the following lemma.
Lemma 1. Let  $z^0$  be a realization of  $Z$ . Given a subset  $\mathcal{L}$  of queries from  $\mathcal{Q}$  (or equivalently, atoms selected from  $D$ ), the conditional entropy of the target  $\langle x, Z \rangle$  given the query answers observed is
$$
h \left(\langle x, Z \rangle \mid \{Q = \langle d _ {Q}, z ^ {0} \rangle : Q \in \mathcal {L} \}\right) = \frac {1}{2} \ln \left(2 \pi e \| \Pi_ {D _ {\mathcal {L}}} ^ {\perp} x \| _ {2} ^ {2}\right), \tag {6}
$$
where  $h$  is the conditional differential entropy,  $d_{Q}$  denotes the atom used to construct the random variable  $Q$ ,  $D_{\mathcal{L}}$  is the sub-matrix of  $D$  consisting of atoms corresponding to queries in  $\mathcal{L}$  and  $\Pi_{D_{\mathcal{L}}}^{\perp}$  is the projection matrix onto the orthogonal complement of the range of  $D_{\mathcal{L}}$ .
The proof of this lemma relies on the joint Gaussianity of all the query answers and the target variable since they are all dot products with the same normal vector  $Z$ . Proof in Appendix §A.1.
Notice that the history of query answers observed, after a finite number of iterations of IP, is just some subset of queries selected from  $\mathcal{Q}$  along with their corresponding answers. Since IP greedily selects queries until the conditional entropy of the target (given history) is minimized (§2.1), Lemma 1 indicates that IP would terminate when  $\| \Pi_{D_{\mathcal{L}}}^{\perp}x\| _2^2\to 0$ , that is, when  $x$  can be "almost exactly" reconstructed from the atoms selected by IP.
The IP-OMP algorithm. Given this choice of query set and target variable, we now state our main result (Theorem 1) proving that IP not only chooses atoms that code for  $x$ , but also uses exactly the same query selection strategy as the atom selection strategy of OMP up to a normalization factor.
Theorem 1. Given an observed signal  $x$ , a realization  $z^0$  of  $Z$ , and the choice of  $\mathcal{Q}$  and the target variable as described in this section, IP proceeds as follows for  $k \geq 1$
$$
Q_{1} = \operatorname *{arg  max}_{Q^{j}\in \mathcal{Q}}I(Q^{j};\langle x,Z\rangle) = \operatorname *{arg  max}_{d^{j}\in D}\frac{\left|\langle d^{j},x\rangle\right|}{\|d^{j}\|_{2}\|x\|_{2}}
$$
$$
Q _ {k + 1} = \underset {Q ^ {j} \in \mathcal {Q}} {\arg \max } I \left(Q ^ {j}; \langle x, Z \rangle \mid \mathcal {H} _ {1: k}\right) = \underset {d ^ {j} \in D, \| \Pi_ {D _ {k}} ^ {\perp} d ^ {j} \| _ {2} \neq 0} {\arg \max } \frac {\left| \left\langle \Pi_ {D _ {k}} ^ {\perp} d ^ {j} , \Pi_ {D _ {k}} ^ {\perp} x \right\rangle \right|}{\left\| \Pi_ {D _ {k}} ^ {\perp} d ^ {j} \| _ {2} \right\| \left\| \Pi_ {D _ {k}} ^ {\perp} x \| _ {2} \right\|}, \tag {IP-OMP}
$$
where  $\mathcal{H}_{1:k} := \{Q_1 = \langle d_1, z^0 \rangle, Q_2 = \langle d_2, z^0 \rangle, \ldots, Q_k = \langle d_k, z^0 \rangle\}$ ,  $d_i$  is the atom (query) selected by  $IP$  in the  $i^{th}$  iteration,  $D_k$  is the sub-matrix of  $D$  consisting of atoms corresponding to queries selected by  $IP$  in the first  $k$  iterations and  $\Pi_{D_k}^{\perp}$  is the projection matrix onto the orthogonal complement of the range of  $D_k$ .
The above result uses the joint Gaussianity of all the query answers and the target variable. A full derivation can be found in Appendix §A.2. Comparing equation 4 and IP-OMP, it is clear that the two algorithms differ by a normalization. To distinguish from the OMP algorithm, we call the
Figure 2: Performance of OMP (solid curves) and IP-OMP (overlaid circles) in the noiseless case as a function of the number of measurements  $m$  for a fixed sparsity level  $s$  and a number of dictionary atoms  $n$ . "Probability of exact recovery" reports the fraction of times (over 1000 simulations) the algorithm exactly recovered the sparse code for a particular setting of  $(m, n, s)$ .
iterative algorithm described above as the IP-OMP algorithm. If one uses the entropy of the posterior  $P(\langle x,Z\rangle \mid \mathcal{H}_{1:\tau})$  criterion for termination after  $\tau$  iterations, then according to Lemma 1 the algorithm would terminate once  $\| \Pi_{D_{\tau}}^{\perp}x\| _2^2\leq \epsilon$ . This is reminiscent of the norm of the residual termination criterion commonly employed by OMP.