SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} [ \langle x, Z \rangle \mid D _ {\mathcal {L}} ^ {T} Z = D _ {\mathcal {L}} ^ {T} z ^ {0} ] = x ^ {T} D _ {\mathcal {L}} (D _ {\mathcal {L}} ^ {T} D _ {\mathcal {L}}) ^ {- 1} D _ {\mathcal {L}} ^ {T} z ^ {0} \\ = \langle D _ {\mathcal {L}} D _ {\mathcal {L}} ^ {\dagger} x, z ^ {0} \rangle . \\ \end{array}
$$
The variance is computed as
$$
\begin{array}{l} \operatorname {v a r} \left(\langle x, Z \rangle \mid \left\{Q = \left\langle d _ {Q}, z ^ {0} \right\rangle : Q \in \mathcal {L} \right\}\right) = x ^ {T} x - x ^ {T} D _ {\mathcal {L}} \left(D _ {\mathcal {L}} ^ {T} D _ {\mathcal {L}}\right) ^ {- 1} D _ {\mathcal {L}} ^ {T} x \\ = x ^ {T} \left(I - D _ {\mathcal {L}} \left(D _ {\mathcal {L}} ^ {T} D _ {\mathcal {L}}\right) ^ {- 1} D _ {\mathcal {L}} ^ {T}\right) x \\ = x ^ {T} \left(I - \Pi_ {D _ {\mathcal {L}}}\right) x \\ = x ^ {T} \Pi_ {D _ {\mathcal {L}}} ^ {\perp} x \\ = \| \Pi_ {D _ {\mathcal {L}}} ^ {\perp} x \| _ {2} ^ {2}, \\ \end{array}
$$
which is exactly the variance parameter for the normal distribution in equation 13. The result in the Lemma is then just the analytical form of the differential entropy of a Gaussian [47, equation 9.9].
Case 2: The covariance matrix is singular. This would be the case when  $x$  can be expressed as a linear combination of the columns (atoms) of  $D_{\mathcal{L}}$ . Intuitively, looking at equation 13, one can conclude that in this case the conditional distribution of  $\langle x,Z\rangle$  given the history would be a Dirac delta distribution since the variance is 0. Consequently, the differential entropy would converge to  $-\infty$ .
This argument can be made rigorous as follows. Consider a family of random variables defined as  $\zeta_{\sigma} = \langle x,Z\rangle +\sigma \eta$  , where  $\eta$  is an independent standard normal distribution. We can conclude from Slutsky's theorem [48, Theorem A.14.9] that
$$
\zeta_ {\sigma} \stackrel {d} {\to} \langle x, Z \rangle \mathrm {a s} \sigma \to 0,
$$
where  $\xrightarrow{d}$  refers to convergence in distribution. Observe that
$$
\forall \sigma > 0 P (\zeta_ {\sigma} \mid \{Q = \langle d _ {Q}, z ^ {0} \rangle : Q \in \mathcal {L} \}) = \mathcal {N} (\langle D _ {\mathcal {L}} D _ {\mathcal {L}} ^ {\dagger} x, z ^ {0} \rangle , \sigma^ {2}),
$$
since  $\| \Pi_{D_{\mathcal{L}}}^{\perp}x\| _2^2 = 0$  (recall in this case  $x$  is a linear combination of columns of  $D_{\mathcal{L}}$ ). This sequence of distributions is known to converge in distribution to the Dirac delta distribution, thus proving the desired result.
Proof. First iteration. Assume that  $x$  is not one-sparse in  $D$  (this case will be covered separately at the end of the proof). For the first iteration  $(k = 1)$ ,
$$
Q_{1} = \operatorname *{arg  max}_{Q^{j}\in \mathcal{Q}}I(Q^{j};\langle x,Z\rangle) = \operatorname *{arg  max}_{d^{j}\in D}I(\langle d^{j},Z\rangle ;\langle x,Z\rangle),
$$
where the second equality is just from definition of the queries and their answers in this case.
$$
I \left(\left\langle d ^ {j}, Z \right\rangle ; \left\langle x, Z \right\rangle\right) = h \left(\left\langle d ^ {j}, Z \right\rangle\right) - h \left(\left\langle d ^ {j}, Z \right\rangle \mid \left\langle x, Z \right\rangle\right), \tag {14}
$$
where  $h(\cdot)$  is the differential entropy [47, Definition 9.1]. Since  $Z$  is Gaussian,  $\langle d^j,Z\rangle$  and  $\langle x,Z\rangle$  are jointly Gaussian. Therefore we can compute the entropies in equation 14 as [47, Example 9.1.2]
$$
h (\langle d ^ {j}, Z \rangle) = \frac {1}{2} \log (2 \pi e \operatorname {v a r} (\langle d ^ {j}, Z \rangle))
$$
$$
h \left(\left\langle d ^ {j}, Z \right\rangle \mid \left\langle x, Z \right\rangle\right) = \frac {1}{2} \log \left(2 \pi e \operatorname {v a r} \left(\left\langle d ^ {j}, Z \right\rangle \mid \left\langle x, Z \right\rangle\right)\right).
$$
Since  $\operatorname{cov}(Z) = I_m$  and  $\mathbb{E}Z = 0$ , we get
$$
\operatorname {v a r} \left(\langle d ^ {j}, Z \rangle\right) = \mathbb {E} \langle d ^ {j}, Z \rangle^ {2} = \| d ^ {j} \| _ {2} ^ {2}.
$$
Considering the joint covariance
$$
\operatorname {c o v} \left(\left( \begin{array}{c} \langle d ^ {j}, Z \rangle \\ \langle x, Z \rangle \end{array} \right)\right) = \left( \begin{array}{c c} \| d ^ {j} \| _ {2} ^ {2} & \langle d ^ {j}, x \rangle \\ \langle x, d ^ {j} \rangle & \| x \| _ {2} ^ {2} \end{array} \right),
$$
and applying a basic identity for Gaussian conditioning [46, Equation A.6], we obtain
$$
\operatorname {v a r} \left(\left\langle d ^ {j}, Z \right\rangle \mid \left\langle x, Z \right\rangle\right) = \left\| d ^ {j} \right\| _ {2} ^ {2} - \frac {\left\langle d ^ {j} , x \right\rangle^ {2}}{\left\| x \right\| _ {2} ^ {2}}.
$$
Thus for iteration  $k = 1$ , IP-OMP amounts to solving
$$
\begin{array}{l} Q _ {1} = \underset {d ^ {j} \in D} {\arg \max } \frac {1}{2} \log \left(\frac {\| d ^ {j} \| _ {2} ^ {2}}{\| d ^ {j} \| _ {2} ^ {2} - \langle d ^ {j} , x \rangle^ {2} / \| x \| _ {2} ^ {2}}\right) \\ = \underset {d ^ {j} \in D} {\arg \max } \frac {| \langle d ^ {j} , x \rangle |}{\| d ^ {j} \| _ {2} \| x \| _ {2}}. \tag {15} \\ \end{array}
$$
Iterations between the first and last. Next consider iterations between the first and last in IP-OMP (the last iteration will be discussed separately). For all  $1 \leq k < \tau - 1$ , the query selection strategy is as follows:
$$
Q _ {k + 1} = \underset {Q ^ {j} \in \mathcal {Q}} {\arg \max } I \left(Q ^ {j}; \langle x, Z \rangle \mid \mathcal {H} _ {1: k}\right) = \underset {d ^ {j} \in D} {\arg \max } I \left(\left\langle d ^ {j}, Z \right\rangle ; \langle x, Z \rangle \mid \mathcal {H} _ {1: k}\right). \tag {16}
$$
As before, we assume that  $d^j$ ,  $x$ , and the columns of  $D_k$  are linearly independent (we will address collinearity issues later). Expanding the Mutual Information (MI) into a difference of differential entropies, we obtain
$$
I \left(\left\langle d ^ {j}, Z \right\rangle ; \left\langle x, Z \right\rangle \mid \mathcal {H} _ {1: k}\right) = h \left(\left\langle d ^ {j}, Z \right\rangle \mid \mathcal {H} _ {1: k}\right) - h \left(\left\langle d ^ {j}, Z \right\rangle \mid \left\langle x, Z \right\rangle , \mathcal {H} _ {1: k}\right). \tag {17}
$$
Notice that  $\langle d^j,Z\rangle ,\langle x,Z\rangle$  and  $D_k^T Z$  are jointly Gaussian. So, similarly to the first iteration, the MI objective simplifies to
$$
I (\langle d ^ {j}, Z \rangle ; \langle x, Z \rangle \mid \mathcal {H} _ {1: k}) = \frac {1}{2} \log (2 \pi e \operatorname {v a r} (\langle d ^ {j}, Z \rangle \mid \mathcal {H} _ {1: k})) - \frac {1}{2} \log (2 \pi e \operatorname {v a r} (\langle d ^ {j}, Z \rangle \mid \langle x, Z \rangle , \mathcal {H} _ {1: k})).
$$
<sup>5</sup>If  $x$  is one-sparse in  $D$ , then this expansion will not hold since the joint distribution of  $\zeta = (\langle d^j,Z\rangle ,\langle x,Z\rangle)^T$  will be degenerate for some  $j$ . However, the mutual information can still be suitably defined. We defer the details of this case to the end of the proof.
The history can be written  $\mathcal{H}_{1:k} = \{D_k^T Z = D_k^T z^0\}$ , so we compute the requisite conditional variances from the joint covariance matrix
$$
\operatorname {c o v} \left(\left( \begin{array}{l} \langle d ^ {j}, Z \rangle \\ \langle x, Z \rangle \\ D _ {k} ^ {T} Z \end{array} \right)\right) = \left( \begin{array}{l l l} \| d ^ {j} \| _ {2} ^ {2} & (d ^ {j}) ^ {T} x & (d ^ {j}) ^ {T} D _ {k} \\ x ^ {T} d ^ {j} & \| x \| _ {2} ^ {2} & x ^ {T} D _ {k} \\ D _ {k} ^ {T} d ^ {j} & D _ {k} ^ {T} x & D _ {k} ^ {T} D _ {k} \end{array} \right). \tag {18}
$$
We once again apply the Gaussian conditioning formula, and rewrite the difference of logs as the log of a quotient to obtain
$$
I \left(\left\langle d ^ {j}, Z \right\rangle ; \left\langle x, Z \right\rangle \mid \mathcal {H} _ {1: k}\right) = \frac {1}{2} \log \left(\frac {\left(d ^ {j}\right) ^ {T} \left[ I - D _ {k} \left(D _ {k} ^ {T} D _ {k}\right) ^ {- 1} D _ {k} ^ {T} \right] d ^ {j}}{\left. (d ^ {j}) ^ {T} \left[ I - [ x D _ {k} ] \binom {x ^ {T} x} {D _ {k} ^ {T} x} \binom {x ^ {T} D _ {k}} {D _ {k} ^ {T} D _ {k}}\right) ^ {- 1} [ x D _ {k} ] ^ {T} \right] d ^ {j}}\right). \tag {19}
$$
Define  $C_k = [D_k \, x]$ , let  $\Pi_{C_k}$  denote the orthogonal projection onto  $\mathrm{range}(C_k)$ , and let  $\Pi_{C_k}^{\perp}$  denote the orthogonal projection onto  $\mathrm{range}(C_k)^{\perp}$ . We can rewrite equation 19 in terms of projections as
$$
I \left(\left\langle d ^ {j}, Z \right\rangle ; \left\langle x, Z \right\rangle \mid \mathcal {H} _ {1: k}\right) = \frac {1}{2} \log \left(\frac {\left\langle d ^ {j} , \Pi_ {k} ^ {\perp} d ^ {j} \right\rangle}{\left\langle d ^ {j} , \Pi_ {C _ {k}} ^ {\perp} d ^ {j} \right\rangle}\right). \tag {20}
$$
Lemma 3 ([49], p. 323). Let  $A$  be partitioned as  $[A_1, A_2]$  with  $A_1 \in \mathbb{R}^{m \times p}$  and  $\mathrm{rank}(A_1) = p$ ,  $A_2 \in \mathbb{R}^{m \times (n - p)}$ , and  $\mathrm{rank}(A_2) = n - p$ . Let  $\Pi_{A_1}$  be the orthogonal projection onto the columns of  $A_1$  and  $B = \Pi_{A_1}^\perp A_2$ . Then the orthogonal projection,  $\Pi_A$ , onto the columns of  $A$  can be written as  $\Pi_A = \Pi_{A_1} + B(B^T B)^{-1} B^T$ .
Applying the above lemma to  $A = C_k$ , we have
$$
\Pi_ {C _ {k}} = \Pi_ {k} + \frac {1}{\| \Pi_ {k} ^ {\perp} x \| _ {2} ^ {2}} \Pi_ {k} ^ {\perp} x x ^ {T} \Pi_ {k} ^ {\perp},
$$
which, recalling the idempotence of  $\Pi_k$ , implies
$$
\begin{array}{l} \Pi_ {C _ {k}} ^ {\perp} = I - \Pi_ {C _ {k}} \\ = \Pi_ {k} ^ {\perp} \left(I - \frac {1}{\| \Pi_ {k} ^ {\perp} x \| _ {2} ^ {2}} x x ^ {T}\right) \Pi_ {k} ^ {\perp}. \\ \end{array}
$$
Thus we can simplify the denominator of equation 20 to
$$
\begin{array}{l} \langle d ^ {j}, \Pi_ {C _ {k}} ^ {\perp} d ^ {j} \rangle = \left\langle d ^ {j}, \Pi_ {k} ^ {\perp} \left(I - \frac {x x ^ {T}}{\| \Pi_ {k} ^ {\perp} x \| _ {2} ^ {2}}\right) \Pi_ {k} ^ {\perp} d ^ {j} \right\rangle \\ = \| \Pi_ {k} ^ {\perp} d ^ {j} \| _ {2} ^ {2} - \frac {\langle \Pi_ {k} ^ {\perp} d ^ {j} , x \rangle^ {2}}{\| \Pi_ {k} ^ {\perp} x \| _ {2} ^ {2}} \\ = \| \Pi_ {k} ^ {\perp} d ^ {j} \| _ {2} ^ {2} - \frac {\langle \Pi_ {k} ^ {\perp} d ^ {j} , \Pi_ {k} ^ {\perp} x \rangle^ {2}}{\| \Pi_ {k} ^ {\perp} x \| _ {2} ^ {2}}, \\ \end{array}
$$
where in the last equation we have used the symmetry and idempotence of  $\Pi_k^\perp$ . Substituting into equation 20, we obtain
$$
\begin{array}{l} \operatorname *{arg  max}_{d^{j}\in D,\| \Pi_{k}^{\perp}d^{j}\|_{2}\neq 0}I(\langle d^{j},Z\rangle ;\langle x,Z\rangle   |\mathcal{H}_{1:k}) = \operatorname *{arg  max}_{d^{j}\in D,\| \Pi_{k}^{\perp}d^{j}\|_{2}\neq 0}\frac{\|\Pi_{k}^{\perp}d^{j}\|_{2}^{2}}{\|\Pi_{k}^{\perp}d^{j}\|_{2}^{2} - \langle\Pi_{k}^{\perp}d^{j},\Pi_{k}^{\perp}x\rangle^{2} / \|\Pi_{k}^{\perp}x\|_{2}^{2}} \\ = \underset {d ^ {j} \in D, \| \Pi_ {k} ^ {\perp} d ^ {j} \| _ {2} \neq 0} {\arg \max } \frac {1}{1 - \langle \Pi_ {k} ^ {\perp} d ^ {j} , \Pi_ {k} ^ {\perp} x \rangle^ {2} / (\| \Pi_ {k} ^ {\perp} x \| _ {2} ^ {2} \| \Pi_ {k} ^ {\perp} d ^ {j} \| _ {2} ^ {2})} \\ = \underset {d ^ {j} \in D, \| \Pi_ {k} ^ {\perp} d ^ {j} \| _ {2} \neq 0} {\arg \max} \frac {| \langle \Pi_ {k} ^ {\perp} d ^ {j} , \Pi_ {k} ^ {\perp} x \rangle |}{\| \Pi_ {k} ^ {\perp} d ^ {j} \| _ {2} \| \Pi_ {k} ^ {\perp} x \| _ {2}}, \\ = \operatorname * {a r g   m a x} _ {d ^ {j} \in D} I (\langle d ^ {j}, Z \rangle ; \langle x, Z \rangle \mid \mathcal {H} _ {1: k}), \\ \end{array}
$$
which is the required query as in equation 16. The last equality is obtained by observing that if for any  $d' \in D$ ,  $\| \Pi_k^\perp d' \|_2 = 0$  then  $d'$  can be written as a linear combination of atoms corresponding to queries selected by IP-OMP in the first  $k$  iteration. As a result, the random variable  $\langle d', Z \rangle$  is a function of the previous  $k$  queries selected, namely  $D_k^T Z$ . Hence,  $\langle d', Z \rangle$  is conditionally independent of  $\langle x, Z \rangle$  given history  $\mathcal{H}_{1:k} = \{D_k^T Z = D_k^T z^0\}$ . Since the mutual information between independent random variables is zero, the fourth equality follows.<sup>6</sup>
Last iteration. The last iteration for IP-OMP needs special care for scenarios in which there is no noise and the observed signal  $x = D\beta$ , for some unknown  $\beta \in \mathbb{R}^n$ . This implies that if the algorithm is carried out for large enough  $\tau - 1$  iterations, then eventually, we would reach that stage such that there exists an atom  $d_{\tau} \in D$ , such that  $x$  will lie in the span of  $[D_{\tau - 1} d_{\tau}]$ . In such scenarios, the joint measure of random variables  $(\langle x, Z \rangle, D_{\tau - 1}^T Z, \langle d_{\tau}, Z \rangle)$  is not absolutely continuous with respect to the Lesbegue measure, and thus is not defined according to equation 17. However, we can still define it as the limiting conditional mutual information for a family of random variables,  $\zeta_{\sigma} = \langle x, Z \rangle + \sigma \eta$ , (with  $\eta \sim \mathcal{N}(0, 1)$  independent of  $Z$ ) with  $\langle d_{\tau}, Z \rangle$  in the limit  $\sigma \to 0$ . Specifically, for any  $\sigma > 0$ , we have
$$
\begin{array}{l} I \left(\left\langle d _ {\tau}, Z \right\rangle ; \zeta_ {\sigma} \mid \mathcal {H} _ {1: \tau - 1}\right) = I \left(\left\langle d _ {\tau}, Z \right\rangle ; \zeta_ {\sigma} \mid D _ {\tau - 1} ^ {T} Z = D _ {\tau - 1} ^ {T} z ^ {0}\right) \tag {21} \\ = h \left(\zeta_ {\sigma} \mid D _ {\tau - 1} ^ {T} Z = D _ {\tau - 1} ^ {T} z ^ {0}\right) - h \left(\zeta_ {\sigma} \mid D _ {\tau - 1} ^ {T} Z = D _ {\tau - 1} ^ {T} z ^ {0}, \langle d _ {\tau}, Z \rangle\right). \\ \end{array}
$$
Since  $\zeta_{\sigma}$  is jointly Gaussian with the random variables,  $D_{\tau -1}^{T}Z$  and  $\langle d_{\tau},Z\rangle$ , equation 21 can be rewritten as, for any  $\sigma >0$
$$
\begin{array}{l} I (\langle d _ {\tau}, Z \rangle ; \zeta_ {\sigma} \mid \mathcal {H} _ {1: \tau - 1}) = \frac {1}{2} \log \left(\frac {\operatorname {v a r} (\zeta_ {\sigma} \mid D _ {\tau - 1} ^ {T} Z = D _ {\tau - 1} ^ {T} z ^ {0})}{\operatorname {v a r} (\zeta_ {\sigma} \mid D _ {\tau - 1} ^ {T} Z = D _ {\tau - 1} ^ {T} z ^ {0} , \langle d _ {\tau} , Z \rangle)}\right) \\ = \frac {1}{2} \log \left(\frac {\| \Pi_ {D _ {\tau - 1}} ^ {\perp} x \| _ {2} ^ {2} + \sigma^ {2}}{\sigma^ {2}}\right). \\ \end{array}
$$
The second equality is obtained by applying known formulae of Gaussian conditioning as done in equation 19. Note that  $\| \Pi_{D_{\tau -1}}^{\perp}\| _2$  can also be written as  $\| \Pi_{\tau -1}^{\perp}\| _2$  following our convention.
Finally, taking the limit  $\sigma \to 0$ , we obtain
$$
I \left(\left\langle d _ {\tau}, Z \right\rangle ; \left\langle x, Z \right\rangle \mid \mathcal {H} _ {1: \tau - 1}\right) := \lim  _ {\sigma \rightarrow 0} I \left(\left\langle d _ {\tau}, Z \right\rangle ; \zeta_ {\sigma} \mid \mathcal {H} _ {1: \tau - 1}\right) = \infty ,
$$
Moreover, notice that since, by assumption,  $x$  now lies in the span of  $[D_{\tau -1}d_{\tau}]$ , we can express it as  $x = [D_{\tau -1}d_{\tau}]\alpha$ , for some  $\alpha \in \mathbb{R}^{\tau}$ . This implies that
$$
\Pi_ {\tau - 1} ^ {\perp} x = \Pi_ {\tau - 1} ^ {\perp} [ D _ {\tau - 1} d _ {\tau} ] \alpha = e \Pi_ {\tau - 1} ^ {\perp} d _ {\tau}
$$
for some scalar  $c \in \mathbb{R}$ . Thus,  $\Pi_{\tau - 1}^{\perp} d_{\tau}$  and  $\Pi_{\tau - 1}^{\perp} x$  are collinear vectors indicating that  $d_{\tau}$  achieves the arg max to the IP-OMP objective. That is,
$$
\max _ {d ^ {j} \in D} \frac {| \langle \Pi_ {\tau - 1} ^ {\perp} d ^ {j} , \Pi_ {\tau - 1} ^ {\perp} x \rangle |}{\| \Pi_ {\tau - 1} ^ {\perp} d ^ {j} \| _ {2} \| \Pi_ {\tau - 1} ^ {\perp} x \| _ {2}} = \frac {| \langle \Pi_ {\tau - 1} ^ {\perp} d _ {\tau} , \Pi_ {\tau - 1} ^ {\perp} x \rangle |}{\| \Pi_ {\tau - 1} ^ {\perp} d _ {\tau} \| _ {2} \| \Pi_ {\tau - 1} ^ {\perp} x \| _ {2}} = 1.
$$
To conclude, we have shown that any atom that results in  $x$  being in the span of that atom plus that atoms already selected so far in the first  $\tau - 1$  iterations would be the arg max of the objective in IP-OMP in the last iteration. This iteration would be the last since all subsequent iterations would have no informative queries left as the conditional mutual information for all remaining queries would be 0, since they would be rendered conditionally independent of the target variable given the history  $H_{1:\tau}$  (as described in footnote 6).
Note that although in this case, the conventional definition of MI is not defined since we do not have a joint density with respect to the Lesbegue measure, we can use the more general definition of mutual information [50, Definition 2.1] which depends on the log of the Radon-Nikodym derivative of the joint measure with respect to the product measure (of the marginals) which is always defined. Since  $\langle d', Z \rangle$  is independent of  $\langle x, Z \rangle$  given the history this Radon-Nikodym derivative would be the constant function 1 and hence MI would be zero.
A.3 Proof of equation 7
Proof. Recall that  $\mathcal{H}_{1:\tau} = \{D_{\tau}^{T}Z = D_{\tau}^{T}z^{0}\}$ . As discussed in equation 12, the joint covariance
$$
\operatorname {c o v} \left(\left( \begin{array}{l} \langle x, Z \rangle \\ D _ {\tau} ^ {T} Z \end{array} \right)\right) = \left( \begin{array}{l l} x ^ {T} x & x ^ {T} D _ {\tau} \\ D _ {\tau} ^ {T} x & D _ {\tau} ^ {T} D _ {\tau} \end{array} \right), \tag {22}
$$
where  $D_{\mathcal{L}}$  is replaced by  $D_{\tau}$  since we are no longer dealing with an arbitrary subset of atoms (denoted by  $\mathcal{L}$ ) but rather with the  $\tau$  atoms selected in the first  $\tau$  iterations of IP-OMP.
Consequently from equation 13 we conclude that
$$
P (\langle x, Z \rangle \mid \mathcal {H} _ {1: \tau}) = \mathcal {N} (\langle D _ {\tau} D _ {\tau} ^ {\dagger} x, z ^ {0} \rangle , \| \Pi_ {D _ {\tau}} ^ {\perp} x \| _ {2} ^ {2}).
$$
The result in equation 7 is then obtained by the fact that the mode of a Gaussian is just its mean.
We conclude this proof with a note that in the case when  $x$  can be expressed as a linear combination of atoms in  $D_{\tau}$ , the distribution  $P(\langle x,Z\rangle \mid \mathcal{H}_{1:\tau})$  is a Dirac delta distribution with all its mass concentrated on  $\langle D_{\tau}D_{\tau}^{\dagger}x,z^{0}\rangle$  (see Case 2 in §A.1). As a result, even in this case, IP-OMP's prediction of the target would be  $\langle D_{\tau}D_{\tau}^{\dagger}x,z^{0}\rangle$ .
A.4 Proof of Lemma 2
Proof. Notice from the query selection equations for IP-OMP that the choice of the next query is independent of the particular realization of  $Z$ . This means that, given  $x$ , for all realizations of  $Z \in \mathbb{R}^m$  ( $\mathbb{R}^m$  is the support of an  $m$ -dimensional standard normal), IP-OMP will select the exact same sequence of  $\tau$  dictionary atoms. Thus, using the result from equation 7 we can conclude that IP-OMP's prediction of the target for any  $z \in \mathbb{R}^m$  will be  $\langle D_{\tau}D_{\tau}^{\dagger}x,z\rangle$ .
B Sparse Recovery Experiments: Details and Further Results
B.1 General Setup
To test the sparse recovery performance of IP-OMP against that of OMP, we experimented with IP-OMP on synthetic data. Our setup is a modified version of Tropp and Gilbert's [24], who studied recovery guarantees for OMP with Gaussian dictionaries. For  $n = 256$  and  $n = 1024$ , we generated random dictionaries  $D_{o}$  and sparse codes  $\beta_{o}$  for varying numbers of measurements  $m$  and sparsity levels  $s$ . Each column of the dictionary  $D_{o}$  was drawn from the uniform distribution on the unit sphere in  $\mathbb{R}^{m}$ . The sparse codes  $\beta_{o}$  were generated by fixing some sparsity level  $s$  and drawing the support  $S$  from all cardinality-  $s$  subsets of  $\{1, \ldots, n\}$  uniformly at random. On the support  $(i \in S)$ , the coefficients  $\beta_{o}'$  were generated according to two schemes:
(i)  $\beta_{o}\sim \mathcal{N}(0,1)$  (Gaussian nonzero coefficients), and
(ii)  $\beta_{o} = 1$  (constant nonzero coefficients).
For  $i \notin S$ , the corresponding coefficients were set to zero for both of the above schemes. The second setting, which is supposed to be more challenging for OMP, matches the generation of sparse coefficient vectors in [24].
For each setting of  $n$  and nonzero coefficients (Gaussian or constant), we report recovery for a different grid of  $m$  values. These values, given in Table 1, were selected to emphasize the transition region in the recovery probability curves. For both small ( $n = 256$ ) settings, we simulated sparsities  $s \in \{4, 10, 16, \ldots, 36\}$ . For both large ( $n = 1024$ ) settings, we simulated sparsities  $s \in \{4, 6, 8, \ldots, 16\}$ .
This setup was used to test the sparse recovery of IP-OMP and OMP in both noiseless and noisy settings. Note that for noisy settings, we only experimented with the Gaussian nonzero coefficients for generating  $\beta_{o}$  on the sampled support (setting (i)) due to resource limitations. We will first elaborate on the noiseless setup and then the noisy case.
<sup>7</sup>In high dimensions, Gaussian vectors concentrate in norm [51, Chapter 3], which justifies applying Tropp and Gilbert's problem settings to spherical dictionaries.
Table 1: Values of  $m$  in sparse recovery experiments.
Nonzero coefficients	n = 256	n = 1024
Gaussian	{4,16,28,...,208}	{5,10,15,...,115}
Constant	{4,16,28,...,256}	{5,10,15,...,205}
B.2 Experiments without Noise
For each  $(m,n,s)$  triple, we generated 1000 dictionary-code-signal triples  $(D_o,\beta_o,x_o)$  as described in Appendix §B.1 and ran both OMP and IP-OMP for  $\tau = s$  iterations. For each trial, we checked whether OMP and IP-OMP recover  $\beta_{o}$  exactly. Writing  $\hat{\beta}_{\mathrm{Alg}}$ , to denote the IP or IP-OMP estimate of  $\beta_{o}$ , we define exact recovery to be  $\mathrm{NMSE}(\hat{\beta}_{\mathrm{Alg}},\beta_o) < 10^{-14}$ . For Gaussian nonzero coefficients, we report the fraction of successful trials for each algorithm and  $(m,n,s)$  triple in Figure 2. For constant nonzero coefficients, we report the same in Figure 6. Both Figure 2 and Figure 6 were inspired by similar figures in [24].
Figure 6: Exact recovery probabilities for OMP and IP-OMP with constant nonzero coefficients. Each solid line represents the recovery probability curve for OMP at a fixed sparsity level, as the number of measurements  $m$  increases. The overlaid circles are the corresponding probabilities of recovery by IP-OMP. The left panel is the small setting ( $n = 256$ ), while the right is the large setting ( $n = 1024$ ).
Qualitatively, the constant nonzero coefficient results in Figure 6 are similar to the Gaussian nonzero coefficient results in Figure 2. As discussed in §3, the estimates  $\hat{\beta}_{\mathrm{Alg}}$  degrade gracefully as the measurement rate  $m / n$  (a measure of the indeterminacy of the problem) tends to zero and sparsity rate  $s / m$  (a measure of the complexity of the solution) tends to one. However, the constant nonzero coefficient case is more challenging to recover (probability curves for each sparsity  $s$  are shifted to the right relative to the Gaussian nonzero coefficient case). The difficulty of the constant nonzero coefficient setting has been noted previously for OMP [24].
B.3 Experiments with Noise
As described in §3, we extended the setup in Appendix §B.1 with the addition of Gaussian noise. That is, for each draw of the uniform spherical dictionary  $D_o$  and sparse code  $\beta_o \sim \mathcal{N}(0, I)$ , we also sampled noise  $e \sim \mathcal{N}(0, \sigma^2 I)$  independent of  $D_o$  and  $\beta_o$ . The variance  $\sigma^2$  of this noise was set to achieve a particular  $\mathbb{E}[\mathrm{SNR}]$  according to equation 23. This was done to ensure that the level of noise added to observed signal is consistent across different settings of  $(m, n, s)$  as described in Table 1. The natural scale for varying the  $\mathbb{E}[\mathrm{SNR}]$  values is logarithmic; in particular, we chose to use the decibel (dB) scale, where we take  $\mathbb{E}[\mathrm{SNR}]$  in dB units to be defined as