SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $b_{1}, \ldots, b_{d - d_{\delta}}$  be a set of orthonormal vectors that are orthogonal to  $\operatorname{span}(V^{\pi_1}, \ldots, V^{\pi_{d_{\delta}}})$  and together with  $V^{\pi_1}, \ldots, V^{\pi_{d_{\delta}}}$  form a basis for  $\operatorname{span}(\{V^{\pi} | \pi \in \Pi\})$ .
We define  $A^{(\mathrm{full})}\in \mathbb{R}^{d\times k}$  as the matrix of replacing the last  $d - d_{\delta}$  rows of  $A$  with  $b_{1},\ldots ,b_{d - d_{\delta}}$  , i.e.,
$$
A ^ {(\text {f u l l})} = \left( \begin{array}{c} V ^ {\pi_ {1} \top} \\ (\alpha_ {1} V ^ {\pi_ {1}} - V ^ {\pi_ {2}}) ^ {\top} \\ \vdots \\ (\alpha_ {d _ {\delta} - 1} V ^ {\pi_ {1}} - V ^ {\pi_ {d _ {\delta}}}) ^ {\top} \\ b _ {1} ^ {\top} \\ \vdots \\ b _ {d - d _ {\delta}} ^ {\top} \end{array} \right).
$$
Observation 1. We have that  $\operatorname{span}(A^{(full)}) = \operatorname{span}(\{V^{\pi}|\pi \in \Pi\})$  and  $\operatorname{rank}(A^{(full)}) = d$ .
Lemma 5. For all  $w \in \mathbb{R}^k$  satisfying  $A^{(full)}w = \mathbf{e}_1$ , we have  $|w \cdot V^\pi - w' \cdot V^\pi| \leq \sqrt{k}\delta \| w' \|_2$  for all  $\pi$ .
We then show that there exists a  $w \in \mathbb{R}^k$  satisfying  $A^{(\mathrm{full})} w = \mathbf{e}_1$  such that  $\left|\widehat{w}^{(\delta)} \cdot V^{\pi} - w \cdot V^{\pi}\right|$  is small for all  $\pi \in \Pi$ .
Lemma 6. If  $|\widehat{\alpha}_i - \alpha_i| \leq \epsilon_\alpha$  and  $\alpha_i \leq C_\alpha$  for all  $i \in [d-1]$ , for every  $\delta \geq 4C_{\alpha}^{\frac{2}{3}}C_Vd^{\frac{1}{3}}\epsilon_{\alpha}^{\frac{1}{3}}$  there exists a  $w \in \mathbb{R}^k$  satisfying  $A^{(full)}w = \mathbf{e}_1$  s.t.  $\left|\widehat{w}^{(\delta)} \cdot V^{\pi} - w \cdot V^{\pi}\right| \leq \mathcal{O}\left(\frac{C_{\alpha}C_{V}^{4}d^{\frac{3}{\delta}}\left\|w'\right\|_{2}^{2}\epsilon_{\alpha}}{\delta^{2}}\right)$  for all  $\pi$ .
We now derive Lemma 2 using the above two lemmas.
Proof of Lemma 2. Let  $w$  be defined in Lemma 6. Then for any policy  $\pi$ , we have
$$
\begin{array}{l} \left| \widehat {w} ^ {(\delta)} \cdot V ^ {\pi} - w ^ {\prime} \cdot V ^ {\pi} \right| \leq \left| \widehat {w} ^ {(\delta)} \cdot V ^ {\pi} - w \cdot V ^ {\pi} \right| + | w \cdot V ^ {\pi} - w ^ {\prime} \cdot V ^ {\pi} | \\ \leq \mathcal {O} \big (\frac {C _ {\alpha} C _ {V} ^ {4} d _ {\delta} ^ {\frac {3}{2}} \| w ^ {\prime} \| _ {2} ^ {2} \epsilon_ {\alpha}}{\delta^ {2}} + \sqrt {k} \delta \| w ^ {\prime} \| _ {2} \big), \\ \end{array}
$$
by applying Lemma 5 and 6.
F Proofs of Lemma 5 and Lemma 6
Proof of Lemma 5. Since  $\mathrm{span}(A^{(\mathrm{full})}) = \mathrm{span}(\{V^{\pi}|\pi \in \Pi \})$ , for every policy  $\pi$ , the value vector can be represented as a linear combination of row vectors of  $A^{(\mathrm{full})}$ , i.e., there exists  $a = (a_{1},\ldots ,a_{d})\in \mathbb{R}^{d}$  s.t.
$$
V ^ {\pi} = \sum_ {i = 1} ^ {d} a _ {i} A _ {i} ^ {(\text {f u l l})} = A ^ {(\text {f u l l}) \top} a. \tag {8}
$$
Now, for any unit vector  $\xi \in \operatorname{span}(b_1, \ldots, b_{d - d_\delta})$ , we have  $\langle V^\pi, \xi \rangle \leq \sqrt{k} \delta$ .
The reason is that at each round  $d_{\delta} + 1$ , we pick an orthonormal basis  $\rho_1, \ldots, \rho_{k - d_{\delta}}$  of  $\operatorname{span}(V^{\pi_1}, \ldots, V^{\pi_{d_{\delta}}})^\perp$  (line 8 in Algorithm 1) and pick  $u_{d_{\delta} + 1}$  to be the one in which there exists a policy with the largest component as described in line 9. Hence,  $|\langle \rho_j, V^\pi \rangle| \leq \delta$  for all  $j \in [k - d_{\delta}]$ .
It follows from Cauchy-Schwarz inequality that  $\langle \xi, V^{\pi} \rangle = \sum_{j=1}^{k - d_{\delta}} \langle \xi, \rho_j \rangle \langle \rho_j, V^{\pi} \rangle \leq \sqrt{k} \delta$ .
Combining with the observation that  $b_{1}, \ldots, b_{d - d_{\delta}}$  are pairwise orthogonal and that each of them is orthogonal to  $\operatorname{span}(V^{\pi_1}, \ldots, V^{\pi_{d_{\delta}}})$  we have
$$
\sum_ {i = d _ {\delta} + 1} ^ {d} a _ {i} ^ {2} = \left| \left\langle V ^ {\pi}, \sum_ {i = d _ {\delta} + 1} ^ {d} a _ {i} b _ {i - d _ {\delta}} \right\rangle \right| \leq \sqrt {\sum_ {i = d _ {\delta} + 1} ^ {d} a _ {i} ^ {2}} \sqrt {k} \delta ,
$$
$$
\sqrt {\sum_ {i = d _ {\delta} + 1} ^ {d} a _ {i} ^ {2}} \leq \sqrt {k} \delta . \tag {9}
$$
Since  $w^{\prime}$  satisfies  $Aw^{\prime} = \mathbf{e}_{1}$ , we have
$$
A ^ {(\text {f u l l})} w ^ {\prime} = (1, 0, \dots , 0, \langle b _ {1}, w ^ {\prime} \rangle , \dots , \langle b _ {d - d _ {s}}, w ^ {\prime} \rangle).
$$
For any  $w \in \mathbb{R}^k$  satisfying  $A^{(\mathrm{full})}w = \mathbf{e}_1$ , consider  $\widetilde{w} = w + \sum_{i=1}^{d - d_{\delta}} \langle b_i, w' \rangle b_i$ . Then we have  $A^{(\mathrm{full})}\widetilde{w} = A^{(\mathrm{full})}w'$ .
Thus, applying (8) twice, we get
$$
\widetilde {w} \cdot V ^ {\pi} = \widetilde {w} ^ {\top} A ^ {(\text {f u l l}) \top} a = w ^ {\prime \top} A ^ {(\text {f u l l}) \top} a = w ^ {\prime} \cdot V ^ {\pi}.
$$
$$
\begin{array}{l} \left| w \cdot V ^ {\pi} - w ^ {\prime} \cdot V ^ {\pi} \right| = \left| w \cdot V ^ {\pi} - \widetilde {w} \cdot V ^ {\pi} \right| \stackrel {(a)} {=} \left| \sum_ {i = 1} ^ {d} a _ {i} (w - \widetilde {w}) \cdot A _ {i} ^ {(\text {f u l l})} \right| \\ = \left| \sum_ {i = d _ {\delta} + 1} ^ {d} a _ {i} \left\langle b _ {i - d _ {\delta}}, w ^ {\prime} \right\rangle \right| \stackrel {(b)} {\leq} \sqrt {\sum_ {i = d _ {\delta} + 1} ^ {d} a _ {i} ^ {2}} \| w ^ {\prime} \| _ {2} \stackrel {(c)} {\leq} \sqrt {k} \delta \| w ^ {\prime} \| _ {2}, \\ \end{array}
$$
where Eq (a) follows from (8), inequality (b) from Cauchy-Schwarz, and inequality (c) from applying (9).
Lemma 6. If  $|\widehat{\alpha}_i - \alpha_i| \leq \epsilon_\alpha$  and  $\alpha_i \leq C_\alpha$  for all  $i \in [d-1]$ , for every  $\delta \geq 4C_\alpha^{\frac{2}{3}}C_V d^{\frac{1}{3}}\epsilon_\alpha^{\frac{1}{3}}$  there exists a  $w \in \mathbb{R}^k$  satisfying  $A^{(full)}w = \mathbf{e}_1$  s.t.  $\left|\widehat{w}^{(\delta)} \cdot V^\pi - w \cdot V^\pi\right| \leq \mathcal{O}\left(\frac{C_\alpha C_V^4 d_{\delta}^{\frac{3}{2}}\left\|w'\right\|_2^2\epsilon_\alpha}{\delta^2}\right)$  for all  $\pi$ .
Before proving Lemma 6, we introduce some notations and a claim.
- For any  $x, y \in \mathbb{R}^k$ , let  $\theta(x, y)$  denotes the angle between  $x$  and  $y$ .
- For any subspace  $U \subset \mathbb{R}^k$ , let  $\theta(x, U) \coloneqq \min_{y \in U} \theta(x, y)$ .
- For any two subspaces  $U, U' \subset \mathbb{R}^k$ , we define  $\theta(U, U')$  as  $\theta(U, U') = \max_{x \in U} \min_{y \in U'} \theta(x, y)$ .
- For any matrix  $M$ , let  $M_{i}$  denote the  $i$ -th row vector of  $M$ ,  $M_{i:j}$  denote the submatrix of  $M$  composed of rows  $i, i + 1, \ldots, j$ , and  $M_{i}$  denote the submatrix composed of all rows  $j \geq i$ .
- Let  $\operatorname{span}(M)$  denote the span of the rows of  $M$ .
Recall that  $\widehat{A} \in \mathbb{R}^{d \times k}$  is defined as
$$
\widehat {A} = \left( \begin{array}{c} V ^ {\pi_ {1} \top} \\ (\widehat {\alpha} _ {1} V ^ {\pi_ {1}} - V ^ {\pi_ {2}}) ^ {\top} \\ \vdots \\ (\widehat {\alpha} _ {d - 1} V ^ {\pi_ {1}} - V ^ {\pi_ {d}}) ^ {\top} \end{array} \right),
$$
which is the approximation of matrix  $A \in \mathbb{R}^{d \times k}$  defined by true values of  $\alpha_{i}$ , i.e.,
$$
A = \left( \begin{array}{c} V ^ {\pi_ {1} \top} \\ (\alpha_ {1} V ^ {\pi_ {1}} - V ^ {\pi_ {2}}) ^ {\top} \\ \vdots \\ (\alpha_ {d - 1} V ^ {\pi_ {1}} - V ^ {\pi_ {d}}) ^ {\top} \end{array} \right).
$$
We denote by  $\widehat{A}^{(\delta)} = \widehat{A}_{1:d_{\delta}}, A^{(\delta)} = A_{1:d_{\delta}} \in \mathbb{R}^{d_{\delta} \times k}$  the sub-matrices comprised of the first  $d_{\delta}$  rows of  $\widehat{A}$  and  $A$  respectively.
Claim 1. If  $|\widehat{\alpha}_i - \alpha_i| \leq \epsilon_\alpha$  and  $\alpha_i \leq C_\alpha$  for all  $i \in [d-1]$ , for every  $\delta \geq 4C_\alpha^{\frac{2}{3}}C_V d^{\frac{1}{3}}\epsilon_\alpha^{\frac{1}{3}}$ , we have
$$
\theta \left(\operatorname {s p a n} \left(A _ {2:} ^ {(\delta)}\right), \operatorname {s p a n} \left(\widehat {A} _ {2:} ^ {(\delta)}\right)\right) \leq \eta_ {\epsilon_ {\alpha}, \delta}, \tag {10}
$$
$$
\theta \left(\operatorname {s p a n} \left(\widehat {A} _ {2:} ^ {(\delta)}\right), \operatorname {s p a n} \left(A _ {2:} ^ {(\delta)}\right)\right) \leq \eta_ {\epsilon_ {\alpha}, \delta}, \tag {11}
$$
where  $\eta_{\epsilon_{\alpha},\delta} = \frac{4C_{\alpha}C_{V}^{2}d_{\delta}\epsilon_{\alpha}}{\delta^{2}}.$
To prove the above claim, we use the following lemma by Balcan et al. (2015).
Lemma 7 (Lemma 3 of Balcan et al. (2015)). Let  $U_{l} = \mathrm{span}(\xi_{1},\dots,\xi_{l})$  and  $\widehat{U}_l = \mathrm{span}(\widehat{\xi}_1,\dots ,\widehat{\xi}_l)$ . Let  $\epsilon_{acc},\gamma_{new}\geq 0$  and  $\epsilon_{acc}\leq \gamma_{new}^2 /(10l)$ , and assume that  $\theta (\widehat{\xi}_i,\widehat{U}_{i - 1})\geq \gamma_{new}$  for  $i = 2,\ldots ,l$ , and that  $\theta (\xi_i,\widehat{\xi}_i)\leq \epsilon_{acc}$  for  $i = 1,\dots,l$ .
$$
\theta (U _ {l}, \widehat {U} _ {l}) \leq 2 l \frac {\epsilon_ {a c c}}{\gamma_ {n e w}}.
$$
Proof of Claim 1. For all  $2 \leq i \leq d_{\delta}$ , we have that
$$
\begin{array}{l} \theta (\widehat {A} _ {i} ^ {(\delta)}, \operatorname {s p a n} (\widehat {A} _ {2: i - 1} ^ {(\delta)})) \geq \theta (\widehat {A} _ {i} ^ {(\delta)}, \operatorname {s p a n} (\widehat {A} _ {1: i - 1} ^ {(\delta)})) \geq \sin (\theta (\widehat {A} _ {i} ^ {(\delta)}, \operatorname {s p a n} (\widehat {A} _ {1: i - 1} ^ {(\delta)}))) \\ \stackrel {(a)} {\geq} \frac {\left| \widehat {A} _ {i} ^ {(\delta)} \cdot u _ {i} \right|}{\left\| \widehat {A} _ {i} ^ {(\delta)} \right\| _ {2}} \stackrel {(b)} {\geq} \frac {\delta}{\left\| \widehat {A} _ {i} ^ {(\delta)} \right\| _ {2}} = \frac {\delta}{\left\| \widehat {\alpha} _ {i - 1} V ^ {\pi_ {1}} - V ^ {\pi_ {i}} \right\| _ {2}} \geq \frac {\delta}{(C _ {\alpha} + 1) C _ {V}}, \\ \end{array}
$$
where Ineq (a) holds as  $u_{i}$  is orthogonal to  $\operatorname{span}(\widehat{A}_{1:i-1}^{(\delta)})$  according to line 8 of Algorithm 1 and Ineq (b) holds due to  $\left| \widehat{A}_{i}^{(\delta)} \cdot u_{i} \right| = |V^{\pi_{i}} \cdot u_{i}| \geq \delta$ . The last inequality holds due to  $\| \widehat{\alpha}_{i-1} V^{\pi_{1}} - V^{\pi_{i}} \|_{2} \leq \widehat{\alpha}_{i-1} \| V^{\pi_{1}} \|_{2} + \| V_{i} \|_{2} \leq (C_{\alpha} + 1) C_{V}$ .
$$
\theta \left(A _ {i} ^ {(\delta)}, \operatorname {s p a n} \left(A _ {2: i - 1} ^ {(\delta)}\right)\right) \geq \frac {\delta}{\left(C _ {\alpha} + 1\right) C _ {V}}.
$$
We continue by decomposing  $V^{\pi_i}$  in the direction of  $V^{\pi_1}$  and the direction perpendicular to  $V^{\pi_1}$ .
For convince, we denote  $v_{i}^{\parallel} \coloneqq V^{\pi_{i}} \cdot \frac{V^{\pi_{1}}}{\|V^{\pi_{1}}\|_{2}}, V_{i}^{\parallel} \coloneqq v_{i}^{\parallel} \frac{V^{\pi_{1}}}{\|V^{\pi_{1}}\|_{2}}, V_{i}^{\perp} \coloneqq V^{\pi_{i}} - V_{i}^{\parallel}$  and  $v_{i}^{\perp} \coloneqq \left\| V_{i}^{\perp}\right\|_{2}$ .
$$
\theta \left(A _ {i} ^ {(\delta)}, \widehat {A} _ {i} ^ {(\delta)}\right) = \theta \left(\alpha_ {i - 1} V ^ {\pi_ {1}} - V ^ {\pi_ {i}}, \widehat {\alpha} _ {i - 1} V ^ {\pi_ {1}} - V ^ {\pi_ {i}}\right) = \theta \left(\alpha_ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} - V _ {i} ^ {\perp}, \widehat {\alpha} _ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} - V _ {i} ^ {\perp}\right).
$$
If  $(\widehat{\alpha}_{i-1}V^{\pi_1} - V_i^{\parallel}) \cdot (\alpha_{i-1}V^{\pi_1} - V_i^{\parallel}) \geq 0$ , i.e.,  $\widehat{\alpha}_{i-1}V^{\pi_1} - V_i^{\parallel}$  and  $\alpha_{i-1}V^{\pi_1} - V_i^{\parallel}$  are in the same direction, then
$$
\begin{array}{l} \theta \left(A _ {i} ^ {(\delta)}, \widehat {A} _ {i} ^ {(\delta)}\right) = \left| \arctan \frac {\left\| \widehat {\alpha} _ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} \right\| _ {2}}{v _ {i} ^ {\perp}} - \arctan \frac {\left\| \alpha_ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} \right\| _ {2}}{v _ {i} ^ {\perp}} \right| \\ \leq \left| \frac {\left\| \widehat {\alpha} _ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} \right\| _ {2}}{v _ {i} ^ {\perp}} - \frac {\left\| \alpha_ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} \right\| _ {2}}{v _ {i} ^ {\perp}} \right| (12) \\ = \frac {\left| \widehat {\alpha} _ {i - 1} - \alpha_ {i - 1} \right| \| V ^ {\pi_ {1}} \| _ {2}}{v _ {i} ^ {\perp}} \\ \leq \frac {\epsilon_ {\alpha} C _ {V}}{\delta}, (13) \\ \end{array}
$$
where Ineq (12) follows from the fact that the derivative of  $\arctan x$  is at most 1, i.e.,  $\frac{\partial \arctan x}{\partial x} = \lim_{a \to x} \frac{\arctan a - \arctan x}{a - x} = \frac{1}{1 + x^2} \leq 1$ .
Inequality (13) holds since  $v_{i}^{\perp} \geq |\langle V^{\pi_{i}}, u_{i} \rangle| \geq \delta$ .
If  $(\widehat{\alpha}_{i-1}V^{\pi_1} - V_i^{\parallel}) \cdot (\alpha_{i-1}V^{\pi_1} - V_i^{\parallel}) < 0$ , i.e.,  $\widehat{\alpha}_{i-1}V^{\pi_1} - V_i^{\parallel}$  and  $\alpha_{i-1}V^{\pi_1} - V_i^{\parallel}$  are in the opposite directions, then we have  $\left\| \widehat{\alpha}_{i-1}V^{\pi_1} - V_i^{\parallel} \right\|_2 + \left\| \alpha_{i-1}V^{\pi_1} - V_i^{\parallel} \right\|_2 = \left\| (\widehat{\alpha}_{i-1} - \alpha_{i-1})V^{\pi_1} \right\|_2 \leq \epsilon_\alpha \| V^{\pi_1} \|_2$ .
$$
\begin{array}{l} \theta \left(\widehat {A} _ {i} ^ {(\delta)}, A _ {i} ^ {(\delta)}\right) = \left| \arctan \frac {\left\| \widehat {\alpha} _ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} \right\| _ {2}}{v _ {i} ^ {\perp}} + \arctan \frac {\left\| \alpha_ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} \right\| _ {2}}{v _ {i} ^ {\perp}} \right| \\ \leq \left| \frac {\left\| \widehat {\alpha} _ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} \right\| _ {2}}{v _ {i} ^ {\perp}} + \frac {\left\| \alpha_ {i - 1} V ^ {\pi_ {1}} - V _ {i} ^ {\parallel} \right\| _ {2}}{v _ {i} ^ {\perp}} \right| \\ \leq \frac {\epsilon_ {\alpha} \left\| V ^ {\pi_ {1}} \right\| _ {2}}{\left| v _ {i} ^ {\perp} \right|} \\ \leq \frac {\epsilon_ {\alpha} C _ {V}}{\delta}. \\ \end{array}
$$
By applying Lemma 7 with  $\epsilon_{\mathrm{acc}} = \frac{\epsilon_{\alpha}C_V}{\delta}$ ,  $\gamma_{\mathrm{new}} = \frac{\delta}{(C_{\alpha} + 1)C_V}$ ,  $(\xi_i,\widehat{\xi}_i) = (A_{i + 1},\widehat{A}_{i + 1})$  (and  $(\xi_i,\widehat{\xi}_i) = (\widehat{A}_{i + 1},A_{i + 1}))$ , we have that when  $\delta \geq 10^{\frac{1}{3}}(C_{\alpha} + 1)^{\frac{2}{3}}C_{V}d_{\delta}^{\frac{1}{3}}\epsilon_{\alpha}^{\frac{1}{3}}$ ,
$$
\theta (\operatorname {s p a n} (A _ {2:} ^ {(\delta)}), \operatorname {s p a n} (\widehat {A} _ {2:} ^ {(\delta)})) \leq \frac {2 d _ {\delta} (C _ {\alpha} + 1) C _ {V} ^ {2} \epsilon_ {\alpha}}{\delta^ {2}} = \eta_ {\epsilon_ {\alpha}, \delta},
$$
$$
\theta (\operatorname {s p a n} \left(\widehat {A} _ {2:} ^ {(\delta)}\right), \operatorname {s p a n} \left(A _ {2:} ^ {(\delta)}\right)) \leq \eta_ {\epsilon_ {\alpha}, \delta}.
$$
This completes the proof of Claim 1 since  $C_{\alpha}\geq 1$
Proof of Lemma 6. Recall that  $\widehat{w}^{(\delta)} = \arg \min_{\widehat{A}^{(\delta)}x = \mathbf{e}_1}\| x\| _2$  is the minimum norm solution to  $\widehat{A}^{(\delta)}x = \mathbf{e}_1$ .
Thus,  $\langle \widehat{w}^{(\delta)},b_i\rangle = 0$  for all  $i\in [d - d_{\delta}]$
Let  $\lambda_1, \ldots, \lambda_{d_{\delta} - 1}$  be any orthonormal basis of  $\operatorname{span}(A_{2:}^{(\delta)})$ .
We construct a vector  $w$  satisfying  $A^{\mathrm{(full)}}w = \mathbf{e}_1$  by removing  $\widehat{w}^{(\delta)}$ 's component in  $\operatorname{span}(A_{2:}^{\delta})$  and rescaling.
$$
w := \frac {\hat {w} ^ {(\delta)} - \sum_ {i = 1} ^ {d _ {\delta} - 1} \left\langle \hat {w} ^ {(\delta)} , \lambda_ {i} \right\rangle \lambda_ {i}}{1 - V ^ {\pi_ {1}} \cdot \left(\sum_ {i = 1} ^ {d _ {\delta} - 1} \left\langle \hat {w} ^ {(\delta)} , \lambda_ {i} \right\rangle \lambda_ {i}\right)}. \tag {14}
$$
It is direct to verify that  $A_1 \cdot w = V^{\pi_1} \cdot w = 1$  and  $A_i \cdot w = 0$  for  $i = 2, \ldots, d_{\delta}$ . As a result,  $A^{(\delta)}w = \mathbf{e}_1$ .
Combining with the fact that  $\widehat{w}^{(\delta)}$  has zero component in  $b_{i}$  for all  $i\in [d - d_{\delta}]$ , we have  $A^{\mathrm{(full)}}w = \mathbf{e}_1$ .
According to Claim 1, we have
$$
\theta (\operatorname {s p a n} (A _ {2:} ^ {(\delta)}), \operatorname {s p a n} (\widehat {A} _ {2:} ^ {(\delta)})) \leq \eta_ {\epsilon_ {\alpha}, \delta}.
$$
Thus, there exist unit vectors  $\widetilde{\lambda}_1,\ldots ,\widetilde{\lambda}_{d_\delta -1}\in \mathrm{span}(\widehat{A}_{2:}^{(\delta)})$  such that  $\theta (\lambda_i,\widetilde{\lambda}_i)\leq \eta_{\epsilon_\alpha ,\delta}$
Since  $\widehat{A}^{(\delta)}\widehat{w}^{(\delta)} = \mathbf{e}_1$ , we have  $\widehat{w}^{(\delta)}\cdot \widetilde{\lambda}_i = 0$  for all  $i = 1,\ldots ,d_{\delta} - 1$ , and therefore,
$$
\left| \widehat {w} ^ {(\delta)} \cdot \lambda_ {i} \right| = \left| \widehat {w} ^ {(\delta)} \cdot \left(\lambda_ {i} - \widetilde {\lambda} _ {i}\right) \right| \leq \left\| \widehat {w} ^ {(\delta)} \right\| _ {2} \eta_ {\epsilon_ {\alpha}, \delta}.
$$
This implies that for any policy  $\pi$
$$
\left| V ^ {\pi} \cdot \sum_ {i = 1} ^ {d _ {\delta} - 1} (\widehat {w} ^ {(\delta)} \cdot \lambda_ {i}) \lambda_ {i} \right| \leq \| V ^ {\pi} \| _ {2} \sqrt {d _ {\delta}} \left\| \widehat {w} ^ {(\delta)} \right\| _ {2} \eta_ {\epsilon_ {\alpha}, \delta} \leq C _ {V} \sqrt {d _ {\delta}} \left\| \widehat {w} ^ {(\delta)} \right\| _ {2} \eta_ {\epsilon_ {\alpha}, \delta}.
$$
Denote by  $\gamma = V^{\pi_1} \cdot \left( \sum_{i=1}^{d_\delta - 1} \left\langle \widehat{w}^{(\delta)}, \lambda_i \right\rangle \lambda_i \right)$ , which is no greater than  $C_V \sqrt{d_\delta} \left\| \widehat{w}^{(\delta)} \right\|_2 \eta_{\epsilon_\alpha, \delta}$ .
$$
\begin{array}{l} \left| \widehat {w} ^ {(\delta)} \cdot V ^ {\pi} - w \cdot V ^ {\pi} \right| \leq \left| \widehat {w} ^ {(\delta)} \cdot V ^ {\pi} - \frac {1}{1 - \gamma} \widehat {w} ^ {(\delta)} \cdot V ^ {\pi} \right| + \left| \frac {1}{1 - \gamma} \widehat {w} ^ {(\delta)} \cdot V ^ {\pi} - w \cdot V ^ {\pi} \right| \\ \leq \frac {\gamma \left\| \widehat {w} ^ {(\delta)} \right\| _ {2} C _ {V}}{1 - \gamma} + \frac {1}{1 - \gamma} \left| \sum_ {i = 1} ^ {d _ {\delta} - 1} \left(\widehat {w} ^ {(\delta)} \cdot \lambda_ {i}\right) \lambda_ {i} \cdot V ^ {\pi} \right| \\ \leq \frac {\gamma \left\| \widehat {w} ^ {(\delta)} \right\| _ {2} C _ {V}}{1 - \gamma} + \frac {C _ {V} \sqrt {d _ {\delta}} \left\| \widehat {w} ^ {(\delta)} \right\| _ {2} \eta_ {\epsilon_ {\alpha} , \delta}}{1 - \gamma} \\ = 2 \left(C _ {V} \left\| \widehat {w} ^ {(\delta)} \right\| _ {2} + 1\right) C _ {V} \sqrt {d _ {\delta}} \left\| \widehat {w} ^ {(\delta)} \right\| _ {2} \eta_ {\epsilon_ {\alpha}, \delta}, \tag {15} \\ \end{array}
$$
where the last equality holds when  $C_V\sqrt{d_\delta}\left\| \widehat{w}^{(\delta)}\right\| _2\eta_{\epsilon_\alpha ,\delta}\leq \frac{1}{2}$
Now we show that  $\left\| \widehat{w}^{(\delta)}\right\| _2\leq C\left\| w'\right\| _2$  for some constant  $C$
Since  $\widehat{w}^{(\delta)}$  is the minimum norm solution to  $\widehat{A}^{(\delta)}x = \mathbf{e}_1$ , we will construct another solution to  $\widehat{A}^{(\delta)}x = \mathbf{e}_1$ , denoted by  $\widehat{w}_0$  in a similar manner to the construction in Eq (14), and show that  $\| \widehat{w}_0\| _2\leq C\| w'\|$ .
Let  $\xi_1, \ldots, \xi_{d_\delta - 1}$  be any orthonormal basis of  $\operatorname{span}(\widehat{A}_{2:}^{(\delta)})$ .
We construct a  $\widehat{w}_0$  s.t.  $\widehat{A}^{(\delta)}\widehat{w}_0 = \mathbf{e}_1$  by removing the component of  $w^{\prime}$  in  $\operatorname {span}(\widehat{A}_{2:}^{(\delta)})$  and rescaling. Specifically, let
$$
\widehat {w} _ {0} = \frac {w ^ {\prime} - \sum_ {i = 1} ^ {d _ {\delta} - 1} \left\langle w ^ {\prime} , \xi_ {i} \right\rangle \xi_ {i}}{1 - \left\langle V ^ {\pi_ {1}} , \left(\sum_ {i = 1} ^ {d _ {\delta} - 1} \left\langle w ^ {\prime} , \xi_ {i} \right\rangle \xi_ {i}\right) \right\rangle}. \tag {16}
$$
Since  $A^{(\delta)}w' = \mathbf{e}_1$ , it directly follows that  $\left\langle \widehat{A}_1, \widehat{w}_0 \right\rangle = \langle V^{\pi_1}, \widehat{w}_0 \rangle = 1$  and that  $\left\langle \widehat{A}_i, \widehat{w}_0 \right\rangle = 0$  for  $i = 2, \ldots, d_{\delta}$ , i.e.,  $\widehat{A}^{(\delta)}\widehat{w}_0 = \mathbf{e}_1$ .
Since Claim 1 implies that  $\theta(\operatorname{span}(\widehat{A}_{2:}^{(\delta)}), \operatorname{span}(A_{2:}^{(\delta)})) \leq \eta_{\epsilon_{\alpha}, \delta}$ , there exist unit vectors  $\widetilde{\xi}_1, \ldots, \widetilde{\xi}_{d_{\delta}-1} \in \operatorname{span}(A_{2:}^{(\delta)})$  such that  $\theta(\xi_i, \widetilde{\xi}_i) \leq \eta_{\epsilon_{\alpha}, \delta}$ .
As  $w'$  has zero component in  $\operatorname{span}(A_{2:}^{(\delta)})$ ,  $w'$  should have a small component in  $\operatorname{span}(\widehat{A}_{2:}^{(\delta)})$ .
$$
| \langle w ^ {\prime}, \xi_ {i} \rangle | = \left| \left\langle w ^ {\prime}, \xi_ {i} - \widetilde {\xi_ {i}} \right\rangle \right| \leq \| w ^ {\prime} \| _ {2} \eta_ {\epsilon_ {\alpha}, \delta},
$$
$$
\left\| \sum_ {i = 1} ^ {d _ {\delta} - 1} \langle w ^ {\prime}, \xi_ {i} \rangle \xi_ {i} \right\| _ {2} \leq \sqrt {d _ {\delta}} \| w ^ {\prime} \| _ {2} \eta_ {\epsilon_ {\alpha}, \delta}.
$$
$$
\left| \left\langle V ^ {\pi_ {1}}, \left(\sum_ {i = 1} ^ {d _ {\delta} - 1} \left\langle w ^ {\prime}, \xi_ {i} \right\rangle \xi_ {i}\right) \right\rangle \right| \leq C _ {V} \sqrt {d _ {\delta}} \| w ^ {\prime} \| _ {2} \eta_ {\epsilon_ {\alpha}, \delta}.
$$
As a result,  $\| \widehat{w}_0\| _2\leq \frac{3}{2}\| w'\| _2$  when  $C_V\sqrt{d_\delta}\| w'\| _2\eta_{\epsilon_\alpha ,\delta}\leq \frac{1}{3}$ , which is true when  $\epsilon_{\alpha}$  is small enough.
According to Lemma 1,  $\epsilon_{\alpha} \leq \frac{4k^{2}\epsilon}{v^{*}}$ ,  $\epsilon_{\alpha} \to 0$  as  $\epsilon \to 0$ .
Thus, we have  $\left\| \widehat{w}^{(\delta)}\right\| _2\leq \left\| \widehat{w}_0\right\| _2\leq \frac{3}{2}\left\| w'\right\| _2$  and  $C_V\sqrt{d_\delta}\left\| \widehat{w}^{(\delta)}\right\| _2\eta_{\epsilon_\alpha ,\delta}\leq \frac{1}{2}$
Combined with Eq (15), we get
$$
\left| \widehat {w} ^ {(\delta)} \cdot V ^ {\pi} - w \cdot V ^ {\pi} \right| = \mathcal {O} \left((C _ {V} \| w ^ {\prime} \| _ {2} + 1) C _ {V} \sqrt {d _ {\delta}} \| w ^ {\prime} \| _ {2} \eta_ {\epsilon_ {\alpha}, \delta}\right).
$$
Since  $C_V\| w'\|_2 \geq |\langle V^{\pi_1}, w' \rangle| = 1$ , by taking  $\eta_{\epsilon_\alpha, \delta} = \frac{4C_\alpha C_V^2 d_\delta \epsilon_\alpha}{\delta^2}$  into the above equation, we have
$$
\left| \widehat {w} ^ {(\delta)} \cdot V ^ {\pi} - w \cdot V ^ {\pi} \right| = \mathcal {O} \left(\frac {C _ {\alpha} C _ {V} ^ {4} d _ {\delta} ^ {\frac {3}{2}} \| w ^ {\prime} \| _ {2} ^ {2} \epsilon_ {\alpha}}{\delta^ {2}}\right),
$$
G Proof of Lemma 3
Proof. Given the output  $(V^{\pi_1},\ldots ,V^{\pi_d})$  of Algorithm 1, we have rank  $\mathrm{(span}(\{V^{\pi_1},\dots ,V^{\pi_{d_\delta}}\})\big) = d_\delta$
For  $i = d_{\delta} + 1, \ldots, d$ , let  $\psi_i$  be the normalized vector of  $V^{\pi_i}$ 's projection into  $\operatorname{span}(V^{\pi_1}, \ldots, V^{\pi_{i-1}})^\perp$  with  $\|\psi_i\|_2 = 1$ .
Then we have that  $\operatorname{span}(V^{\pi_1},\ldots ,V^{\pi_{i - 1}},\psi_i) = \operatorname{span}(V^{\pi_1},\ldots ,V^{\pi_i})$  and that  $\{\psi_i|i = d_\delta + 1,\dots ,d\}$  are orthonormal.
For every policy  $\pi$ , the value vector can be represented as a linear combination of  $\widehat{A}_1, \ldots, \widehat{A}_{d_\delta}, \psi_{d_\delta + 1}, \ldots, \psi_d$ , i.e., there exists a unique  $a = (a_1, \ldots, a_d) \in \mathbb{R}^d$  s.t.  $V^\pi = \sum_{i=1}^{d_\delta} a_i \widehat{A}_i + \sum_{i=d_\delta + 1}^d a_i \psi_i$ .
Since  $\psi_{i}$  is orthogonal to  $\psi_{j}$  for all  $j\neq i$  and  $\psi_{i}$  is are orthogonal to  $\operatorname {span}(\widehat{A}_1,\ldots ,\widehat{A}_{d_\delta})$ , we have  $a_{i} = \langle V^{\pi},\psi_{i}\rangle$  for  $i\geq d_{\delta} + 1$ .