SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2 Expanding and compressing trajectories
1: compute  $V^{\pi}(s,h)$  for all  $s\in S,h\in [H]$  by dynamic programming according to Eq (4)   
2:  $F^{(0)} = \{(s_0)\}$  and  $\beta^{(0)}(s_0) = 1$    
3: for  $t = 0,\dots ,H - 1$  do   
4:  $J_{F^{(t)}}\gets \{J(\tau \circ s)|\tau \in F^{(t)},s\in S\}$  and  $p_{F^{(t)},\beta^{(t)}}(\tau \circ s)\gets \beta^{(t)}(\tau)P(s|s_t^\tau ,\pi (s_t^\tau))$  for  $\tau \in F^{(t)},s\in S$  // expanding step   
5:  $(J^{(t + 1)},\beta^{(t + 1)})\gets \mathbf{C}4(J_{F^{(t)}},p_{F^{(t)},\beta^{(t)}})$  and  $F^{(t + 1)}\gets \{\tau |J(\tau)\in J^{(t + 1)}\}$  // compressing step   
6: end for   
7: output  $F^{(H)}$  and  $\beta^{(H)}$
Theorem 3. Algorithm 2 outputs  $F^{(H)}$  and  $\beta^{(H)}$  satisfying that  $|F^{(H)}| \leq k + 1$  and  $\sum_{\tau \in F^{(H)}} \beta^{(H)}(\tau) \Phi(\tau) = V^{\pi}$  in time  $\mathcal{O}(k^4 H |\mathcal{S}| + kH |\mathcal{S}|^2)$ .
The proof of Theorem 3 follows immediately from the construction of the algorithm. According to Eq (5), we have  $V^{\pi} = \sum_{\tau \in F^{(1)}} \beta^{(1)}(\tau) J(\tau)$ . Then we can show that the output of Algorithm 2 is a valid weighted trajectory set by induction on the length of representative trajectories. C4 guarantees that  $|F^{(t)}| \leq k + 1$  for all  $t = 1, \ldots, H$ , and thus, we only keep track of at most  $k + 1$  trajectories at each step and achieve the computation guarantee in the theorem. Combined with Theorem 1, we derive the following Corollary.
Corollary 1. Running the algorithm in Theorem 1 with weighted trajectory set representation returned by Algorithm 2 gives us the same guarantee as that of Theorem 1 in time  $\mathcal{O}(k^2 H|\mathcal{S}|^2 |\mathcal{A}| + (k^5 H|\mathcal{S}| + k^2 H|\mathcal{S}|^2)\log (\frac{k}{\epsilon}))$
In this paper, we designed efficient algorithms for learning users' preferences over multiple objectives from comparative feedback. The efficiency is expressed in both the running time and number of queries (both polynomial in  $H$ ,  $|S|$ ,  $|\mathcal{A}|$ ,  $k$  and logarithmic in  $1 / \epsilon$ ). The learned preferences of a user can then be used to reduce the problem of finding a personalized optimal policy for this user to a (finite horizon) single scalar reward MDP, a problem with a known efficient solution. As we have focused on minimizing the policy comparison queries, our algorithms are based on polynomial time pre-processing calculations that save valuable comparison time for users.
The results in Section 3 are of independent interest and can be applied to a more general learning setting, where for some unknown linear parameter  $w^{*}$ , given a set of points  $X$  and access to comparison queries of any two points, the goal is to learn  $\arg \max_{x\in X}\langle w^{*},x\rangle$ . E.g., in personalized recommendations for coffee beans in terms of the coffee profile described by the coffee suppliers (body, aroma, crema, roast level,...), while users could fail to describe their optimal coffee beans profile, adopting the methodology in Section 3 can retrieve the ideal coffee beans for a user using comparisons (where the mixing with "do nothing" is done by diluting the coffee with water and the optimal coffee for a given profile is the one closest to it).
When moving from the explicit representation of policies as mappings from states to actions to a more natural policy representation as a weighted trajectory set, we then obtained the same optimality guarantees in terms of the number of queries. While there could be other forms of policy representations (e.g., a small subset of common states), one advantage of our weighted trajectory set representation is that it captures the essence of the policy multi-objective value in a clear manner via  $\mathcal{O}(k)$  trajectories and weights. The algorithms provided in Section 4 are standalone and could also be of independent interest for explainable RL (Alharin et al., 2020). For example, to exemplify the multi-objective performance of generic robotic vacuum cleaners (this is beneficial if we only have e.g., 3 of them—we can apply the algorithms in Section 4 to generate weighted trajectory set representations and compare them directly without going through the algorithm in Section 3.).
An interesting direction for future work is to relax the assumption that the MDP is known in advance. One direct way is to first learn the model (in model-based RL), then apply our algorithms in the learned MDP. The sub-optimality of the returned policy will then depend on both the estimation error of the model and the error introduced by our algorithms (which depends on the parameters in the learned model).
This work was supported in part by the National Science Foundation under grants CCF-2212968 and ECCS-2216899 and by the Defense Advanced Research Projects Agency under cooperative agreement HR00112020003. The views expressed in this work do not necessarily reflect the position or the policy of the Government and no official endorsement should be inferred. Approved for public release; distribution is unlimited.
This project was supported in part by funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement number 882396), by the Israel Science Foundation (grant number 993/17), Tel Aviv University Center for AI and Data Science (TAD), the Eric and Wendy Schmidt Fund, and the Yandex Initiative for Machine Learning at Tel Aviv University.
We would like to thank all anonymous reviewers, especially Reviewer Gnoo, for their constructive comments.
Ailon, N. (2012). An active learning algorithm for ranking from pairwise preferences with an almost optimal query complexity. Journal of Machine Learning Research, 13(1):137-164.
Ailon, N., Karnin, Z. S., and Joachims, T. (2014). Reducing dueling bandits to cardinal bandits. In ICML, volume 32, pages 856-864.
Akrour, R., Schoenauer, M., and Sebag, M. (2012). APRIL: Active preference learning-based reinforcement learning. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 116-131.
Alharin, A., Doan, T.-N., and Sartipi, M. (2020). Reinforcement learning interpretation methods: A survey. IEEE Access, 8:171058-171077.
Balcan, M., Blum, A., and Vempala, S. S. (2015). Efficient representations for lifelong learning and autoencoding. In Proceedings of the Annual Conference on Learning Theory (COLT), pages 191-210.
Balcan, M.-F., Vitercik, E., and White, C. (2016). Learning combinatorial functions from pairwise comparisons. In Proceedings of the Annual Conference on Learning Theory (COLT).
Barrett, L. and Narayanan, S. (2008). Learning all optimal policies with multiple criteria. In Proceedings of the International Conference on Machine Learning (ICML), pages 41-47.
Bengs, V., Busa-Fekete, R., El Mesaoudi-Paul, A., and Hüllermeier, E. (2021). Preference-based online learning with dueling bandits: A survey. J. Mach. Learn. Res.
Bhatia, K., Pananjady, A., Bartlett, P., Dragan, A., and Wainwright, M. J. (2020). Preference learning along multiple criteria: A game-theoretic perspective. In Advances in Neural Information Processing Systems (NeurIPS), pages 7413-7424.
Biyik, E. and Sadigh, D. (2018). Batch active preference-based learning of reward functions. In Proceedings of the Conference on Robot Learning (CoRL), pages 519-528.
Blum, A. and Shao, H. (2020). Online learning with primary and secondary losses. Advances in Neural Information Processing Systems, 33:20427-20436.
Chatterjee, K. (2007). Markov decision processes with multiple long-run average objectives. In Proceedings of the International Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS), pages 473-484.
Chatterjee, K., Majumdar, R., and Henzinger, T. A. (2006). Markov decision processes with multiple objectives. In Proceedings of the Annual Symposium on Theoretical Aspects of Computer Science (STACS), pages 325-336.
Ren, W., Liu, J., and Shroff, N. B. (2018). PAC ranking from pairwise and listwise queries: Lower bounds and upper bounds. arXiv preprint arXiv:1806.02970.
Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67-113.
Rothkopf, C. A. and Dimitrakakis, C. (2011). Preference elicitation and inverse reinforcement learning. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 34-48.
Sadigh, D., Dragan, A. D., Sastry, S., and Seshia, S. A. (2017). Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS).
Saha, A. and Gopalan, A. (2018). Battle of bandits. In Uncertainty in Artificial Intelligence.
Saha, A. and Gopalan, A. (2019). PAC Battling Bandits in the Plackett-Luce Model. In Algorithmic Learning Theory, pages 700-737.
Saha, A., Koren, T., and Mansour, Y. (2021). Dueling convex optimization. In International Conference on Machine Learning, pages 9245-9254. PMLR.
Sui, Y., Zhuang, V., Burdick, J., and Yue, Y. (2017). Multi-dueling bandits with dependent arms. In Conference on Uncertainty in Artificial Intelligence, UAI'17.
Sui, Y., Zoghi, M., Hofmann, K., and Yue, Y. (2018). Advancements in dueling bandits. In *IJCAI*, pages 5502-5510.
Wang, N., Wang, H., Karimzadehgan, M., Kveton, B., and Boutilier, C. (2022). Imo^3: Interactive multi-objective off-policy optimization. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22.
Wilson, A., Fern, A., and Tadepalli, P. (2012). A Bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems (NeurIPS).
Wirth, C., Akrour, R., Neumann, G., and Furnkranz, J. (2017a). A survey of preference-based reinforcement learning methods. J. Mach. Learn. Res.
Wirth, C., Akrour, R., Neumann, G., and Furnkranz, J. (2017b). A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46.
Wirth, C., Furnkranz, J., and Neumann, G. (2016). Model-free preference-based reinforcement learning. In Proceedings of the National Conference on Artificial Intelligence (AAAI).
Yona, G., Moran, S., Elidan, G., and Globerson, A. (2022). Active learning with label comparisons. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI).
Zoghi, M., Whiteson, S., Munos, R., Rijke, M. d., et al. (2014). Relative upper confidence bound for the  $k$ -armed dueling bandit problem. In JMLR Workshop and Conference Proceedings, number 32, pages 10-18. JMLR.
Multi-objective sequential decision making There is a long history of work on multi-objective sequential decision making (Rojiers et al., 2013), with one key focus being the realization of efficient algorithms for approximating the Pareto front (Chatterjee et al., 2006; Chatterjee, 2007; Marinescu et al., 2017). Instead of finding a possibly optimal policy, we concentrate on specific user preferences and find a policy that is optimal for that specific user. Just like the ideal car for one person could be a Chevrolet Spark (small) and for another, it is a Ford Ranger (a truck).
In the context of multi-objective RL (Hayes et al., 2022), the goal can be formulated as one of learning a policy for which the average return vector belongs to a target set (hence the term "multi-criteria" RL), which existing work has treated as a stochastic game (Mannor and Shimkin, 2001, 2004). Other works seek to maximize (in expectation) a scalar version of the reward that may correspond to a weighted sum of the multiple objectives (Barrett and Narayanan, 2008; Chen et al., 2019) as we consider here, or a nonlinear function of the objectives (Cheung, 2019). Multi-objective online learning has also been studied, see (Mannor et al., 2014; Blum and Shao, 2020) for example.
The parameters that define this scalarization function (e.g., the relative objective weights) are often unknown and vary with the task setting or user. In this case, preference learning (Wirth et al., 2017a) is commonly used to elicit the value of these parameters. Doumpos and Zopounidis (2007) describe an approach to eliciting a user's relative weighting in the context of multi-objective decision-making. Bhatia et al. (2020) learn preferences over multiple objectives from pairwise queries using a game-theoretic approach to identify optimal randomized policies. In the context of RL involving both scalar and vector-valued objectives, user preference queries provide an alternative to learning from demonstrations, which may be difficult for people to provide (e.g., in the case of robots with high degrees-of-freedom), or explicit reward specifications (Cheng et al., 2011; Rothkopf and Dimitrakakis, 2011; Akfour et al., 2012; Wilson et al., 2012; Furnkranz et al., 2012; Jain et al., 2015; Wirth et al., 2016; Christiano et al., 2017; Wirth et al., 2017b; Ibarz et al., 2018; Lee et al., 2021). These works typically assume that noisy human preferences over a pair of trajectories are correlated with the difference in their utilities (i.e., the reward acts as a latent term predictive of preference). Many contemporary methods estimate the latent reward by minimizing the cross-entropy loss between the reward-based predictions and the human-provided preferences (i.e., finding the reward that maximizes the likelihood of the observed preferences) (Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021; Knox et al., 2022; Pacchiano et al., 2022).
Wilson et al. (2012) describe a Bayesian approach to policy learning whereby they query a user for their preference between a pair of trajectories and use these preferences to maintain a posterior distribution over the latent policy parameters. The task of choosing the most informative queries is challenging due to the continuous space of trajectories, and is generally NP-hard (Ailon, 2012). Instead, they assume access to a distribution over trajectories that accounts for their feasibility and relevance to the target policy, and they describe two heuristic approaches to selecting trajectory queries based on this distribution. Finally, Sadigh et al. (2017) describe an approach to active preference-based learning in continuous state and action spaces. Integral to their work is the ability to synthesize dynamically feasible trajectory queries. Biyik and Sadigh (2018) extend this approach to the batch query setting.
Comparative feedback in other problems Comparative feedback has been studied in other problems in learning theory, e.g., combinatorial functions (Balcan et al., 2016). One closely related problem is active ranking/learning using pairwise comparisons (Jamieson and Nowak, 2011; Kane et al., 2017; Saha et al., 2021; Yona et al., 2022). These works usually consider a given finite sample of points. Kane et al. (2017) implies a lower bound of the number of comparisons is linear in the cardinality of the set even if the points satisfy the linear structural constraint as we assume in this work. In our work, the points are value vectors generated by running different policies under the same MDP and thus have a specific structure. Besides, we allow comparison of policies not in the policy set. Thus, we are able to obtain the query complexity sublinear in the number of policies. Another related problem using comparative/preference feedback isueling bandits that aim to learn through pairwise feedback (Ailon et al., 2014; Zoghi et al., 2014) (see also Bengs et al. (2021); Sui et al. (2018) for surveys), or more generally any subsetwise feedback (Sui et al., 2017; Saha and Gopalan, 2018, 2019; Ren et al., 2018). However, unlikeueling bandits, we consider noiseless comparative feedback.
B Proof of Lemma 1
Proof. We first show that, according to our algorithm (lines 1-6 of Algorithm 1), the returned  $\pi_1$  satisfies that
$$
\left\langle w ^ {*}, V ^ {\pi_ {1}} \right\rangle \geq \max _ {i \in [ k ]} \left\langle w ^ {*}, V ^ {\pi^ {\mathbf {e} _ {i}}} \right\rangle - \epsilon .
$$
This can be proved by induction over  $k$ . In the base case of  $k = 2$ , it's easy to see that the returned  $\pi_1$  satisfies the above inequality. Suppose the above inequality holds for any  $k \leq n - 1$  and we prove that it will also hold for  $k = n$ . After running the algorithm over  $j = 2, \dots, k - 1$  (line 2), the returned policy  $\pi_{e^*}$  satisfies that
$$
\left\langle w ^ {*}, V ^ {\pi^ {e ^ {*}}} \right\rangle \geq \max  _ {i \in [ k - 1 ]} \left\langle w ^ {*}, V ^ {\pi^ {\mathbf {e} _ {i}}} \right\rangle - \epsilon .
$$
Then there are two cases,
- If  $\left\langle w^{*}, V^{\pi^{\epsilon^{*}}} \right\rangle < \left\langle w^{*}, V^{\pi^{\mathbf{e}_{k}}} \right\rangle - \epsilon$ , we will return  $\pi_{1} = \pi^{\mathbf{e}_{k}}$  and also,  $\left\langle w^{*}, V^{\pi^{\mathbf{e}_{k}}} \right\rangle \geq \left\langle w^{*}, V^{\pi^{\mathbf{e}_{i}}} \right\rangle - \epsilon$  for all  $i \in [k]$ .
- If  $\left\langle w^{*}, V^{\pi^{e^{*}}} \right\rangle \geq \left\langle w^{*}, V^{\pi^{\mathbf{e}_{k}}} \right\rangle - \epsilon$ , then we will return  $\pi^{e^{*}}$  and it satisfies  $\left\langle w^{*}, V^{\pi^{\mathbf{e}^{*}}} \right\rangle \geq \max_{i \in [k]} \left\langle w^{*}, V^{\pi^{\mathbf{e}_{i}}} \right\rangle - \epsilon$ .
As  $\pi^{\mathbf{e}_i}$  is the optimal personalized policy when the user's preference vector is  $\mathbf{e}_i$ , we have that
$$
v ^ {*} = \left\langle w ^ {*}, V ^ {\pi^ {*}} \right\rangle = \sum_ {i = 1} ^ {k} w _ {i} ^ {*} \left\langle V ^ {\pi^ {*}}, \mathbf {e} _ {i} \right\rangle \leq \sum_ {i = 1} ^ {k} w _ {i} ^ {*} \left\langle V ^ {\pi^ {\mathbf {e} _ {i}}}, \mathbf {e} _ {i} \right\rangle \leq \left\langle w ^ {*}, \sum_ {i = 1} ^ {k} V ^ {\pi^ {\mathbf {e} _ {i}}} \right\rangle ,
$$
where the last inequality holds because the entries of  $V^{\pi^{\mathbf{e}_i}}$  and  $w^*$  are non-negative. Therefore, there exists  $i \in [k]$  such that  $\langle w^*, V^{\pi^{\mathbf{e}_i}} \rangle \geq \frac{1}{k} \langle w^*, V^{\pi^*} \rangle = \frac{1}{k} v^*$ .
$$
\left\langle w ^ {*}, V ^ {\pi_ {1}} \right\rangle \geq \max  _ {i \in [ k ]} \left\langle w ^ {*}, V ^ {\pi^ {\mathbf {e} _ {i}}} \right\rangle - \epsilon \geq \frac {1}{k} v ^ {*} - \epsilon \geq \frac {1}{2 k} v ^ {*},
$$
when  $\epsilon \leq \frac{v^*}{2k}$ . By rearranging terms, we have  $\frac{v^*}{\langle w^*, V^{\pi_1} \rangle} \leq 2k$ .
By setting  $C_{\alpha} = 2k$ , we have  $\left|\widehat{\alpha}_i - \alpha_i\right|\langle w^*,V^{\pi_1}\rangle \leq C_\alpha \epsilon = 2k\epsilon$  and thus,  $\left|\widehat{\alpha}_i - \alpha_i\right| \leq \frac{4k^2\epsilon}{v^*}$ .
C Pseudo Code of Computation of the Basis Ratios
The pseudo code of searching  $\widehat{\alpha}_i$ 's is described in Algorithm 3.
Algorithm 3 Computation of Basis Ratios
1: input:  $(V^{\pi_1},\ldots ,V^{\pi_d})$  and  $C_{\alpha}$   
2: for  $i = 1,\dots ,d - 1$  do  
3: let  $l = 0,h = 2C_{\alpha}$  and  $\widehat{\alpha}_i = C_\alpha$   
4: while True do  
5: if  $\widehat{\alpha}_i > 1$  then  
6: compare  $\pi_1$  and  $\frac{1}{\widehat{\alpha}_i}\pi_{i + 1} + (1 - \frac{1}{\widehat{\alpha}_i})\pi_0$ ; if  $\pi_1\succ \frac{1}{\widehat{\alpha}_i}\pi_{i + 1} + (1 - \frac{1}{\widehat{\alpha}_i})\pi_0$  then  $h\gets \widehat{\alpha}_i$ $\widehat{\alpha}_i\gets \frac{l + h}{2}$ ; if  $\pi_1\succ \frac{1}{\widehat{\alpha}_i}\pi_{i + 1} + (1 - \frac{1}{\widehat{\alpha}_i})\pi_0$  then  $l\gets \widehat{\alpha}_i,\widehat{\alpha}_i\gets \frac{l + h}{2}$   
7: else  
8: compare  $\pi_{i + 1}$  and  $\widehat{\alpha}_i\pi_1 + (1 - \widehat{\alpha}_i)\pi_0$ ; if  $\widehat{\alpha}_i\pi_1 + (1 - \widehat{\alpha}_i)\pi_0\succ \pi_{i + 1}$  then  $h\gets \widehat{\alpha}_i$ $\widehat{\alpha}_i\gets \frac{l + h}{2}$ ; if  $\widehat{\alpha}_i\pi_1 + (1 - \widehat{\alpha}_i)\pi_0\succ \pi_{i + 1}$  then  $l\gets \widehat{\alpha}_i,\widehat{\alpha}_i\gets \frac{l + h}{2}$   
9: end if  
10: if "indistinguishable" is returned then  
11: break  
12: end if  
13: end while  
14: end for  
15: output:  $(\widehat{\alpha}_1,\dots ,\widehat{\alpha}_{d - 1})$
Proof. Theorem 1 follows by setting  $\epsilon_{\alpha} = \frac{4k^2\epsilon}{v^*}$  and  $C_{\alpha} = 2k$  as shown in Lemma 1 and combining the results of Lemma 2 and 3 with the triangle inequality.
Specifically, for any policy  $\pi$ , we have
$$
\begin{array}{l} \left| \langle \widehat {w}, V ^ {\pi} \rangle - \langle w ^ {\prime}, V ^ {\pi} \rangle \right| \leq \left| \langle \widehat {w}, V ^ {\pi} \rangle - \left\langle \widehat {w} ^ {(\delta)}, V ^ {\pi} \right\rangle \right| + \left| \left\langle \widehat {w} ^ {(\delta)}, V ^ {\pi} \right\rangle - \langle w ^ {\prime}, V ^ {\pi} \rangle \right| \\ \leq \mathcal {O} \left(\left(\sqrt {k} + 1\right) ^ {d - d _ {\delta}} C _ {\alpha} \left(\frac {C _ {\alpha} C _ {V} ^ {4} d _ {\delta} ^ {\frac {3}{2}} \| w ^ {\prime} \| _ {2} ^ {2} \epsilon_ {\alpha}}{\delta^ {2}} + \sqrt {k} \delta \| w ^ {\prime} \| _ {2}\right)\right) \tag {7} \\ \leq \mathcal {O} ((\sqrt {k} + 1) ^ {d - d _ {\delta} + 3} \| w ^ {\prime} \| _ {2} \left(\frac {C _ {V} ^ {4} k ^ {4} \| w ^ {\prime} \| _ {2} \epsilon}{v ^ {*} \delta^ {2}} + \delta)\right). \\ \end{array}
$$
Since  $\| w^{\prime}\| = \frac{\|w^{*}\|}{\langle w^{*},V^{\pi_{1}}\rangle}$  and  $\langle w^{*},V^{\pi_{1}}\rangle \geq \frac{v^{*}}{2k}$  from Lemma 1, we derive
$$
\begin{array}{l} v ^ {*} - \left\langle w ^ {*}, V ^ {\pi^ {\hat {w}}} \right\rangle = \left\langle w ^ {*}, V ^ {\pi_ {1}} \right\rangle \left(\left\langle w ^ {\prime}, V ^ {\pi^ {*}} \right\rangle - \left\langle w ^ {\prime}, V ^ {\pi^ {\hat {w}}} \right\rangle\right) \\ \leq \langle w ^ {*}, V ^ {\pi_ {1}} \rangle \left(\left\langle \widehat {w}, V ^ {\pi^ {*}} \right\rangle - \left\langle \widehat {w}, V ^ {\pi^ {\widehat {w}}} \right\rangle + \mathcal {O} \left((\sqrt {k} + 1) ^ {d - d _ {\delta} + 3} \| w ^ {\prime} \| _ {2} \left(\frac {C _ {V} ^ {4} k ^ {4} \| w ^ {\prime} \| _ {2} \epsilon}{v ^ {*} \delta^ {2}} + \delta\right)\right)\right) \\ \leq \mathcal {O} \left(\langle w ^ {*}, V ^ {\pi_ {1}} \rangle (\sqrt {k} + 1) ^ {d - d _ {\delta} + 3} \| w ^ {\prime} \| _ {2} \left(\frac {C _ {V} ^ {4} k ^ {4} \| w ^ {\prime} \| _ {2} \epsilon}{v ^ {*} \delta^ {2}} + \delta\right)\right) \\ = \mathcal {O} \left(\left(\sqrt {k} + 1\right) ^ {d - d _ {\delta} + 3} \| w ^ {*} \| _ {2} \left(\frac {C _ {V} ^ {4} k ^ {5} \| w ^ {*} \| _ {2} \epsilon}{v ^ {* 2} \delta^ {2}} + \delta\right)\right) \\ = \mathcal {O} \left(\left(\frac {C _ {V} ^ {2} \| w ^ {*} \| _ {2} ^ {2}}{v ^ {*}}\right) ^ {\frac {2}{3}} (\sqrt {k} + 1) ^ {d + \frac {1 6}{3}} \epsilon^ {\frac {1}{3}}\right). \\ \end{array}
$$
The first inequality follows from
$$
\left\langle w ^ {\prime}, V ^ {\pi^ {*}} \right\rangle - \left\langle w ^ {\prime}, V ^ {\pi^ {\hat {w}}} \right\rangle = \left\langle w ^ {\prime}, V ^ {\pi^ {*}} \right\rangle - \left\langle w ^ {\prime}, V ^ {\pi^ {\hat {w}}} \right\rangle
$$
$$
\left. \right. + \left(\left\langle \widehat {w}, V ^ {\pi^ {*}} \right\rangle - \left\langle \widehat {w}, V ^ {\pi^ {*}} \right\rangle\right) + \left(\left\langle \widehat {w}, V ^ {\pi^ {\widehat {w}}} \right\rangle - \left\langle \widehat {w}, V ^ {\pi^ {\widehat {w}}} \right\rangle\right),
$$
and applying (7) twice- once for  $\pi^{*}$  and once for  $\pi^{\widehat{w}}$ . The last inequality follows by setting  $\delta = \left(\frac{C_V^4k^5\|w^*\|_2\epsilon}{v^{*2}}\right)^{\frac{1}{3}}$ .
Lemma 2. If  $|\widehat{\alpha}_i - \alpha_i| \leq \epsilon_\alpha$  and  $\alpha_i \leq C_\alpha$  for all  $i \in [d-1]$ , for every  $\delta \geq 4C_\alpha^{\frac{2}{3}}C_V d^{\frac{1}{3}}\epsilon_\alpha^{\frac{1}{3}}$ , we have  $\left|\langle \widehat{w}^{(\delta)}, V^\pi \rangle - \langle w', V^\pi \rangle\right| \leq O\left(\frac{C_\alpha C_V^4 d_\delta^{\frac{3}{2}} \|w' \|_2^2 \epsilon_\alpha}{\delta^2} + \sqrt{k}\delta \|w' \|_2\right)$  for all  $\pi$ , where  $w' = \frac{w^*}{\langle w^*, V^{\pi_1} \rangle}$ .
To prove Lemma 2, we will first define a matrix,  $A^{(\mathrm{full})}$
Given the output  $(V^{\pi_1},\dots ,V^{\pi_d})$  of Algorithm 1, we have rank  $(\mathrm{span}(\{V^{\pi_1},\ldots ,V^{\pi_{d_\delta}}\})) = d_\delta$