SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Choice of  $\pi_2,\dots ,\pi_d$  . After finding  $\pi_{1}$  we next search the remaining  $d - 1$  polices  $\pi_2,\ldots ,\pi_d$  sequentially (lines 8-13 of Algorithm 1). For  $i = 2,\dots ,d$  we find a direction  $u_{i}$  such that (i) the vector  $u_{i}$  is orthogonal to the space of current value vectors  $\mathrm{span}(V^{\pi_1},\dots ,V^{\pi_{i - 1}})$  and (ii) there exists a policy  $\pi_{i}$  such that  $V^{\pi_i}$  has a significant component in the direction of  $u_{i}$  Condition (i) is used to guarantee that the policy  $\pi_{i}$  has a value vector linearly independent of  $\mathrm{span}(V^{\pi_1},\dots ,V^{\pi_{i - 1}})$  .Condition (ii) is used to cope with the error caused by inaccurate approximation of the ratios  $\widehat{\alpha}_i$  Intuitively, when  $\| \alpha_iV^{\pi_1} - V^{\pi_{i + 1}}\| _2\ll \epsilon$  the angle between  $\widehat{\alpha}_iV^{\pi_1} - V^{\pi_{i + 1}}$  and  $\alpha_{i}V^{\pi_{1}} - V^{\pi_{i + 1}}$  could be very large, which results in an inaccurate estimate of  $w^{*}$  in the direction of  $\alpha_{i}V^{\pi_{1}} - V^{\pi_{i + 1}}$  For example, if  $V^{\pi_1} = \mathbf{e}_1$  and  $V^{\pi_i} = \mathbf{e}_1 + \frac{1}{w_i^*}\epsilon \mathbf{e}_i$  for  $i = 2,\dots ,k$  then  $\pi_1,\pi_i$  are "indistinguishable" and the estimate ratio  $\widehat{\alpha}_{i - 1}$  can be 1. Then the estimate of  $w^{*}$  by solving linear equations in Eq (1) is  $(1,0,\dots ,0)$  ,which could be far from the true  $w^{*}$  .Finding  $u_{i}$  's in which
policy values have a large component can help with this problem.
Algorithm 1 provides a more detailed description of this procedure. Note that if there are  $n$  different users, we will run Algorithm 1 at most  $k$  times instead of  $n$  times. The reason is that Algorithm
1 only utilizes preference comparisons while searching for  $\pi_1$  (lines 1-6), and not for  $\pi_2, \ldots, \pi_k$  (which contributes to the  $k^2$  factor in computational complexity). As there are at most  $k$  candidates for  $\pi_1$ , namely  $\pi^{e_1}, \ldots, \pi^{e_k}$ , we execute lines 1-6 of Algorithm 1 for  $n$  rounds and lines 8-13 for only  $k$  rounds.
3.2 Computation of Basis Ratios
As we have mentioned before, comparing basis policies alone does not allow for the exact computation of the  $\alpha_{i}$  ratios as comparing  $\pi_1,\pi_i$  can only reveal which is better but not how much. To this end, we will use the "do nothing" policy to approximate every ratio  $\alpha_{i}$  up to some additive error  $|\widehat{\alpha}_i - \alpha_i|$  using binary search over the parameter  $\widehat{\alpha}_i\in [0,C_\alpha ]$  for some  $C_\alpha \geq 1$  (to be determined later) and comparison queries of policy  $\pi_{i + 1}$  with policy  $\widehat{\alpha}_i\pi_1 + (1 - \widehat{\alpha}_i)\pi_0$  if  $\widehat{\alpha}_i\leq 1$  (or comparing  $\pi_{1}$  and  $\frac{1}{\widehat{\alpha}_i}\pi_{i + 1} + (1 - \frac{1}{\widehat{\alpha}_i})\pi_0$  instead if  $\widehat{\alpha}_i > 1)$ . Notice that the personalized value of  $\widehat{\alpha}_i\pi_1 + (1 - \widehat{\alpha}_i)\pi_0$  is identical to the personalized value of  $\pi_{1}$  multiplied by  $\widehat{\alpha}_i$ . We stop once  $\widehat{\alpha}_i$  is such that the user returns "indistinguishable". Once we stop, the two policies have roughly the same personalized value,
$$
\left. \text {i f} \widehat {\alpha} _ {i} \leq 1, \left| \widehat {\alpha} _ {i} \langle w ^ {*}, V ^ {\pi_ {1}} \rangle - \langle w ^ {*}, V ^ {\pi_ {i + 1}} \rangle \right| \leq \epsilon ; \text {i f} \widehat {\alpha} _ {i} > 1, \left| \langle w ^ {*}, V ^ {\pi_ {1}} \rangle - \frac {1}{\widehat {\alpha} _ {i}} \langle w ^ {*}, V ^ {\pi_ {i + 1}} \rangle \right| \leq \epsilon . \right. \tag {3}
$$
Eq (1) combined with the above inequality implies that  $|\widehat{\alpha}_i - \alpha_i| \langle w^*, V^{\pi_1} \rangle \leq C_\alpha \epsilon$ . Thus, the approximation error of each ratio is bounded by  $|\widehat{\alpha}_i - \alpha_i| \leq \frac{C_\alpha \epsilon}{\langle w^*, V^{\pi_1} \rangle}$ . To make sure the procedure will terminate, we need to set  $C_\alpha \geq \frac{v^*}{\langle w^*, V^{\pi_1} \rangle}$  since  $\alpha_i$ 's must lie in the interval  $[0, \frac{v^*}{\langle w^*, V^{\pi_1} \rangle}]$ . Upon stopping binary search once Eq (3) holds, it takes at most  $\mathcal{O}(d \log(C_\alpha \langle w^*, V^{\pi_1} \rangle / \epsilon))$  comparison queries to estimate all the  $\alpha_i$ 's.
Due to the carefully picked  $\pi_1$  in Algorithm 1, we can upper bound  $\frac{v^*}{\langle w^*, V^{\pi_1} \rangle}$  by  $2k$  and derive an upper bound for  $|\widehat{\alpha}_i - \alpha_i|$  by selecting  $C_\alpha = 2k$ .
Lemma 1. When  $\epsilon \leq \frac{v^*}{2k}$ , we have  $\frac{v^*}{\langle w^*, V^{\pi_1} \rangle} \leq 2k$ , and  $|\widehat{\alpha}_i - \alpha_i| \leq \frac{4k^2 \epsilon}{v^*}$  for every  $i \in [d]$ .
In what follows we set  $C_{\alpha} = 2k$ . The pseudo code of the above process of estimating  $\alpha_{i}$ 's is deferred to Algorithm 3 in Appendix C.
3.3 Preference Approximation and Personalized Policy
We move on to present an algorithm that estimates  $w^{*}$  and calculates a nearly optimal personalized policy. Given the  $\pi_{i}$ 's returned by Algorithm 1 and the  $\widehat{\alpha}_{i}$ 's returned by Algorithm 3, consider a matrix  $\widehat{A} \in \mathbb{R}^{d \times k}$  with 1st row  $V^{\pi_1\top}$  and the  $i$ th row  $(\widehat{\alpha}_{i-1}V^{\pi_1} - V^{\pi_i})^\top$  for every  $i = 2,\dots,d$ . Let  $\widehat{w}$  be a solution to  $\widehat{A}x = \mathbf{e}_1$ . We will show that  $\widehat{w}$  is a good estimate of  $w' := \frac{w^{*}}{\langle w^{*},V^{\pi_1}\rangle}$  and that  $\pi^{\widehat{w}}$  is a nearly optimal personalized policy. In particular, when  $\epsilon$  is small, we have  $|\langle \widehat{w},V^{\pi}\rangle -\langle w',V^{\pi}\rangle| = O(\epsilon^{\frac{1}{3}})$  for every policy  $\pi$ . Putting this together, we derive the following theorem.
Theorem 1. Consider the algorithm of computing  $\widehat{A}$  and any solution  $\widehat{w}$  to  $\widehat{A}x = \mathbf{e}_1$  and outputting the policy  $\pi^{\widehat{w}} = \arg \max_{\pi \in \Pi} \langle \widehat{w}, V^{\pi} \rangle$ , which is the optimal personalized policy for preference vector  $\widehat{w}$ . Then the output policy  $\pi^{\widehat{w}}$  satisfying that  $v^{*} - \left\langle w^{*}, V^{\pi^{\widehat{w}}} \right\rangle \leq \mathcal{O}\left(\left(\sqrt{k} + 1\right)^{d + \frac{14}{3}} \epsilon^{\frac{1}{3}}\right)$  using  $\mathcal{O}(k \log (k / \epsilon))$  comparison queries.
Computation Complexity We remark that Algorithm 1 solves Eq (2) for the optimal policy in scalar reward MDP at most  $\mathcal{O}(k^2)$  times. Using, e.g., Finite Horizon Value iteration to solve for the optimal policy takes  $\mathcal{O}(H|\mathcal{S}|^2|\mathcal{A}|)$  steps. However, while the time complexity it takes to return the optimal policy for a single user is  $\mathcal{O}(k^2H|\mathcal{S}|^2|\mathcal{A}| + k\log(\frac{k}{\epsilon}))$ , considering  $n$  different users rather than one results in overall time complexity of  $\mathcal{O}((k^3 + n)H|\mathcal{S}|^2|\mathcal{A}| + nk\log(\frac{k}{\epsilon}))$ .
Proof Technique. The standard technique typically starts by deriving an upper bound on  $\| \widehat{w} - w^{*} \|_{2}$  and then uses this bound to upper bound  $\sup_{\pi} |\langle \widehat{w}, V^{\pi} \rangle - \langle w^{*}, V^{\pi} \rangle|$  as  $C_V \| \widehat{w} - w^{*} \|_{2}$ . However,
<sup>7</sup>We write  $\widehat{\alpha}_i\pi_1 + (1 - \widehat{\alpha}_i)\pi_0$  to indicate that  $\pi_1$  is used with probability  $\widehat{\alpha}_i$ , and that  $\pi_0$  is used with probability  $1 - \widehat{\alpha}_i$ .
this method fails to achieve a non-vacuous bound in case there are two basis policies that are close to each other. For instance, consider the returned basis policy values:  $V^{\pi_1} = (1,0,0)$ ,  $V^{\pi_2} = (1,1,0)$ , and  $V^{\pi_3} = (1,0,\eta)$  for some  $\eta > 0$ . When  $\eta$  is extremely small, the estimate  $\widehat{w}$  becomes highly inaccurate in the direction of  $(0,0,1)$ , leading to a large  $\| \widehat{w} - w^* \|_2$ . Even in such cases, we can still obtain a non-vacuous guarantee since the selection of  $V^{\pi_3}$  (line 11 of Algorithm 1) implies that no policy in  $\Pi$  exhibits a larger component in the direction of  $(0,0,1)$  than  $\pi_3$ .
Proof Sketch. The analysis of Theorem 1 has two parts. First, as mentioned in Sec 3.1, when  $\| \alpha_{i}V^{\pi_{1}} - V^{\pi_{i + 1}}\|_{2}\ll \epsilon$ , the error of  $\widehat{\alpha}_{i + 1}$  can lead to inaccurate estimate of  $w^{*}$  in direction  $\alpha_{i}V^{\pi_{1}} - V^{\pi_{i + 1}}$ . Thus, we consider another estimate of  $w^{*}$  based only on some  $\pi_{i + 1}$ 's with a relatively large  $\| \alpha_{i}V^{\pi_{1}} - V^{\pi_{i + 1}}\|_{2}$ . In particular, for any  $\delta >0$ , let  $d_{\delta}\coloneqq \min_{i\geq 2:\max (|v^{u_i}|,|v^{-u_i}|)\leq \delta}i - 1$ . That is to say, for  $i = 2,\ldots ,d_{\delta}$ , the policy  $\pi_{i}$  satisfies  $\langle u_i,V^{\pi_i}\rangle >\delta$  and for any policy  $\pi$ , we have  $\langle u_{d_{\delta} + 1},V^{\pi}\rangle \leq \delta$ . Then, for any policy  $\pi$  and any unit vector  $\xi \in \operatorname {span}(V^{\pi_1},\dots ,V^{\pi_{d_\delta}})^\perp$ , we have  $\langle \xi ,V^{\pi}\rangle \leq \sqrt{k}\delta$ . This is because at round  $d_{\delta} + 1$ , we pick an orthonormal basis  $\rho_{1},\ldots ,\rho_{k - d_{\delta}}$  of  $\operatorname {span}(V^{\pi_1},\ldots ,V^{\pi_{d_\delta}})^\perp$  (line 8 in Algorithm 1) and pick  $u_{d_{\delta} + 1}$  to be the one in which there exists a policy with the largest component as described in line 9. Hence,  $|\langle \rho_j,V^{\pi}\rangle |\leq \delta$  for all  $j\in [k - d_{\delta}]$ . Then, we have  $\langle \xi ,V^{\pi}\rangle = \sum_{j = 1}^{k - d_{\delta}}\langle \xi ,\rho_j\rangle \langle \rho_j,V^{\pi}\rangle \leq \sqrt{k}\delta$  by Cauchy-Schwarz inequality. Let  $\widehat{A}^{(\delta)}\in \mathbb{R}^{d_{\delta}\times k}$  be the sub-matrix comprised of the first  $d_{\delta}$  rows of  $\widehat{A}$ . Then we consider an alternative estimate  $\widehat{w}^{(\delta)} = \arg \min_{x:\widehat{A}^{(\delta)}x = e_1}\| x\| _2$ , the minimum norm solution of  $x$  to  $\widehat{A}^{(\delta)}x = e_1$ . We upper bound  $\sup_{\pi}\left|\left\langle \widehat{w}^{(\delta)},V^{\pi}\right\rangle -\left\langle w^{\prime},V^{\pi}\right\rangle \right|$  in Lemma 2 and  $\sup_{\pi}\left|\left\langle \widehat{w},V^{\pi}\right\rangle -\left\langle \widehat{w}^{(\delta)},V^{\pi}\right\rangle \right|$  in Lemma 3. Then we are done with the proof of Theorem 1.
Lemma 2. If  $|\widehat{\alpha}_i - \alpha_i| \leq \epsilon_\alpha$  and  $\alpha_i \leq C_\alpha$  for all  $i \in [d-1]$ , for every  $\delta \geq 4C_\alpha^{\frac{2}{3}}C_V d^{\frac{1}{3}}\epsilon_\alpha^{\frac{1}{3}}$ , we have  $\left|\langle \widehat{w}^{(\delta)}, V^{\pi} \rangle - \langle w', V^{\pi} \rangle\right| \leq \mathcal{O}\left(\frac{C_\alpha C_V^4 d_{\delta}^{\frac{3}{2}} \|w' \|_2^2 \epsilon_\alpha}{\delta^2} + \sqrt{k} \delta \|w' \|_2\right)$  for all  $\pi$ , where  $w' = \frac{w^*}{\langle w^*, V^{\pi_1} \rangle}$ .
Since we only remove the rows in  $\widehat{A}$  corresponding to  $u_{i}$ 's in the subspace where no policy's value has a large component,  $\widehat{w}$  and  $\widehat{w}^{(\delta)}$  are close in terms of  $\sup_{\pi} \left| \langle \widehat{w}, V^{\pi} \rangle - \langle \widehat{w}^{(\delta)}, V^{\pi} \rangle \right|$ .
Lemma 3. If  $|\widehat{\alpha}_i - \alpha_i| \leq \epsilon_\alpha$  and  $\alpha_i \leq C_\alpha$  for all  $i \in [d-1]$ , for every policy  $\pi$  and every  $\delta \geq 4C_\alpha^{\frac{2}{3}}C_V d^{\frac{1}{3}}\epsilon_\alpha^{\frac{1}{3}}$ , we have  $\left|\widehat{w} \cdot V^\pi - \widehat{w}^{(\delta)} \cdot V^\pi\right| \leq \mathcal{O}((\sqrt{k} + 1)^{d - d_\delta} C_\alpha \epsilon^{(\delta)})$ , where  $\epsilon^{(\delta)} = \frac{C_\alpha C_V^4 d_{\delta}^{\frac{3}{2}} \| w' \|_2^2 \epsilon_\alpha}{\delta^2} + \sqrt{k} \delta \| w' \|_2$  is the upper bound in Lemma 2.
Note that the result in Theorem 1 has a factor of  $k^{\frac{d}{2}}$ , which is exponential in  $d$ . Usually, we consider the case where  $k = \mathcal{O}(1)$  is small and thus  $k^{d} = \mathcal{O}(1)$  is small. We get rid of the exponential dependence on  $d$  by applying  $\widehat{w}^{(\delta)}$  to estimate  $w^{*}$  directly, which requires us to set the value of  $\delta$  beforehand. The following theorem follows directly by assigning the optimal value for  $\delta$  in Lemma 2.
Theorem 2. Consider the algorithm of computing  $\widehat{A}$  and any solution  $\widehat{w}^{(\delta)}$  to  $\widehat{A}^{(\delta)}x = \mathbf{e}_1$  for  $\delta = k^{\frac{5}{3}}\epsilon^{\frac{1}{3}}$  and outputting the policy  $\pi^{\widehat{w}^{(\delta)}} = \arg \max_{\pi \in \Pi}\left\langle \widehat{w}^{(\delta)},V^{\pi}\right\rangle$ . Then the policy  $\pi^{\widehat{w}^{(\delta)}}$  satisfies that  $v^{*} - \left\langle w^{*},V^{\pi^{\widehat{w}^{(\delta)}}}\right\rangle \leq \mathcal{O}\left(k^{\frac{13}{6}}\epsilon^{\frac{1}{3}}\right)$ .
Notice that the algorithm in Theorem 2 needs to set the hyperparameter  $\delta$  beforehand while we don't have to set any hyperparameter in Theorem 1. The improper value of  $\delta$  could degrade the performance of the algorithm. But we can approximately estimate  $\epsilon$  by binary searching  $\eta \in [0,1]$  and comparing  $\pi_1$  against the scaled version of itself  $(1 - \eta)\pi_{1}$  until we find an  $\eta$  such that the user cannot distinguish between  $\pi_1$  and  $(1 - \eta)\pi_{1}$ . Then we can obtain an estimate of  $\epsilon$  and use the estimate to set the hyperparameter.
We remark that though we think of  $k$  as a small number, it is unclear whether the dependency on  $\epsilon$  in Theorems 1 and 2 is optimal. The tight dependency on  $\epsilon$  is left as an open problem. We briefly discuss a potential direction to improve this bound in Appendix I.
4 Learning from Weighted Trajectory Set Representation
In the last section, we represented policies using their explicit form as state-action mappings. However, such a representation could be challenging for users to interpret. For example, how safe is a car
described by a list of  $|S|$  states and actions such as "turning left"? In this section, we design algorithms that return a more interpretable policy representation—a weighted trajectory set.
Recall the definition in Section 2, a weighted trajectory set is a small set of trajectories from the support of the policy and corresponding weights, with the property that the expected return of the trajectories in the set (according to the weights) is exactly the value of the policy (henceforth, the exact value property).<sup>8</sup> As these sets preserve all the information regarding the multi-objective values of policies, they can be used as policy representations in policy comparison queries of Algorithm 3 without compromising on feedback quality. Thus, using these representations obtain the same optimality guarantees regarding the returned policy in Section 3 (but would require extra computation time to calculate the sets).
There are two key observations on which the algorithms in this section are based:
(1) Each policy  $\pi$  induces a distribution over trajectories. Let  $q^{\pi}(\tau)$  denote the probability that a trajectory  $\tau$  is sampled when selecting actions according to  $\pi$ . The expected return of all trajectories under  $q^{\pi}$  is identical to the value of the policy, i.e.,  $V^{\pi} = \sum_{\tau} q^{\pi}(\tau) \Phi(\tau)$ . In particular, the value of a policy is a convex combination of the returns of the trajectories in its support. However, we avoid using this convex combination to represent a policy since the number of trajectories in the support of a policy could be exponential in the number of states and actions.
(2) The existence of a small weighted trajectory set is implied by Carathéodory's theorem. Namely, since the value of a policy is in particular a convex combination of the returns of the trajectories in its support, Carathéodory's theorem implies that there exist  $k + 1$  trajectories in the support of the policy and weights for them such that a convex combination of their returns is the value of the policy. Such a  $(k + 1)$ -sized set will be the output of our algorithms.
We can apply the idea behind Carathéodory's theorem proof to compress trajectories as follows. For any  $(k + 2)$ -sized set of  $k$ -dimensional vectors  $\{\mu_1, \dots, \mu_{k+2}\}$ , for any convex combination of them  $\mu = \sum_{i=1}^{k+2} p_i \mu_i$ , we can always find a  $(k+1)$ -sized subset such that  $\mu$  can be represented as the convex combination of the subset by solving a linear equation. Given an input of a probability distribution  $p$  over a set of  $k$ -dimensional vectors,  $M$ , we pick  $k+2$  vectors from  $M$ , reduce at least one of them through the above procedure. We repeat this step until we are left with at most  $k+1$  vectors. We refer to this algorithm as C4 (Compress Convex Combination using Carathéodory's theorem). The pseudocode is described in Algorithm 4, which is deferred to Appendix J due to space limit. The construction of the algorithm implies the following lemma immediately.
Lemma 4. Given a set of  $k$ -dimensional vectors  $M \subset \mathbb{R}^k$  and a distribution  $p$  over  $M$ ,  $C4(M, p)$  outputs  $M' \subset M$  with  $|M'| \leq k + 1$  and a distribution  $q \in \mathrm{Simplex}^{M'}$  satisfying that  $\mathbb{E}_{\mu \sim q}[\mu] = \mathbb{E}_{\mu \sim p}[\mu]$  in time  $\mathcal{O}(|M|k^3)$ .
So now we know how to compress a set of trajectories to the desired size. The main challenge is how to do it efficiently (in time  $\mathcal{O}(\mathrm{poly}(H|S||\mathcal{A}|)))$ . Namely, since the set of all trajectory returns from the support of the policy could be of size  $\Omega(|S|^H)$ , using it as input to C4 Algorithm is inefficient. Instead, we will only use C4 as a subroutine when the number of trajectories is small.
We propose two efficient approaches for finding weighted trajectory representations. Both approaches take advantage of the property that all trajectories are generated from the same policy on the same MDP. First, we start with a small set of trajectories of length of 1, expand them, compress them, and repeat until we get the set of trajectory lengths of  $H$ . The other is based on the construction of a layer graph where a policy corresponds to a flow in this graph and we show that finding representative trajectories is equivalent to flow decomposition.
In the next subsection, we will describe the expanding and compressing approach and defer the flow decomposition based approach to Appendix K due to space considerations. We remark that the flow decomposition approach has a running time of  $\mathcal{O}(H^2 |\mathcal{S}|^2 +k^3 H|\mathcal{S}|^2)$  (see appendix for
Without asking for the exact value property, one could simply return a sample of  $\mathcal{O}(\log k / (\epsilon^{\prime})^{2})$  trajectories from the policy and uniform weights. With high probability, the expected return of every objective is  $\epsilon^\prime$ -far from its expected value. The problem is that this set does not necessarily capture rare events. For example, if the probability of a crash for any car is between  $(\epsilon^{\prime})^{4}$  and  $(\epsilon^{\prime})^{2}$ , depending on the policy, users that only care about safety (i.e., no crashes) are unlikely to observe any "unsafe" trajectories at all, in which case we would miss valuable feedback.
details), which underperforms the expanding and compressing approach (see Theorem 3) whenever  $|S|H + |S|k^3 = \omega(k^4 + k|S|)$ . For presentation purposes, in the following, we only consider the deterministic policies. Our techniques can be easily extended to random policies.<sup>9</sup>
Expanding and Compressing Approach.
The basic idea is to find  $k + 1$  trajectories of length 1 to represent  $V^{\pi}$  first and then increase the length of the trajectories without increasing the number of trajectories. For policy  $\pi$ , let  $V^{\pi}(s,h) = \mathbb{E}_{S_0 = s}\left[\sum_{t = 0}^{h - 1}R(S_t,\pi (S_t))\right]$  be the value of  $\pi$  with initial state  $S_0 = s$  and time horizon  $h$ . Since we study the representation for a fixed policy  $\pi$  in this section, we slightly abuse the notation and represent a trajectory by  $\tau^{\pi} = (s,s_{1},\ldots ,s_{H})$ . We denote the state of trajectory  $\tau$  at time  $t$  as  $s_t^\tau = s_t$ . For a trajectory prefix  $\tau = (s,s_1,\dots ,s_h)$  of  $\pi$  with initial state  $s$  and  $h\leq H$  subsequent states, the return of  $\tau$  is  $\Phi (\tau) = R(s,\pi (s)) + \sum_{t = 1}^{h - 1}R(s_t,\pi (s_t))$ . Let  $J(\tau)$  be the expected return of trajectories (of length  $H$ ) with the prefix being  $\tau$ , i.e.,
$$
J (\tau) := \Phi (\tau) + V \left(s _ {h} ^ {\tau}, H - h\right).
$$
For any  $s \in S$ , let  $\tau \circ s$  denote the trajectory of appending  $s$  to  $\tau$ . We can solve  $V^{\pi}(s, h)$  for all  $s \in S, h \in [H]$  by dynamic programming in time  $\mathcal{O}(kH|\mathcal{S}|^2)$ . Specifically, according to definition, we have  $V^{\pi}(s, 1) = R(s, \pi(s))$  and
$$
V ^ {\pi} (s, h + 1) = R (s, \pi (s)) + \sum_ {s ^ {\prime} \in \mathcal {S}} P \left(s ^ {\prime} \mid s, \pi (s)\right) V ^ {\pi} \left(s ^ {\prime}, h\right). \tag {4}
$$
Thus, we can represent  $V^{\pi}$  by
$$
V ^ {\pi} = R (s _ {0}, \pi (s _ {0})) + \sum_ {s \in \mathcal {S}} P (s | s _ {0}, \pi (s _ {0})) V ^ {\pi} (s, H - 1) = \sum_ {s \in \mathcal {S}} P (s | s _ {0}, \pi (s _ {0})) J (s _ {0}, s).
$$
By applying C4, we can find a set of representative trajectories of length 1,  $F^{(1)} \subset \{(s_0,s)|s \in S\}$  with  $|F^{(1)}| \leq k + 1$  and weights  $\beta^{(1)} \in \mathrm{Simplex}^{F^{(1)}}$  such that
$$
V ^ {\pi} = \sum_ {\tau \in F ^ {(1)}} \beta^ {(1)} (\tau) J (\tau). \tag {5}
$$
Supposing that we are given a set of trajectories  $F^{(t)}$  of length  $t$  with weights  $\beta^{(t)}$  such that  $V^{\pi} = \sum_{\tau \in F^{(t)}} \beta^{(t)}(\tau) J(\tau)$ , we can first increase the length of trajectories by 1 through Eq (4) and obtain a subset of  $\{\tau \circ s | \tau \in F^{(t)}, s \in S\}$ , in which the trajectories are of length  $t + 1$ . Specifically, we have
$$
V ^ {\pi} = \sum_ {\tau \in F ^ {(t)}, s \in \mathcal {S}} \beta^ {(t)} (\tau) P \left(s \mid s _ {t} ^ {\tau}, \pi \left(s _ {t} ^ {\tau}\right)\right) J (\tau \circ s). \tag {6}
$$
Then we would like to compress the above convex combination through C4 as we want to keep track of at most  $k + 1$  trajectories of length  $t + 1$  due to the computing time. More formally, let  $J_{F^{(t)}} \coloneqq \{J(\tau \circ s) | \tau \in F^{(t)}, s \in S\}$  be the set of expected returns and  $p_{F^{(t)},\beta^{(t)}} \in \mathrm{Simplex}^{F^{(t)} \times S}$  with  $p_{F^{(t)},\beta^{(t)}}(\tau \circ s) = \beta^{(t)}(\tau)P(s|s_t^\tau, \pi(s_t^\tau))$  be the weights appearing in Eq (6). Here  $p_{F^{(t)},\beta^{(t)}}$  defines a distribution over  $J_{F^{(t)}}$  with the probability of drawing  $J(\tau \circ s)$  being  $p_{F^{(t)},\beta^{(t)}}(\tau \circ s)$ . Then we can apply C4 over  $(J_{F^{(t)},p_{F^{(t)},\beta^{(t)}}})$  and compress the representative trajectories  $\{\tau \circ s | \tau \in F^{(t)}, s \in S\}$ . We start with trajectories of length 1 and repeat the process of expanding and compressing until we get trajectories of length  $H$ . The details are described in Algorithm 2.
<sup>9</sup>In Section 3, we consider a special type of random policy that is a mixed strategy of a deterministic policy (the output from an algorithm that solves for the optimal policy for an MDP with scalar reward) with the "do nothing" policy. For this specific random policy, we can find weighted trajectory representations for both policies and then apply Algorithm 4 to compress the representation.