SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} V ^ {\pi} = \sum_ {\tau} q ^ {\pi} (\tau) \Phi (\tau) = \sum_ {\tau} q ^ {\pi} (\tau) \left(\sum_ {t = 0} ^ {H - 1} R \left(s _ {t} ^ {\tau}, \pi \left(s _ {t} ^ {\tau}\right)\right)\right) \\ = \sum_ {t = 0} ^ {H - 1} \sum_ {\tau} q ^ {\pi} (\tau) R \left(s _ {t} ^ {\tau}, \pi \left(s _ {t} ^ {\tau}\right)\right) \\ = \sum_ {t = 0} ^ {H - 1} \sum_ {(s, s ^ {\prime}) \in E ^ {(t)}} R (s, \pi (s)) \sum_ {\tau : (s _ {t} ^ {\tau}, s _ {t + 1} ^ {\tau}) = (s, s ^ {\prime})} q ^ {\pi} (\tau) \\ = \sum_ {t = 0} ^ {H - 1} \sum_ {(s, s ^ {\prime}) \in E ^ {(t)}} R (s, \pi (s)) f (s, s ^ {\prime}) \\ = \sum_ {t = 0} ^ {H - 1} \sum_ {(s, s ^ {\prime}) \in E ^ {(t)}} R (s, \pi (s)) \sum_ {(\tau , f _ {\tau}) \in Q: (s _ {t} ^ {\tau}, s _ {t + 1} ^ {\tau}) = (s, s ^ {\prime})} f _ {\tau} \\ \end{array}
$$
$$
\begin{array}{l} = \sum_ {(\tau , f _ {\tau}) \in Q} f _ {\tau} \left(\sum_ {t = 0} ^ {H - 1} R \left(s _ {t} ^ {\tau}, \pi \left(s _ {t} ^ {\tau}\right)\right)\right) \\ = \sum_ {(\tau , f _ {\tau}) \in Q} f _ {\tau} \Phi (\tau). \\ \end{array}
$$
Computational complexity: Solving  $f$  takes  $\mathcal{O}(|E|)$  time. The algorithm will run for  $\mathcal{O}(|E|)$  iterations and each iteration takes  $\mathcal{O}(H)$  time. Since  $|E| = \mathcal{O}(|S|^2 H)$ , the total running time of Algorithm 5 is  $\mathcal{O}(|S|^2 H^2)$ . C4 will take  $\mathcal{O}(k^3 |E|)$  time.
M Proof of Theorem 3
Proof. Correctness: C4 guarantees that  $\left|F^{(H)}\right|\leq k + 1$
We will prove  $\sum_{\tau \in F^{(H)}}\beta_{\tau}^{(H)}\Phi (\tau) = V^{\pi}$  by induction on  $t = 1,\ldots ,H$
Recall that for any trajectory  $\tau$  of length  $h$ ,  $J(\tau) = \Phi(\tau) + V(s_h^\tau, H - h)$  was defined as the expected return of trajectories (of length  $H$ ) with the prefix being  $\tau$ .
In addition, recall that  $J_{F^{(t)}} = \{J(\tau \circ s) | \tau \in F^{(t)}, s \in S\}$  and  $p_{F^{(t)},\beta^{(t)}}$  was defined by letting  $p_{F^{(t)},\beta^{(t)}}(\tau \circ s) = \beta^{(t)}(\tau)P(s|s_t^\tau, \pi(s_t^\tau))$ .
For the base case, we have that at  $t = 1$
$$
\begin{array}{l} V ^ {\pi} = R \left(s _ {0}, \pi \left(s _ {0}\right)\right) + \sum_ {s \in \mathcal {S}} P \left(s \mid s _ {0}, \pi \left(s _ {0}\right)\right) V ^ {\pi} (s, H - 1) = \sum_ {s \in \mathcal {S}} P \left(s \mid s _ {0}, \pi \left(s _ {0}\right)\right) J \left(\left(s _ {0}\right) \circ s\right) \\ = \sum_ {s \in \mathcal {S}} p _ {F ^ {(0)}, \beta^ {(0)}} ((s _ {0}) \circ s) J ((s _ {0}) \circ s) = \sum_ {\tau \in F ^ {(1)}} \beta^ {(1)} (\tau) J (\tau). \\ \end{array}
$$
Suppose that  $V^{\pi} = \sum_{\tau' \in F^{(t)}} \beta^{(t)}(\tau') J(\tau')$  holds at time  $t$ , then we prove the statement holds at time  $t + 1$ .
$$
\begin{array}{l} V ^ {\pi} = \sum_ {\tau^ {\prime} \in F ^ {(t)}} \beta^ {(t)} (\tau^ {\prime}) J (\tau^ {\prime}) = \sum_ {\tau^ {\prime} \in F ^ {(t)}} \beta^ {(t)} (\tau^ {\prime}) (\Phi (\tau^ {\prime}) + V ^ {\pi} (s _ {t} ^ {\tau^ {\prime}}, H - t)) \\ = \sum_ {\tau^ {\prime} \in F ^ {(t)}} \beta^ {(t)} \left(\tau^ {\prime}\right) \left(\Phi \left(\tau^ {\prime}\right) + \sum_ {s \in \mathcal {S}} P \left(s \mid s _ {t} ^ {\tau^ {\prime}}, \pi \left(s _ {t} ^ {\tau^ {\prime}}\right)\right) \left(R \left(s _ {t} ^ {\tau^ {\prime}}, \pi \left(s _ {t} ^ {\tau^ {\prime}}\right)\right) + V ^ {\pi} (s, H - t)\right)\right) \\ = \sum_ {\tau^ {\prime} \in F ^ {(t)}} \sum_ {s \in \mathcal {S}} \beta^ {(t)} (\tau^ {\prime}) P (s | s _ {t} ^ {\tau^ {\prime}}, \pi (s _ {t} ^ {\tau^ {\prime}})) \left(\Phi (\tau^ {\prime}) + R (s _ {t} ^ {\tau^ {\prime}}, \pi (s _ {t} ^ {\tau^ {\prime}})) + V ^ {\pi} (s, H - t)\right) \\ = \sum_ {\tau^ {\prime} \in F ^ {(t)}} \sum_ {s \in \mathcal {S}} \beta^ {(t)} (\tau^ {\prime}) P (s | s _ {t} ^ {\tau^ {\prime}}, \pi (s _ {t} ^ {\tau^ {\prime}})) (\Phi (\tau^ {\prime} \circ s) + V ^ {\pi} (s, H - t)) \\ = \sum_ {\tau^ {\prime} \in F ^ {(t)}} \sum_ {s \in \mathcal {S}} p _ {F ^ {(t)}, \beta^ {(t)}} \left(\tau^ {\prime} \circ s\right) J \left(\tau^ {\prime} \circ s\right) \\ = \sum_ {\tau \in F ^ {(t + 1)}} \beta^ {(t + 1)} (\tau) J (\tau). \\ \end{array}
$$
By induction, the statement holds at  $t = H$  by induction, i.e.,  $V^{\pi} = \sum_{\tau \in F(H)} \beta_{\tau}^{(H)} J(\tau) = \sum_{\tau \in F^{(H)}} \beta_{\tau}^{(H)} \Phi(\tau)$ .
Computational complexity: Solving  $V^{\pi}(s,h)$  for all  $s\in S, h\in [H]$  takes time  $\mathcal{O}(kH|\mathcal{S}|^2)$ . In each round, we need to call C4 for  $\leq (k + 1)|S|$  vectors, which takes  $\mathcal{O}(k^4 |\mathcal{S}|)$  time. Thus, we need  $\mathcal{O}(k^4 H|\mathcal{S}| + kH|\mathcal{S}|^2)$  time in total.
N Example of maximizing individual objective
Observation 2. Assume there exist  $k > 2$  policies that together assemble  $k$  linear independent value vectors. Consider the  $k$  different policies  $\pi_1^*, \ldots, \pi_k^*$  that each  $\pi_i^*$  maximizes the objective  $i \in [k]$ . Then, their respective value vectors  $V_1^*, \ldots, V_k^*$  are not necessarily linearly independent. Moreover, if  $V_1^*, \ldots, V_k^*$  are linearly depended it does not mean that  $k$  linearly independent value vectors do not exist.
Proof. For simplicity, we show an example with a horizon of  $H = 1$  but the results could be extended to any  $H \geq 1$ . We will show an example where there are 4 different value vectors, where 3 of them are obtained by the  $k = 3$  policies that maximize the 3 objectives and have linear dependence.
Consider an MDP with a single state (also known as Multi-arm Bandit) with 4 actions with deterministic reward vectors (which are also the expected values of the 4 possible policies in this case):
$$
r (1) = \left( \begin{array}{c} 8 \\ 4 \\ 2 \end{array} \right), \quad r (2) = \left( \begin{array}{c} 1 \\ 2 \\ 3 \end{array} \right), \quad r (3) = \left( \begin{array}{c} 8 5 / 1 2 \\ 2 5 / 6 \\ 3 5 / 1 2 \end{array} \right) \approx \left( \begin{array}{c} 7. 0 8 3 \\ 4. 1 6 7 \\ 2. 9 1 6 7 \end{array} \right), \quad r (4) = \left( \begin{array}{c} 1 \\ 3 \\ 2 \end{array} \right).
$$
Denote  $\pi^a$  as the fixed policy that always selects action  $a$ . Clearly, policy  $\pi^1$  maximizes the first objective, policy  $\pi^2$  the third, and policy  $\pi^3$  the second ( $\pi^4$  do not maximize any objective). However,
-  $r(3)$  linearly depends on  $r(1)$  and  $r(2)$  as
$$
\frac {5}{6} r (1) + \frac {5}{1 2} r (2) = r (3).
$$
- In addition,  $r(4)$  is linearly independent in  $r(1), r(2)$ : Assume not. Then, there exists  $\beta_1, \beta_2 \in \mathbb{R}$  s.t.:
$$
\beta_ {1} \cdot r (1) + \beta_ {2} \cdot r (2) = \left( \begin{array}{c} 8 \beta_ {1} + \beta_ {2} \\ 4 \beta_ {1} + 2 \beta_ {2} \\ 2 \beta_ {1} + 3 \beta_ {2} \end{array} \right) = \left( \begin{array}{c} 1 \\ 3 \\ 2 \end{array} \right) = r (4).
$$
Hence, the first equations imply  $\beta_{2} = 1 - 8\beta_{1}$ , and  $4\beta_{1} + 2 - 16\beta_{1} = 3$ , hence  $\beta_{1} = -\frac{1}{12}$  and  $\beta_{2} = \frac{5}{3}$ . Assigning in the third equation yields  $-\frac{1}{6} + 5 = 2$  which is a contradiction.