SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Crucially, a pseudo metric differs from a metric in that if  $d(x,y) = 0$  it may be the case that  $x \neq y$ .
Definition 4 (Diffuse Metric). A diffuse metric,  $d: X \times X \to \mathbb{R}_{\geq 0}$  has the following properties for some  $x, y, z \in X$ :
1.  $d(x,x)\geq 0$
2. Symmetry:  $d(x,y) = d(y,x)$
3. Triangle inequality:  $d(x,z) \leq d(x,y) + d(y,z)$
Crucially, a diffuse metric differs from a pseudo metric in that self-distances may be non-zero.
For readers interested in distances that admit non-zero self-distances, we refer them to material on partial metrics [Matthews, 1992]. We make the following note as Castro et al. [2022]: the original definition of partial metrics (see Matthews [1992]) uses a different triangle inequality criterion than the one in Definition 4 and is too strict (i.e. diffuse metrics violate this triangle inequality criterion), so we consider the diffuse metric definition presented in this paper.
We now present background material on the Wasserstein and related distances.
Definition 5 (Wasserstein Distance [Villani, 2008]). Let  $d: X \times X \to \mathbb{R}_{\geq 0}$  be a distance function and  $\Omega$  the set of all joint distributions with marginals  $\mu$  and  $\lambda$  over the space  $X$ , then we have:
$$
W (d) (\mu , \lambda) = \left(\inf  _ {\omega \in \Omega} \mathbb {E} _ {x _ {1}, x _ {2} \sim \omega} [ d (x _ {1}, x _ {2}) ]\right) \tag {4}
$$
Definition 6 (Dual formulation of the Wasserstein Distance [Villani, 2008]). Let  $d: X \times X \to \mathbb{R}_{\geq 0}$  be a distance function and marginals  $\mu$  and  $\lambda$  over the space  $X$ , then we have:
$$
W (d) (\mu , \lambda) = \sup  _ {f \in L i p _ {1, d} (X)} \mathbb {E} _ {x _ {1} \sim \mu} [ f (x _ {1}) ] - \mathbb {E} _ {x _ {2} \sim \lambda} [ f (x _ {2}) ] \tag {5}
$$
where  $\operatorname{Lip}_{1,d}(X)$  denotes the 1-Lipschitz functions  $f: X \to \mathbb{R}$  such that  $|f(x_1) - f(x_2)| \leq d(x_1, x_2)$ .
Definition 7 (Łukaszyk-Karmowski distance [Łukaszyk, 2004]). Let  $d: X \times X \to \mathbb{R}_{\geq 0}$  be a distance function and marginals  $\mu$  and  $\lambda$  over the space  $X$ , then we have:
$$
D _ {L K} (d) (\mu , \lambda) = \left(\mathbb {E} _ {x _ {1} \sim \mu , x _ {2} \sim \lambda} [ d (x _ {1}, x _ {2}) ]\right) \tag {6}
$$
We then have the following fact:  $W(d)(\mu, \lambda) \leq D_{\mathrm{LK}}(d)(\mu, \lambda)$  i.e. the Wasserstein distance is upper-bounded by the Lukaszyk-Karmowski distance [Castro et al., 2022].
Proof. Consider  $d, d' \in \mathbb{R}^{\mathcal{X} \times \mathcal{X}}$ , then we have:
$$
\begin{array}{l} \left| \left| \left(\mathcal {F} ^ {\pi_ {e}} d\right) \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right) - \left(\mathcal {F} ^ {\pi_ {e}} d ^ {\prime}\right) \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right) \right| \right| | _ {\infty} \\ = \left\| \gamma \mathbb {E} _ {s _ {1} ^ {\prime}, s _ {2} ^ {\prime} \sim P, a _ {1} ^ {\prime}, a _ {2} ^ {\prime} \sim \pi_ {e}} \left[ d \left(s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}\right) - d ^ {\prime} \left(s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}\right) \right] \right\| | _ {\infty} \\ = | \gamma | \cdot | | \mathbb {E} _ {s _ {1} ^ {\prime}, s _ {2} ^ {\prime} \sim P, a _ {1} ^ {\prime}, a _ {2} ^ {\prime} \sim \pi_ {e}} [ d (s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}) - d ^ {\prime} (s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}) ] | | _ {\infty} \\ \leq \gamma \max  _ {s _ {1} ^ {\prime}, a _ {1} ^ {\prime}, s _ {2} ^ {\prime}, a _ {2} ^ {\prime}} | d (s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}) - d ^ {\prime} (s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}) ] | = \gamma | | d - d ^ {\prime} | | _ {\infty} \\ \end{array}
$$
Proof. Since  $\mathcal{F}^{\pi_e}$  is a contraction mapping and that  $\mathbb{R}^{\mathcal{X}\times \mathcal{X}}$  is complete under the  $L^{\infty}$  norm, by Banach's fixed-point theorem,  $\lim_{t\to \infty}\mathcal{F}_t^{\pi_e}(d) = d_{\pi_e}$ .
Proof. To prove that  $d_{\pi_e}$  is a diffuse metric, we need to show it has the following properties for  $(s_1, a_1), (s_2, a_2), (s_3, a_3) \in \mathcal{X}$ . We follow Castro et al. [2022]'s strategy (see Proposition 4.10) to prove that a distance function is a diffuse metric. Recall that  $d_{\pi_e}(s_1, a_1; s_2, a_2) \coloneqq |r(s_1, a_1) - r(s_2, a_2)| + \gamma \mathbb{E}_{s'_1, s'_2 \sim P, a'_1, a'_2 \sim \pi_e}[d_{\pi_e}(s'_1, a'_1; s'_2, a'_2)]$ .
1. Non-negativity i.e.  $d_{\pi_e}(s_1, a_1; s_2, a_2) \geq 0$ . Since  $|r(s_1, a_1) - r(s_2, a_2)| \geq 0$ , recursively rolling out the definition of  $d_{\pi_e}$  means that  $d_{\pi_e}(s_1, a_1; s_2, a_2)$  is a sum of discounted non-negative terms.
2. Symmetry i.e.  $d_{\pi_e}(s_1, a_1; s_2, a_2) = d_{\pi_e}(s_2, a_2; s_1, a_1)$ . Since  $|r(s_1, a_1) - r(s_2, a_2)| = |r(s_2, a_2) - r(s_1, a_1)|$ , unrolling  $d_{\pi_e}(s_1, a_1; s_2, a_2)$  and  $d_{\pi_e}(s_2, a_2; s_1, a_1)$  recursively results in the discounted sum of the same terms.
3. Triangle inequality i.e.  $d_{\pi_e}(s_1, a_1; s_2, a_2) \leq d_{\pi_e}(s_1, a_1; s_3, a_3) + d_{\pi_e}(s_2, a_2; s_3, a_3)$ . To show this fact, we will first consider an initialization to the distance function  $d_0(s_1, a_1; s_2, a_2) = 0, \forall (s_1, a_1), (s_2, a_2) \in \mathcal{X}$  and consider repeated applications of the operator  $\mathcal{F}^{\pi_e}$  to  $d_0$ , which we know will make  $d_0$  converge to  $d_{\pi_e}$  (Proposition 2). We will show by induction that each successive update  $d_{t+1} = \mathcal{F}^{\pi_e}(d_t)$  satisfies the triangle inequality, which implies that  $d_{\pi_e}$  satisfies the triangle inequality.
We have the base the case at  $t = 0$  trivially holding true due to the initialization of  $d_0$ . Now let the inductive hypothesis be true for all  $t > 1$  i.e.  $d_t(s_1, a_1; s_2, a_2) \leq d_t(s_1, a_1; s_3, a_3) + d_t(s_3, a_3; s_2, a_2)$  for any  $(s_1, a_1), (s_2, a_2), (s_3, a_3) \in \mathcal{X}$ . However, we know that:
$$
\begin{array}{l} d _ {t + 1} (s _ {1}, a _ {1}; s _ {2}, a _ {2}) = | r (s _ {1}, a _ {1}) - r (s _ {2}, a _ {2}) | + \gamma \mathbb {E} _ {s _ {1} ^ {\prime}, s _ {2} ^ {\prime} \sim P, a _ {1} ^ {\prime}, a _ {2} ^ {\prime} \sim \pi_ {e}} [ d _ {t} (s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}) ] \\ \stackrel {(a)} {=} | r (s _ {1}, a _ {1}) - r (s _ {2}, a _ {2}) | + r (s _ {3}, a _ {3}) - r (s _ {3}, a _ {3}) + \gamma \mathbb {E} _ {s _ {1} ^ {\prime}, s _ {2} ^ {\prime} \sim P, a _ {1} ^ {\prime}, a _ {2} ^ {\prime} \sim \pi_ {e}} [ d _ {t} (s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}) ] \\ \stackrel {(b)} {\leq} \left| r \left(s _ {1}, a _ {1}\right) - r \left(s _ {3}, a _ {3}\right) \right| + \left| r \left(s _ {2}, a _ {2}\right) - r \left(s _ {3}, a _ {3}\right) \right| \\ + \gamma \mathbb {E} _ {s _ {1} ^ {\prime}, s _ {2} ^ {\prime} \sim P, a _ {1} ^ {\prime}, a _ {2} ^ {\prime} \sim \pi_ {e}} \left[ d _ {t} \left(s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}\right) \right] \\ \stackrel {(c)} {\leq} \left| r \left(s _ {1}, a _ {1}\right) - r \left(s _ {3}, a _ {3}\right) \right| + \left| r \left(s _ {2}, a _ {2}\right) - r \left(s _ {3}, a _ {3}\right) \right| \\ + \gamma \mathbb {E} _ {s _ {1} ^ {\prime}, s _ {2} ^ {\prime}, s _ {3} ^ {\prime} \sim P, a _ {1} ^ {\prime}, a _ {2} ^ {\prime}, a _ {3} ^ {\prime} \sim \pi_ {e}} \left[ d _ {t} \left(s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {3} ^ {\prime}, a _ {3} ^ {\prime}\right) + d _ {t} \left(s _ {3} ^ {\prime}, a _ {3} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}\right) \right] \\ = \left| r \left(s _ {1}, a _ {1}\right) - r \left(s _ {3}, a _ {3}\right) \right| + \gamma \mathbb {E} _ {s _ {1} ^ {\prime}, s _ {3} ^ {\prime} \sim P, a _ {1} ^ {\prime}, a _ {3} ^ {\prime} \sim \pi_ {e}} \left[ d _ {t} \left(s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {3} ^ {\prime}, a _ {3} ^ {\prime}\right) \right] \\ + \left| r \left(s _ {2}, a _ {2}\right) - r \left(s _ {3}, a _ {3}\right) \right| + \gamma \mathbb {E} _ {s _ {2} ^ {\prime}, s _ {3} ^ {\prime} \sim P, a _ {2} ^ {\prime}, a _ {3} ^ {\prime} \sim \pi_ {e}} \left[ d _ {t} \left(s _ {3} ^ {\prime}, a _ {3} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}\right) \right] \\ = d _ {t + 1} \left(s _ {1}, a _ {1}; s _ {3}, a _ {3}\right) + d _ {t + 1} \left(s _ {2}, a _ {2}; s _ {3}, a _ {3}\right) \\ \end{array}
$$
$$
d _ {t + 1} \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right) \leq d _ {t + 1} \left(s _ {1}, a _ {1}; s _ {3}, a _ {3}\right) + d _ {t + 1} \left(s _ {2}, a _ {2}; s _ {3}, a _ {3}\right)
$$
where (a) is due to adding and subtracting  $r(s_3, a_3)$ , (b) is due to Jensen's inequality, (c) is due to application of the inductive hypothesis. Thus, the triangle inequality is satisfied for all  $t \geq 0$ , and given that  $d_{t+1} \to d_{\pi_e}$ , we have that  $d_{\pi_e}$  also satisfies the triangle inequality.
Proof. To prove this fact, we follow Castro et al. [2022] (see Proposition 4.8) and use a co-inductive argument [Kozen, 2006]. We will show that if  $|q^{\pi_e}(s_1,a_1) - q^{\pi_e}(s_2,a_2)| \leq d(s_1,a_1; s_2,a_2)$  holds true for some specific symmetric  $d \in \mathbb{R}^{\mathcal{X} \times \mathcal{X}}$ , then the statement also holds true for  $\mathcal{F}^{\pi_e}(d)$ , which means it will hold for  $d_{\pi_e}$ .
We have that for any  $(s,a)\in \mathcal{X}$ ,  $\max_{s,a}\frac{-|r(s,a)|}{1 - \gamma} \leq q^{\pi_e}(s,a) \leq \max_{s,a}\frac{|r(s,a)|}{1 - \gamma}$ . Thus, for any  $(s_1,a_1),(s_2,a_2) \in \mathcal{X}$ , we have that  $|q^{\pi_e}(s_1,a_1) - q^{\pi_e}(s_2,a_2)| \leq 2\max_{s,a}\frac{|r(s,a)|}{1 - \gamma}$ . We can then assume that our specific symmetric  $d$  is the constant function  $d(s_1,a_1;s_2,a_2) = 2\max_{s,a}\frac{|r(s,a)|}{1 - \gamma}$  which satisfies our requirement that  $|q^{\pi_e}(s_1,a_1) - q^{\pi_e}(s_2,a_2)| \leq d(s_1,a_1;s_2,a_2)$ .
Therefore, we have  $q^{\pi_e}(s_1,a_1) - q^{\pi_e}(s_2,a_2)$
$$
\begin{array}{l} = r(s_{1},a_{1}) - r(s_{2},a_{2}) + \gamma \sum_{s^{\prime}_{1}\in \mathcal{S}}\sum_{a^{\prime}_{1}\in \mathcal{A}}P(s^{\prime}_{1}|s_{1},a_{1})\pi_{e}(a^{\prime}_{1}|s^{\prime}_{1})q^{\pi_{e}}(s^{\prime}_{1},a^{\prime}_{1}) - \gamma \sum_{s^{\prime}_{2}\in \mathcal{S}}\sum_{a^{\prime}_{2}\in \mathcal{A}}P(s^{\prime}_{2}|s_{2},a_{2})\pi_{e}(a^{\prime}_{2}|s^{\prime}_{2})q^{\pi_{e}}(s^{\prime}_{2},a^{\prime}_{2}) \\ \leq |r(s_{1},a_{1}) - r(s_{2},a_{2})| + \gamma \sum_{s^{\prime}_{1},s^{\prime}_{2}\in \mathcal{S}}\sum_{a^{\prime}_{1},a^{\prime}_{2}\in \mathcal{A}}P(s^{\prime}_{1}|s_{1},a_{1})\pi_{e}(a^{\prime}_{1}|s^{\prime}_{1})P(s^{\prime}_{2}|s_{2},a_{2})\pi_{e}(a^{\prime}_{2}|s^{\prime}_{2})(q^{\pi_{e}}(s^{\prime}_{1},a^{\prime}_{1}) - q^{\pi_{e}}(s^{\prime}_{2},a^{\prime}_{2})) \\ \stackrel {(a)} {\leq} | r (s _ {1}, a _ {1}) - r (s _ {2}, a _ {2}) | + \gamma \sum_ {s _ {1} ^ {\prime}, s _ {2} ^ {\prime} \in \mathcal {S}} \sum_ {a _ {1} ^ {\prime}, a _ {2} ^ {\prime} \in \mathcal {A}} P (s _ {1} ^ {\prime} | s _ {1}, a _ {1}) \pi_ {e} (a _ {1} ^ {\prime} | s _ {1} ^ {\prime}) P (s _ {2} ^ {\prime} | s _ {2}, a _ {2}) \pi_ {e} (a _ {2} ^ {\prime} | s _ {2} ^ {\prime}) d (s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}) \\ = \mathcal {F} ^ {\pi_ {e}} (d) \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right) \\ \end{array}
$$
where (a) follows from the induction hypothesis. Similarly, by symmetry, we can show that  $q^{\pi_e}(s_2,a_2) - q^{\pi_e}(s_1,a_1) \leq \mathcal{F}^{\pi_e}(d)(s_1,a_1;s_2,a_2)$ . Thus, we have it that  $|q^{\pi_e}(s_1,a_1) - q^{\pi_e}(s_2,a_2)| \leq d_{\pi_e}(s_1,a_1;;s_2,a_2)$ .
Proof. The proof closely follows that of Lemma 8 of Kemertas and Aumentado-Armstrong [2021], which is in turn based on Theorem 5.1 of Ferns et al. [2004]. The main difference between their theorems and ours is that the former is based on state representations and the latter is based on optimal state-value functions, while ours is focused on state-action representations for  $\pi_{\mathrm{e}}$ .
We first remark that this new aggregated MDP,  $\widetilde{\mathcal{M}}$ , can be viewed as a Markov reward process (MRP) where the "states" are aggregated state-action pairs of the original MDP,  $\mathcal{M}$ . We now define the reward function and transition dynamics of the clustered MRP  $\widetilde{\mathcal{M}}$ , where  $|\phi(x)|$  is the size of the cluster  $\phi(x)$ . Note that  $\mathbb{P}$  denotes the probability of the event.
$$
\tilde {r} (\phi (x)) = \frac {1}{| \phi (x) |} \sum_ {y \in \phi (x)} r (y)
$$
$$
\widetilde {P} (\phi (x ^ {\prime}) | \phi (x)) = \frac {1}{| \phi (x) |} \sum_ {y \in \phi (x)} \mathbb {P} (\phi (x ^ {\prime}) | y)
$$
Then we have:  $|q^{\pi_{\mathrm{e}}}(x) - \tilde{q}^{\pi_{\mathrm{e}}}(\phi (x))|$
where (a) is due to the definition of  $\tilde{\mathbf{r}}$  and  $\widetilde{P}$ , (b) is due to multiplying and dividing by  $|\phi(x)|$ , (c) is due to re-arranging terms, (d.1) is due to Jensen's inequality, (d.2 and d.3) are disaggregating the sums over clustered state-action into sums over original state-action by expanding  $\mathbb{P}(\phi(x')|y) = \sum_{x \in \phi(x')} P(x|y)$  for each clustered state-action,  $\phi(x')$ , (e) is grouping the terms, (f) is by adding and subtracting  $\frac{1}{|\phi(x)|} \sum_{y \in \phi(x)} P(x'|y) q^{\pi_{\mathrm{e}}} (x')$ , (g) is since the infinity norm of the difference of the action-values is greater than the expected difference, (h) is re-writing the expression in terms of expectations.
From Theorem 1 we know  $q^{\pi_e}$  is 1-Lipschitz with respect to the distance function  $d_{\pi_e}$ . Notice that (h) contains the dual formulation of the Wasserstein distance where  $f = q^{\pi_e}$  (see Definition 6). We
can then re-write (h) in terms of original definition of the Wasserstein distance:
$$
\begin{array}{l} | q ^ {\pi_ {\mathrm {e}}} (x) - \tilde {q} ^ {\pi_ {\mathrm {e}}} (\phi (x)) | \leq \frac {1}{| \phi (x) |} \sum_ {y \in \phi (x)} (| r (x) - r (y) | + \gamma W (d _ {\pi_ {e}}) (P (\cdot | x), P (\cdot | y)) + \gamma \| q - \tilde {q} \| _ {\infty}) \\ \stackrel {(i)} {\leq} \frac {1}{| \phi (x) |} \sum_ {y \in \phi (x)} \left(| r (x) - r (y) | + \gamma D _ {\mathrm {L K}} \left(d _ {\pi_ {e}}\right) \left(x ^ {\prime}, y ^ {\prime}\right) + \gamma \| q - \widetilde {q} \| _ {\infty}\right) \\ \stackrel {(j)} {=} \frac {1}{| \phi (x) |} \sum_ {y \in \phi (x)} (| r (x) - r (y) | + \gamma \mathbb {E} _ {x ^ {\prime} \sim \mathbb {P} ^ {\pi_ {e}}, y ^ {\prime} \sim \mathbb {P} ^ {\pi_ {e}}} [ d _ {\pi_ {e}} (x ^ {\prime}, y ^ {\prime}) ] + \gamma \| q - \widetilde {q} \| _ {\infty}) \\ \stackrel {(k)} {=} \frac {1}{| \phi (x) |} \sum_ {y \in \phi (x)} \left(d _ {\pi_ {\mathrm {e}}} (x, y) + \gamma \| q - \widetilde {q} \| _ {\infty}\right) \\ \stackrel {(l)} {\leq} 2 \epsilon + \gamma \left\| q - \widetilde {q} \right\| _ {\infty} \\ \end{array}
$$
$$
\left| q ^ {\pi_ {\mathrm {e}}} (x) - \tilde {q} ^ {\pi_ {\mathrm {e}}} (\phi (x)) \right| \stackrel {(m)} {\leq} \frac {2 \epsilon}{1 - \gamma}, \forall x \in \mathcal {X}
$$
where (i) is due the fact that the Lukaszyk-Karmowski,  $D_{\mathrm{LK}}$ , upper bounds the Wasserstein distance, (j) is using Definition 7, (k) is due to the definition of  $d_{\pi_{\mathrm{e}}}$ , and (l) is due the fact that the maximum distance between any two  $x,y \in \phi(x)$  is at most  $2\epsilon$ , which is greater than the average distance between any one point to every other point in the cluster, and (m) is due to  $\|q - \widetilde{q}\|_{\infty} \leq \frac{2\epsilon}{1 - \gamma}$ .
Proof. From Lemma 1 we have that  $|q^{\pi_{\mathrm{e}}}(s_0, a_0) - q^{\pi_{\mathrm{e}}}(\phi(s_0, a_0))| \leq \frac{2\epsilon}{(1 - \gamma)}$ .
$$
\begin{array}{l} \left| \mathbb {E} _ {s _ {0}, a _ {0} \sim \pi_ {\mathrm {e}}} [ q ^ {\pi_ {\mathrm {e}}} (s _ {0}, a _ {0}) ] - \mathbb {E} _ {s _ {0}, a _ {0} \sim \pi_ {\mathrm {e}}} [ q ^ {\pi_ {\mathrm {e}}} (\phi (s _ {0}, a _ {0})) ] \right| = \left| \mathbb {E} _ {s _ {0}, a _ {0} \sim \pi_ {\mathrm {e}}} [ q ^ {\pi_ {\mathrm {e}}} (s _ {0}, a _ {0}) - q ^ {\pi_ {\mathrm {e}}} (\phi (s _ {0}, a _ {0})) ] \right| \\ \stackrel {(a)} {\leq} \mathbb {E} _ {s _ {0}, a _ {0} \sim \pi_ {\mathrm {e}}} \left[ \left| q ^ {\pi_ {\mathrm {e}}} \left(s _ {0}, a _ {0}\right) - q ^ {\pi_ {\mathrm {e}}} \left(\phi \left(s _ {0}, a _ {0}\right)\right) \right| \right] \\ \stackrel {(b)} {\leq} \mathbb {E} _ {s _ {0}, a _ {0} \sim \pi_ {\mathrm {e}}} \left(\frac {2 \epsilon}{1 - \gamma}\right) \\ = \frac {2 \epsilon}{(1 - \gamma)}, \\ \end{array}
$$
where (a) follows from Jensen's inequality and (b) follows from Lemma 1.
C ROPE Pseudo-code
Algorithm 1 ROPE+FQE
1: Input: policy to evaluate  $\pi_e$ , batch  $\mathcal{D}$ , encoder parameters class  $\Omega$ , action-value parameter class  $\Xi$ , encoder function  $\phi : S \times \mathcal{A} \to \mathbb{R}^d$ , action-value function  $q : S \times \mathcal{A} \to \mathbb{R}$ .
2:  $\hat{\omega} \coloneqq \arg \min_{\omega \in \Omega}$
$$
\mathbb {E} _ {\left(s _ {1}, a _ {1}, s _ {1} ^ {\prime}\right), \left(s _ {2}, a _ {2}, s _ {2} ^ {\prime}\right) \sim \mathcal {D}} \left[ \rho \left(\left| r \left(s _ {1}, a _ {1}\right) - r \left(s _ {2}, a _ {2}\right) \right| + \gamma \mathbb {E} _ {a _ {1} ^ {\prime}, a _ {2} ^ {\prime} \sim \pi_ {\mathrm {e}}} \left[ \tilde {d} _ {\bar {\omega}} \left(s _ {1} ^ {\prime}, a _ {1} ^ {\prime}; s _ {2} ^ {\prime}, a _ {2} ^ {\prime}\right) \right] - \tilde {d} _ {\omega} \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right)\right) \right]
$$
{ROPE training phase; where  $\tilde{d}_{\omega}(s_1, a_1; s_2, a_2) := \frac{||\phi_{\omega}(s_1, a_1)||_2^2 + ||\phi_{\omega}(s_2, a_2)||_2^2}{2} + \beta \theta(\phi_{\omega}(s_1, a_1), \phi_{\omega}(s_2, a_2))$ ,  $\bar{\omega}$  are fixed parameters of target network, and  $\rho$  is the Huber loss. See Section 3.1 for more details.}
3:  $\hat{\xi} \coloneqq \arg \min_{\xi \in \Xi} \mathbb{E}_{(s,a,s')\sim \mathcal{D}}\left[\rho\left(r(s,a) + \gamma \mathbb{E}_{a'\sim \pi_{\mathrm{e}}}[q_{\hat{\xi}}(\phi_{\hat{\omega}}(s',a'))] - q_{\xi}(\phi_{\hat{\omega}}(s,a))\right)\right]$  {FQE using fixed encoder  $\phi_{\hat{\omega}}$  from Step 2, where  $\rho$  is the Huber loss.}
4: Return  $q_{\hat{\xi}}$  {Estimated action-value function of  $\pi_e$ ,  $q^{\pi_e}$ .}
D Empirical Results
We now include additional experiments that were deferred from the main text.
D.1 Gridworld Visualizations
In Section 4.2.1, we visualize how ROPE and on-policy MICO group state-action pairs. We now consider two additional metrics that group state-action pairs:
1. Policy similarity metric [Agarwal et al., 2021a]:  $d_{\mathrm{PSM}}(s_1, a_1; s_2, a_2) \coloneqq |\pi_{\mathrm{e}}(a_1|s_1) - \pi_{\mathrm{e}}(a_2|s_2)| + \gamma \mathbb{E}_{a_1', a_2'} \sim \pi_{\mathrm{e}}[d_{\mathrm{PSM}}((s_1', a_1'), (s_2', a_2'))]$ . This metric measures short- and long-term similarity based on how  $\pi_{\mathrm{e}}$  acts in different states, not in terms of the rewards and returns it receives.
2. Random policy similarity metric [Dadashi et al., 2021]:  $d_{\mathrm{RAND}}(s_1, a_1; s_2, a_2) \coloneqq |r(s_1, a_1) - r(s_2, a_2)| + \gamma \mathbb{E}_{a' \sim \mathcal{U}(\mathcal{A})}[d_{\mathrm{RAND}}((s_1', a'), (s_2', a'))]$ . Similar to  $d_{\pi_{\mathrm{e}}}$ , but considers behavior of a random policy that samples actions uniformly.
(b)  $d_{\mathrm{PSM}}$
Figure 5: Figure (a):  $q^{\pi_e}$  for  $\pi_e$ . Center and right: group clustering according to PSM (Figure (b)) and random-policy metric (Figure (c)) (center number in each triangle is group ID). Two state-action pairs are grouped together if their distance according to the specific metric is 0. The top right cell is blank since it is the terminal state, which is not grouped.
(c)  $d_{\mathrm{RAND}}$
From Figure 5, we reach the same conclusion as we did in Section 4.2.1: that existing state-action similarity metrics are unsuitable for learning  $q^{\pi_{\mathrm{e}}}$  due to how they group state-action pairs.
D.2 Deep OPE Experiments
We now present additional details on our empirical setup and additional experiments.
D.2.1 Additional Empirical Setup Details
Before applying any of the algorithms, we normalize the states of the dataset to make the each feature dimension have 0 mean and 1 standard deviation.
FQE Training Details In all experiments and all datasets, we use a neural network as FQE's action-value function with 2 layers and 256 neurons using RELU activation function. We use mini-batch gradient descent to train the FQE network with mini-batch sizes of 512 and for  $300\mathrm{K}$  gradient steps. We use the Adam optimizer with learning rate  $1e^{-5}$  and weight decay  $1e^{-2}$ . FQE minimizes the Huber loss. The only changes for FQE-DEEP are that it uses a neural network size of 4 layers with 256 neurons and trains for  $500\mathrm{K}$  gradient steps. Preliminary results with lower learning rates such as  $5e^{-6}$  and  $1e^{-6}$  did not make a difference. FQE uses an exponentially-moving average target network with  $\tau = 0.005$  updated every epoch.
ROPE and BCRL Details In all experiments and datasets, we use a neural network as the state-action encoder for ROPE with 2 layers and 256 neurons with the RELU activation. We use minibatch gradient descent to train the the encoder network with mini-batch sizes of 512 and for  $300\mathrm{K}$
gradient steps. For ROPE and BCRL, we hyperparameter sweep the output dimension of the encoder. Additionally, for ROPE, we sweep over the angular distance scalar,  $\beta$ . For the output dimension, we sweep over dimensions:  $\{|X|/3, |X|/2, |X|\}$ , where  $|X|$  is the dimension of the original state-action space of the environment. For  $\beta$ , we sweep over  $\{0.1, 1, 10\}$ . The best performing hyperparameter set is the one that results in lowest RMAE (from  $\rho(\pi_{\mathrm{e}})$ ) at the end of FQE training. ROPE uses an exponentially-moving average target network with  $\tau = 0.005$  updated every epoch. Finally, the output of ROPE's encoder is fed through a LayerNorm [Ba et al., 2016] layer, followed by a TANH layer. ROPE minimizes the Huber loss.
When computing  $d^{\pi_{\mathrm{e}}}\approx \tilde{d}_{\omega}$  ROPE uses the same procedure as MICO (appendix C.2. of Castro et al. [2022]):
$$
\tilde {d} _ {\omega} \left(s _ {1}, a _ {1}; s _ {2}, a _ {2}\right) := \frac {\left| \left| \phi_ {\omega} \left(s _ {1} , a _ {1}\right) \right| \right| _ {2} ^ {2} + \left| \left| \phi_ {\bar {\omega}} \left(s _ {2} , a _ {2}\right) \right| \right| _ {2} ^ {2}}{2} + \beta \theta \left(\phi_ {\omega} \left(s _ {1}, a _ {1}\right), \phi_ {\bar {\omega}} \left(s _ {2}, a _ {2}\right)\right)
$$
where it applies the target network parameters,  $\bar{\omega}$ , on the  $(s_2,a_2)$  pair for stability. For the angular distance  $\theta (\phi_{\omega}(s_1,a_1),\phi_{\omega}(s_2,a_2))$ , we have the cosine-similarity and the angle as below. Note in practice, for numerical stability, a small constant (e.g.  $1e^{-6}$  or  $5e^{-5}$ ) may have to be added when computing the square-root.
$$
\mathbf {C S} (\phi_ {\omega} (s _ {1}, a _ {1}), \phi_ {\omega} (s _ {2}, a _ {2})) = \frac {\langle \phi_ {\omega} (s _ {1} , a _ {1}) , \phi_ {\omega} (s _ {2} , a _ {2}) \rangle}{| | \phi_ {\omega} (s _ {1} , a _ {1}) | | | | \phi_ {\omega} (s _ {2} , a _ {2}) | |}
$$
$$
\theta \left(\phi_ {\omega} \left(s _ {1}, a _ {1}\right), \phi_ {\omega} \left(s _ {2}, a _ {2}\right)\right) = \arctan 2 \left(\sqrt {1 - \mathrm {C S} \left(\phi_ {\omega} \left(s _ {1} , a _ {1}\right) , \phi_ {\omega} \left(s _ {2} , a _ {2}\right)\right) ^ {2}}, \mathrm {C S} \left(\phi_ {\omega} \left(s _ {1}, a _ {1}\right), \phi_ {\omega} \left(s _ {2}, a _ {2}\right)\right)\right)
$$