SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 10. Define  $\mathcal{F}^{\pi}:\mathbb{M}\to \mathbb{M}$  by  $\mathcal{F}^{\pi}(d)(s,u) = |R_s^\pi -R_u^\pi | + \gamma \mathcal{W}_1(d)(T_s^\pi ,T_u^\pi)$ , then  $\mathcal{F}^{\pi}$  has a least fixed point  $d_{\sim}^{\pi}$ , and  $d_{\sim}^{\pi}$  is a  $\pi$ -bisimulation metric.
Although the Wasserstein distance is a powerful metric to calculate the distance between two probability distributions, it requires to enumerate all states which is impossible in RL tasks of continuous state space. Various extensions have been proposed [54, 6, 52] to reduce the computational complexity. DBC [54] extend bisimulation metrics to learn state representation, via minimizing the  $\ell_1$ -norm distance of representations and the bisimulation metrics, meanwhile modeling the latent dynamics as Gaussian and utilizing  $W_{2}$  distance to compute it, which can be formulated as a closed-form result. However, DBC has several issues like loss function mismatch and specific requirements for Gaussian modeling, which limits its application and performance.
C.2 MICo distance
MICo distance [6], tackles the above issue by restricting the coupling class to the independent coupling to avoid intractable Wasserstein distance computation. The MICo operator and its associated theoretical guarantee are given as:
Theorem 11. [6] Given a policy  $\pi$ , MICO distance  $\mathcal{F}^{\pi}$  is defined as:
$$
\mathcal {F} ^ {\pi} U \left(s _ {i}, s _ {j}\right) = \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma \mathbb {E} _ {s _ {i} ^ {\prime} \sim T _ {s} ^ {\pi}, s _ {j} ^ {\prime} \sim T _ {s _ {j}} ^ {\pi}} \left[ U \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right] \tag {16}
$$
has a fixed point  $U^{\pi}$ .
By considering the Wasserstein distance in the definition of bisimulation metrics can be upper-bounded by taking a restricted class of couplings of the transition distributions, MICo restricts the coupling class precisely to the singleton containing the independent coupling, utilizing the Independent Couple sampling strategy to bypass the computation of the Wasserstein distance. However, MICo distance  $U$  requires to be a Lukaszyk-Karmowski metric, which does not satisfy the identity of indiscernibles. As a result, the approximated distance on the learned embedding space based on the MICo distance, which involves a Lukaszyk-Karmowski metric to measure the distance between dynamics, may suffer from the violation issue of the identity of indiscernibles.
C.3 SimSR operator
To avoid the potential representation collapse, SimSR [52] develop a more concise update operator to learn state representation more effectively. Coupling with cosine distance, SimSR defines its operator as:
Theorem 12. [52] Given a policy  $\pi$ , Simple State Representation (SimSR) is updated as:
$$
\mathcal {F} ^ {\pi} \overline {{c o s}} _ {\phi} \left(s _ {i}, s _ {j}\right) = \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma \mathbb {E} _ {s _ {i} ^ {\prime} \sim T _ {s _ {i}} ^ {\pi}, s _ {j} ^ {\prime} \sim T _ {s _ {j}} ^ {\pi}} \left[ \overline {{c o s}} _ {\phi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right] \tag {17}
$$
has the same fixed point as MICO.
Further, considering the latent dynamics can be beneficial to representation learning, they additionally develop a form of operator including dynamics modeling:
Theorem 13. [52] Given a policy  $\pi$ , and a latent dynamics model  $\hat{T}$ , SimSR is updated as
$$
\mathcal {F} ^ {\pi} \overline {{c o s}} _ {\phi} \left(s _ {i}, s _ {j}\right) = \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma \mathbb {E} _ {z _ {i} ^ {\prime} \sim \hat {T} _ {\phi (s _ {i})} ^ {\pi}, z _ {j} ^ {\prime} \sim \hat {T} _ {\phi (s _ {j})} ^ {\pi}} \left[ \overline {{c o s}} \left(z _ {i} ^ {\prime}, z _ {j} ^ {\prime}\right) \right]. \tag {18}
$$
If latent dynamics are specified,  $\mathcal{F}^{\pi}$  has a fixed point.
When considering MICo distance and the basic version of SimSR, we can notice that they have a similar recursive iteration formulation. And therefore both works can be generalized under:
$$
\mathcal {F} ^ {\pi} G ^ {\pi} \left(s _ {i}, s _ {j}\right) = \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma_ {\substack {s _ {i} ^ {\prime} \sim T _ {s _ {i}} ^ {\pi} \\ s _ {j} ^ {\prime} \sim T _ {s _ {j}} ^ {\pi}}} \left[ G ^ {\pi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right], \tag{19}
$$
while the instantiation of  $G$  varies in these two approaches.
C.4 Lifted MDP
The connection between bisimulation-based operators and lifted MDP can be referred to [6]. We provide the corresponding Lemma here for reference.
Lemma 2. (Lifted MDP) The bisimulation-based update operator  $\mathcal{F}^{\pi}$  for  $\mathcal{M}$ , is the Bellman evaluation operator for a specific lifted MDP.
Proof. Given the MDP specified by the tuple  $(\mathcal{S},\mathcal{A},T,R)$ , we construct a lifted MDP  $(\widetilde{\mathcal{S}},\widetilde{\mathcal{A}},\widetilde{T},\widetilde{R})$  by taking the state space to be  $\widetilde{S} = S^2$ , the action space to be  $\widetilde{\mathcal{A}} = \mathcal{A}^2$ , the transition dynamics to be given by  $\widetilde{T}_{\widetilde{s}}^{\widetilde{a}}(\widetilde{s}') = \widetilde{T}_{(s_i,s_j)}^{(a_i,a_j)}((s_i',s_j')) = T_{s_i}^{a_i}(s_i')T_{s_j}^{a_j}(s_j')$  for all  $(s_i,s_j),(s_i',s_j') \in S^2$ ,  $a_i,a_j \in \mathcal{A}$  and the action-independent rewards to be  $\widetilde{R}_{\widetilde{s}} = \widetilde{R}_{(s_i,s_j)} = |r_{s_i}^\pi - r_{s_j}^\pi|$  for all  $s_i, s_j \in S$ . The Bellman evaluation operator  $\widetilde{\mathcal{F}}^{\widetilde{\pi}}$  for this lifted MDP at discount rate  $\gamma$  under the policy  $\tilde{\pi}(\tilde{a}|\tilde{s}) = \tilde{\pi}(a_i,a_j|s_i,s_j) = \pi(a_i|s_i)\pi(a_j|s_j)$  is given by (for all  $G^\pi \in \mathbb{R}^{S \times S}$  and  $(s_i,s_j) \in S \times S$ ):
$$
(\widetilde {\mathcal {F}} ^ {\tilde {\pi}} \tilde {G} ^ {\pi}) (\tilde {s}) = \widetilde {R} _ {\tilde {s}} + \gamma \sum_ {\tilde {s} ^ {\prime} \in \tilde {\mathcal {S}}} \widetilde {T} _ {\tilde {s}} ^ {\tilde {a}} (\tilde {s} ^ {\prime}) \tilde {\pi} (\tilde {a} | \tilde {s}) \tilde {G} ^ {\pi} (\tilde {s} ^ {\prime})
$$
$$
\begin{array}{l} (\widetilde {\mathcal {F}} ^ {\widetilde {\pi}} G ^ {\pi}) (s _ {i}, s _ {j}) = \widetilde {R} _ {(s _ {i}, s _ {j})} + \gamma \sum_ {(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \in \mathcal {S} ^ {2}} \widetilde {T} _ {(s _ {i}, s _ {j})} ^ {(a _ {i}, a _ {j})} ((s _ {i} ^ {\prime}, s _ {j} ^ {\prime})) \widetilde {\pi} (a _ {i}, a _ {j} | s _ {i}, s _ {j}) G ^ {\pi} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \\ = \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma \sum_ {\left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \in \mathcal {S} ^ {2}} T _ {s _ {i}} ^ {\pi} \left(s _ {i} ^ {\prime}\right) T _ {s _ {j}} ^ {\pi} \left(s _ {j} ^ {\prime}\right) G ^ {\pi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) = \left(\mathcal {F} _ {M} ^ {\pi} G ^ {\pi}\right) \left(s _ {i}, s _ {j}\right). \square \\ \end{array}
$$
C.5 Expectile Regression
Expectile regression, a method in statistics, is an extension of quantile regression that provides a more detailed analysis of a distribution's tail. This technique aims to estimate the expectiles of a conditional distribution, which are like percentiles but with respect to the mean, not the median. In essence, expectile regression can help capture the structure of data variability and analyze extreme observations in a more precise manner than quantile regression. The  $\tau \in (0,1)$  expectile of some random variable  $X$  is defined as a solution to the asymmetric least squares problem:
$$
\underset {m _ {\tau}} {\arg \min } \mathbb {E} _ {x \sim X} \left[ L _ {2} ^ {\tau} (x - m _ {\tau}) \right], \tag {20}
$$
Figure 5: The asymmetric squared loss used for expectile regression. Larger  $\tau$  gives more weight to positive differences.
where  $L_2^\tau(u) = |\tau - \mathbb{1}(u < 0)|u^2$ . That is, for  $\tau > 0.5$ , this asymmetric loss function downweights the contributions of  $x$  values smaller than  $m_\tau$  while giving more weights to larger values. Figure 5 shows the illustration of this asymmetric loss. More detailed descriptions can be found in [28, 36].
D.1 Connection between bisimulation error and bisimulation Bellman residual
In this section, we will revise some definitions a bit for obtaining the equivalence between bisimulation error and bisimulation Bellman residual. We first define bisimulation error  $\Delta_{\phi}^{\pi}$  that measures the distance of the approximation  $G_{\phi}^{\pi}$  to the fixed point  $G_{\sim}^{\pi}$  as:
$$
\Delta_ {\phi} ^ {\pi} := G _ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) - G _ {\sim} ^ {\pi} \left(s _ {i}, s _ {j}\right). \tag {21}
$$
And define bisimulation Bellman residual  $\epsilon_{\phi}^{\pi}$  as:
$$
\epsilon_ {\phi} ^ {\pi} := G _ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) - \mathcal {F} ^ {\pi} G _ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right). \tag {22}
$$
Notably, this is slightly different from the notation in Section 4 given the fact that we do not apply absolute value here. Then, we can have the following theorems.
Theorem 14. (The bisimulation Bellman residual can be defined as a function of the bisimulation error)
$$
\epsilon_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) = \Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) - \gamma \mathbb {E} _ {s _ {i} ^ {\prime} \sim T _ {s _ {i}} ^ {\pi}} \left[ \Delta_ {\phi} ^ {\pi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right], \tag {23}
$$
Proof. This follows directly from the bisimulation update operator:
$$
\begin{array}{l} \epsilon_ {\phi} ^ {\pi} (s _ {i}, s _ {j}) = G _ {\phi} ^ {\pi} (s _ {i}, s _ {j}) - \mathcal {F} ^ {\pi} G _ {\phi} ^ {\pi} (s _ {i}, s _ {j}) \\ = G _ {\sim} ^ {\pi} \left(s _ {i}, s _ {j}\right) + \Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) - \mathcal {F} ^ {\pi} \left(G _ {\sim} ^ {\pi} \left(s _ {i}, s _ {j}\right) + \Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right)\right) \tag {24} \\ = \Delta_ {\phi} ^ {\pi} (s _ {i}, s _ {j}) - \gamma \mathbb {E} _ {s _ {i} ^ {\prime} \sim T _ {s _ {i}} ^ {\pi}} [ \Delta_ {\phi} ^ {\pi} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) ] \\ \end{array}
$$
Theorem 15. (The bisimulation error can be defined as a function of the bisimulation Bellman residual). For any state pair  $(s_i, s_j) \in S \times S$ , the approximation error  $\Delta_{\phi}^{\pi}(s_i, s_j)$  can be defined as a function of the Bellman bisimulation error  $\epsilon_{\phi}$
$$
\Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) = \frac {1}{1 - \gamma} \mathbb {E} _ {\left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \sim \mu_ {\pi}} \left[ \epsilon_ {\phi} ^ {\pi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right]. \tag {25}
$$
Proof. Our proof follows similar steps to the proof of Lemma 6.1 in [26] and Theorem 1 in [17]. First by definition:
$$
\begin{array}{l} \Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) := G _ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) - G _ {\sim} ^ {\pi} \left(s _ {i}, s _ {j}\right) \tag {26} \\ \Rightarrow G _ {\sim} ^ {\pi} \left(s _ {i}, s _ {j}\right) = G _ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) - \Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) \\ \end{array}
$$
Then we can decompose the error:
$$
\begin{array}{l} \Delta_ {\phi} ^ {\pi} (s _ {i}, s _ {j}) = G _ {\phi} ^ {\pi} (s _ {i}, s _ {j}) - G _ {\sim} ^ {\pi} (s _ {i}, s _ {j}) \\ = G _ {\phi} ^ {\pi} (s _ {i}, s _ {j}) - \left(| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} | + \gamma \underset { \begin{array}{c} s _ {j} ^ {\prime} \sim T _ {s _ {j}} ^ {\pi} \\ \end{array} } {\mathbb {E}} _ {s _ {i} ^ {\prime} \sim T _ {s _ {j}} ^ {\pi}} [ G _ {\sim} ^ {\pi} (s _ {i} ^ {\prime}, s _ {i} ^ {\prime}) ]\right) \\ = G_{\phi}^{\pi}(s_{i},s_{j}) - \left(|r_{s_{i}}^{\pi} - r_{s_{j}}^{\pi}| + \gamma \mathbb{E}_{\substack{s_{i}^{\prime}\sim T_{s_{i}}^{\pi}\\ s_{j}^{\prime}\sim T_{s_{j}}^{\pi}}} [G_{\phi}^{\pi}(s_{i}^{\prime},s_{j}^{\prime}) - \Delta_{\phi}^{\pi}(s_{i}^{\prime},s_{j}^{\prime})]\right) \\ = G _ {\phi} ^ {\pi} (s _ {i}, s _ {j}) - \left(| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} | + \gamma \mathbb {E} _ {s _ {i} ^ {\prime} \sim T _ {s _ {i}} ^ {\pi}} [ G _ {\phi} ^ {\pi} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) ]\right) + \gamma \mathbb {E} _ {s _ {i} ^ {\prime} \sim T _ {s _ {i}} ^ {\pi}} [ \Delta_ {\phi} ^ {\pi} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) ] \\ = \epsilon_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) + \gamma \mathbb {E} _ {s _ {i} ^ {\prime} \sim T _ {s _ {i}} ^ {\pi}} \left[ \Delta_ {\phi} ^ {\pi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right] \tag {27} \\ \end{array}
$$
By considering the operator  $G$  as the Bellman evaluation operator for the lifted MDP (See Section C.4), we can rewrite the formula as:
$$
\Delta_ {\tilde {\phi}} ^ {\tilde {\pi}} (\tilde {x}) = \epsilon_ {\tilde {\phi}} ^ {\tilde {\pi}} (\tilde {x}) + \gamma \mathbb {E} _ {\tilde {x} ^ {\prime} \sim T _ {\tilde {x}} ^ {\tilde {\pi}}} \left[ \Delta_ {\tilde {\phi}} ^ {\tilde {\pi}} \left(\tilde {x} ^ {\prime}\right) \right]. \tag {28}
$$
Then we can treat  $\Delta_{\bar{\phi}}^{\bar{\pi}}(\tilde{x})$  as a value function and  $e_{\bar{\phi}}^{\bar{\pi}}(\tilde{x})$  as reward, we can see that:
$$
\Delta_ {\phi} ^ {\tilde {\pi}} (\tilde {x}) = \frac {1}{1 - \gamma} \mathbb {E} _ {\tilde {x} ^ {\prime} \sim T _ {\tilde {x}} ^ {\tilde {\pi}}} \left[ \epsilon_ {\phi} ^ {\tilde {\pi}} \left(\tilde {x} ^ {\prime}\right) \right]. \tag {29}
$$
Then we can obtain
$$
\Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) = \frac {1}{1 - \gamma} \mathbb {E} _ {\left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \sim \mu_ {\pi}} \left[ \epsilon_ {\phi} ^ {\pi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right]. \tag {30}
$$
D.2 Thoerem 3
$$
\Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) \leq \frac {1}{1 - \gamma} \mathbb {E} _ {\left(s _ {i}, s _ {j}\right) \sim \mu_ {\pi}} \left[ \epsilon_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) \right]. \tag {31}
$$
Proof. We start from Equation 25 in Section D.1.
$$
\begin{array}{l} \Delta_ {\phi} ^ {\pi} (s _ {i}, s _ {j}) = \frac {1}{1 - \gamma} \mathbb {E} _ {(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \sim \mu_ {\pi}} \left[ \epsilon_ {\phi} ^ {\pi} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \right] \\ \Rightarrow \left| \Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) \right| = \frac {1}{1 - \gamma} \left| \mathbb {E} _ {\left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \sim \mu_ {\pi}} \left[ \epsilon_ {\phi} ^ {\pi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right] \right| \tag {32} \\ \leq \frac {1}{1 - \gamma} \mathbb {E} _ {(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \sim \mu_ {\pi}} \left[ \left| \epsilon_ {\phi} ^ {\pi} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \right| \right]. \\ \end{array}
$$
Then when we define bisimulation error  $\Delta_{\phi}^{\pi}(s_i,s_j)\coloneqq |\Delta_{\phi}^{\pi}(s_i,s_j)|$  and bisimulation Bellman residual  $\epsilon_{\phi}^{\pi}(s_i',s_j')\coloneqq |\epsilon_{\phi}^{\pi}(s_i',s_j')|$ , we have
$$
\Delta_ {\phi} ^ {\pi} \left(s _ {i}, s _ {j}\right) \leq \frac {1}{1 - \gamma} \mathbb {E} _ {\left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \sim \mu_ {\pi}} \left[ \epsilon_ {\phi} ^ {\pi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right]. \tag {33}
$$
D.3 Proposition 4
Proposition 4. (The expected bisimulation residual is not sufficient over incomplete datasets). If there exists states  $s_i'$  and  $s_j'$  not contained in dataset  $\mathcal{D}$ , where the occupancy  $\mu_{\pi}(s_i'|s_i, a_i) > 0$  and  $\mu_{\pi}(s_j'|s_j, a_j) > 0$  for some  $s_i \in \mathcal{D}$ ,  $s_j \in \mathcal{D}$ , then there exists a bisimulation measurement and  $C > 0$  such that
Proof. This is a direct consequence of Theorem 15. Let  $\mathcal{D}'$  contain the set of state pairs  $(s_i', s_j')$  not contained in the dataset  $\mathcal{D}$ , where the next-state pair occupancy  $\mu_{\pi}(s_i', s_j'|s_i, a_i, s_j, a_j) > 0$ . Let  $\mathcal{D}_{\mathrm{unique}}$  be the set of unique state pairs in  $\mathcal{D}$ . It follows that
$$
\begin{array}{l} \Delta_ {\phi} ^ {\pi} (s _ {i}, s _ {j}) = \frac {1}{1 - \gamma} \mathbb {E} _ {(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \sim \mu_ {\pi}} \left[ \epsilon_ {\phi} ^ {\pi} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \right] \\ = \frac {1}{1 - \gamma} \sum_ {\left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \sim \mathcal {D} _ {\text {u n i q u e}}} \mu_ {\pi} \left(\left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \mid s _ {i}, a _ {i}, s _ {j}, a _ {j}\right) \epsilon_ {\phi} ^ {\pi} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) + \tag {34} \\ \frac {1}{1 - \gamma} \sum_ {(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \sim \mathcal {D} ^ {\prime}} \mu_ {\pi} ((s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) | s _ {i}, a _ {i}, s _ {j}, a _ {j}) \epsilon_ {\phi} ^ {\pi} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) \\ \end{array}
$$
Recall that  $\epsilon_{\phi}^{\pi}(s_i,s_j) = \Delta_{\phi}^{\pi}(s_i,s_j) - \gamma \mathbb{E}_{\substack{s_i'\sim T_{s_i}^{\pi}\\ s_j'\sim T_{s_j}^{\pi}}}\left[\Delta_{\phi}^{\pi}(s_i',s_j')\right]$ , and there exists at least one
$G(s_{i},s_{j})$ , such that  $(s_i,s_j)\in \mathcal{D}'$ . Since the sets  $\mathcal{D}$  and  $\mathcal{D}'$  are distinct, it follows that there exists a measurement  $G$  such that  $\epsilon_{\phi}^{\pi}(s_i,s_j) = 0$  for all  $(s_i,s_j)\in \mathcal{D}$ , but  $\frac{1}{1 - \gamma}\sum_{(s_i',s_j')\sim \mathcal{D}'}\mu_{\pi}(s_i',s_j'|s_i,a_i,s_j,a_j)\epsilon_{\phi}^{\pi}(s_i',s_j') = C$ .
D.4 Lemma 5
Lemma 5. For any  $\tau \in [0,1)$ ,  $\mathcal{F}_{\tau}^{\pi}$  is a  $\gamma_{\tau}$ -contraction, where  $\gamma_{\tau} = 1 - 2\alpha (1 - \gamma)\min \{\tau ,1 - \tau \}$ .
Proof. Note that  $\mathcal{F}_{1/2}^{\pi}$  is the standard bisimulation operator for  $\pi$ , of which the fixed point is  $G_{\sim}^{\pi}$ . To keep the notation succinct, we will replace  $G^{\pi}$  with  $G$ . For any  $G_1, G_2$ ,
$$
\begin{array}{l} \mathcal {F} _ {1 / 2} ^ {\pi} G _ {1} (s _ {i}, s _ {j}) - \mathcal {F} _ {1 / 2} ^ {\pi} G _ {2} (s _ {i}, s _ {j}) \\ = \left(G _ {1} \left(s _ {i}, s _ {j}\right) + \alpha \mathbb {E} ^ {\pi} [ \delta_ {i} ]\right) - \left(G _ {2} \left(s _ {i}, s _ {j}\right) + \alpha \mathbb {E} ^ {\pi} [ \delta_ {j} ]\right) \\ = (1 - \alpha) \left(G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right)\right) + \alpha \mathbb {E} _ {\pi} \left[ (1 - \gamma) \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma G _ {1} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right. \\ \left. - (1 - \gamma) \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| - \gamma G _ {2} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right] \tag {35} \\ = (1 - \alpha) \left(G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right)\right) + \alpha \mathbb {E} _ {\pi} \left[ \gamma G _ {1} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) - \gamma G _ {2} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) \right] \\ \leq (1 - \alpha) \| G _ {1} - G _ {2} \| _ {\infty} + \alpha \gamma \| G _ {1} - G _ {2} \| _ {\infty} \\ = (1 - \alpha (1 - \gamma)) \| G _ {1} - G _ {2} \| _ {\infty}. \\ \end{array}
$$
When  $\tau \neq \frac{1}{2}$ , we introduce two more operators to simplify the analysis:
$$
\begin{array}{l} \left(\mathcal {F} _ {+} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) = G \left(s _ {i}, s _ {j}\right) + \mathbb {E} ^ {\pi} [ \delta ] _ {+} \\ \left(\mathcal {T} ^ {\pi} G _ {-}\right) \left(s _ {i}, s _ {j}\right) = G \left(s _ {i}, s _ {j}\right) + \mathbb {E} ^ {\pi} [ \delta ] \end{array} \tag {36}
$$
$$
\left(\mathcal {F} _ {-} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right) = G \left(s _ {i}, s _ {j}\right) + \mathbb {E} ^ {\pi} [ \delta ] _ {-}
$$
Now we show that both operators meet the Banach-fixed point theorem (e.g.  $\| \mathcal{F}_+^\pi G_1 - \mathcal{F}_+^\pi G_2\|_\infty \leq \| G_1 - G_2\|_\infty$ ). For any  $G_{1},G_{2}$
$$
\begin{array}{l} \left(\mathcal {F} _ {+} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {+} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right) \\ = G _ {1} - G _ {2} + \mathbb {E} ^ {\pi} \left[ \left[ \delta_ {i} \right] _ {+} - \left[ \delta_ {j} \right] _ {+} \right] \tag {37} \\ = \mathbb {E} ^ {\pi} \left[ G _ {1} + \left[ \delta_ {i} \right] _ {+} - \left(G _ {2} + \left[ \delta_ {j} \right] _ {+}\right) \right] \\ \end{array}
$$
The relationship between  $G_{1} + [\delta_{i}]_{+}$  and  $G_{2} + [\delta_{j}]_{+}$  exists in four cases:
-  $\delta_{i} \geq 0, \delta_{j} \geq 0$ , then
$$
G _ {1} + \left[ \delta_ {i} \right] _ {+} - \left(G _ {2} + \left[ \delta_ {j} \right] _ {+}\right) = \gamma \left(G _ {1} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) - G _ {2} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right)\right). \tag {38}
$$
-  $\delta_{i} < 0, \delta_{j} < 0$ , then
$$
G _ {1} + \left[ \delta_ {i} \right] _ {+} - \left(G _ {2} + \left[ \delta_ {j} \right] _ {+}\right) = G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right). \tag {39}
$$
-  $\delta_{i} \geq 0, \delta_{j} < 0$ , then
$$
\begin{array}{l} G _ {1} + \left[ \delta_ {i} \right] _ {+} - \left(G _ {2} + \left[ \delta_ {j} \right] _ {+}\right) \\ = (1 - \gamma) \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma G _ {1} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) - G _ {2} \left(s _ {i}, s _ {j}\right) \\ <   (1 - \gamma) \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma G _ {1} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right) - ((1 - \gamma) \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + \gamma G _ {2} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right)) \tag {40} \\ = \gamma (G _ {1} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime}) - G _ {2} (s _ {i} ^ {\prime}, s _ {j} ^ {\prime})), \\ \end{array}
$$
where the inequality comes from  $G_{2}(s_{i}, s_{j}) > (1 - \gamma) |r_{s_{i}}^{\pi} - r_{s_{j}}^{\pi}| + \gamma G_{2}(s_{i}', s_{j}')$ .
-  $\delta_{i} < 0, \delta_{j} \geq 0$ , then
$$
\begin{array}{l} G _ {1} + \left[ \delta_ {i} \right] _ {+} - \left(G _ {2} + \left[ \delta_ {j} \right] _ {+}\right) \\ = G _ {1} \left(s _ {i}, s _ {j}\right) - \left((1 - \gamma) \left| r _ {s _ {i}} ^ {\pi} - r _ {s _ {j}} ^ {\pi} \right| + G _ {2} \left(s _ {i} ^ {\prime}, s _ {j} ^ {\prime}\right)\right) \tag {41} \\ \leq G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right), \\ \end{array}
$$
where the inequality comes from  $G_{2}(s_{i}, s_{j}) \leq (1 - \gamma) |r_{s_{i}}^{\pi} - r_{s_{j}}^{\pi}| + \gamma G_{2}(s_{i}', s_{j}')$ .
As a result, we have  $(\mathcal{F}_{+}^{\pi}G_{1})(s_{i},s_{j}) - (\mathcal{F}_{+}^{\pi}G_{2})(s_{i},s_{j})\leq \| G_{1} - G_{2}\|_{\infty}$ . Combine  $\mathcal{F}_{+}^{\pi}$  and  $\mathcal{F}_{-}^{\pi}$ , we can rewrite  $\mathcal{F}_{\tau}^{\pi}$  as:
$$
\begin{array}{l} \mathcal {F} _ {\tau} ^ {\pi} G (s _ {i}, s _ {j}) = G (s _ {i}, s _ {j}) + 2 \alpha \mathbb {E} ^ {\pi} [ \tau [ \delta ] _ {+} + (1 - \tau) [ \delta ] _ {-} ] \\ = (1 - 2 \alpha) G \left(s _ {i}, s _ {j}\right) + 2 \alpha \tau \left(G \left(s _ {i}, s _ {j}\right) + \mathbb {E} ^ {\pi} \left[ [ \delta ] _ {+} \right] + 2 \alpha (1 - \tau) \left(G \left(s _ {i}, s _ {j}\right) + \mathbb {E} ^ {\pi} \left[ [ \delta ] _ {-} \right]\right) \right. \\ = (1 - 2 \alpha) G \left(s _ {i}, s _ {j}\right) + 2 \alpha \tau \left(\mathcal {F} _ {+} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) + 2 \alpha (1 - \tau) \left(\mathcal {F} _ {-} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right). \tag {42} \\ \end{array}
$$
What's more