SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {F} _ {\frac {1}{2}} ^ {\pi} G (s _ {i}, s _ {j}) = G (s _ {i}, s _ {j}) + \alpha \mathbb {E} ^ {\pi} [ \delta ] \\ = G \left(s _ {i}, s _ {j}\right) + \alpha \left(\left(\mathcal {F} _ {+} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) + \left(\mathcal {F} _ {-} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - 2 \alpha G \left(s _ {i}, s _ {j}\right)\right) \tag {43} \\ = (1 - 2 \alpha) G \left(s _ {i}, s _ {j}\right) + \alpha \left(\left(\mathcal {F} _ {+} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) + \left(\mathcal {F} _ {-} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right)\right). \\ \end{array}
$$
When  $\tau >\frac{1}{2}$  , for any  $G_{1}$  and  $G_{2}$  ..
$$
\begin{array}{l} \left(\mathcal {F} _ {\tau} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {\tau} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right) \\ = (1 - 2 \alpha) \left(G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right)\right) + 2 \alpha \tau \left(\left(\mathcal {F} _ {+} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {+} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right)\right) \\ + 2 \alpha (1 - \tau) \left(\left(\mathcal {F} _ {-} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {-} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right)\right) \\ = (1 - 2 \alpha - 2 (1 - 2 \alpha) (1 - \tau)) (G _ {1} (s _ {i}, s _ {j}) - G _ {2} (s _ {i}, s _ {j})) _ {2} (1 - \tau) ((\mathcal {F} _ {\frac {1}{2}} ^ {\pi} G _ {1}) (s _ {i}, s _ {j}) - (\mathcal {F} _ {\frac {1}{2}} ^ {\pi} G _ {2}) (s _ {i}, s _ {j})) \\ - 2 \alpha (1 - 2 \tau) \left(\left(\mathcal {F} _ {+} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {+} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right)\right) \\ \leq (1 - 2 \alpha - 2 (1 - 2 \alpha) (1 - \tau)) \| G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right) \| _ {\infty} \\ + 2 (1 - \tau) (1 - \alpha (1 - \gamma)) \| G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right) \| _ {\infty} \\ - 2 \alpha (1 - 2 \tau) \| G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right) \| _ {\infty} \\ = (1 - 2 \alpha (1 - \tau) (1 - \gamma)) \| G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right) \| _ {\infty} \tag {44} \\ \end{array}
$$
When  $\tau < \frac{1}{2}$ , for any  $G_{1}$  and  $G_{2}$ :
$$
\begin{array}{l} \left(\mathcal {F} _ {\tau} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {\tau} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right) \\ = (1 - 2 \alpha) \left(G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right)\right) + 2 \alpha \tau \left(\left(\mathcal {F} _ {+} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {+} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right)\right) \\ + 2 \alpha (1 - \tau) \left(\left(\mathcal {F} _ {-} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {-} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right)\right) \\ = (1 - 2 \alpha - 2 \tau (1 - 2 \alpha)) \left(G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right)\right) + 2 \tau \left(\left(\mathcal {F} _ {\frac {1}{2}} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {\frac {1}{2}} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right)\right) \\ + 2 \alpha (1 - 2 \tau) \left(\left(\mathcal {F} _ {-} ^ {\pi} G _ {1}\right) \left(s _ {i}, s _ {j}\right) - \left(\mathcal {F} _ {-} ^ {\pi} G _ {2}\right) \left(s _ {i}, s _ {j}\right)\right) \\ \leq (1 - 2 \alpha - 2 \tau (1 - 2 \alpha)) \| G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right) \| _ {\infty} \\ + 2 \tau (1 - \alpha (1 - \gamma)) \| G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right) \| _ {\infty} \\ + 2 \alpha (1 - 2 \tau) \| G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right) \| _ {\infty} \\ = (1 - 2 \alpha \tau (1 - \gamma)) \| G _ {1} \left(s _ {i}, s _ {j}\right) - G _ {2} \left(s _ {i}, s _ {j}\right) \| _ {\infty}. \tag {45} \\ \end{array}
$$
D.5 Lemma 6
Proof. We denote  $G_{\tau'}$  is the fixed point of applying the operator  $\mathcal{F}_{\tau'}^{\pi}$ , and  $G_{\tau}$  is the fixed point of applying the operator  $\mathcal{F}_{\tau}$ . Based on Equation 6, we have:
$$
\begin{array}{l} \mathcal {F} _ {\tau^ {\prime}} ^ {\pi} G (s _ {i}, s _ {j}) - \mathcal {F} _ {\tau} ^ {\pi} G (s _ {i}, s _ {j}) \\ = (1 - 2 \alpha) G \left(s _ {i}, s _ {j}\right) + 2 \alpha \tau^ {\prime} \mathcal {F} _ {+} ^ {\pi} G \left(s _ {i}, s _ {j}\right) + 2 \alpha \left(1 - \tau^ {\prime}\right) \mathcal {F} _ {-} ^ {\pi} G \left(s _ {i}, s _ {j}\right) \\ - \left((1 - 2 \alpha) G \left(s _ {i}, s _ {j}\right) + 2 \alpha \tau \mathcal {F} _ {+} ^ {\pi} G \left(s _ {i}, s _ {j}\right) + 2 \alpha (1 - \tau) \mathcal {F} _ {-} ^ {\pi} G \left(s _ {i}, s _ {j}\right)\right) \tag {46} \\ = 2 \alpha (\tau^ {\prime} - \tau) \left(\mathcal {F} _ {+} ^ {\pi} G (s _ {i}, s _ {j}) - \mathcal {F} _ {-} ^ {\pi} G (s _ {i}, s _ {j})\right) \\ = 2 \alpha (\tau^ {\prime} - \tau) \mathbb {E} ^ {\pi} [ [ \delta ] _ {+} - [ \delta ] _ {-} ] \geq 0. \\ \end{array}
$$
Therefore  $G_{\tau^{\prime}} > G_{\tau}$
D.6 Theorem 7
$$
\lim  _ {\tau \rightarrow 1} G _ {\tau} \left(s _ {i}, s _ {j}\right) = \max  _ {\substack {a _ {i} \in \mathcal {A}, a _ {j} \in \mathcal {A}\\\text {s.t.} \pi_ {\beta} \left(a _ {i} \mid s _ {i}\right) > 0, \pi_ {\beta} \left(a _ {j} \mid s _ {j}\right) > 0}} G _ {\sim} ^ {*} \left(\left(s _ {i}, a _ {i}\right), \left(s _ {j}, a _ {j}\right)\right). \tag{47}
$$
where  $G_{\sim}^{*}((s_{i},a_{i}),(s_{j},a_{j}))$  is a fixed-point measurement constrained to the dataset and defined on state-action space  $S\times \mathcal{A}$  as
$$
G _ {\sim} ^ {*} \left(\left(s _ {i}, a _ {i}\right), \left(s _ {j}, a _ {j}\right)\right) = \left| r \left(s _ {i}, a _ {i}\right) - r \left(s _ {j}, a _ {j}\right) \right| + \gamma \mathbb {E} _ {\substack {s _ {i} ^ {\prime} \sim T _ {s _ {i}} ^ {\pi_ {\beta}} \\ s _ {j} ^ {\prime} \sim T _ {s _ {j}} ^ {\pi_ {\beta}}}} \left[ \max  _ {\substack {a _ {i} ^ {\prime} \in \mathcal {A}, a _ {j} ^ {\prime} \in \mathcal {A} \\ s. t. \pi_ {\beta} \left(a _ {i} ^ {\prime} \mid s _ {i} ^ {\prime}\right) > 0, \pi_ {\beta} \left(a _ {j} ^ {\prime} \mid s _ {j} ^ {\prime}\right) > 0}} G _ {\sim} ^ {*} \left(\left(s _ {i} ^ {\prime}, a _ {i} ^ {\prime}\right), \left(s _ {j} ^ {\prime}, a _ {j} ^ {\prime}\right)\right) \right].
$$
Proof. First, we can easily proof that  $G_{\sim}^{*}(s_{i},a_{i},s_{j},a_{j})$  is a fixed point. Define the corresponding operator of  $G_{\sim}^{*}$  is  $F^{*}$ , we can know that  $F^{*}$  is a contraction. Then, we have
Corollary 16. For any  $\tau, s_i, s_j \in S$  we have
$$
G _ {\tau} \left(s _ {i}, s _ {j}\right) \leq \max  _ { \begin{array}{l} a _ {i} \in \mathcal {A}, a _ {j} \in \mathcal {A} \\ s. t. \pi_ {\beta} \left(a _ {i} \mid s _ {i}\right) > 0, \pi_ {\beta} \left(a _ {j} \mid s _ {j}\right) > 0 \end{array} } G _ {\sim} ^ {*} \left(\left(s _ {i}, a _ {i}\right), \left(s _ {j}, a _ {j}\right)\right) \tag {49}
$$
Proof. The proof follows from the observation that convex combination is smaller than maximum.  $\square$
Besides, we also have
Lemma 17. Let  $X$  be a real-valued random variable with a bounded support and supremum of the support is  $x^{*}$ . Then,
$$
\lim  _ {\tau \to 1} m _ {\tau} = x ^ {*}
$$
Proof. Same as the Lemma 1 in [28]. One can show that expectiles of a random variable have the same supremum  $x^{*}$ . Moreover, for all  $\tau_{1}$  and  $\tau_{2}$  such that  $\tau_{1} < \tau_{2}$ , we get  $m_{\tau_1} \leq m_{\tau_2}$ . Therefore, the limit follows from the properties of bounded monotonically non-decreasing functions.
Combining Corollary 16 and Lemma 17, we can obtain the above.
D.7 Theorem 8
Theorem 8. (Value bound based on on-policy bisimulation measurements in terms of encoder error). Given an MDP  $\widetilde{\mathcal{M}}$  constructed by aggregating states in an  $\omega$ -neighborhood, and an encoder  $\phi$  that maps from states in the original MDP  $\mathcal{M}$  to these clusters, the value functions for the two MDPs are bounded as
$$
\left| V ^ {\pi} \left(s _ {i}\right) - \widetilde {V} ^ {\pi} \left(\phi \left(s _ {i}\right)\right) \right| \leq \frac {2 \omega + \hat {\Delta}}{c _ {r} (1 - \gamma)}. \tag {50}
$$
Proof. Let the reward function be bounded as  $R \in [0,1]$ ,  $\phi : S \to \widetilde{S}$ , and  $\phi(s_i) = \phi(s_j) \Rightarrow \hat{G}_{\phi}^{\pi}(s_i, s_j) = |\phi(s_i) - \phi(s_j)| \leq 2\omega$ , we can conduct an aggregate MDP  $\widetilde{M} = (\widetilde{S}, \mathcal{A}, \widetilde{T}, \widetilde{R})$ . Let  $\xi$  be
a measure on  $S$ . Following Lemma 8 in [27], we have that:
$$
\begin{array}{l} \left| V ^ {\pi} \left(s _ {i}\right) - \widetilde {V} ^ {\pi} \left(\phi \left(s _ {i}\right)\right) \right| \leq \frac {c _ {r} ^ {- 1}}{\xi (\phi (s))} \int_ {z \in \phi (s)} c _ {R} | r ^ {\pi} (s) - r ^ {\pi} (z) | \\ + (1 - \gamma) \left| \int_ {s ^ {\prime} \in \mathcal {S}} \left(T ^ {\pi} \left(s ^ {\prime} | s\right) - T ^ {\pi} \left(s ^ {\prime} | z\right)\right) \frac {c _ {r} \gamma}{1 - \gamma} V ^ {\pi} \left(s ^ {\prime}\right) d s ^ {\prime} \right| d \xi (z) + \gamma \| V ^ {\pi} - \widetilde {V} ^ {\pi} \| _ {\infty} \\ \leq \frac {c _ {r} ^ {- 1}}{\xi (\phi (s))} \int_ {z \in \phi (s)} G _ {\sim} ^ {\pi} (s, z) d \xi (z) + \gamma \| V ^ {\pi} - \widetilde {V} ^ {\pi} \| _ {\infty} \tag {51} \\ \end{array}
$$
Thus, taking the supremum on the LHS, we have:
$$
\begin{array}{l} (1 - \gamma) \left| V ^ {\pi} (s _ {i}) - \widetilde {V} ^ {\pi} (\phi (s _ {i})) \right| \leq \frac {c _ {r} ^ {- 1}}{\xi (\phi (s))} \int_ {z \in \phi (s)} G _ {\sim} ^ {\pi} (s, z) d \xi (z) \\ \leq \frac {c _ {r} ^ {- 1}}{\xi (\phi (s))} \int_ {z \in \phi (s)} \hat {G} _ {\phi} ^ {\pi} (s, z) + \| G _ {\sim} ^ {\pi} - \hat {G} _ {\phi} ^ {\pi} \| _ {\infty} d \xi (z) \tag {52} \\ = \frac{c_{r}^{-1}}{\xi(\phi(s))}\int \limits_{z\in \phi (s)}(2\omega +\hat{\Delta})d\xi (z) \\ = c _ {r} ^ {- 1} (2 \omega + \hat {\Delta}). \\ \end{array}
$$
$$
\left| V ^ {\pi} \left(s _ {i}\right) - \widetilde {V} ^ {\pi} \left(\phi \left(s _ {i}\right)\right) \right| \leq \frac {2 \omega + \hat {\Delta}}{c _ {r} (1 - \gamma)}, \tag {53}
$$
E Understanding of Theorem 7
$$
\lim  _ {\tau \rightarrow 1} G _ {\tau} \left(s _ {i}, s _ {j}\right) = \max  _ {\substack {a _ {i} \in \mathcal {A}, a _ {j} \in \mathcal {A}\\s. t. \pi_ {\beta} \left(a _ {i} \mid s _ {i}\right) > 0, \pi_ {\beta} \left(a _ {j} \mid s _ {j}\right) > 0}} G _ {\sim} ^ {*} \left(\left(s _ {i}, a _ {i}\right), \left(s _ {j}, a _ {j}\right)\right). \tag{54}
$$
$$
G _ {\sim} ^ {*} \left(\left(s _ {i}, a _ {i}\right), \left(s _ {j}, a _ {j}\right)\right) = \left| r \left(s _ {i}, a _ {i}\right) - r \left(s _ {j}, a _ {j}\right) \right| + \gamma \mathbb {E} _ {s _ {i} ^ {\prime} \sim T _ {s _ {i}} ^ {\pi_ {\beta}}} \max  _ {\substack {a _ {i} ^ {\prime} \in \mathcal {A}, a _ {j} ^ {\prime} \in \mathcal {A} \\ s _ {j} ^ {\prime} \sim T _ {s _ {j}} ^ {\pi_ {\beta}}}} G _ {\sim} ^ {*} \left(\left(s _ {i} ^ {\prime}, a _ {i} ^ {\prime}\right), \left(s _ {j} ^ {\prime}, a _ {j} ^ {\prime}\right)\right). \tag{55}
$$
Given the MDP specified by the tuple  $(\mathcal{S},\mathcal{A},T,R)$ , we construct a lifted MDP  $(\widetilde{\mathcal{S}},\widetilde{\mathcal{A}},\widetilde{T},\widetilde{R})$ , by taking the state space to be  $\widetilde{S} = S^2$ , the action space to be  $\widetilde{\mathcal{A}} = \mathcal{A}^2$ , the transition dynamics to be given by  $\widetilde{T}_{\widetilde{s}}^{\widetilde{a}}(\widetilde{s}') = \widetilde{T}_{(s_i,s_j)}^{(a_i,a_j)}((s_i',s_j')) = T_{s_i}^{a_i}(s_i')T_{s_j}^{a_j}(s_j')$  for all  $(s_i,s_j),(s_i',s_j') \in S^2$ ,  $a_i,a_j \in \mathcal{A}$  and the action-independent rewards to be  $\widetilde{R}_{\widetilde{s}} = \widetilde{R}_{(s_i,s_j)} = |r_{s_i}^\pi - r_{s_j}^\pi|$  for all  $s_i, s_j \in S$ . The Bellman evaluation operator  $\widetilde{\mathcal{F}}^{\widetilde{\pi}}$  for this lifted MDP at discount rate  $\gamma$  under the policy  $\tilde{\pi}(\tilde{a}|\tilde{s}) = \tilde{\pi}(a_i,a_j|s_i,s_j) = \pi(a_i|s_i)\pi(a_j|s_j)$  is given by (for all  $G^\pi \in \mathbb{R}^{\mathcal{S} \times \mathcal{S}}$  and  $(s_i,s_j) \in S \times S$ ):
$$
\left(\widetilde {\mathcal {F}} ^ {\tilde {\pi}} Q ^ {*}\right) \left(\tilde {s}, \tilde {a}\right) = \widetilde {R} _ {\tilde {s}, \tilde {a}} + \gamma \sum_ {\tilde {s} \in \tilde {\mathcal {S}}} \widetilde {T} _ {\tilde {s}} ^ {\tilde {a}} \left(\tilde {s} ^ {\prime}\right) \max  _ {\tilde {a} \in \tilde {A}} Q ^ {*} \left(\tilde {s} ^ {\prime}, \tilde {a} ^ {\prime}\right). \tag {56}
$$
Though similar, Equation 55 has more constraints as it requires the possibility of  $\pi_{\beta}(a_i'|s_i')$  and  $\pi_{\beta}(a_j'|s_j')$  are larger than zero in the dataset. As such, we may also change the Equation 56 to:
$$
\left(\tilde {\mathcal {F}} ^ {\tilde {\pi}} Q ^ {*}\right) \left(\tilde {s}, \tilde {a}\right) = \tilde {R} _ {\tilde {s}, \tilde {a}} + \gamma \sum_ {\tilde {s} \in \tilde {\mathcal {S}}} \tilde {T} _ {\tilde {s}} ^ {\tilde {a}} \left(\tilde {s} ^ {\prime}\right) \max  _ {\substack {\tilde {a} ^ {\prime} \in \mathcal {A} \\ \text {s.t.} \tilde {\pi} _ {\beta} \left(\tilde {a} ^ {\prime} \mid \tilde {s} ^ {\prime}\right) > 0}} Q ^ {*} \left(\tilde {s} ^ {\prime}, \tilde {a} ^ {\prime}\right). \tag{57}
$$
This is, indeed, equivalent to the in-sample-style Q function in [28]. Intuitively,  $G_{\sim}^{*}((s_{i},a_{i}),(s_{j},a_{j}))$  can be interpreted as the optimal state-action value function  $Q^{*}(\tilde{s},\tilde{a})$  in a lifted MDP  $\widetilde{M}$ . Then  $G_{\sim}^{\pi}((s_i,a_i),(s_j,a_j))$  is the state-action value function  $Q^{\pi}(\tilde{s},\tilde{a})$  that associated with policy  $\pi$ , and  $G_{\sim}(s_i,s_j)$  as a state value function  $V(\tilde{s})$ . And therefore, we can connect our expectile-based bisimulation operator to the lifted MDP, where we can use the conventional analytics tools in RL to analyze bisimulation operators.
F.1 Ablation Study - Value of Expectile
Figure 6: Performance comparison on 12 D4RL tasks over 10 seeds with one standard error shaded in the default setting.
Here we present the ablation study of setting different expectile  $\tau \in \{0.3, 0.4, \dots, 0.7\}$  in Figure 6 to investigate the effect of the critical hyper-parameter in EBS. The experimental results demonstrate that the final performance gradually improves with a larger  $\tau$ . Notably, the most superior performance is achieved when  $\tau$  equals 0.6. However, when  $\tau$  further increases to 0.7, the agent's performance suffers a sharp decline. We hypothesize that this could be due to the value function possibly exploding when  $\tau$  is set to larger values, subsequently leading to poorer performance outcomes. This is as expected since the over-large  $\tau$  leads to the overestimation error caused by neural networks. The experimental results demonstrate that we can balance a trade-off between minimizing the expected bisimulation residual and evaluating "optimal" measurement solely on the dataset by choosing a suitable  $\tau$ .
F.2 Ablation Study - Effectiveness of Reward Scaling
In the experiment, we set  $\gamma$  as 0.99 and  $c_{r}$  will be  $1 - \gamma = 0.01$  accordingly in  $\mathbf{RS}$ . In this ablation experiment, we considered different combinations of min-max normalization/standardization and
Figure 7: Ablation studies on 6 D4RL tasks over 3 seeds with one standard error shaded.
Figure 8: Ablation studies on 6 D4RL tasks over 3 seeds on MICo.
various value of  $c_{r}$  (including 1, 0.1, 0.01, and 0.001). The results in Figure 7 are consistent with our analysis in Section 5.2. The last two show better gains. As  $RS$  has tighter bounds, it excels in most datasets, validating our theory.
F.3 Case Study on MICo
As we illustrate in Section 5.2 and Section 6, Results in Figure 8 show that an unsuitable reward scaling dramatically decreases the performance while applying EBS will increase the performance in many datasets.
G Additional Related Works
Here we present a brief introduction of all the baselines we used in the experiments:
TD3BC [16] add a behavior cloning term to regularize the policy of the TD3 [18] algorithm, achieves a state-of-the-art performance in Offline settings.
$\mathbf{DrQ} + \mathbf{BC}$  [35] combining data augmentation techniques with the TD3+BC method, which applies TD3 in the offline setting with a regularizing behavioral-cloning term to the policy loss. The policy objective is:  $\pi = \operatorname*{argmax}_{\pi} \mathbb{E}_{(s,a) \sim \mathcal{D}} \left[ \lambda Q(s,\pi (s)) - (\pi (s) - a)^2 \right]$
DRIML [38] and HOMER [39] (Time Contrastive methods) learn representations which can discriminate between adjacent observations in a rollout and pairs of random observations.
CURL [31] (Augmentation Contrastive method) learns a representation that is invariant to a class of data augmentations while being different across random example pairs.
Inverse Model [44] (One-Step Inverse Models) predict the action taken conditioned on the previous and resulting observations.
H.1 The severity of the proposed problem
How do bisimulation-based objectives perform in other (online or goal-conditioned) settings? Various methods, such as DBC [54], MICo [6], SimSR [52], and PSE [1], have consistently demonstrated positive results in online settings, regardless of the presence of distractors. This evidence supports the efficacy of bisimulation techniques in online settings. Additionally, GCB [23] excelled in goal-conditioned environments, ExTra [46] showcased the power of bisimulation metric in exploration, and HiP-BMDP [55] successfully incorporated bisimulation into multi-task settings, highlighting its superior performance, all mostly in online settings too, with little work in offline RL. These studies suggest that when tailored to specific environments, bisimulation methods can excel. Despite these works, bisimulation methods have had little success when extended to offline settings, and our motivation is to tackle this problem.
While bisimulation objectives used in the offline setting are directly affected by missing transitions, many other representation objectives may not. When referring to state representation learning, using bisimulation in offline settings presents challenges due to the two issues we outlined: the presence of missing transitions and inappropriate reward scales. Concurrently, there exists other representation objectives, like CURL [31], ATC [49], which focus on pairs of states without the explicit necessity for transition information. As a consequence, they do not explicitly require accounting for missing transitions or reward scaling in their objectives. This absence of direct influence sets them apart from bisimulation-based methods. Yet, we consider that bisimulation-based techniques have a theoretical edge and have proven effective in online settings. Thus, we deem that our work is impactful in that it delivers a proof that bisimulation can be successful offline.
Compounded effect for bisimulation principle in offline settings. In online scenarios, state representations and policies are updated concurrently, while in offline settings, state representation is pre-trained before policy learning, with the two phases completely decoupled. Errors during representation learning in offline settings can have a compounded effect on policy learning, leading to significant issues. This is the reason that missing transitions is particularly harmful to the bisimulation principle in offline settings. Although reward scales affect bisimulation universally, as offline settings require pretraining state embedding, any major discrepancy between this fixed representation and the policy parameter space can further undermine the learning process. Hence, the proposed solutions hold promise for enhancing bisimulation's efficiency in offline settings.
H.2 Suitability of different techniques
EBS We provide EBS as a general method, which is applicable to all bisimulation-based objectives, given that they all adhere to the foundational principle of bisimulation. This principle revolves around the contraction mapping properties similar to the value iteration. Whenever there's an intent to employ bisimulation in offline scenarios, with an aim to reduce the Bellman residual for approximating the fixed point, the outlined challenge emerges. Consequently, EBS holds the potential to enhance any bisimulation-based method, regardless of the distance they use.
RS In essence, the given theoretical analysis is applicable across all bisimulation-based objectives. However, the precise settings for hinge on the foundational distance. For instance, SimSR uses the cosine distance which has definitive bounds. As a result, we need to infer the ideal setting from Equation 10 and Theorem 8. In contrast, the MICo-like distance and DBC employ L-K distance and L1 distance respectively, having bounds ranging from . Consequently, they can adapt to more value settings. We propose our approach as a general method/principle to employ a novel bisimulation metric or distance, especially in the context of offline RL.
Table 3: The exact values of bisimulation error and bisimulation bellman residual
Transition number	100	500	1000	2000
Bisimulation error	0.2792	0.2891	0.2880	0.2915
Bisimulation Bellman residual	0.003	0.0009	0.0032	0.0016
I Empirical estimation of bisimulation error
In this section, we would like to conduct a toy experiment to empirically show that the bisimulation error could possibly be larger than bisimulation bellman residual in fixed/finite datasets.
Data collection To collect the evaluation dataset, we utilize TD3 [18] (a deterministic algorithm) instead of SAC [22] to avoid stochasticity. Firstly, we train a TD3 agent using the rlkit [45] codebase until convergence. Then, we collect 10k transitions and select specific transitions (such as 10, 100, 1000, 5000...) from these 10k transitions with uniform probability to form the evaluation dataset  $\mathcal{D}$ . For determining termination, we follow the settings described in [18] and [22], considering a state terminal only if termination occurs before 1000 timesteps. If termination occurs before 1000 timesteps, we set  $\gamma = 0$ ; otherwise, we set  $\gamma = 0.99$ .
Computation Given an evaluation dataset  $\mathcal{D}$ , the bisimulation Bellman residual  $\epsilon_{\phi}$  is computed by  $\frac{1}{|\mathcal{D}|} \sum_{((s_i, a_i), (s_j, a_j)) \sim \mathcal{D}} (G_{\phi}(s_i, s_j) - (|r(s_i, a_i) - r(s_j, a_j) + \gamma G_{\phi}(s_i, s_j)))^2$ , and the bisimulation error  $\Delta_{\phi}$  is computed by  $\frac{1}{\mathcal{D}} \sum_{(s_i, s_j) \sim \mathcal{D}} (G_{\phi}(s_i, s_j) - G_{\sim}(s_i, s_j))^2$ , where  $G_{\sim}$  denotes the corresponding fixed point measurement. Since directly computing  $G_{\sim}$  is challenging, we compute  $|V^{\pi}(s_i) - V^{\pi}(s_j)|$  instead, as they should be equal when considering the measurement is the fixed point and the transition is deterministic. To compute  $V^{\pi}(s_i)$ , we reset the MuJoCo environment to the specific state  $s_i$  and ran the policy for 1000 timesteps. Since the environment and policy are deterministic, a single trajectory is sufficient to estimate the true value.
The results are presented in Table 3, which indicates that the bisimulation error on finite datasets is indeed larger than the bisimulation bellman residual.