Title: Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bisimulation metrics

Reinforcement Learning

Representation Learning

Offline RL
Abstract: While bisimulation-based approaches hold promise for learning robust state representations for Reinforcement Learning (RL) tasks,  their efficacy in offline RL tasks has not been up to par. In some instances, their performance has even significantly underperformed alternative methods. We aim to understand why bisimulation methods succeed in online settings, but falter in offline tasks. Our analysis reveals that missing transitions in the dataset are particularly harmful to the bisimulation principle, leading to ineffective estimation. We also shed light on the critical role of reward scaling in bounding the scale of bisimulation measurements and of the value error they induce. Based on these findings, we propose to apply the expectile operator for representation learning to our offline RL setting, which helps to prevent overfitting to incomplete data. Meanwhile, by introducing an appropriate reward scaling strategy, we avoid the risk of feature collapse in representation space. We implement these recommendations on two state-of-the-art bisimulation-based algorithms, MICo and SimSR, and demonstrate performance gains on two benchmark suites: D4RL and Visual D4RL. Codes are provided at \url{https://github.com/zanghyu/Offline_Bisimulation}.

Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning

Abstract

While bisimulation-based approaches hold promise for learning robust state representations for Reinforcement Learning (RL) tasks, their efficacy in offline RL tasks has not been up to par. In some instances, their performance has even significantly underperformed alternative methods. We aim to understand why bisimulation methods succeed in online settings, but falter in offline tasks. Our analysis reveals that missing transitions in the dataset are particularly harmful to the bisimulation principle, leading to ineffective estimation. We also shed light on the critical role of reward scaling in bounding the scale of bisimulation measurements and of the value error they induce. Based on these findings, we propose to apply the expectile operator for representation learning to our offline RL setting, which helps to prevent overfitting to incomplete data. Meanwhile, by introducing an appropriate reward scaling strategy, we avoid the risk of feature collapse in representation space. We implement these recommendations on two state-of-the-art bisimulation-based algorithms, MICo and SimSR, and demonstrate performance gains on two benchmark suites: D4RL and Visual D4RL. Codes are provided at https://github.com/zanghyu/Offline_Bisimulation .

1 Introduction

Reinforcement learning (RL) algorithms often require a significant amount of data to achieve optimal performance [40, 48, 22]. In scenarios where collecting data is costly or impractical, Offline RL methods offer an attractive alternative by learning effective policies from previously collected data [29, 43, 32, 35, 16, 24]. However, capturing the complex structure of the environment from limited data remains a challenge for Offline RL [4]. This involves pre-training the state representation on offline data and then learning the policy upon the fixed representations [51, 47, 41, 53]. Though driven by various motivations, previous methods can be mainly categorized into two classes: i) implicitly shaping the agent's representation of the environment via prediction and control of some aspects of the environment through auxiliary tasks , e.g. , maximizing the diversity of visited states [34, 10], exploring attentive contrastive learning on sub-trajectories [51], or capturing temporal information about the environment [47]; ii) utilizing behavioral metrics , such as bisimulation metrics [11, 13, 5], to capture complex structure in the environment by measuring the similarity of behavior on the

representations [52, 7]. The former methods have proven their effectiveness theoretically and empirically in Offline settings [41, 47, 51], while the adaptability of the latter approaches in the context of limited datasets remains unclear. This paper tackles this question.

Bisimulation-based approaches, as their name suggests, utilize the bisimulation metrics update operator to construct an auxiliary loss and learn robust state representations. These representations encapsulate the behavioral similarities between states by considering the difference between their rewards and dynamics. While the learned representations possess several desirable properties, such as smoothness [19], visual invariance [54, 1, 52], and task adaptation [56, 37, 46, 8], bisimulation-based objectives in most approaches are required to be coupled with the policy improvement procedure [54, 6, 52]. In Offline RL, pretraining state representations via bisimulation-based methods is supposed to be cast as a special case of on-policy bisimulation metric learning where the behavior policy is fixed so that good performance should ensue. However, multiple recent studies [51, 21] suggest that bisimulation-based algorithms yield significantly poorer results on Offline tasks compared to a variety of (self-)supervised objectives.

In this work, we highlight problems with using the bisimulation principle as an objective in Offline settings. We aim to provide a theoretical understanding of the performance gap in bisimulation-based approaches between online and offline settings:' why do bisimulation approaches perform well in Online RL tasks but tend to fail in Offline RL ones? ' By establishing a connection between the Bellman and bisimulation operators, we uncover that missing transitions, which often occur in Offline settings, can cause the bisimulation principle to be compromised. This means that the bisimulation estimator can be ineffective in finite datasets. Moreover, we notice that the scale of the reward impacts the upper bounds of both the bisimulation measurement 2 fixed point and the value error. This scaling term, if not properly handled, can potentially lead to representation collapse.

To alleviate the aforementioned issues, we propose to learn state representations based on the expectile operator. With this asymmetric operator predicting expectiles of the representation distribution, we can achieve a balance between the behavior measurement and the greedy assignment of the measurement over the dataset. This results in a form of regularization over the bisimulation measurement, thus preventing overfitting to the incomplete data, and implicitly avoiding out-of-distribution estimation errors. Besides, by considering the specific properties of different bisimulation measurements, we investigate the representation collapse issue for the ones that are instantiated with bounded distances ( e.g. , cosine distance) and propose a way to scale rewards that reduces collapse. We integrate these improvements mainly on two bisimulation-based baselines, MICo [7] and SimSR [52], and show the effectiveness of the proposed modifications.

The primary contributions of this work are as follows:

We investigate the potential harm of directly applying the bisimulation principle in Offline settings, prove that the bisimulation estimator can be ineffective in finite datasets, and emphasize the essential role of reward scaling.

We propose theoretically motivated modifications on two representative bisimulation-based baselines, including an expectile-based operator and a tailored reward scaling strategy. These proposed changes are designed to address the challenges encountered when applying the bisimulation principle in offline settings.

We demonstrate the superior performance our approach yields through an empirical study on two benchmark suites, D4RL [15] and Visual D4RL [35].

2 Related Work

State representation learning in Offline RL Pretraining representations has been recently studied in Offline RL settings, where several studies presented its effectiveness [3, 47, 41, 25]. In this paradigm, we learn state representations on pre-collected datasets before value estimation or policy improvement steps are run. The learned representation can then be used for subsequent policy learning, either online or offline. Some typical auxiliary tasks for pretraining state representations include capturing the dynamical [42] and temporal [47] information of the environment, exploring

attentive contrastive learning on sub-trajectories [51], or improving policy performance by applying data augmentations techniques to the pixel-based inputs [9, 35].

Bisimulation-based methods The pioneer works by [20, 33] aim to overcome the curse of dimensionality by defining equivalence relations between states to reduce system complexity. However, these approaches are impractical as they usually demand an exact match of transition distributions. To address this issue, [12, 14] propose a bisimulation metric to aggregate similar states. This metric quantifies the similarity between two states and serves as a distance measure to allow efficient state aggregation. Unfortunately, it remains computationally expensive as it requires a full enumeration of states. Later, [5] devise an on-policy bisimulation metric for policy evaluation, providing a scalable method for computing state similarity. Building upon this, [54] develop a metric to learn state representations by modeling the latent dynamic transition as Gaussian. [6] further investigate the independent couple sampling strategy to reduce the computational complexity of representation learning, whereas [52] propose to learn state representations built on the cosine distance to alleviate a representation collapse issue. Despite the promising results obtained, one of the major remaining challenges in this paradigm is its dependency on coupling state representation learning with policy training. This is not always suitable for Offline settings, given that obtaining on-policy reward and transition differences is infeasible due to our inability to gather additional agent-environment interactions. To adapt bisimulation-based approaches to Offline settings, one solution is to consider the policy over the dataset as a specific behavior policy, and then apply the bisimulation principle on it to learn state representations in a pretraining stage, thus disentangling policy training from bisimulation-based learning. Notably, although there exist recent studies [51, 42] investigating the potential of bisimulation-based methods to pretrain state representations, it has not yielded satisfactory results yet [51].

3 Preliminaries

3.1 Offline RL

We consider the standard Markov decision process (MDP) framework, in which the environment is given by a tuple M = ( S , A , T, r, γ ) , with state space S , action space A , transition function T that decides the next state s ′ ∼ T ( ·| s, a ) , reward function r ( s, a ) bounded by [ R min , R max ] , and a discount factor γ ∈ [0 , 1) . The agent in state s ∈ S selects an action a ∈ A according to its policy, mapping states to a probability distribution over actions: a ∼ π ( ·| s ) . We make use of the state value function V π ( s ) = E M ,π [ ∑ ∞ t =0 γ t r ( s t , a t ) | s 0 = s ] to describe the long term discounted reward of policy π starting at state s . In the sequel, we use T a s and r a s to denote T ( ·| s, a ) and r ( s, a ) , respectively. In Offline RL, we are given a fixed dataset of environment interactions that include N transition samples, i.e. D = { s i , a i , s ′ i , r i } N i =1 . We assume that the dataset D is composed of trajectories generated i.i.d. under the control of a behavior policy π β , whose state occupancy is denoted by µ β ( s ) .

3.2 Bisimulation-based Update Operator

The concept of bisimulation is used to establish equivalence relations on states. This is done recursively by considering two states as equivalent if they have the same distribution over state transitions and the same immediate reward [30, 20]. Since bisimulation considers worst-case differences between states, it commonly results in 'pessimistic' outcomes. To address this limitation, the π -bisimulation metric was proposed in [5]. This new metric only considers actions induced by a given policy π rather than all actions when measuring the behavior distance between states:

Theorem 1. [5] Let M be the set of all measurements on S . Define F π : M → M by

where s i , s j ∈ S , r π s i = ∑ a ∈A π ( a | s i ) r a s i , T π s i = ∑ a ∈A π ( a | s i ) T a s i , and W ( g ) is the Wasserstein distance with cost function g between distributions. Then F π has a least fixed point g π ∼ , and g π ∼ is a π -bisimulation metric.

Although it is feasible to compute the behavior difference measurement g π ∼ by applying the operator F π iteratively (which is guaranteed to converge to a fixed point since F π is a contraction), this approach comes at a high computational complexity due to the Wasserstein distance on the right-hand

side of the equation. To tackle this issue, MICo [6] proposed using an independent couple sampling strategy instead of optimizing the overall coupling of the distributions T π s i and T π s j , resulting in a novel measurement to evaluate the difference between states. Additionally, SimSR [52] further explored the potentiality of combining the cosine distance with bisimulation-based measurements to learn state representations. Both works can be generalized as:

and F π has a least fixed point G π ∼ 3 . The instantiation of G varies in different approaches [6, 52]. For example, in SimSR [52], the cosine distance is used to instantiate G on the embedding space, and the dynamics difference is computed by the cosine distance between the next-state pair ( s ′ i , s ′ j ) sampled from a transition model of the environment. A more detailed description can be found in Appendix C.

Lemma 2. [6] ( Lifted MDP ) The bisimulation-based update operator F π for M is the Bellman evaluation operator for a specific lifted MDP .

Due to this interpretation of the bisimulation-based update operator as the Bellman evaluation operator in a lifted MDP, we can derive certain conclusions about bisimulation by drawing inspiration from policy evaluation methods. In the next section, we will borrow analytical ideas from [17] to prove that the bisimulation-based objective may be ineffective for finite datasets. We summarize all notations in Appendix A and provide all proofs in Appendix D.

4 Ineffective Bisimulation Estimators in Finite Datasets

The high-level idea of bisimulation-based state representation learning is to learn state embeddings such that when states are projected onto the embedding space, their behavioral similarity is maintained. We denote our parameterized state encoder by ϕ : S → R n and a distance D ( · , · ) in the embedding space R n by G π ϕ ( s i , s j ) . = D ( ϕ ( s i ) , ϕ ( s j )) . For instance, D ( · , · ) may be the Łukaszyk-Karmowski distance [6] or the cosine distance [52]. To avoid unnecessary confusion, we defer implementation details to Section 5.

When considering bisimulation-based state representations, the goal is to acquire stable state representations under policy π via the measurement G π ∼ . The primary focus is usually to minimize a loss over the bisimulation error , denoted by ∆ π ϕ , which measures the distance between the approximation G π ϕ and the fixed point G π ∼ :

However, since the fixed point G π ∼ is unobtainable without full knowledge of the underlying MDP, this approximation error is often unknown. Recall that in Lemma 2, we have shown that we can connect a bisimulation-based update operator to a lifted MDP. Taking inspiration from Bellman evaluation for the value function, we define the bisimulation Bellman residual ϵ π ϕ as:

Then, we can connect the bisimulation Bellman residual with the bisimulation error by the following: Theorem 3. (Bisimulation error upper-bound). Let µ π ( s ) denote the stationary distribution over states, let µ π ( · , · ) denote the joint distribution over synchronized pairs of states ( s i , s j ) sampled independently from µ π ( · ) . For any state pair ( s i , s j ) ∈ S × S , the bisimulation error ∆ π ϕ ( s i , s j ) can be upper-bounded by a sum of expected bisimulation Bellman residuals ϵ π ϕ :

Thereafter, the bisimulation Bellman residual is used as a surrogate objective to approximate the fixed point G π ∼ when learning our state representation. Indeed, the minimization of the bisimulation Bellman residual objective over all pairs ( s ′ i , s ′ j ) ∼ µ π leads to the minimization of the corresponding bisimulation error. This ensures that if the expected on-policy bisimulation Bellman residual ( i.e. ,

E µ π [ ϵ π ϕ ] , and we will use the term 'expected bisimulation residual' in following) minimization objective is zero, then the bisimulation error must be zero for the state pairs under the same policy. However, when the dataset is limited, rather than an infinite transition set covering the whole MDP, minimizing the expected bisimulation residual will no longer be sufficient to guarantee a zero bisimulation error.

Proposition 4. (The expected bisimulation residual is not sufficient over incomplete datasets). If there exists states s ′ i and s ′ j not contained in dataset D , where the occupancy µ π ( s ′ i | s i , a i ) > 0 and µ π ( s ′ j | s j , a j ) > 0 for some ( s i , s j ) ∼ µ π , then there exists a bisimulation measurement G π ϕ and a constant C > 0 such that

For all ( ˆ s i , ˆ s j ) ∈ D , the bisimulation Bellman residual ϵ π ϕ ( ˆ s i , ˆ s j ) = 0 .

There exists ( s i , s j ) ∈ D , such that the bisimulation error ∆ π ϕ ( s i , s j ) = C .

As an example, if we only have ( s i , a i , r, s ′ i ) and ( s j , a j , r, s ′ j ) in a dataset, where both rewards equal to zero for state s i and s j , and if we choose G π ϕ ( s i , s j ) = C , and G π ϕ ( s ′ i , s ′ j ) = 1 γ C , then the bisimulation Bellman residual is ϵ π ϕ ( s i , s j ) = 0 , while the bisimulation error ∆ π ϕ = G π ϕ ( s i , s j ) -0 = C is strictly positive. Note that this failure case does not involve modifying the environment in an extremely adversarial manner, it simply occurs when we are required to estimate the representation of states with subsequent states that are missing from the dataset. Since the distance between the missing states can be arbitrarily large as they are out-of-distribution, directly minimizing the Bellman bisimulation error could achieve the minimal Bellman bisimulation error over the dataset, while not necessarily improving the state representation.

In the context of Offline RL, since the dataset is finite, bisimulation-based representation learning ought to be conceptualized as a pretraining process over the behavior policy π β of the dataset D . However, the failure case above indicates that applying the bisimulation operator F π β and minimizing the associated Bellman bisimulation error does not necessarily ensure the sufficiency of the learned representation for downstream tasks. Ideally, if we had access to the fixed-point measurement G π β ∼ , then we could directly minimize the error between the approximation G and the fixed-point G π β ∼ . However, given the static and incomplete nature of the dataset, acquiring the fixed-point G π β ∼ explicitly is not feasible. From another perspective, the failure stems from out-of-distribution estimation errors. Assuming we could estimate the bisimulation exclusively with in-sample learning , this issue could be intuitively mitigated. As such, we resort to expectile regression as a regularizer, allowing us to circumvent the need for out-of-sample / unseen state pairs.

5 Method

In this section, we describe how we adapt existing bisimulation-based representation approaches to offline RL. We use the expectile-based operator to learn state representations that optimize the behavior measurement over the dataset, while avoiding overfitting to the incomplete data. In addition, we analyze the impact of reward scaling and propose as a consequence to normalize the reward difference in the bisimulation Bellman residual in order to satisfy the specific nature of different instantiations of the bisimulation measurement while keeping a lower value error. The pseudo-code of our method is shown in Algorithms in Appendix B.

5.1 Expectile-based Bisimulation Operator

The efficacy of expectile regression in achieving in-sample learning has already been demonstrated in previous research [28, 36]. Consequently, we will first describe our proposed expectile -based operator, and subsequently show how expectile regression can effectively address the aforementioned challenge. Specifically, we consider the update operator as follows:

where ˆ ϵ is the estimated one-step bisimulation Bellman residual, π β is the behavior policy, G ¯ ϕ is the target encoder, updated using an exponential moving average, and [ · ] + = max( · , 0) . Since the expectile operator in Equation 6 does not have a closed-form solution, in practice, we minimize it through gradient descent steps:

where α is the step size. The fixed-point of the measurement obtained using this expectile-based operator is denoted as G τ . Although the utilization of the expectile statistics is well established, its application for estimating bisimulation measurement is not particularly intuitive. In the following, we will show how expectile-based operator can be helpful in addressing the aforementioned issue. First, it is worth noting that when τ = 1 / 2 , this operator becomes the bisimulation expectation of the behavior policy, i.e. , E µ π β [ˆ ϵ ] . Next, we shall consider how this operator performs when τ → 1 . We show that under certain assumptions, our method indeed approximates an 'optimal' measurement in terms of the given dataset. We first prove a technical lemma stating that the update operator is still a contraction, and then prove a lemma relating different expectiles, finally we derive our main result regarding the 'optimality' of our method.

Lemma5. For any τ ∈ [0, 1), F π is a γ -contraction, where γ = 1 -2 α (1 -γ ) min { τ, 1 -τ } < 1

Lemma 6. For any τ, τ ∈ [0 , 1) with τ ≥ τ , and for all s , s ∈ S and any α , we have G ′ ≥ G

τ τ τ . ′ ′ i j τ τ .

Theorem 7. In deterministic MDP and fixed finite dataset, we have:

where G ∗ ∼ (( s i , a i ) , ( s j , a j )) is a fixed-point measurement constrained to the dataset and defined on the state-action space S × A as

Intuitively, G ∼ (( s i , a i ) , ( s j , a j )) can be interpreted as a state-action value function Q (˜ s, ˜ a ) in a lifted MDP ˜ M , and G ∼ ( s i , s j ) as a state value function V (˜ s ) . We defer the detailed explanation to Appendix E.

Theorem 7 illustrates that, as τ → 1 , we are effectively approximating the maximum G ∼ (( s i , a i ) , ( s j , a j )) over actions a ′ i , a ′ j from the dataset. When we set τ = 1 , the expectilebased bisimulation operator achieves fully in-sample learning: we only consider state pairs that have corresponding actions in the dataset. For instance, only when we have ( s ′ i , a ′ i ) ∈ D and ( s ′ j , a ′ j ) ∈ D , can we apply the measurement of G ∗ ∼ . As such, by manipulating τ , we balance a trade-off between minimizing the expected bisimulation residual (for τ = 0 . 5 ) and evaluating G ∗ ∼ (( s i , a i ) , ( s j , a j )) solely on the dataset (for τ = 1 ), thereby sidestepping the failure case outlined in Proposition 4 in an implicit manner.

5.2 Reward Scaling

Most previous works [5, 54, 6, 52] have overlooked the impact of reward scaling in the bisimulation operator. To demonstrate its importance, we investigate a more general form of the bisimulation operator in Equation 2, given as:

We then can derive the following:

0

250

500

750

1000

1250

1500

1750

2000

Timesteps

100

50

0

50

100

150

200

250

SimSR w/ RS

SimSR w/o RS

0

250

500

750

1000

1250

1500

1750

2000

Timesteps

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Numerical Value

w/ RS

w/o RS

Accordingly, we have G π ∼ ( s i , s j ) ≤ c r · ( R max -R min ) 1 -c k . Adopting the conventional settings of c r = 1 and c k = γ as suggested in [6, 52], could possibly result in a relatively large upper bound of G π ∼ between states. This is due to the common practice of setting γ at 0 . 99 . However, when bisimulation operators are instantiated with bounded distances, e.g., cosine distance, such a setting may be unsuitable. Therefore, it becomes important to tighten the upper bound.

Besides, we can also derive the value bound between the ground truth value function and the approximated value function:

Theorem 8. (Value bound based on on-policy bisimulation measurements in terms of approximation error). Given an MDP ˜ M constructed by aggregating states in an ω -neighborhood, and an encoder ϕ that maps from states in the original MDP M to these clusters, the value functions for the two MDPs are bounded as

where ˆ ∆ := ∥ ˆ G π ∼ -ˆ G π ϕ ∥ ∞ is the approximation error.

In essence, Equation 10 and Theorem 8 reveal that: (i) there is a positive correlation between the reward scale c r and the upper bound of the fixed-point G π ∼ , and (ii) a larger reward scale c r facilitates a more accurate approximation of the value function ˜ V π ( ϕ ( s )) to its ground-truth value V π ( s ) . It is important to note that c r also impacts the value of ˆ ∆ , as depicted in Figure 7(Right) 4 . Therefore, it is crucial to first ensure the alignment with the instantiation of the bisimulation measurement, and then choose the largest possible c r to minimize the value error. For instance, as the SimSR operator [52] uses the cosine distance, c k = γ is predetermined. We should thus set c r ∈ [0 , 1 -γ ] , and apply min-max normalization to the reward function. This can make G π ∼ ≤ 1 and therefore be consistent with the maximum value of 1 of the cosine distance. To achieve a tighter bound in Equation11, we should then maximize the reward scale, setting c r to 1 -γ . Figure 7 illustrates the effectiveness of this reward scaling.

6 Experiments

6.1 Performance Comparison in D4RL Benchmark

Implementation Details Weanalyze our proposed method on the D4RL benchmark [15] of OpenAI gym MuJoCo tasks [50] which includes a variety of datasets that have been commonly used in the

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

100

walker2d-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

100

hopper-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

100

halfcheetah-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

100

walker2d-medium-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

100

hopper-medium-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

20

40

60

80

walker2d-medium-replay-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

20

40

60

hopper-medium-replay-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

10

20

30

40

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

20

40

60

80

walker2d-medium-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

20

40

60

hopper-medium-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

20

40

Timesteps

SimSR

MICo

SimSR+RS

MICo+EBS

SimSR+RS+EBS

td3bc(baseline)

0

20

40

60

80

Normalized scores

0.0

0.1

0.2

0.3

0.4

0.5

Density

SimSR

SimSR+RS

SimSR+RS+EBS

MICo

MICo+EBS

Offline RL community. To illustrate the effectiveness of our method, we implement it on top of two bisimulation-based approaches, MICo [6] and SimSR [52]. It is worth noting that there are two versions of SimSR depending on its use of a latent dynamics model: SimSR_basic follows the dynamics that the environment provides, and SimSR_full constructs latent dynamics for sampling successive latent states. We opt for SimSR_basic as our backbone, as it exhibits superior and more stable performance in the D4RL benchmark tasks compared to SimSR_full. Additionally, to explore the impact of bisimulation-based representation learning on the downstream performance of policy learning, we build these approaches on top of the Offline RL method TD3BC [16]. We examine three environments: halfcheetah, hopper, and walker2d, with four datasets per task: expert, medium-expert, medium-replay, and medium. We first pretrain the encoder during 100 k timesteps, then freeze it, pass the raw state through the frozen encoder to obtain the representations that serve as input for the Offline RL algorithm. Further details on the experiment setup are included in Appendix F.

Analysis Figure 2 illustrates the performance of two approaches and their variants in the D4RL tasks. We use EBS to represent the scheme of employing the expectile-based operator, while RS

denotes the reward scaling scheme. The latter includes both min-max reward normalization and penalization coefficient with (1 -γ ) in the bisimulation operator. As discussed in Section 5.2, the role of reward scaling varies depending on the specific instantiation of G 5 . We observe that without RS , SimSR almost fails in every dataset, which aligns with our understanding of the critical role reward scaling plays. The results also illustrate that EBS effectively enhances the downstream performance of the policy for both SimSR and MICo. It is noteworthy that in this experiment, we set τ = 0 . 6 for the expectile in SimSR and τ = 0 . 7 in MICo across all datasets, demonstrating the robustness of this hyperparameter. Regarding SimSR, when RS is applied ( SimSR+RS ), the performance is comparable to the TD3BC baseline, while the incorporation of the expectile-based operator ( SimSR+RS+EBS ) further enhances final performance and sample efficiency. Besides, we additionally present the IQM normalized return of all variants in Figure 3, illustrating our performance gains over the backbones. Further, we have also constructed an ablation study to investigate the impact of different settings of τ , the results show that a suitable expectile τ is crucial for control tasks. We present the corresponding results in Appendix E.

6.2 Performance Comparison in V-D4RL Benchmark

Implementation details We also evaluate our method on a visual observation setting of DMControl suite (DMC) tasks, V-D4RL benchmark [35]. Similar to the previous experiment, we add the proposed schemes on top of MICo and SimSR. In the experiments, we notice that the latent dynamics modeling can help to boost performance for the visual setting, hence we use SimSR_full as the backbone. Additionally, we also notice that MICo often gives really poor performance in the V-D4RL benchmark, while adding latent dynamics alleviates the issue. Therefore, we boost MICo with explicit dynamics modeling for a fair comparison. To compare the performance with the other representation approaches, we include 4 competitive representation learning approaches for Offline RL, including DRIML [38], HOMER [39],

0.0

0.2

0.4

0.6

0.8

1.0

1e5

0

100

200

300

400

500

600

cheetah_run-expert

0.0

0.2

0.4

0.6

0.8

1.0

1e5

0

200

400

600

walker_walk-expert

0.0

0.2

0.4

0.6

0.8

1.0

1e5

0

100

200

300

400

500

600

cheetah_run-medium-expert

0.0

0.2

0.4

0.6

0.8

1.0

1e5

0

200

400

600

800

walker_walk-medium-expert

SimSR+RS+EBS

SimSR

MICo+RS

MICo

Timesteps

CURL [31], and Inverse model [44]. Detailed descriptions of these approaches can be found in Appendix G.

Analysis We evaluate all aforementioned approaches by integrating the pre-trained encoder from each into an Offline RL method DrQ+BC [35], which combines data augmentation techniques with TD3BC. The results in Table 1 and Figure 4 illustrate the effectiveness of our proposed method, the numerical improvements are underlined with red upward arrows. Compared to the other baselines, while SimSR+RS+EBS does not achieve the highest score in all datasets, it achieves the best overall performance. Besides, our modifications on MICo and SimSR consistently show significant improvements. This indicates that our proposed method is not only applicable to raw-state inputs but also compatible with pixel-based observations.

7 Discussion

Limitations and Future Work While τ remains constant in our D4RL experiments, optimal performance may arise under different τ settings, contingent on the specific attributes of the dataset. Therefore, to yield the best outcomes, one might need to set various τ to identify the most suitable value. However, this process could consume substantial computational resources. Another area of potential study involves evaluating the effectiveness of our approach in off-policy settings, given that off-policy settings may also lead to similar failure cases.

Conclusion In this work, we highlight the effectiveness of the bisimulation operator over incomplete datasets and emphasize the crucial role of reward scaling in Offline settings. By employing the expectile operator in bisimulation, we manage to strike a balance between behavior measurement and greedy assignment of the measurement over datasets. We also propose a reward scaling strategy to reduce the risk of representation collapse in specific bisimulation-based measurements. Empirical studies show the effectiveness of our proposed modifications.

Acknowledgments

This work was partially supported by the NSFC under Grants 92270125 and 62276024, as well as the National Key R&D Program of China under Grant No.2022YFC3302101.

References

Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G. Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 29304-29320, 2021.

Sanjeev Arora, Simon S. Du, Sham M. Kakade, Yuping Luo, and Nikunj Saunshi. Provable representation learning for imitation learning via bi-level optimization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 367-376. PMLR, 2020.

David Brandfonbrener, Rémi Tachet des Combes, and Romain Laroche . Incorporating explicit uncertainty estimates into deep offline reinforcement learning. In Proceedings of the 5th Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM) , 2022.

Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 10069-10076. AAAI Press, 2020.

Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. Mico: Improved representations via sampling-based state similarity for markov decision processes. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 30113-30126, 2021.

Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. Mico: Learning improved representations via sampling-based state similarity for markov decision processes. CoRR , abs/2106.08229, 2021.

Pablo Samuel Castro and Doina Precup. Using bisimulation for policy transfer in mdps. In Maria Fox and David Poole, editors, Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010 . AAAI Press, 2010.

Xin Chen, Sam Toyer, Cody Wild, Scott Emmons, Ian Fischer, Kuang-Huei Lee, Neel Alex, Steven H Wang, Ping Luo, Stuart Russell, Pieter Abbeel, and Rohin Shah. An empirical investigation of representation learning for imitation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In Deborah L. McGuinness and George Ferguson, editors, Proceedings of the Nineteenth National Conference on Artificial Intelligence, Sixteenth Conference on Innovative Applications of Artificial Intelligence, July 25-29, 2004, San Jose, California, USA , pages 950-951. AAAI Press / The MIT Press, 2004.

Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In David Maxwell Chickering and Joseph Y. Halpern, editors, UAI '04, Proceedings of the 20th Conference in Uncertainty in Artificial Intelligence, Banff, Canada, July 7-11, 2004 , pages 162-169. AUAI Press, 2004.

Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM J. Comput. , 40(6):1662-1714, 2011.

Norman Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In Nevin L. Zhang and Jin Tian, editors, Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI 2014, Quebec City, Quebec, Canada, July 23-27, 2014 , pages 210-219. AUAI Press, 2014.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR , abs/2004.07219, 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 20132-20145, 2021.

Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. Why should I trust you, bellman? the bellman error is a poor replacement for value error. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 6918-6943. PMLR, 2022.

Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1582-1591. PMLR, 2018.

Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 2170-2179. PMLR, 2019.

Robert Givan, Thomas L. Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artif. Intell. , 147(1-2):163-223, 2003.

Pengjie Gu, Mengchen Zhao, Chen Chen, Dong Li, Jianye Hao, and Bo An. Learning pseudometric-based action representations for offline reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 7902-7918. PMLR, 2022.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1856-1865. PMLR, 2018.

Philippe Hansen-Estruch, Amy Zhang, Ashvin Nair, Patrick Yin, and Sergey Levine. Bisimulation makes analogies in goal-conditioned reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 8407-8426. PMLR, 2022.

Zhang-Wei Hong, Rémi Tachet des Combes, Pulkit Agrawal, and Romain Laroche . Harnessing mixed offline reinforcement learning datasets via trajectory weighting. In Proceedings of the 11th International Conference on Learning Representations (ICLR) , 2023.

Riashat Islam, Manan Tomar, Alex Lamb, Yonathan Efroni, Hongyu Zang, Aniket Didolkar, Dipendra Misra, Xin Li, Harm van Seijen, Remi Tachet des Combes, and John Langford. Agent-controller representations: Principled offline rl with rich exogenous information, 2022.

Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Claude Sammut and Achim G. Hoffmann, editors, Machine Learning, Proceedings of the Nineteenth International Conference (ICML 2002), University of New South Wales, Sydney, Australia, July 8-12, 2002 , pages 267-274. Morgan Kaufmann, 2002.

Mete Kemertas and Tristan Aumentado-Armstrong. Towards robust bisimulation metric learning. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 4764-4777, 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Marco A. Wiering and Martijn van Otterlo, editors, Reinforcement Learning , volume 12 of Adaptation, Learning, and Optimization , pages 45-73. Springer, 2012.

Kim Guldstrand Larsen and Arne Skou. Bisimulation through probabilistic testing. In Conference Record of the Sixteenth Annual ACM Symposium on Principles of Programming Languages, Austin, Texas, USA, January 11-13, 1989 , pages 344-352. ACM Press, 1989.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised representations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 5639-5650. PMLR, 2020.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR , abs/2005.01643, 2020.

Lihong Li, Thomas J. Walsh, and Michael L. Littman. Towards a unified theory of state abstraction for mdps. In International Symposium on Artificial Intelligence and Mathematics, ISAIM 2006, Fort Lauderdale, Florida, USA, January 4-6, 2006 , 2006.

Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 18459-18473, 2021.

Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. CoRR , abs/2206.04779, 2022.

Xiaoteng Ma, Yiqin Yang, Hao Hu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, Bin Liang, and Qihan Liu. Offline reinforcement learning with value-based episodic memory. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Bogdan Mazoure, Ahmed M. Ahmed, R. Devon Hjelm, Andrey Kolobov, and Patrick MacAlpine. Cross-trajectory representation learning for zero-shot generalization in RL. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Bogdan Mazoure, Remi Tachet des Combes, Thang Doan, Philip Bachman, and R. Devon Hjelm. Deep reinforcement and infomax learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 6961-6971. PMLR, 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat. , 518(7540):529-533, 2015.

Ofir Nachum and Mengjiao Yang. Provable representation learning for imitation with contrastive fourier features. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 30100-30112, 2021.

Ofir Nachum and Mengjiao Yang. Provable representation learning for imitation with contrastive fourier features. CoRR , abs/2105.12272, 2021.

Kimia Nadjahi*, Romain Laroche *, and Rémi Tachet des Combes. Safe policy improvement with soft baseline bootstrapping. In Proceedings of the 17th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD) , 2019.

Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 2778-2787. PMLR, 2017.

Vitchyr Pong, Ashvin Nair, Murtaza Dalal, and Steven Lin. Rlkit, 2020.

Anirban Santara, Rishabh Madan, Pabitra Mitra, and Balaraman Ravindran. Extra: Transferguided exploration. In Amal El Fallah Seghrouchni, Gita Sukthankar, Bo An, and Neil YorkeSmith, editors, Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS '20, Auckland, New Zealand, May 9-13, 2020 , pages 1987-1989. International Foundation for Autonomous Agents and Multiagent Systems, 2020.

Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data-efficient reinforcement learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 12686-12699, 2021.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nat. , 550(7676):354-359, 2017.

Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 9870-9879. PMLR, 2021.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012 , pages 5026-5033. IEEE, 2012.

Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 11784-11794. PMLR, 2021.

Hongyu Zang, Xin Li, and Mingzhong Wang. Simsr: Simple distance-based state representations for deep reinforcement learning. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022 , pages 8997-9005. AAAI Press, 2022.

Hongyu Zang, Xin Li, Jie Yu, Chen Liu, Riashat Islam, Remi Tachet des Combes, and Romain Laroche. Behavior prior representation learning for offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Multi-task reinforcement learning as a hidden-parameter block MDP. CoRR , abs/2007.07206, 2020.

Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Learning robust state abstractions for hidden-parameter block mdps. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Contents

Appendix

A Notation

Table 2 summarizes our notation.

16: end for

B Algorithm

We provide the algorithm in Algorithm 1, and a pytorch-like implementation build on top of SimSR in Algorithm 2.

Algorithm 1 Proposed Implementation

1: Stage 1 Preprocessing:

3: Stage 2 Pretraining the encoder:

2: Min-Max reward normalization: ¯ r = r -r min r max -r min

4: Initialize encoder parameter ϕ , expectile τ , learning rate α , discount factor γ .

5: for each gradient step do

6: Apply reward scaling when computing ˆ ϵ :

7: Update encoder ϕ :

8: end for

9: Stage 3 Training value function and policy network:

10: Initialize value function parameter ψ , policy network parameter θ , learning rate λ V and λ π .

11: for each gradient step do

12: Sample tuple ( s, a, s ′ , ¯ r ) from dataset D

13: Encode the states to representation space: z = ϕ ( s ) , z ′ = ϕ ( s ′ )

14: Update value function with ( z, a, z ′ , ¯ r ) :

15: Update policy network with ( z, a, z ′ , ¯ r ) :

Algorithm 2 SimSR+RS+EBS Pseudocode, PyTorch-like

C Technical backgrounds

C.1 Bisimulation metric

Bisimulation measures equivalence relations on MDPs with a recursive form: two states are deemed equivalent if they share the equivalent distributions over the next equivalent states and they have the same immediate reward [30, 20]. However, since bisimulation considers equivalence for all actions, including bad ones, it commonly results in 'pessimistic' outcomes. Instead, [5] developed π -bisimulation which removes the requirement of considering each action and only needs to consider the actions induced by a policy π .

Definition 9. [5] Given an MDP M , an equivalence relation E π ⊆ S × S is a π -bisimulation relation if whenever ( s , u ) ∈ E π the following properties hold:

where S E π is the state space S partitioned into equivalence classes defined by E π . Two states s, u ∈ S are π -bisimilar if there exists a π -bisimulation relation E π such that ( s, u ) ∈ E π .

However, π -bisimulation is still too stringent to be applied at scale as π -bisimulation relation emphasizes the equivalence is a binary property: either two states are equivalent or not, thus becoming too sensitive to perturbations in the numerical values of the model parameters. The problem becomes even more prominent when deep frameworks are applied.

Thereafter, they proposed a π -bisimulation metric to leverage the absolute value between the immediate rewards w.r.t. two states and the 1 -Wasserstein distance ( W 1 ) between the transition distributions conditioned on the two states and the policy π to formulate such measurement:

Theorem 10. Define F π : M → M by F π ( d )( s, u ) = | R π s -R π u | + γ W 1 ( d )( T π s , T π u ) , then F π has a least fixed point d π ∼ , and d π ∼ is a π -bisimulation metric.

Although the Wasserstein distance is a powerful metric to calculate the distance between two probability distributions, it requires to enumerate all states which is impossible in RL tasks of continuous state space. Various extensions have been proposed [54, 6, 52] to reduce the computational complexity. DBC [54] extend bisimulation metrics to learn state representation, via minimizing the ℓ 1 -norm distance of representations and the bisimulation metrics, meanwhile modeling the latent dynamics as Gaussian and utilizing W 2 distance to compute it, which can be formulated as a closedform result. However, DBC has several issues like loss function mismatch and specific requirements for Gaussian modeling, which limits its application and performance.

C.2 MICo distance

MICo distance [6], tackles the above issue by restricting the coupling class to the independent coupling to avoid intractable Wasserstein distance computation. The MICo operator and its associated theoretical guarantee are given as:

Theorem 11. [6] Given a policy π , MICo distance F π is defined as:

has a fixed point U π .

By considering the Wasserstein distance in the definition of bisimulation metrics can be upperbounded by taking a restricted class of couplings of the transition distributions, MICo restricts the coupling class precisely to the singleton containing the independent coupling, utilizing the Independent Couple sampling strategy to bypass the computation of the Wasserstein distance. However, MICo distance U requires to be a Łukaszyk-Karmowski metric, which does not satisfy the identity of indiscernibles. As a result, the approximated distance on the learned embedding space based on the MICo distance, which involves a Łukaszyk-Karmowski metric to measure the distance between dynamics, may suffer from the violation issue of the identity of indiscernibles.

C.3 SimSR operator

To avoid the potential representation collapse, SimSR [52] develop a more concise update operator to learn state representation more effectively. Coupling with cosine distance, SimSR defines its operator as:

Theorem 12. [52] Given a policy π , Simple State Representation (SimSR) is updated as:

has the same fixed point as MICo.

Further, considering the latent dynamics can be beneficial to representation learning, they additionally develop a form of operator including dynamics modeling:

Theorem 13. [52] Given a policy π , and a latent dynamics model ˆ T , SimSR is updated as

If latent dynamics are specified, F π has a fixed point.

When considering MICo distance and the basic version of SimSR, we can notice that they have a similar recursive iteration formulation. And therefore both works can be generalized under:

while the instantiation of G varies in these two approaches.

C.4 Lifted MDP

The connection between bisimulation-based operators and lifted MDP can be referred to [6]. We provide the corresponding Lemma here for reference.

Lemma 2. ( Lifted MDP ) The bisimulation-based update operator F π for M , is the Bellman evaluation operator for a specific lifted MDP .

Proof. Given the MDP specified by the tuple ( S , A , T, R ) , we construct a lifted MDP ( ˜ S , ˜ A , ˜ T, ˜ R ) , by taking the state space to be ˜ S = S 2 , the action space to be ˜ A = A 2 , the transition dynamics to be given by ˜ T ˜ a ˜ s (˜ s ′ ) = ˜ T ( a i ,a j ) ( s i ,s j ) (( s ′ i , s ′ j )) = T a i s i ( s ′ i ) T a j s j ( s ′ j ) for all ( s i , s j ) , ( s ′ i , s ′ j ) ∈ S 2 , a i , a j ∈ A , and the action-independent rewards to be ˜ R ˜ s = ˜ R ( s i ,s j ) = | r π s i -r π s j | for all s i , s j ∈ S . The Bellman evaluation operator ˜ F ˜ π for this lifted MDP at discount rate γ under the policy ˜ π (˜ a | ˜ s ) = ˜ π ( a i , a j | s i , s j ) = π ( a i | s i ) π ( a j | s j ) is given by (for all G π ∈ R S×S and ( s i , s j ) ∈ S × S ):

C.5 Expectile Regression

Expectile regression, a method in statistics, is an extension of quantile regression that provides a more detailed analysis of a distribution's tail. This technique aims to estimate the expectiles of a conditional distribution, which are like percentiles but with respect to the mean, not the median. In essence, expectile regression can help capture the structure of data variability and analyze extreme observations in a more precise manner than quantile regression. The τ ∈ (0 , 1) expectile of some random variable X is defined as a solution to the asymmetric least squares problem:

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

u

0.0

0.2

0.4

0.6

0.8

Loss

Expectile loss for different

=0.1

=0.4

=0.5

=0.6

=0.9

where L τ 2 ( u ) = | τ -✶ ( u < 0) | u 2 . That is, for τ > 0 . 5 , this asymmetric loss function downweights the contributions of x values smaller than m τ while giving more weights to larger values. Figure 5 shows the illustration of this asymmetric loss. More detailed descriptions can be found in [28, 36].

D Proof

D.1 Connection between bisimulation error and bisimulation Bellman residual

In this section, we will revise some definitions a bit for obtaining the equivalence between bisimulation error and bisimulation Bellman residual. We first define bisimulation error ∆ π ϕ that measure the distance of the approximation G π ϕ to the fixed point G π ∼ as:

And define bisimulation Bellman residual ϵ π ϕ as:

Notably, this is slightly different from the notation in Section 4 given the fact that we do not apply absolute value here. Then, we can have the following theorems.

Theorem 14. (The bisimulation Bellman residual can be defined as a function of the bisimulation error)

Proof. This follows directly from the bisimulation update operator:

Theorem 15. (The bisimulation error can be defined as a function of the bisimulation Bellman residual). For any state pair ( s i , s j ) ∈ S × S , the approximation error ∆ π ϕ ( s i , s j ) can be defined as a function of the Bellman bisimulation error ϵ ϕ

Proof. Our proof follows similar steps to the proof of Lemma 6.1 in [26] and Theorem 1 in [17]. First by definition:

Then we can decompose the error:

By considering the operator G as the Bellman evaluation operator for the lifted MDP (See Section C.4), we can rewrite the formula as:

Then we can treat ∆ ˜ π ϕ (˜ x ) as a value function and ϵ ˜ π ϕ (˜ x ) as reward, we can see that:

Then we can obtain

D.2 Thoerem 3

Theorem 3. (Bisimulation error upper-bound). Let µ π ( s ) denote the stationary distribution over states, let µ π ( · , · ) denote the joint distribution over synchronized pairs of states ( s i , s j ) sampled independently from µ π ( · ) . For any state pair ( s i , s j ) ∈ S × S , the bisimulation error ∆ π ϕ ( s i , s j ) can be upper-bounded by a sum of expected bisimulation Bellman residuals ϵ π ϕ :

Proof. We start from Equation 25 in Section D.1.

Then when we define bisimulation error ∆ π ϕ ( s i , s j ) := | ∆ π ϕ ( s i , s j ) | and bisimulation Bellman residual ϵ π ϕ ( s ′ i , s ′ j ) := | ϵ π ϕ ( s ′ i , s ′ j ) | , we have

D.3 Proposition 4

Proposition 4. (The expected bisimulation residual is not sufficient over incomplete datasets). If there exists states s ′ i and s ′ j not contained in dataset D , where the occupancy µ π ( s ′ i | s i , a i ) > 0 and µ π ( s ′ j | s j , a j ) > 0 for some s i ∈ D , s j ∈ D , then there exists a bisimulation measurement and C > 0 such that

For all ( ˆ s i , ˆ s j ) ∈ D , the bisimulation Bellman residual ϵ π ϕ ( ˆ s i , ˆ s j ) = 0 .

There exists ( s i , s j ) ∈ D , such that the bisimulation error ∆ π ϕ ( s i , s j ) = C .

Proof. This is a direct consequence of Theorem 15. Let D ′ contain the set of state pairs ( s ′ i , s ′ j ) not contained in the dataset D , where the next-state pair occupancy µ π ( s ′ i , s ′ j | s i , a i , s j , a j ) > 0 . Let D unique be the set of unique state pairs in D . It follows that

Recall that ϵ π ϕ ( s i , s j ) = ∆ π ϕ ( s i , s j ) -γ E s ′ i ∼ T π s i s ′ j ∼ T π s j [∆ π ϕ ( s ′ i , s ′ j )] , and there exists at least one

G ( s i , s j ) , such that ( s i , s j ) ∈ D ′ . Since the sets D and D ′ are distinct, it follows that there exists a measurement G such that ϵ π ϕ ( s i , s j ) = 0 for all ( s i , s j ) ∈ D , but 1 1 -γ ∑ ( s ′ i ,s ′ j ) ∼D ′ µ π ( s ′ i , s ′ j | s i , a i , s j , a j ) ϵ π ϕ ( s ′ i , s ′ j ) = C .

D.4 Lemma 5

Lemma 5. For any τ ∈ [0, 1), F π τ is a γ τ -contraction, where γ τ = 1 -2 α (1 -γ ) min { τ, 1 -τ } .

Proof. Note that F π 1 / 2 is the standard bisimulation operator for π , of which the fixed point is G π ∼ . To keep the notation succinct, we will replace G π with G . For any G 1 , G 2 ,

̸

When τ = 1 2 , we introduce two more operators to simplify the analysis:

Now we show that both operators meet the Banach-fixed point theorem (e.g. ∥F π + G 1 -F π + G 2 ∥ ∞ ≤ ∥ G 1 -G 2 ∥ ∞ ). For any G 1 , G 2 :

The relationship between G 1 +[ δ i ] + and G 2 +[ δ j ] + exists in four cases:

δ i ≥ 0 , δ j ≥ 0 , then

δ i < 0 , δ j < 0 , then

δ i ≥ 0 , δ j < 0 , then

where the inequality comes from G 2 ( s i , s j ) > (1 -γ ) | r π s i -r π s j | + γG 2 ( s ′ i , s ′ j ) .

δ i < 0 , δ j ≥ 0 , then

where the inequality comes from G 2 ( s i , s j ) ≤ (1 -γ ) | r π s i -r π s j | + γG 2 ( s ′ i , s ′ j ) .

As a result, we have ( F π + G 1 )( s i , s j ) -( F π + G 2 )( s i , s j ) ≤ ∥ G 1 -G 2 ∥ ∞ . Combine F π + and F π -, we can rewrite F π τ as:

What's more

1

When τ < 1 , for any G and G :

D.5 Lemma 6

Lemma 6. For any τ, τ ′ ∈ [0 , 1) with τ ′ ≥ τ , and for all s i , s j ∈ S and any α , we have G τ ′ ≥ G τ .

Proof. We denote G τ ′ is the fixed point of applying the operator F π τ ′ , and G τ is the fixed point of applying the operator F τ . Based on Equation 6, we have:

D.6 Theorem 7

Theorem 7. In deterministic MDP and fixed finite dataset, we have:

where G ∗ ∼ (( s i , a i ) , ( s j , a j )) is a fixed-point measurement constrained to the dataset and defined on state-action space S × A as

Proof. First, we can easily proof that G ∗ ∼ ( s i , a i , s j , a j ) is a fixed point. Define the corresponding operator of G ∗ ∼ is F ∗ , we can know that F ∗ is a contraction. Then, we have

Corollary 16. For any τ , s i , s j ∈ S we have

Proof. The proof follows from the observation that convex combination is smaller than maximum.

Besides, we also have

Lemma 17. Let X be a real-valued random variable with a bounded support and supremum of the support is x ∗ . Then,

Proof. Same as the Lemma 1 in [28]. One can show that expectiles of a random variable have the same supremum x ∗ . Moreover, for all τ 1 and τ 2 such that τ 1 < τ 2 , we get m τ 1 ≤ m τ 2 . Therefore, the limit follows from the properties of bounded monotonically non-decreasing functions.

Combining Corollary 16 and Lemma 17, we can obtain the above.

D.7 Theorem 8

Theorem 8. (Value bound based on on-policy bisimulation measurements in terms of encoder error). Given an MDP ˜ M constructed by aggregating states in an ω -neighborhood, and an encoder ϕ that maps from states in the original MDP M to these clusters, the value functions for the two MDPs are bounded as

where ˆ ∆ := ∥ ˆ G π ∼ -ˆ G π ϕ ∥ ∞ is the approximation error.

Proof. Let the reward function be bounded as R ∈ [0 , 1] , ϕ : S → ˜ S , and ϕ ( s i ) = ϕ ( s j ) ⇒ ˆ G π ϕ ( s i , s j ) = | ϕ ( s i ) -ϕ ( s j ) | ≤ 2 ω , we can conduct an aggregat MDP ˜ M = ( ˜ S , A , ˜ T, ˜ R ) . Let ξ be

a measure on S . Following Lemma 8 in [27], we have that:

Thus, taking the supremum on the LHS, we have:

Therefore,

E Understanding of Theorem 7

Theorem 7. In deterministic MDP and fixed finite dataset, we have:

where G ∗ ∼ (( s i , a i ) , ( s j , a j )) is a fixed-point measurement constrained to the dataset and defined on the state-action space S × A as

Given the MDP specified by the tuple ( S , A , T, R ) , we construct a lifted MDP ( ˜ S , ˜ A , ˜ T, ˜ R ) , by taking the state space to be ˜ S = S 2 , the action space to be ˜ A = A 2 , the transition dynamics to be given by ˜ T ˜ a ˜ s (˜ s ′ ) = ˜ T ( a i ,a j ) ( s i ,s j ) (( s ′ i , s ′ j )) = T a i s i ( s ′ i ) T a j s j ( s ′ j ) for all ( s i , s j ) , ( s ′ i , s ′ j ) ∈ S 2 , a i , a j ∈ A , and the action-independent rewards to be ˜ R ˜ s = ˜ R ( s i ,s j ) = | r π s i -r π s j | for all s i , s j ∈ S . The Bellman evaluation operator ˜ F ˜ π for this lifted MDP at discount rate γ under the policy ˜ π (˜ a | ˜ s ) = ˜ π ( a i , a j | s i , s j ) = π ( a i | s i ) π ( a j | s j ) is given by (for all G π ∈ R S×S and ( s i , s j ) ∈ S × S ):

Though similar, Equation 55 has more constraints as it requires the possibility of π β ( a ′ i | s ′ i ) and π β ( a ′ j | s ′ j ) are larger than zero in the dataset. As such, we may also change the Equation 56 to:

This is, indeed, equivalent to the in-sample -style Q function in [28]. Intuitively, G ∗ ∼ (( s i , a i ) , ( s j , a j )) can be interpreted as the optimal state-action value function Q ∗ (˜ s, ˜ a ) in a lifted MDP ˜ M . Then G π ∼ (( s i , a i ) , ( s j , a j )) is the state-action value function Q π (˜ s, ˜ a ) that associated with policy π , and G ∼ ( s i , s j ) as a state value function V (˜ s ) . And therefore, we can connect our expectile-based bisimulation operator to the lifted MDP, where we can use the conventional analytics tools in RL to analyze bisimulation operators.

F Additional Experiments

F.1 Ablation Study - Value of Expectile

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

100

walker2d-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

100

hopper-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

halfcheetah-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

25

50

75

100

walker2d-medium-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

25

50

75

100

hopper-medium-expert-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

20

40

60

80

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

20

40

60

80

walker2d-medium-replay-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

20

40

60

hopper-medium-replay-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

10

20

30

40

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

20

40

60

80

walker2d-medium-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

0

20

40

60

hopper-medium-v2

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e5

20

30

40

Timesteps

=0.3

=0.4

=0.5

=0.6

=0.7

Here we present the ablation study of setting different expectile τ ∈ { 0 . 3 , 0 . 4 , · · · , 0 . 7 } in Figure 6 to investigate the effect of the critical hyper-parameter in EBS. The experimental results demonstrate that the final performance gradually improves with a larger τ . Notably, the most superior performance is achieved when τ equals 0.6. However, when τ further increases to 0.7, the agent's performance suffers a sharp decline. We hypothesize that this could be due to the value function possibly exploding when τ is set to larger values, subsequently leading to poorer performance outcomes. This is as expected since the over-large τ leads to the overestimation error caused by neural networks. The experimental results demonstrate that we can balance a trade-off between minimizing the expected bisimulation residual and evaluating 'optimal' measurement solely on the dataset by choosing a suitable τ .

F.2 Ablation Study - Effectiveness of Reward Scaling

In the experiment, we set γ as 0.99 and c r will be 1 -γ = 0 . 01 accordingly in RS . In this ablation experiment, we considered different combinations of min-max normalization/standardization and

1

2

3

4

1e5

0

20

40

60

80

100

walker2d-expert-v2

1

2

3

4

1e5

0

20

40

60

80

100

hopper-expert-v2

1

2

3

4

1e5

0

20

40

60

80

halfcheetah-expert-v2

1

2

3

4

1e5

0

20

40

60

80

100

walker2d-medium-expert-v2

1

2

3

4

1e5

0

20

40

60

80

100

hopper-medium-expert-v2

1

2

3

4

1e5

0

20

40

60

80

Timesteps

base

cr

=0.01

standardization,

cr

=1

standardization,

cr

=0.01

min-max norm,

cr

=1

min-max norm,

cr

=0.1

RS: min-max norm,

cr

=0.01

min-max norm,

cr

=0.001

1

2

3

4

1e5

0

20

40

60

80

100

walker2d-expert-v2

1

2

3

4

1e5

0

20

40

60

80

100

hopper-expert-v2

1

2

3

4

1e5

0

5

10

15

20

25

30

35

halfcheetah-expert-v2

1

2

3

4

1e5

0

20

40

60

80

walker2d-medium-expert-v2

1

2

3

4

1e5

0

20

40

60

80

hopper-medium-expert-v2

1

2

3

4

1e5

0

10

20

30

40

50

Timesteps

MICo

MICo+RS

MICo+EBS

MICo+RS+EBS

various value of c r (including 1, 0.1, 0.01, and 0.001). The results in Figure 7 are consistent with our analysis in Section 5.2. The last two show better gains. As RS has tighter bounds, it excels in most datasets, validating our theory.

F.3 Case Study on MICo

As we illustrate in Section 5.2 and Section 6, Results in Figure 8 show that an unsuitable reward scaling dramatically decreases the performance while applying EBS will increase the performance in many datasets.

G Additional Related Works

Here we present a brief introduction of all the baselines we used in the experiments:

TD3BC [16] add a behavior cloning term to regularize the policy of the TD3 [18] algorithm, achieves a state-of-the-art performance in Offline settings.

DrQ+BC [35] combining data augmentation techniques with the TD3+BC method, which applies TD3 in the offline setting with a regularizing behavioral-cloning term to the policy loss. The policy objective is: π = argmax π E ( s,a ) ∼D [ λQ ( s, π ( s )) -( π ( s ) -a ) 2 ]

DRIML [38] and HOMER [39] (Time Contrastive methods) learn representations which can discriminate between adjacent observations in a rollout and pairs of random observations.

CURL [31] (Augmentation Contrastive method) learns a representation that is invariant to a class of data augmentations while being different across random example pairs.

Inverse Model [44] (One-Step Inverse Models) predict the action taken conditioned on the previous and resulting observations.

H Additional Discussion

H.1 The severity of the proposed problem

How do bisimulation-based objectives perform in other (online or goal-conditioned) settings? Various methods, such as DBC [54], MICo [6], SimSR [52], and PSE [1], have consistently demonstrated positive results in online settings, regardless of the presence of distractors. This evidence supports the efficacy of bisimulation techniques in online settings. Additionally, GCB [23] excelled in goal-conditioned environments, ExTra [46] showcased the power of bisimulation metric in exploration, and HiP-BMDP [55] successfully incorporated bisimulation into multi-task settings, highlighting its superior performance, all mostly in online settings too, with little work in offline RL. These studies suggest that when tailored to specific environments, bisimulation methods can excel. Despite these works, bisimulation methods have had little success when extended to offline settings, and our motivation is to tackle this problem.

While bisimulation objectives used in the offline setting are directly affected by missing transitions, many other representation objectives may not. When referring to state representation learning, using bisimulation in offline settings presents challenges due to the two issues we outlined: the presence of missing transitions and inappropriate reward scales. Concurrently, there exists other representation objectives, like CURL [31], ATC [49], which focus on pairs of states without the explicit necessity for transition information. As a consequence, they do not explicitly require accounting for missing transitions or reward scaling in their objectives. This absence of direct influence sets them apart from bisimulation-based methods. Yet, we consider that bisimulation-based techniques have a theoretical edge and have proven effective in online settings, Thus, we deem that our work is impactful in that it delivers a proof that bisimulation can be successful offline.

Compounded effect for bisimulation principle in offline settings. In online scenarios, state representations and policies are updated concurrently, while in offline settings, state representation is pre-trained before policy learning, with the two phases completely decoupled. Errors during representation learning in offline settings can have a compounded effect on policy learning, leading to significant issues. This is the reason that missing transitions is particularly harmful to the bisimulation principle in offline settings. Although reward scales affect bisimulation universally, as offline settings require pretraining state embedding, any major discrepancy between this fixed representation and the policy parameter space can further undermine the learning process. Hence, the proposed solutions hold promise for enhancing bisimulation's efficiency in offline settings.

H.2 Suitability of different techniques

EBS We provide EBS as a general method, which is applicable to all bisimulation-based objectives, given that they all adhere to the foundational principle of bisimulation. This principle revolves around the contraction mapping properties similar to the value iteration. Whenever there's an intent to employ bisimulation in offline scenarios, with an aim to reduce the Bellman residual for approximating the fixed point, the outlined challenge emerges. Consequently, EBS holds the potential to enhance any bisimulation-based method, regardless of the distance they use.

RS In essence, the given theoretical analysis is applicable across all bisimulation-based objectives. However, the precise settings for hinge on the foundational distance. For instance, SimSR uses the cosine distance which has definitive bounds. As a result, we need to infer the ideal setting from Equation 10 and Theorem 8. In contrast, the MICo-like distance and DBC employ L-K distance and L1 distance respectively, having bounds ranging from . Consequently, they can adapt to more value settings. We propose our approach as a general method/principle to employ a novel bisimulation metric or distance, especially in the context of offline RL.

I Empirical estimation of bisimulation error

In this section, we would like to conduct a toy experiment to empirically show that the bisimulation error could possibly be larger than bisimulation bellman residual in fixed/finite datasets.

Data collection To collect the evaluation dataset, we utilize TD3 [18] (a deterministic algorithm) instead of SAC [22] to avoid stochasticity. Firstly, we train a TD3 agent using the rlkit [45] codebase until convergence. Then, we collect 10k transitions and select specific transitions (such as 10, 100, 1000, 5000...) from these 10k transitions with uniform probability to form the evaluation dataset D . For determining termination, we follow the settings described in [18] and [22], considering a state terminal only if termination occurs before 1000 timesteps. If termination occurs before 1000 timesteps, we set γ = 0 ; otherwise, we set γ = 0 . 99 .

Computation Given an evaluation dataset D , the bisimulation Bellman residual ϵ ϕ is computed by 1 |D| ∑ (( s i ,ai ) , ( s j ,a j )) ∼D ( G ϕ ( s i , s j ) -( | r ( s i , a i ) -r ( s j , a j ) + γG ϕ ( s i , s j ))) 2 , and the bisimulation error ∆ ϕ is computed by 1 D ∑ ( s i,s j ) ∼D ( G ϕ ( s i , s j ) -G ∼ ( s i , s j )) 2 , where G ∼ denotes the corresponding fixed point measurement. Since directly computing G ∼ is challenging, we compute | V π ( s i ) -V π ( s j ) | instead, as they should be equal when considering the measurement is the fixed point and the transition is deterministic. To compute V π ( s i ) , we reset the MuJoCo environment to the specific state s i and ran the policy for 1000 timesteps. Since the environment and policy are deterministic, a single trajectory is sufficient to estimate the true value.

The results are presented in Table 3, which indicates that the bisimulation error on finite datasets is indeed larger than the bisimulation bellman residual.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBisimulation methods tend to fail when applied to offline RL. The authors performed a bisimulation error analysis and concluded that missing transitions, inevitable when training with a fixed offline dataset, lead to inaccurate metric predictions. Therefore, training policies based on the representation learned with inaccurate predictions will perform poorly. The paper proposes a $\\tau$-expectile-based bisimulation operator to overcome this challenge. This operator achieves fully in-sample learning in the limit $\\tau\\to 1$. The authors also note that reward scaling is crucial when the underlying state-space metric is bounded. Experiments conducted on state-based and pixel-based offline RL problems show that both expectile-based bisimulation operator and reward scaling have a significant impact on the final quality of the learned representation.\n\nSTRENGTHS:\nThe authors mathematically analyzed the properties of the original bisimulation operator and the novel expectile-based bisimulation operator. The mathematical techniques used in the proofs and the careful analysis of the error bounds will be very helpful to other researchers trying to work on applying bisimulation-metric-based representation learning to offline RL. The expectile-based based bisimulation operator the authors proposed is also interesting and effective under various settings, as the experimental results reveal. Finally, the paper is overall well-written and easy to understand.\n\nWEAKNESSES:\nIt is doubtful that learning a bisimulation operator in the offline setting generally fails because of the reasons the paper presents. To facilitate learning by diversifying trajectories, a lot of RL environments impose a limit on the episode length. An environment with a time limit of $T$ should have states that an agent can arrive within $T+1$ timesteps but not in $T$ timesteps. Those states satisfy the conditions of Proposition 4, which means learning a bisimulation operator will fail. This is not the case.\n\nAlso, according to line 238, unlike the original bisimulation operator, learning the expectile-based bisimulation operator will succeed because we only consider state pairs with corresponding actions in the dataset. Suppose this is the reason why learning an expectile-based bisimulation operator succeeds. Then why not just omit the last and the second-last states of each trajectory in the dataset during training while using the original bisimulation operator? Then the next-state pairs of every state-pairs left will have corresponding actions in the dataset. It is hard to imagine such a simple solution will work.\n\nQUESTIONS:\nIs the proposed algorithm also effective in the online RL setting? Also, it would be helpful for the readers if the paper introduces the definition of $G^\\pi$ and a *lifted MDP* before mentioning them in equation (2) and line 139, respectively.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work, and I do not believe this work will have a potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates why bisimulation-based methods, which are known to work well in the online setting, fail in the offline case. It hypothesizes that missing transitions and reward scaling issues are a source of this failure, and proposes a way to mitigate these issues. It includes experiments on various benchmarking tasks.\n\nSTRENGTHS:\n1. According to me, this is an important problem to solve. Recently bisimulation-based representation learning methods have become popular but have focused mostly on the online setting. The insights from this paper are illuminating, especially Theorem 3 and Proposition 4.\n2. I like that they considered the MICO distance since unlike other metrics, MICO is a diffuse metric and has additional complications due to the non-zero self-distances.\n\nWEAKNESSES:\nIt appears that the approach is limited to a single and known behavior policy. I wonder how this could be used in the multiple and unknown behavior policy setting, which is starting to become a more popular setting in offline RL.\n\nQUESTIONS:\n1. I’d point authors to this new paper by the MICO authors: https://openreview.net/forum?id=nHfPXl1ly7 to see how their approach fits in with this kernel-based bisimulation approach.\n2. How do reward normalization schemes impact the results? For example, subtracting min and dividing by max-min (where min/max are the min/max rewards in the dataset), or 0/1 mean/std normalization etc? Are these beneficial when computing the bisimulation metric?\n3. Why is the Effective Dimension a good metric in the offline RL case (Figure 1)? The paper just plots the results without justifying the usage. I ask this by considering other offline RL metrics for representations such as distribution shift and coverage [1, 2] in terms of the state-action feature representations.\n4. From the experiments, it appears that MICO and SimSR were applied as-is. I wonder if it would be useful to apply an off-policy version of say MICO by either sampling from the policy getting trained or using some version of importance sampling?\n5. It would be interesting to include some preliminary results in the appendix on off-policy evaluation (not just control) (keep the main results in tact; I am referring more to an exploratory analysis for the appendix).\n6. How is maximization and subject-to constraints (eqn 8) satisfied in continuous action setting?\n\n[1] Instabilities of Offline RL with Pre-Trained Neural Representation. Wang et al.\n\n[2] Learning Bellman Complete Representations for Offline Policy Evaluation. Chang et al.\n\nLIMITATIONS:\nSee above.\n\nN/A societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the pitfall of bisimulation-based methods in offline RL. It identifies missing transitions in the finite dataset as a significant problem for bisimulation-based methods, leading to ineffective estimation. The paper also highlights the importance of reward scaling in bounded cosine distance method.\n\nTo address these issues, the paper proposes using the expectile operator for the measurement learning in offline RL, as expectile operator helps prevent overfitting to incomplete data. Additionally, the paper analyzes the upper bound of metric w.r.t. the reward scale and proposes a reward scaling strategy to reduce representation collapse.\n\nSTRENGTHS:\nThis paper establishes a connection between the Bellman and bisimulation operators to understand the gap for bisimulation-based approaches in online and offline RL settings. It defines the bisimulation Bellman residual and the bisimulation error and analyzes the gap of between minimizing them. \n\nThis paper also analyzes the bounds of the bimiluation-based metric and how the reward scaling affects the cosine distance. It demonstrates that setting the reward coefficient to $1-\\gamma$ improves performance compared to using SimSR.\n\nWEAKNESSES:\nThe motivation for utilizing the expectile operator could be explained more clearly. While expectile regression is applicable to both online [1] and offline RL, its relevance to addressing Proposition 4 and preventing overfitting in the offline RL setting requires further elaboration.\n\nThe authors propose reward scaling as a solution to mitigate the limitation imposed by the cosine distance, which has a range of [0, 2]. This range imposes an upper bound on the value of $c_r$. However, Theorem 8 suggests that a larger value of $c_r$ leads to a more accurate approximation of the value function. An alternative approach to overcome the limitations of the cosine distance would be to use other distance metrics such as L1 or L2 distance. Hence, it would be beneficial for the authors to clarify the rationale behind selecting cosine distance over these alternatives.\n\nIn the comparison experiments between SimSR+RS(+EBS) and MICo(+EBS), it is possible that RS improves performance due to the numerical stability of neural networks. By upper bounding the range of $G$ to [0, 1], which can be stably predicted by neural networks, RS might contribute to the observed performance enhancement.\n\nThe bisimulation Bellman residual $\\epsilon_{\\phi}^{\\pi}$ may not be minimized to zero if $G_{\\phi}^{\\pi}$ is a cosine distance. Consider the case of $s_i = s_j$. $G_{\\phi}^{\\pi}(s_i, s_j) = 0$ because it is cosine distance. However, $FG_{\\phi}^{\\pi}(s_i, s_j)$ is a Łukaszyk–Karmowski distance because  $\\gamma  E_{s^{\\prime}_i, s^{\\prime}_j} G_\\phi^\\pi(s^{\\prime}_i, s^{\\prime}_j)$ \n\nis a Łukaszyk–Karmowski distance as $s^{\\prime}_i$ and $s^{\\prime}_j$ are sampled [2],\n\n and $|r_i^{\\pi} - r_j^{\\pi}|$ is untractable and usually alternated to $E_{r_i, r_j}|r_i - r_j|$, which is again a Łukaszyk–Karmowski distance[3]. Therefore, $FG_{\\phi}^{\\pi}(s_i, s_j) \\geq G_{\\phi}^{\\pi}(s_i, s_j)$ if $s_i = s_j$, the equality is taken only if $G_{\\phi}^{\\pi}(s^{\\prime}_i, s^{\\prime}_j) = 0$ for all $s^{\\prime}_i$ and $s^{\\prime}_j$ and \n\n$r_i = r_j$ for all $r_i$ and $r_j$. This can destroy the Proposition 4 because not all $\\epsilon_{\\phi}^{\\pi}$ is zero. However, MICo distance does not meet this issue.\n\n------\nReference\n\n[1] Rowland et al. Statistics and Samples in Distributional Reinforcement Learning. ICML 2019.\n\n[2] Castro et al. MICo: Improved representations via sampling-based state similarity for Markov decision processes. NeurIPS 2021.\n\n[3] Chen et al. Learning Representations via a Robust Behavioral Metric for Deep Reinforcement Learning. NeurIPS 2022.\n\nQUESTIONS:\n1. Lemma 5 and 6 require a citation to [4].\n1. How does the value of $c_r$ affect the value bound according to Theorem 8 and Theorem 15?\n1. How is Eq. (29) derived from Eq. (28) in Theorem 15 in the Supplementary Material?\n1. It is confusing that the main text defines $\\Delta$ and $\\epsilon$ as absolute values, while the supplementary material defines them as differences. The authors should use different notations (rather than the same $\\Delta$ and $\\epsilon$) in the supplementary material. \n1. What is the value of $\\tau$ in Table 1?\n1. Some implementation details are missing, including the learning rate and network architecture.\n\n-----\nReference\n\n[4] Ma et al. Offline Reinforcement Learning with Value-based Episodic Memory. ICLR 2022.\n\nLIMITATIONS:\nThe authors should clarify the motivation of expectile operator and refine the theoretical analysis with cosine distance.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on understanding and ameliorating the relatively poor performance of bisimulation-based representations in offline RL. The authors identify two main issues: overfitting to incomplete data and reward scaling. When the offline dataset is missing states, the representation that’s learned may not actually reflect the true bisimulation metric of the underlying MDP even if the associated Bellman residual over the dataset is zero. Their approach is to regularize learning with expectile regression to avoid overfitting to the incomplete data. With regards to reward scaling, the authors show that a larger reward scale can produce a more accurate approximation of the value function. Experiments are carried out on continuous control tasks with both proprioceptive and visual inputs.\n\nSTRENGTHS:\n- The theoretical analysis seems carefully done and gives useful, practical insights, and the connection of the bisimulation update operator to policy evaluation is interesting. \n\n- The careful evaluation of the proposed approach in the D4RL experiments is fairly convincing that the modifications proposed by the authors produce some improvement in performance.\n\nWEAKNESSES:\n- In the D4RL results, SimSR + RS + EBS only clearly outperforms the other approaches on 3, possibly 4, of the 12 settings. \n\n- It would be helpful to have either more qualitative analysis of the existing experiments or a toy experiment to provide intuition as to the resulting behavioral differences and success/failure cases induced by the authors’ approach rather than just performance plots. \n\n- Running the V-D4RL benchmark over only 3 seeds seems rather low, and no measures of variance are provided. This stands in contrast to the D4RL results, for which not only full learning curves with error shading are shown, but also IQM measurements. \n\n- More discussion (possibly in the appendix) regarding the variation in performance across different datasets would be helpful.\n\n- Clarity could be improved in some areas, particularly in providing more background to the introduction of the expectile-based operator how it aids in in-sample learning / prevents overfitting to incomplete data. More precise language would also be helpful in the reward scaling section.\n\nQUESTIONS:\n- Can the authors provide learning curves + IQM analysis for the V-D4RL experiment? \n\n- Could the authors run experiments with MICo + RS (and MICo + RS + EBS) as well? If RS is not applicable to MICo for some reason, could the authors add a note either in the appendix or main text explaining why?\n\nLIMITATIONS:\nThe authors do an adequate job acknowledging the limitations of their approach, but perhaps more discussion would be helpful in the appendix. I don’t see any direct negative societal implications of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of bisimulation-based representations in offline reinforcement learning tasks, which are less effective than alternative methods. To address this issue, the paper proposes a tailored reward scaling strategy and an auxiliary loss function based on bisimulation metrics to learn robust state representations. The experiments show that the proposed method outperforms existing methods on a range of locomotion tasks in the DeepMind Control Suite. Ablation studies suggest that the proposed reward scaling strategy and bisimulation-based loss function are critical to the performance of the method. Overall, I believe this work makes a valuable contribution to the field of reinforcement learning by improving the effectiveness of bisimulation-based representations in offline RL tasks.\n\nSTRENGTHS:\n**Motivation and intuition**\n- The motivation for addressing the pitfalls of bisimulation-based representations in offline reinforcement learning is convincing. The authors clearly explain the challenges faced by bisimulation-based approaches in offline RL tasks and the need for solutions to improve their performance. Their analysis reveals that missing transitions in the dataset, which often occur in offline settings, are particularly harmful to the bisimulation principle\n\n**Novelty**\n- The idea of utilizing expectile operators to prevent overfitting to incomplete data is intuitive and convincing, which is theoretically supported by the paper's analysis. This paper presents an effective way to implement this idea, which has not been explored in previous works.\n\n**Technical contribution**\n- The proposed solutions, an expectile-based operator and a tailored reward scaling strategy, for addressing the pitfalls of bisimulation-based representations in offline RL tasks seem compelling, especially when dealing with limited datasets. The authors provide a thorough analysis of the performance gains achieved by their proposed solutions in various benchmark suites.\n\n**Clarity**\n- The overall writing is clear and well-organized. The authors utilize figures well to illustrate the ideas, and Figure 1 clearly shows the results with and without reward scaling. \n- The paper gives clear descriptions in both theoretical and intuitive ways. The notations, formulations, and theorems are well-explained in the appendix, making it easy for readers to follow the technical details.\n\n**Experimental results**\n- The experimental results are impressive and demonstrate the effectiveness of the proposed framework in improving the performance of bisimulation-based approaches in offline RL tasks. The authors provide clear visualizations of the results, and Figure 3 particularly provides a clear comparison of the performance gains achieved by the proposed framework over the two state-of-the-art bisimulation-based algorithms, MICo and SimSR.\n\n**Reproducibility**\n- The code is provided, which helps understand the details of the proposed framework.\n- Given the clear description in the main paper and the details provided in the appendix, I believe reproducing the results is possible.\n\nWEAKNESSES:\n**Ablation study**\nThe proposed framework comprises two components: an expectile-based operator and a tailored reward scaling strategy. Conducting an ablation study to verify the proposed methods to some naive ones would be good. Take the tailored reward scaling strategy for example, one possible naive reward scaling strategy could be to multiply the reward signal by a constant factor, such as 0.1 or 10, and observe the effect on the performance of the bisimulation-based RL algorithm. Another possible method could be to clip the reward signal to a certain range, such as [-1, 1], and observe the effect on the performance.\n\n**Limitation**\nI notice that the authors say that Since MICo does not necessitate a particular upper bound, RS may harm its performance, leading them to exclude the MICo+RS results from the main experiments. It would be good to elaborate more on when the two methods, an expectile-based operator and a tailored reward scaling strategy, can help improve the performance of given algorithms.\n\n**Experiment setup**\n I think that adding manipulation and navigation environments to the experimental evaluation of the proposed method would be a valuable addition to the paper. While the locomotion environments used in the current evaluation provide a good testbed for evaluating the proposed method, it would be beneficial to evaluate the method in a broader range of environments to better understand its generalizability and effectiveness across different types of tasks instead of only evaluating on locomotion environments.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary\n-------\n\nThis submission\\'s goal is to understand why bisimulation methods\nsuffers in the offline RL setting. The authors go on to propose two\nmethods that help learn a better bisimulation metric, that result in\nrepresentations that more faithfully capture state abstractions from\noffline datasets. The two methods are 1) (EBS) an expectile operator\nthat regularizes the learned state representation to in-sample learning\nand 2) (RS) a reward scaling where they tune the hyperparameter on the\nreward difference (with theory for further motivation). Experiments are\nconducted in offline RL using D4RL with either states or pixel-based\nobservations. The empirical results provide some evidence that RS+EBS\nimprove the learned bisimulation metric, and hence the state\nrepresentation.\n\nDecision\n--------\n\nThe theoretical results, at the intersection of bisimulation and offline\nRL theory, is interesting and novel. Besides the theory, however, I\nremain unconvinced by the empirical evidence and the severity of the\nproposed problem. As a result, I lean towards rejection. In the D4RL\nexperiments, it does not seem that there is a benefit to using the\nbisimulation metric at all. While there is a purported benefit in the\nvisual domain, it is not clear that this is due to RS+EBS or the fact\nthat there are two additional tunable hyperparameters. Moreover, the two\nmethods do not really address a novel problem. Missing transitions, and\nEBS, is an issue for offline RL in general, not necessarily isolated to the bisimulation metrics. Whereas the\nreward scale is an issue with the bisimulation metric, and not necessarily isolated to the offline RL setting.\nSo, while I like the spirit of the paper, I do not find the empirical evidence\nconvincing.\n\nSTRENGTHS:\nStrengths\n---------\n\n-   Interesting theoretical insights, combining 1) bisimulation theory\n    which casts a metric as a fixed point in a lifted MDP and 2) offline\n    RL theory which shows limitations of RL algorithms on limited\n    datasets.\n-   Presentation and initial motivation of the problem, bisimulation in\n    the offline setting, is clear. There is value in understanding the\n    role of bisimulation metrics, especially in the case of offline RL\n    where the learned representation is known to be important for\n    transfer.\n\nWEAKNESSES:\nWeaknesses\n----------\n\n-   While motivation is initially clear, it seems ultimately misguided.\n    The results in the reference (\\[42\\] in paper, Yang and\n    Nachum, 2021) does not suggest that bisimulation is particularly\n    poor in the offline setting. The bisimulation results are also poor\n    in the online setting, and the paper continues to show that several\n    methods perform worse in the offline setting, which is unsurprising.\n-   The two methods used to improve bisimulation in the offline setting\n    are either not unique to issue with bisimulation (missing\n    transitions) or not seemingly relevant to the offline setting\n    (reward scaling). Both of these methods do seemingly help improve\n    learning the bisimulation metric, but the empirical results are not\n    convincing that this is helpful over the baselines.\n-   Empirical results on D4RL are not entirely convincing, and the\n    reasons for excluding MICo+RS are not clear. The results in the\n    pixel-based D4RL, while seemingly an improvement on average, are\n    difficult to evaluate without confidence intervals and with only 3\n    runs.\n\nQUESTIONS:\nDetailed Comments & Questions\n-----------------\n\n-   Line 38: \\\"objectives in most approaches are required to be coupled\n    with the policy improvement procedure\\\" What about their approaches\n    require coupling with the policy improvement procedure and why is\n    this important? Is this because they are using a $π^*$-bisimulation?\n\n-   Line 39-43: The point you are making here is that bisimulation\n    metrics do not work well in offline RL. The points about special\n    cases of online RL seem unrelated and are distracting from this main\n    point. Moreover, reference \\[19\\] provides no empirical evidence for\n    or against bisimulation metrics in offline RL. Moreover, the\n    empirical evidence in \\[42\\] suggests that all methods do worse in\n    the offline RL setting, and does not seem to suggest that\n    bisimulation in particular suffers in the offline setting.\n\n-   Line 48: The problem of missing transitions is an inherent problem\n    to all offline RL methods.\n\n-   Line 50: The problem of reward scale seems unrelated to offline RL?\n\n-   Lien 55: \\\"we can achieve a balance between the behavior measurement\n    and the greedy assignment of the measurement over the dataset.\\\" I\n    do not understand this statement. Are you claiming that the\n    expectile operator balances a type of exploration-exploitation\n    trade-off? If so, it is not clear how or why.\n\n-   Line 244: \\\"Most previous works \\[4, 45, 5, 43\\] have overlooked the\n    impact of reward scaling in the bisimulation\\\" The DBC paper in\n    reference 45 does in fact have a hyperparameter for reward scaling.\n    I think this sentence needs more qualification, because most\n    combined losses include a hyperparameter for tuning their relative\n    importance.\n\n-   Line 248: I have a hard time interpreting the conclusion based off\n    of the inequality following (10). The result is an upperbound on the\n    fixed point and both $c_k$ and $c_r$ are free variables on the right\n    hand of the inequality. But, what does this inequality tells us\n    about the trade-off of setting, for example, $c_k = 0$ thus\n    upperbounding the fixed point by 0? Different settings of $c_k$ and\n    $c_r$ lead to different fixed points, $G^\\pi_\\tilde$. By using this\n    inequality to set an upperbound on the distance, you are also\n    biasing the metric. For example, for smaller $c_k$, you are\n    considering states similar if the resulting immediate behavior is\n    similar and putting less weight on differences in long-term\n    behavior.\n\n-   \\\" However, when bisimulation operators are instantiated with\n    bounded distances, e.g., cosine distance, such a setting may be\n    unsuitable.\\\" I don\\'t see why this is inherently undesirable. The\n    bisimulation distance aggregates the cosine distance across a\n    trajectory via the bellman recursion. I do not see why they need to\n    have the same upper bound. What you are attempting here seems like,\n    in an RL analogy, putting the rewards and the value on the same\n    scale.\n\n-   \\\"with the maximum value of 1 of the cosine distance. To achieve a\n    tighter bound in Equation11, we should then maximize the reward\n    scale, setting cr to 1 − γ.\\\" I do not think that c~r~ is a free\n    parameter that can be chosen to tighten the bound. This is because\n    $c_r$ also determines the fixed point of the bisimulation metric and\n    the learned representation. But, accepting your result for the\n    moment, I do not see why you set $c_r$ to be $1-\\gamma$ and not some\n    arbitrarily large number.\n\n-   Analysis of Figure 2: I am not sure what to take away from these\n    results. Does the TD3BC baseline use any of the learned\n    representations, or is it trained from scratch from the raw state?\n    The goal of the paper is understanding and addressing the pitfalls\n    of bisimulation representations in offline RL. But, these result\n    suggest that bisimulation representations are neither needed nor\n    helpful for the downstream task. The motivation is undermined: even\n    if bisimulation representations are augmented with your suggested\n    changes (expectiles + reward scaling), we get roughly the same\n    performance as not using any bisimulation at all.\n\n-   MICo + RS: what does it mean that MICo does not necessitate an\n    upper-bound?\n\n-   Table 1: The results are not entirely convincing because the\n    pixel-based offline RL experiments are known to have high variance.\n    Without reporting confidence intervals, and by uisng only 3 seeds, I\n    do not find this evaluation convincing.\n\n-   Ablation: The reward scale coefficient, while motivated by theory,\n    is not ablated.\n\nMinor Comments\n--------------\n\n-   Line 32: No question has been posed yet, so it is not clear what\n    question this paper is answering. A question does occur a few\n    paragraphs down, so maybe this statement is an artifact?\n-   Figure 2: Some lines are dashed while others are not, and this is\n    not indicated in the legend.\n\nLIMITATIONS:\nThe authors outline limitations in the expectile operator hyperparameter. Further, I see no potential negative societal concerns.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-27"}