Title: Reining Generalization in Offline Reinforcement Learning via Representation Distinction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Offline Reinforcement Learning
Abstract: Offline Reinforcement Learning (RL) aims to address the challenge of distribution shift between the dataset and the learned policy, where the value of out-of-distribution (OOD) data may be erroneously estimated due to overgeneralization. It has been observed that a considerable portion of the benefits derived from the conservative terms designed by existing offline RL approaches originates from their impact on the learned representation. This observation prompts us to scrutinize the learning dynamics of offline RL, formalize the process of generalization, and delve into the prevalent overgeneralization issue in offline RL. We then investigate the potential to rein the generalization from the representation perspective to enhance offline RL. Finally, we present  Representation Distinction (RD), an innovative plug-in method for improving offline RL algorithm performance by explicitly differentiating between the representations of in-sample and OOD state-action pairs generated by the learning policy. Considering scenarios in which the learning policy mirrors the behavioral policy and similar samples may be erroneously distinguished, we suggest a dynamic adjustment mechanism for RD based on an OOD data generator to prevent data representation collapse and further enhance policy performance. We demonstrate the efficacy of our approach by applying RD to specially-designed backbone algorithms and widely-used offline RL algorithms. The proposed RD method significantly improves their performance across various continuous control tasks on D4RL datasets, surpassing several state-of-the-art offline RL algorithms.

Reining Generalization in Offline Reinforcement Learning via Representation Distinction

Abstract

Offline Reinforcement Learning (RL) aims to address the challenge of distribution shift between the dataset and the learned policy, where the value of out-ofdistribution (OOD) data may be erroneously estimated due to overgeneralization. It has been observed that a considerable portion of the benefits derived from the conservative terms designed by existing offline RL approaches originates from their impact on the learned representation. This observation prompts us to scrutinize the learning dynamics of offline RL, formalize the process of generalization, and delve into the prevalent overgeneralization issue in offline RL. We then investigate the potential to rein the generalization from the representation perspective to enhance offline RL. Finally, we present Representation Distinction (RD), an innovative plug-in method for improving offline RL algorithm performance by explicitly differentiating between the representations of in-sample and OOD state-action pairs generated by the learning policy. Considering scenarios in which the learning policy mirrors the behavioral policy and similar samples may be erroneously distinguished, we suggest a dynamic adjustment mechanism for RD based on an OOD data generator to prevent data representation collapse and further enhance policy performance. We demonstrate the efficacy of our approach by applying RD to designed backbone algorithms and widely-used offline RL algorithms. The proposed RD method significantly improves their performance across various continuous control tasks on D4RL datasets, surpassing several state-of-the-art offline RL algorithms.

1 Introduction

Reinforcement learning (RL) is a machine learning paradigm centered on training intelligent agents to make decisions through interactions with an environment. In this learning process, an agent develops a policy-a mapping from states to actions-via trial and error, with the goal of maximizing cumulative rewards over time. However, acquiring interaction data in many real-world applications can be expensive, time-consuming, or even hazardous. This spurred the advancement of offline Reinforcement Learning, which has garnered considerable interest in the research community due to its ability to learn effective policies from pre-existing data collected by unknown behavioral policy

without necessitating online interactions with the environment. Offline RL can result in substantial savings in resources, time, and the risk associated with online exploration.

Despite its appealing potential, offline RL encounters challenges in learning optimal policies from limited and suboptimal datasets. The primary challenge in offline RL is the distribution shift between the dataset and the learned policy. As the agent updates its policy, it may come across out-of-distribution (OOD) state-action pairs absent from the limited support of the fixed offline dataset [1]. Conventional RL algorithms, such as SAC [2] or TD3 [3], could yield overly optimistic Q-value estimations for these unseen state-action pairs due to overgeneralization, leading to catastrophic performance. Existing methods in offline RL seek to address this challenge by incorporating techniques like policy constraint [4, 5, 6, 7], conservative value estimates, [8, 9], and uncertainty estimation [10, 11, 12].

Recently, [13] elucidated that the ability of existing offline reinforcement learning (RL) methods to handle OOD actions is significantly attributed to their impact on the learned representations of state-action pairs. The learned representation can capture the underlying structure and essential aspects of the state-action space, enabling the agent to identify similarities and patterns across various state-action pairs, thus improving agent's learning efficacy and generalization capacity. Although there are a few works [14] also study representation learning in offline RL, they focus on how different kinds of pretrained representation matter in downstream policy learning, neglecting the understanding of the essential interplay between representation and policy during the offline co-learning process.

In this paper, we first present a view dubbed Backup-Generalization Cycle to gain a tangible comprehension of how a typical offline RL algorithm learns. We subsequently formalize how generalization happens, and discuss the overgeneralization issue prevalent in offline RL. Finally, we introduce a novel method called Representation Distinction (RD) to mitigate overgeneralization from in-sample state-action pairs to OOD ones. Specifically, we distinctly differentiate the data sourced from the dataset and the data generated by the learning policy from the representation perspective. In cases where the learning policy's performance aligns with the behavioral policy and the generated data largely overlaps with the dataset, we devise an OOD data generator to produce data with lower Q-values than the current policy. We then gradually shift our focus from differentiating in-sample data and data generated by the learning policy to differentiating data generated by the learning policy and data generated by the OOD actor in the representation space, preventing similar samples from being erroneously distinguished.

To demonstrate the efficacy of our proposed RD method in enhancing offline RL policy performance by reining the generalization, we apply it to two backbone algorithms, which are modified versions of the original SAC and TD3 algorithms by incorporating an ensemble of critics and an uncertainty regularization term in the policy update process. By integrating RD into these simple algorithms, our proposed method RD can improve the performance of the backbone agents on D4RL datasets across various continuous control tasks and outperforms several state-of-the-art offline RL algorithms. We also apply RD on two widely-used baselines TD3BC and CQL and improve their performance significantly. In-depth analysis based on visualization and statistical results that demonstrates the efficacy of RD in obtaining more differentiable representation compared to its variants is also provided.

The main contributions of this work can be summarized as follows:

We introduce a view called Backup-Generalization Cycle to foster an understanding of typical offline value function learning, and highlights the necessity of reining generalization to enhance offline RL.

We proposed a practical plug-in method Representation Distinction (RD) from the perspective of representation to enhance the performance of offline RL methods by inhibiting overgeneralization among state-action pairs sourced from different distributions.

We evaluate the effectiveness of RD by applying it to designed backbone algorithms and existing widely-used algorithms and enhance their performance significantly.

2 Related Works

Offline RL Offline RL algorithms strive to train RL agents using pre-collected datasets. Nevertheless, the distribution shift between the behavior policy and the policy being learned may lead to issues, as OOD actions are sampled from the learned policy and incorporated into the learned

critic. To address this challenge, various approaches have been proposed. Some earlier methods aim to constrain the learned policy to remain close to the behavior policy, which can be accomplished through explicit policy regularization [4, 5, 6], implicit policy constraints [15, 16, 17, 18, 19], or by employing auxiliary behavioral cloning losses [7]. Alternative approaches penalize the Q-value of OOD actions to discourage their selection [8, 9, 10, 11, 12]. Moreover, model-based methods that train with conservative penalties have been suggested [20, 21, 22, 23, 24, 25].

Representation in Offline RL. Prior research has aimed to analyze various aspects of the representations induced by TD-based methods using function approximation, predominantly in the standard online RL setting [26, 27, 28, 29, 30]. More recently, this line of inquiry has emerged in the offline RL setting [31, 32, 33, 34, 13]. [31] investigates which representations can result in stable convergence of TD in a linear setting. [32, 33] explore the learning dynamics of Q-learning in an overparameterized setting, observing excessively low-rank and aliased feature representations at the fixed points identified by TD-learning. [34] studies the extent of impact of different interventions on the causal link between the effective rank and offline RL agent performance and points out that there is no strong relationship between them. [13] highlights that a substantial portion of the benefits of existing offline RL approaches, which aim to avoid OOD actions, actually originates from their effects on the learned representations. The authors also identify specific metrics that facilitate effective evaluation of the quality of the learned representation. [14] pretrain the representation using different auxiliary loss and then fix the representation to apply it to downstream policy learning. Nevertheless, a thorough examination of the explicit representation distinction between in-sample and OOD state-action pairs in the context of offline RL is lacking. Our approach promotes orthogonality between the representation vectors of in-sample and OOD data, offering a more practical solution for inhibiting overgeneralization, thus enhancing offline RL.

3 Preliminary

Markov Decision Process (MDP) is a mathematical framework that models decision-making processes in stochastic environments. It is defined by a tuple ( S, A, P, R, γ ) , where S is the set of states, A is the set of actions, P is the state transition probability function, R is the reward function, and γ is the discount factor. Q-learning is the major approach for obtaining the optimal π ( s ) , which learns a Qvalue function Q ( s, a ) that represents the expected cumulative discounted rewards when starting from the state s taking the action a and executing the policy π thereafter. the Q-value function is evaluated by iterating the Bellman operator as T Q ( s, a ) = E s ′ ∼ P ( ·| s,a ) [ r ( s, a ) + γ E a ′ ∼ π ( s ′ ) Q ( s ′ , a ′ ) ] . The goal of RL is to find an optimal policy π ( s ) that maximizes the cumulative discounted rewards.

Offline Reinforcement Learning [1], aims to learn a policy from a fixed dataset of interaction samples, without further interaction with the environment. The dataset D consists of transition tuples ( s, a, r, s ′ ) collected from interactions between one or multiple behavior policies and the environment. The goal is to learn an optimal policy using only this dataset.

To deal with large state-action space, deep RL resorts to deep neural networks for function approximation. Typically, a Q-network parameterized by ϕ can be viewed as Q ϕ ( s, a ) = w ⊤ Φ( s, a ) , where Φ( s, a ) ∈ R d is regarded as the representation obtained by the penultimate layer of Q-network while w ∈ R d is the linear weight [35, 36, 33, 13]. In conventional online and offline RL learning process, the representation and the linear weight learn together. A few recent works[37, 38, 39, 40] also introduce auxiliary objectives to strengthen or regularize the representation with different purposes. We use this view for the presentation of our analysis and proposed method in the following sections.

4 Reining Generalization in Offline RL

In this section, we focus on the learning dynamics of offline RL. First, we present a view called Backup-Generalization Cycle (Sec. 4.1) to gain a tangible comprehension of how a typical offline RL algorithm learns. We then take a further step to formalize how generalization happens, and discuss the overgeneralization issue in offline RL (Sec. 4.2). Finally, we propose the idea of kernel control to address the overgeneralization issue (Sec. 4.3), with the practical implementation deferred to Sec. 5.

𝑄 𝑠, 𝑎

𝑟 + 𝐶(𝑄(𝑠 ′

, 𝜋(𝑠

′

))

Backup

Generalization

𝑄 𝑠□, 𝑎 □

𝑟 + 𝐶(𝑄 𝑠⋆, 𝜋(𝑠 ⋆

)

)

𝑄 𝑠∘, 𝑎

∘

𝑟 + 𝐶(𝑄 𝑠□, 𝜋(𝑠 □ )

)

𝑄 𝑠△, 𝑎△

𝑟 + 𝐶(𝑄 𝑠∘, 𝜋(𝑠 ∘

)

)

𝑄 𝑠⋆, 𝑎

⋆

4.1 Backup-Generalization Cycle in Offline RL

Offline RL learns from a static offline dataset with no access to online interaction over the course of policy learning. To gain a better understanding of the intricacies of offline deep RL, for the first, we introduce a view called Backup-Generalization Cycle. This view, as depicted in Fig. 1, fosters an understanding of typical offline value function learning via two key components: Backup and Generalization :

Typical offline deep RL algorithms employ various forms of backups to learn the Q -function network. Here we use C ( · ) to denote a generic form of the target value such as Clipped Double Q -Learning [3]. It is notable that only the values of state-action pairs in the offline dataset are directly updated.

Since the values of all state-action pairs that are not in the dataset will never be directly updated, changes to the values of such ( s, a ) / ∈ D are solely indirectly instigated by the backups on ( s, a ) ∈ D , i.e., through generalization . Note that the state-action pairs for the target values of backup are highly likely to be absent from the dataset, i.e., ( s ′ , π ( s ′ )) / ∈ D , as the learning of current policy π .

Hence, the typical offline learning process of value function can be regarded as a consequence of the complex interplay between backup and generalization. This dynamic interplay forms a cycle: (1) the backups on ( s, a ) ∈ D consistently influence the values of ( s, a ) / ∈ D (highly likely including ( s ′ , π ( s ′ )) ); (2) the consistently changing Q ( s ′ , π ( s ′ )) participates in the backups on ( s, a ) / ∈ D ; the two kinds of dynamics iterate and twine during the learning process.

There are also some similar opinions mentioned in a few recent works on offline RL [41, 29]. We utilize the Backup-Generalization Cycle to distinctly separate and better analyze these two contributing factors. In the next subsection, we concentrate on the overgeneralization issue with a formal characterization of value generalization among state-action pairs.

4.2 Overgeneralization in Offline RL

According to the Backup-Generalization Cycle presented above, generalization plays a significant role for ( s, a ) / ∈ D . In contrast to backup, which is relatively explicit and controllable, generalization is implicit and intricate. In the following of this subsection, we analyze value generalization among state-action pairs during the offline RL learning process.

For a starting case, we consider how the Q function update caused by typical Temporal-Difference (TD) learning on a single state-action pair ( s, a ) ∈ D (denoted as ϕ → ϕ ′ ), affects the Q -value of an arbitrary state-action pair (¯ s, ¯ a ) . The post-update parameter ϕ ′ can be formalized as follows:

where learning rate is omitted for convenience. We further formalize the post-update Q -value of (¯ s, ¯ a ) by Taylor expansion at the pre-update parameter ϕ :

Now, we can characterize the generalization from ( s, a ) to (¯ s, ¯ a ) by plugging Eq.1 into Eq.2:

Behavioral Policy Distribution

Learning Policy Distribution

Learned OOD Policy Distribution

Over-Inhibition

𝑄 𝑠□, 𝑎 □

𝑟 + 𝐶(𝑄 𝑠⋆, 𝜋(𝑠 ⋆

)

)

𝑄 𝑠∘, 𝑎

∘

𝑟 + 𝐶(𝑄 𝑠□, 𝜋(𝑠 □ )

)

𝑟 + 𝐶(𝑄 𝑠⋆, 𝜋𝑜𝑜𝑑 (𝑠 ⋆

)

)

𝑄 𝑠□, 𝑎 □

𝑟 + 𝐶(𝑄 𝑠⋆, 𝜋(𝑠 ⋆

)

)

Learning

Generalization

Generalization

Policy-Dataset

Generalization Inhibition

Policy-OOD

Generalization Inhibition

𝑄(𝑠,∙)

𝑎

𝑄(𝑠,∙)

𝑎

Over-Inhibition

where k ϕ (¯ s, ¯ a, s, a ) . = ∇ ϕ Q ϕ (¯ s, ¯ a ) ⊤ ∇ ϕ Q ϕ ( s, a ) and it is called Neural Tangent Kernel in [27]. Eq.3 elucidates the change of the Q -value of any state-action pair (¯ s, ¯ a ) can be mainly characterized by the kernel k ϕ (¯ s, ¯ a, s, a ) and the TD-error T Q ϕ ( s, a ) -Q ϕ ( s, a ) . Apparently, the kernel controls the extent of generalization: for small values of k ϕ (¯ s, ¯ a, s, a ) , the generalization effect is minor and the classic tabular learning (i.e., no generalization) can be viewed as a special case when k ϕ and the higher-order term are 0; for large values of k ϕ (¯ s, ¯ a, s, a ) , the generalization effect is prominent and Q ϕ ′ (¯ s, ¯ a ) can be largely changed in either the same or the opposite direction of TD-error. Intriguingly, Eq. 3 indicates that we can control the generalization by mainly adjusting the kernel k ϕ (¯ s, ¯ a, s, a ) .

It is widely deemed that the crux of offline RL is the overestimation of OOD data, i.e., ( s, a ) / ∈ D . Recall the Backup-Generalization Cycle in Fig. 1. The overestimation propagates directly via backups and further spreads via generalization. Most existing works on offline RL focus on eliminating or suppressing overestimation by conservative value estimation [8, 10] or distribution constraint [4, 5], leading to various forms of C ( · ) . These works can be categorized as overestimation-control backups. However, since the source of overestimation is improper generalization, i.e., overgeneralization issue, we argue that it is inadequate to only cut the downstream while leave the upstream uncontrolled. Naturally, the generalization should also be carefully controlled as a preposed complement to the overestimation-control backups. Therefore, we take a further step and propose a novel method to control the generalization in the next subsection.

4.3 Reining Generalization via Kernel Control

Since the control of generalization (i.e., the upstream ) is neglected in most previous offline RL works, we propose a novel method to address the overgeneralization issue by controlling the kernel in Eq. 3. The key idea is a two-stage generalization control: Policy-Dataset Generalization Inhibition and Policy-OOD Generalization Inhibition. Fig. 2 illustrates the concept of the two stages and an intermediate scene between them, which are detailed below.

As illustrated in the left of Fig. 1, the first stage of generalization control is to inhibit the generalization from behavior policy distribution (or dataset) to learning policy distribution, i.e., suppress Q ( s, a ) Generalization - - - - - - - → Q ( s, π ( s )) for ( s, a ) ∈ D . Such kind of generalization is called extrapolation in the offline RL literature [1]. Extrapolation is a main component of generalization as its consequence Q ( s, π ( s )) consistently participates in the backups. Thus, overestimation caused by improper extrapolation severely handicaps the learning. For a stable and effective learning process, we argue that extrapolation should be inhibited, especially at the early learning stage, where the learning policy is usually largely different with the behavior policy. This can be achieved by encouraging the kernel to 0, i.e., min ϕ |∇ ϕ Q ϕ ( s, a ) ⊤ ∇ ϕ Q ϕ ( s, π ( s )) | . Consequently, training on ( s, a ) ∈ D induces only a minor change to Q ϕ ( s, π ( s )) .

With effective suppression of overestimation achieved by the first stage of generalization inhibition, we further consider an intermediate scene where the learning policy evolves and resembles the behavioral policy as the training proceeds, leading to a distribution overlap. In this scene, the effect of the first stage could become over-inhibition. In another word, min ϕ |∇ ϕ Q ϕ ( s, a ) ⊤ ∇ ϕ Q ϕ ( s, π ( s )) | for a ≈ π ( s ) could lead to a sharp and bumpy Q -value landscape as shown in the middle of Fig. 2. It adversely impacts the robustness of Q function, further resulting in a brittle policy. Hence, it's unwise

to impose the Policy-Dataset Generalization Inhibition all along the learning process. For some empirical evidence of this, we observe performance decay in some of our experiments in Table 3 and Fig.4. A natural remedy is to gradually remove the inhibition. However, it leaves value generalization uncontrolled once again, exposing the policy learning under the risk of collapse due to overestimation.

To address the dilemma above, we introduce the second stage, Policy-OOD Generalization Inhibition, as illustrated in the right of Fig. 2. In this stage, we make use of an additional policy π ood that outputs OOD actions . The OOD action here is defined as an action with a lower Q -value than the action selected by the learning policy for the same state. The actual effect of the second stage is suppression of the generalization between the learning policy distribution and OOD policy distribution. Such a means of kernel control can be viewed as a special way of increasing the action gap [42]. It improves the robustness of value function learning by discouraging erroneous generalization on suboptimal actions. Formally, this is achieved by min ϕ |∇ ϕ Q ϕ ( s, π ( s )) ⊤ ∇ ϕ Q ϕ ( s, π ood ( s )) | .

Overall, our kernel control method consists of the two stages of generalization inhibition, which effectively controls the generalization over the entire course of learning, providing a favorable upstream for better offline RL.

5 Representation Distinction in Offline RL

In this section, we devise a practical algorithm to rein the generalization in offline RL following the design of two stages of generalization inhibition. A heuristic method to achieve smooth transition among them is also proposed. We term our algorithm of reining generalization as Representation Distinction (RD) and apply them to both existing and designed backbone algorithms.

5.1 Practical Algorithm Design

We first focus on the phase depicted in the left of Fig.2. A natural approach is to directly minimize E s,a ∼D | k ϕ ( s, π ( s ) , s, a ) | across the dataset. However, due to the high dimensionality of ∇ ϕ Q ϕ ( s, a ) , the direct computation of E s,a ∼D | k ϕ ( s, π ( s ) , s, a ) | involves the computation and backpropagation through per-example gradient dot products, which is computationally prohibitive. To mitigate this, we adopt the method proposed by [33] and approximate ∆( ϕ ) with the contribution solely from the last layer parameters, i.e., E s,a ∼D |∇ w Q ϕ ( s, π ( s )) ⊤ ∇ w Q ϕ ( s, a ) | . Consequently, we derive the following loss function:

where Φ( s, a ) signifies the representation of state-action pairs as mentioned in Section 3. By minimizing L 1 , we encourage the learned Q-function to yield representations that are as orthogonal as possible between data from dataset and π . In this manner, the representations of ( s, a ) and ( s, π ( s )) can be distinctly discerned.

To substantiate the efficacy of inhibiting generalization between in-sample data and data derived from the learning policy π , we conduct rudimentary experiments by directly minimizing L 1 during the training of the standard off-policy SAC algorithm [2] on both halfcheetah-medium and halfcheetah-medium-expert. The evaluation results depicted in Fig.3 demonstrate that enhancing the representation alone, devoid

0

250

500

Training Steps (1e3)

-20

0

20

40

60

80

Normalized Returns

SAC w/o RD

SAC with RD

0

250

500

750

1000

Training Steps (1e3)

0

20

40

60

80

Normalized Returns

SAC w/o RD

SAC with RD

of explicit pessimism, can achieve commendable performance, substantially surpassing the baseline SAC. These outcomes underline the potency of inhibiting generalization to eliminate overestimation in offline setting.

Further, we consider the phase in the right of Fig.2. To improve the robustness of the Q function by circumventing the excessive generalization of the Q-function to OOD data, we introduce a simple OOD data generator that creates data with lower Q-values compared to the data produced by current policy π . Specifically, the OOD data generator π ood is trained to generate OOD state-action pairs that satisfy the following condition: α ∗ Q ϕ ( s, a ) < Q ϕ ( s, π ood ( s )) < β ∗ Q ϕ ( s, a ) , where Q ϕ ( s, a ) and Q ϕ ( s, π ood ( s )) are assumed to be positive for ease of presentation, and 0 < α < β < 1 is the predefined hyperparameters. In practice, the optimization of π ood is implemented via the following margin ranking loss:

As the offline RL training progresses, we gradually shift our focus from distinguishing in-sample data and data generated by π to distinguishing data generated by π and data generated by the OOD actor π ood in the representation space, i.e., L 2 = E s ∼D | Φ( s, π ood ( s )) ⊤ Φ( s, π ( s )) | . In practice, a simple heuristic approach is proposed to help balance L 1 and L 2 :

where w gradually increases according to w = tanh( step_num M ) , step_num refers to the current training step, and M is a hyperparameter. During the update of π ood, the gradient towards Q ϕ is detached.

We would like to emphasize that the OOD generator is not required to be meticulously trained, as we only expect the Q value of the generated OOD data to be appropriately lower than that of the data generated by the current policy. This is due to the fact that the OOD data is not employed to directly influence the policy's decision-making process. Instead, it serves to inhibit the excessive generalization of the Q function, thereby enhancing the robustness.

5.2 Applying RD in Offline RL Algorithms

While the introducing of RD to the offline learning process offers benefits, it alone cannot boost the performance of a standard online RL algorithm to a cutting-edge level, as depicted in Fig.3. This is because merely reining generalization from the representation perspective does not necessarily prevent overestimation of the Q-value. Our objective is to construct RD as a universally applicable plug-in to enhance any offline RL algorithm. Thus, to evaluate RD's versatility as a plug-in, we apply it in both existing algorithms and two designed rudimentary backbone algorithms named SAC-N-Unc and TD3-N-Unc. By modifying the original SAC/TD3 algorithm to employ a critic ensemble of number N and incorporate an uncertainty regularization term within the policy update process, we derive these backbone algorithms. It should be noted that these algorithms are not the original contribution of this paper as many similar uncertainty-based methods [10, 43, 12, 44, 45, 46] have been proposed. Details of the backbone algorithms are provided in the appendix.

Note that RD is algorithm-agnostic, meaning it can be applied to a wide range of actor-critic offline RL algorithms. Incorporating RD into offline RL algorithms is also straightforward, requiring only

the addition of two steps to the conventional policy updating scheme, as illustrated in Algorithm 1 in the appendix. In each training iteration, the OOD data generator is first trained to generate data with lower Q-values than the current policy. Then, the Q-function is updated by minimizing the loss function L = ϵ ∗ L RD + L critic, and the policy is updated based on L actor. Here, ϵ represents a weighted factor, and L critic and L actor refer to the optimization objectives of any offline RL algorithm. A psedudocode using RD in offline RL is given in Algorothm 1. In the experimental section, we demonstrate the versatility of RD by integrating it into both backbone algorithms and existing offline RL algorithms.

6 Experiments

We evaluate RD by applying it on TD3BC [7], CQL [8], SAC-N-Unc and TD3-N-Unc through a series of experiments on D4RL [47] gym MuJoCo-v2 and Adroit-v1 datasets, where the former is a dataset commonly used in previous work for continuous control tasks, and the latter poses a significant challenge for most offline RL methods due to its sparse reward property. We compare our method with BC and several model-free offline RL algorithms, including DT [48], TD3BC [7], CQL [8], IQL [49], EDAC [10], and Diffusion-QL [50]. We obtain the results of the baselines by re-running the official codes or directly extracting them from the original papers. In our experiments, all the re-run baselines and our algorithm are executed with five random seeds, and we report the average normalized results of the final ten evaluations. We report the performance at 1M gradient step for TD3BC and CQL and that at 3M gradient step for SAC-N-Unc and TD3-N-Unc on MuJoCo tasks. For Adroit tasks, we report results at 500K gradient step. Note that due to the space limitation, we abbreviate the names of the datasets from {RANDOM, MEDIUM, MEDIUM-REPLAY, MEDIUM-EXPERT, EXPERT} to {R, M, MR, ME, E} in all the tables.

6.1 Main Results

To demonstrate the applicability of RD, we integrate RD into the training regime of TD3BC, CQL, TD3-N-Unc and SAC-N-Unc. As observed from Table 1, both TD3BC and CQL exhibit improved overall performance when RD is applied. It is noteworthy as the core idea of CQL to increase the Q value of the data in the dataset and diminish the Q value of the data generated by a mixed policy comprising the learning policy and random policy, which incorporates the representation distinction of state-action pairs sourced from different distributions. Therefore in practice, we apply the insights of RD to gradually transfer the original Q value restriction to the Q-value differentiation between the mixed policy and the designed OOD policy. As a result, CQL's performance is significantly improved. Moreover, the backbone algorithms themselves yield substantial advantages from the application of RD. Table 2 further reveals that backbone algorithms equipped with RD can achieve the best overall performance and attains state-of-the-art performance on several datasets. Additional evidence supporting the efficacy of RD in enhancing convergence speed and its potential to reduce the quantity of Q ensembles is provided in the appendix.

6.2 Comparison and discussion with techniques that inhibit overestimation

We compare our method RD with the existing techniques that inhibit potential overestimation including DR3 [41] and Layer Norm [51]. In addition to the regularizer, all the other parameters are kept the same across different methods to ensure fairness. As shown in Table 3, RD is more helpful in helping improving the backbone algorithm than DR3 and Layer Norm.

RD and DR3 differ mainly from two perspectives. From the angle of the framework derived from, DR3 is derived from the theoretical characterizing implicit regularization in TD-Learning, which is a generalization of the implicit regularization from Supervised Learning [52, 53] to TD-Learning in RL setting. In contrast, RD is derived from the proposed Backup-Generalization framework. From the angle of regularization effect, DR3 is proposed to directly counter the implicit regularization of TD-Learning while RD is to suppress the generalization between in-sample data and OOD data. With some heuristics, DR3 regularizer arrives at a similar form, i.e., to minimize the NTK between consecutive state-action pairs in backup, to RD regularizer. Apparently, DR3 regularizer can be a special case of RD regularizer in terms of the definition of out-of-sample actions. The regularization effect of DR3 is similar to the Policy-Dataset Generalization Inhibition shown by Figure 2. Such a regularization can induce over-inhibition of generalization when the distribution of current policy overlaps with the offline dataset as illustrated. We also show in the following that DR3 and PolicyDataset Generalization Inhibition (i.e., PDD in the following part) achieve similar empirical results.

6.3 Ablation Studies and In-depth Analysis

We design ablation experiments to elucidate the significance of RD's components. We compare RD against several variants including, including Policy-Dataset Distinction (PDD), Random-Dataset Distinction (RDD), Policy-OOD Distinction (POD), and Policy-Dataset Distinction with Dynamically Adjusted Weight (PDDDAW). PDD refers to the auxiliary representation loss calculated using data generated by the learning policy and that from the dataset. Likewise, RDD utilizes data generated by a random policy versus that from the dataset, while POD employs data produced by the learning policy against that by the OOD actor. As for PDDDAW, its PDD component is reweighted using the same heuristic w adjustment method as RD, but the POD component is persistently weighted by zero. We draw a comparison between TD3-N-Unc + RD and these variants, as tabulated in Table 3. The results demonstrate that RD, fundamentally a dynamic amalgamation of PDD and POD, attains superior overall performance, underscoring the indispensability of each RD component.

0

500

1000

1500

2000

2500

3000

Training Steps (1e3)

0

20

40

60

80

100

Normalized Returns

halfcheetah-expert-v2

PDD

RD

Q-value

Q-value

Figure 6: T-SNE visualization of representation via RD.

It is worth noting that using PDD can help backbone algorithms achieve relatively satisfactory performance. However, as depicted in Fig 4, algorithm using PDD occasionally suffers performance degradation during the later stages of training, consequently impairing the overall efficacy of the approach. To elucidate this phenomenon, the final models trained via RD and PDD, as shown in Fig 4, are saved and the quality of the representations obtained by these models are compared. A total of 100 state-action pairs are sampled from the halfcheetah-expert, halfcheetah-medium, and halfcheetah-random datasets, respectively, and then consolidated. Using t-Distributed Stochastic Neighbor Embedding (t-SNE), the distribution of the representations of these 300 state-action pairs derived from both models is charted. Each sample is marked with distinctive symbols (stars for random data, squares for medium data, and circles for expert data) to differentiate samples from diverse datasets. Moreover, varying colors are assigned to individual samples corresponding to the Q-value estimated by the proficiently trained Q-value network acquired by RD, with brighter colors signifying higher Q-values. As depicted in Fig 5 and 6, the RD-derived representation exhibits superior differentiation between state-action pairs with high and low Q-values, whereas the PDD-derived representation frequently misclassifies expert and medium data.

To better demonstrate that the representations learned by RD are superior, we calculate which dataset the top five nearest samples to each sample belong to, and compute the mean over all samples from the same dataset in Table 4. For instance, the EXPERT-EXPERT metric of representation learned via RD indicates that 94% of the five nearest samples to the expert samples are also expert samples. We can conclude that representation learned via RD can better help samples from the same dataset to cluster closer. More results are provided in the appendix.

7 Conclusions and Limitations

This paper analyze the important role of generalization in offline RL and presents a novel plug-in method, Representation Distinction (RD), to enhance the performance of offline RL algorithms by reining the generalization. By explicitly differentiating between in-sample and OOD state-action pairs, the generalization across data could be properly inhibited. Our extensive experiments demonstrate the efficacy of the RD method when applied to several offline RL algorithms, significantly enhancing their performance across various continuous control tasks on D4RL datasets. In conclusion, our work contributes a valuable perspective to the field of offline RL by focusing on reining generalization from the representation perspective and providing a flexible solution that can be applied to a variety of existing algorithms.

However, our work is not without limitations. For example, currently RD primarily focus on continuous control tasks, and its applicability to discrete control tasks is yet to be explored. In addition, the transition between the two phases of RD is a heuristic method, which could be improved by designing adaptive adjustment methods. Future research directions may include addressing the aforementioned limitation.

References

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv:2005.01643 [cs, stat] , November 2020. arXiv: 2005.01643.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.04745 , 2022.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems , 34:7436-7447, 2021.

Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202.11566 , 2022.

Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. arXiv preprint arXiv:2206.02829 , 2022.

Xinyang Geng, Kevin Li, Abhishek Gupta, Aviral Kumar, and Sergey Levine. Effective offline rl needs going beyond pessimism: Representations and distributional shift. In Decision Awareness in Reinforcement Learning Workshop at ICML 2022 , 2022.

Mengjiao Yang and Ofir Nachum. Representation matters: offline pretraining for sequential decision making. In International Conference on Machine Learning , pages 11784-11794. PMLR, 2021.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 , 2020.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning , pages 1719-1735. PMLR, 2021.

Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable advantage-weighted policy optimization for offline rl. arXiv preprint arXiv:2203.08949 , 2022.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142, 2020.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems , 33:21810-21823, 2020.

Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. arXiv preprint arXiv:2008.05556 , 2020.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems , 34:28954-28967, 2021.

Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204.12581 , 2022.

Kaiyang Guo, Yunfeng Shao, and Yanhui Geng. Model-based offline reinforcement learning with pessimism-modulated dynamics belief. arXiv preprint arXiv:2210.06692 , 2022.

Hamid Maei, Csaba Szepesvari, Shalabh Bhatnagar, Doina Precup, David Silver, and Richard S Sutton. Convergent temporal-difference learning with arbitrary smooth function approximation. Advances in neural information processing systems , 22, 2009.

Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint arXiv:1903.08894 , 2019.

Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In International Conference on Machine Learning , pages 767-777. PMLR, 2020.

Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems , 33:18560-18572, 2020.

Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204.09560 , 2022.

Chenjun Xiao, Bo Dai, Jincheng Mei, Oscar A Ramirez, Ramki Gummadi, Chris Harris, and Dale Schuurmans. Understanding and leveraging overparameterization in recursive value estimation. In International Conference on Learning Representations , 2022.

Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations , 2021.

Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations , 2022.

Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski, Georg Ostrovski, Mehrdad Farajtabar, Matt Hoffman, Razvan Pascanu, and Arnaud Doucet. An empirical study of implicit regularization in deep offline rl. arXiv preprint arXiv:2207.02099 , 2022.

Wesley Chung, Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for nonlinear value function approximation. In International Conference on Learning Representations , 2019.

Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Marc G Bellemare, and David Silver. The value-improvement path: Towards better representations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7160-7168, 2021.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning , volume 119, pages 5639-5650, 2020.

Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In Advances in Neural Information Processing Systems , 2020.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning , volume 139, pages 11920-11931, 2021.

Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations , 2021.

Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations .

Marc G Bellemare, Georg Ostrovski, Arthur Guez, Philip Thomas, and Rémi Munos. Increasing the action gap: New operators for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30, 2016.

Alex Beeson and Giovanni Montana. Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning. arXiv preprint arXiv:2303.14716 , 2023.

Hongchang Zhang, Boyuan Wang, Yixiu Mao, Jianzhun Shao, Yuhang Jiang, Yi Xu, and Xiangyang Ji. Pessimistic policy iteration for offline reinforcement learning, 2023.

Kamyar Ghasemipour, Shixiang (Shane) Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 18267-18281. Curran Associates, Inc., 2022.

Jordi Smit, Canmanie T Ponnambalam, Matthijs TJ Spaan, and Frans A Oliehoek. Pebl: Pessimistic ensembles for offline deep reinforcement learning. In Robust and Reliable Autonomy in the Wild Workshop at the 30th International Joint Conference of Artificial Intelligence , 2021.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , 2022.

Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 , 2022.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on learning theory , pages 483-513. PMLR, 2020.

Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems , 34:27449-27461, 2021.

Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as a 'Launchpad' , 2022.

Peer reviews:
comments: SUMMARY:
Authors consider overgeneralization in offline RL which is new and propose the modification which helps to solve the problem and  can be applied to any algorithm.

STRENGTHS:
Proposed method boosted performance of different algorithms by a notable margin in most of the cases and it can be integrated in any algorithm while being simple. Good ablation study.

WEAKNESSES:
The problem epxeriments is that algorithms are tested using MuJoCo datasets and  2 pen datasets and  appeares not to be sufficient. For example, in CORL (https://arxiv.org/abs/2210.07105) benchmarks which you mention and  in original work (https://arxiv.org/abs/2110.01548), it is seen that EDAC performs great on the MuJoCo but fails on AntMaze (the recent update of CORL tested even more datasets and EDAC seems to fail on Adroit too). There are already some algorithms which are competitive or outperform EDAC on MuJoCo while working better on other datasets, see https://arxiv.org/abs/2206.02829 as an example. Some algorithms can performe not that good on MuJoCo but outperform others on different datasets, see concurrent work (https://arxiv.org/abs/2305.09836) or IQL in CORL.

I suggest to run more experiments using AntMaze or all of the Adroit datasets to verify that your approach helps. We can't conclude that it helps without checking more datasets. Wihout it I'm not sure that paper should be accepted.

QUESTIONS:
How does you modification affect training runtime? Is the change significant or not?

LIMITATIONS:
Limitations are not clear without additional benchmarking or runtime information.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigated the effect of generalization in overestimation of critic values and devised an algorithm that can mitigate overgeneralization. The proposed method, Representation Distinction (RD), is orthogonal to most offline RL algorithms and thus be applied to most of them. Experimental results show that RD significantly enhances the performance of baseline algorithms.\n\nSTRENGTHS:\n### Originality\nThe paper introduces the Backup-Generalization framework to analyze the overestimation of critic values.\nThe authors devised a novel method of learning a suboptimal policy to sample OOD actions and use them in the later stage of training. They also proposed a simple and effective heuristic that allows a natural transition from PDD to POD as the training progresses.\n### Quality\nMost of the claims are technically sound. The authors have analyzed the performance of their method on popular offline RL benchmarks and conducted multiple interesting ablation studies.\n### Clarity\nThe paper is overall well-written and easy to understand.\n### Significance\nMost works on offline RL focus on the backup step. This work instead points out the role of generalization in overestimation of critic values, which seems like a promising research direction. Also, the proposed method can be applied to most existing offline RL algorithms to improve their performance.\n\nWEAKNESSES:\n1. Line 315~316 is difficult to understand. A pseudocode of CQL+RD will be helpful.\n2. A theoretical analysis of how RD differs from DR3 (Kumar et al. 2022) and why using OOD actions is better than just optimizing the DR3 auxiliary loss, $\\mathbb{E}_{(s, a, s', a')\\sim\\mathcal{D}}\\left[\\Phi(s', a')^\\intercal\\Phi(s, a)\\right]$, is missing from the paper. Adding this will improve the originality of the paper, as this work looks like a slight modification of DR3 at first glance. Also, the authors do not compare their algorithm with DR3 variants of CQL, TD3-BC, and SAC. Although I expect their performance to be on par with the PDD variants, I believe they should be included in the experiments section.\n3. TD3-N-Unc and SAC-N-Unc use an ensemble of N critics. It is unfair to compare their performance with other baselines that only use one critic.\n\n### References\nAviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Value-based deep reinforcement learning requires explicit regularization. In *International Conference on Learning Representations*, 2022.\n\nQUESTIONS:\nTable 3 reports the average normalized score of TD3-N-Unc on the Walker2D-Expert domain to be $4.1\\pm 7.1$\nand Table 1 reports the average normalized score of TD3-N-Unc on the Waler2D-Medium domain to be $69.9\\pm 35.2$. Why does TD3-N-Unc perform a lot better on the medium dataset?\n\n&nbsp;\n\n### Minor suggestions\n\nEq. (4) It would be easier for the readers to follow if the authors explicitly mention that $\\nabla_{\\mathbf{w}}Q_\\phi(s, a)=\\Phi(s, a)$.\n\nLine 315: gruadually $\\to$ gradually\n\nThere are duplicate references: [33] and [41] are both Kumar et al. (2022).\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper explains the necessity of reining generalization in offline RL and proposed a new method Representation Distinction (RD) based on this principle. RD shows good performance on D4RL benchmarks.

STRENGTHS:
- This paper presented a view called Backup-Generalization Cycle, which explains that the generalization issue should be considered in the training phase.
- This experiment shown in Figure 3 is interesting, which demonstrates overgeneralization is an important issue in offline RL and the method is able to mitigate it.
- By applying RD to existing algorithms, their performance is improved significantly.

WEAKNESSES:
- There is a gap between analysis and practical implementation. Therefore, it is difficult to identify whether the performance improvement comes from the overgeneralization issue. 
- The algorithm introduces some hyperparameters which may make it a little difficult to tune.
- The difference between PDD and RD is not significant.

QUESTIONS:
- The experiment in Figure 3 is only conducted on halfcheetah environment, which is not very convincing as halfcheetah has some special features. Can the authors provide more results on other environments?
- In section 6.2, the authors only demonstrated the representation of  PDD and RD. While I'm curious about the representation of the original algorithm.

LIMITATIONS:
The limitation of this work is that it is limited to continuous control tasks.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a flexible plug-in method called Representation Distinction (RD) to address the overgeneralization issue in offline Reinforcement Learning (RL) algorithms. The authors formalizes the process of generalization and investigates the potential to rein the generalization from the representation perspective to enhance offline RL, performing both Policy-Dataset and Policy OOD Generalization Inhibition with awareness of the two possible phases in the offline learning course. The proposed method explicitly differentiates between the representations of in-sample and out-of-distribution state-action pairs generated by the learning policy, which significantly improves the performance of the backbone and widely-used offline RL algorithms across various continuous control tasks on D4RL datasets.\n\nSTRENGTHS:\n+ **Clarity**: the paper is well-structured and clear to follow.\n+ **Flexibility and Good Performance**: the authors propose a novel method called Representation Distinction (RD) to improve the performance of offline reinforcement learning algorithms, which explicitly differentiates between the representations of in-sample and out-of-distribution (OOD) state-action pairs generated by the learning policy, demonstrating the efficacy of the proposed approach by flexibly applying RD to specially-designed backbone algorithms and widely-used offline RL algorithms.\n+ **Insight**: the paper provides an insightful, nuanced and systematic formalization of the process of generalization in offline RL and then investigates the prevalent overgeneralization issue in offline RL.\n\nWEAKNESSES:\n+ **Missing Citations or Clear Indications**: some recent papers also utilize NTK to analyze the genralization ability of offline RL on similar considerations with RD [1, 2, 3]. It is advisable to provide more comparisons and at least analyses on the way different NTK-relevant approaches address generalization issues in offline RL.\n+ **Evaluation**: questions about experimental setups and results are listed in the next section.\n\n[1] Kumar et al. DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization. ICLR 2022.\n\n[2] Ghasemipour et al. Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters. NeurIPS 2022.\n\n[3] Li et al. When data geometry meets deep function: Generalizing offline reinforcement learning. ICLR 2023.\n\nQUESTIONS:\n**Methodology**\n+ It is recommended to include more comparisons and conduct thorough analyses on how RD and other existing NTK-relevant approaches address generalization issues in offline RL. This would enhance the comprehensiveness and depth of the discussion.\n+ As far as I am concerned, RD via kernel control is doing value regularization implicitly on representation level. It would be beneficial to provide a clearer and more insightful explanation of the potential advantages of applying RD via kernel control instead of value regularization and policy constraint, particularly in the context of offline RL algorithms. \n\n**Evaluation**\n+ There appears to be an inconsistency between the results of SAC/TD3-N-Unc+RD in Table 1 and Table 2. It is important to address this discrepancy and provide clarification or additional information to ensure the accuracy and reliability of the reported findings.\n+ Considering that well-established techniques that inhibit potential overestimation induced by extrapolation already applied into reinforcement learning, such as Lipchitz Regularization with Spectral Norm [4] and Layer Norm [5], are flexible and widely applicable, it would be worth discussing and experimenting with the possibility of leveraging these existing and simpler methods to effectively constrain and smooth the critic approximation, thereby preventing overgeneralization.\n+ The comparison between curves obtained from a single seed in Figure 4 lacks experimental validity and may raise concerns about the reliability of the results. It is essential to address this limitation and consider conducting experiments with multiple seeds to improve the robustness and credibility of the presented findings.\n\n[4] Gogianu et al.  Spectral normalisation for deep reinforcement learning: An optimisation perspective. ICML 2021.\n\n[5] Ba et al. Layer normalization. Advances in NeurIPS 2016 Deep Learning Symposium, 2016.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}