Reining Generalization in Offline Reinforcement Learning via Representation Distinction
Authors: Yi Ma
Hongyao Tang
Dong Li
Zhaopeng Meng
Venue: NeurIPS 2023 poster (2023)
Keywords: Offline Reinforcement Learning

Abstract: Offline Reinforcement Learning (RL) aims to address the challenge of distribution shift between the dataset and the learned policy, where the value of out-of-distribution (OOD) data may be erroneously estimated due to overgeneralization. It has been observed that a considerable portion of the benefits derived from the conservative terms designed by existing offline RL approaches originates from their impact on the learned representation. This observation prompts us to scrutinize the learning dynamics of offline RL, formalize the process of generalization, and delve into the prevalent overgeneralization issue in offline RL. We then investigate the potential to rein the generalization from the representation perspective to enhance offline RL. Finally, we present  Representation Distinction (RD), an innovative plug-in method for improving offline RL algorithm performance by explicitly differentiating between the representations of in-sample and OOD state-action pairs generated by the learning policy. Considering scenarios in which the learning policy mirrors the behavioral policy and similar samples may be erroneously distinguished, we suggest a dynamic adjustment mechanism for RD based on an OOD data generator to prevent data representation collapse and further enhance policy performance. We demonstrate the efficacy of our approach by applying RD to specially-designed backbone algorithms and widely-used offline RL algorithms. The proposed RD method significantly improves their performance across various continuous control tasks on D4RL datasets, surpassing several state-of-the-art offline RL algorithms.

Reining Generalization in Offline Reinforcement Learning via Representation Distinction
Yi Ma
Hongyao Tang"
College of Intelligence and Computing Tianjin University mayi@t @t ju. edu. cn
Université de Montréal Mila tang www.yaommilal...oomoooommmmmmaam queeee
Dong Li
Zhaopeng Meng
Noah's Ark Lab, Huawei Technology dongleecsu@gmail.co com
College of Intelligence and Computing
Tianjin University mengzpētju adu..edu.com cn
Abstract
Offline Reinforcement Learning (RL) aims to address the challenge of distribu- tion shift between the dataset and the learned policy, where the value of out-of- distribution (OOD) data may be erroneously estimated due to overgeneralization It has been observed that considerable portion of the benefits derived from the conservative terms designed by existing offline RL approaches originates from their impact on the learned representation. This observation prompts us to scrutinize the learning dynamics of offline RL, formalize the process of generalization, and delve into the prevalent overgeneralization issue in offline RI.. We then investigate the potential to rein the generalization from the representation perspective to enhance offline RI. Finally, we present Representation Distinction (RD), an innovative plug-in method for improving offline RL algorithm performance by explicitly diffcrentiating between the representations of in-sample and OOD state-action pairs generated by the learning policy. Considering scenarios in which the learn- ing policy mirrors the behavioral policy and similar samples may be erroneously distinguished, we suggest dynamic adjustment mechanism for RD based OOD data generator to prevent data representation collapse and further enhance policy performance. We demonstrate the efficacy of our approach by applying RD to designed backbone algorithms and widely-used offline RI algorithms. The proposed RD method significantly improves their performance across various tinuous control tasks on D4RL datasets, surpassing several state-of-the-an offline RL algorithms.
Introduction
Reinforcement learning (RL) is machine learning paradigm centered on training intelligent agents to make decisions through interactions with an environment. In this learning process, an agent develops a policy-a mapping from states actions-via. via trial and error, with the goal of maximizing cumulative rewards over time. However, acquiring interaction data in many real-world applications can be expensive, time-consuming, or even hazardous. This spurred the advancement of offline Reinforcement Learning, which has garnered considerable interest in the research community due to its ability learn effective policies from pre-existing data collected by unknown behavioral policy
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

without necessitating online interactions with the environment. Offline RL can result in substantial savings in resources, time, and the risk associated with online exploration.
Despite its appealing potential, offline RI. encounters challenges in learning optimal policies from lim- ited and suboptimal datasets. The primary challenge in offline RI. is the distribution shift between the dataset and the learned policy. As the agent updates its policy, it may come across out-of-distribution (OOD) state-action pairs absent from the limited support of the fixed offline dataset [. Conventional RL algorithms, such as SAC (2) or TD3 [3, could yield overly optimistic Q-value estimations for these unseen state-action pairs due to overgeneralization, leading to catastrophic performance. Existing methods in offline RL seek to address this challenge by incorporating techniques like policy
constraint 4 [ 6
7 conservative value estimates, [8 [9, and uncertainty estimation [10 [2]
13 elucidated that the ability of existing offline reinforcement learning (RI.) methods to handle OOD actions is significantly attributed their impact on the learned representations of state-action pairs. The learned representation can capture the underlying structure and essential aspects of the state-action space, enabling the agent to identify similarities and patterns across various state-action pairs, thus improving agent's 's learning efficacy and generalization capacity. Although there are few works [4 also study representation learning in offline RL, they focus on how different kinds of pretrained representation matter in downstream policy learning, neglecting the understanding of the essential interplay between representation and policy during the offline co-learning process.
Recently,
In this paper, we first present a view dubbed Tackup-Generalizn hension of how typical offline RI algorithm learns. We subsequently formalize how generalization happens, and discuss the overgeneralization issue prevalent in offiin RI. Finally, we introduce a novel method called Representation Distinction (RD) to mitigate overgeneralization from in-sample state-action pairs to OOD ones. Specifically, we distinctly differentiate the data sourced from the dataset and the data generated by the learning policy from the representation perspective. In cases where the learning policy's 's performance aligns with the behavioral policy and the generated data largely overlaps with the dataset, we devise OOD data generator produce data with lower Q-values than the current policy. We then gradually shift our focus from differentiating in- in-sample data and data generated by the learning policy to differentiating data generated by the learning policy and data generated by the OOD actor in the representation space, preventing similar samples from being erroneously distinguished.
Cycle to gain a tangible compre-
To demonstrate the efficacy of our proposed RD method in enhancing offline RL, policy performance by reining the generalization, we apply it two backbone algorithms, which are modified versions of the original SAC and TD3 algorithms by incorporating an ensemble of critics and an uncertainty regularization term in the policy update process. By integrating RD into these simple algorithms, our proposed method RD can improve the performance of the backbone agents on D4RL datasets across various continuous control tasks and outperforms several state-of-the-an offline RL algorithms. We also apply RD on two widely-used baselines TD3BC and CQL and improve their performance significantly. In-depth analysis based on visualization and statistical results that demonstrates the efficacy of RD in obtaining more differentiable representation compared to its variants is also provided.
The main contributions of this work can be summarized as follows:
We introduce view called Backup-Generalizzttin Cycle to foster an understanding of typical offline value function learning, and highlights the necessity of reining generalization to enhance offline RI.
We proposed a practical plug-in method Representation Distinction (RD) from the per- spective of representation to enhance the performance of offline RL methods by inhibiting overgeneralization among state-action pairs sourced from different distributions.
We evaluate the effectiveness of RD by applying it to designed backbone algorithms and existing widely-used algorithms and enhance their performance significantly.
2 Related Works
Offfine RL Offline RL algorithms strive to train RL agents using pre-collected datasets. Never- theless, the distribution shift between the behavior policy and the policy being learned may lead to issues, as OOD actions are sampled from the learned policy and incorporated into the learned

---

critic. To address this challenge, various approaches have been proposed. Some earlier methods aim to constrain the learned policy to remain close to the behavior policy. which can be accomplished
through explicit policy regularization 4 S
19 or by employing auxiliary behavioral cloning losses 77]. Alternative approaches penalize the Q-value of
6 implicit policy constraints [15 16
I7
18]
OOD actions to discourage their selection 8 9 10 II 12 Moreover, model-based methods that
train with conservative penalties have been suggested 20 21 22
23 24 25).
Representation in Offline RI.. Prior research has aimed to analyze various aspects of the rep- resentations induced by TD)-based methods using function approximation, predominantly in the standard online RL setting [26
27
28
29
30
More recently, this line of inquiry has emerged in the investigates which representations can result in stable
offline RL setting [31 32
33
34
13I.
32 explore the learning dynamics of Q-learning in an overparameterized setting, observing excessively low-rank and aliased feature representations at the fixed points identified by TD-learning. 34 studies the extent of impact of different interventions on the causal link between the effective rank and offline RL agent performance and points out that there is no strong relationship between them. [13] highlights that a substantial portion of the benefits of existing offline RL approaches, which aim to avoid OOD actions, actually originates from their effects on the learned representations. The authors also identify specific metrics that facilitate effective evaluation of the quality of the learned representation. [1] pretrain the representation using different auxiliary loss and then fix the representation to apply it to downstream policy learning. Nevertheless, a thorough examination of the explicit representation distinction between in-sample and OOD state-action pairs in the context of offline RL is lacking. Our approach promotes orthogonality between the representation vectors of in-sample and OOD data, offering more practical solution for inhibiting overgeneralization, thus enhancing offline RL.
convergence of TD in linear setting.
3 Preliminary
Markov Decision Process (MDP) is a mathematical framework that models decision-making processes in stochastic environments. It is defined by tuple (S, A, P. R2 7), where S is the set of states, A is the set of actions, P is the state transition probability function, R is the reward function, and y is the discount factor. Q-learning is the major approach for obtaining the optimal rTss, which learns a Q- value function ((s, a) that represents the expected cumulative discounted rewards when starting from the state taking the action a and executing the policy π thereafter. the Q-value function evaluated by iterating the Bellman operator as TQ(s, a) = EEs~P(-)s,a)) (rss, a) + + (8', a²)]. The goal of RL is to find an optimal policy 7 (s) that maximizes the cumulative discounted rewards.
Offline Reinforcement Learning [1 aims to learn policy from fixed dataset of interaction samples, without further interaction with the environment. The dataset consists of transition tuples (s, a, T, s') collected from interactions between one multiple behavior policies and the environment. The goal is to learn an optimal policy using only this dataset.
To deal with large state-action space, deep RL resorts to deep neural networks for function approxi- mation. Typically, a Q-network parameterized by Φ can be viewed as Qs a) w T (bss a). where (s, a) E I2d is regarded as the representation obtained by the penultimate layer of Q-network while E R is the linear weight [35 36 33 13] In conventional online and offline RL learning process,
the representation and the linear weight learn together. A few recent works 37 38 39 40) also
introduce auxiliary objectives to strengthen regularize the representation with different purposes.
We use this view for the presentation of our analysis and proposed method in the following sections.
4 Reining Generalization in Offline RL
In this section, we focus on the learning dynamics of offline RL. First, we present view called Backuu--eneralizztton Cycle (Sec. to gain a tangible comprehension of how a typical offline RL algorithm learns. We then take a further step to formalize how generalization happens, and discuss the overgeneralization issue in offline RI. (Sec. (4.2. Finally, we propose the idea of kernel control to
address the overgeneralization issue (Sec. 4.3 with the practical implementation deferred to Sec.
3

---

(()
=(((....Tss))
Backup r ((Q((s,m(ss))
((s.a))
++C(U(m m(sn))
@ a)
((12.02)
r+ r(C(s
Generalization
as..
Figure 1: An illustration oenera--Generllzzttinnn Cycle. $0. 80, $ denotes four states in the offline dataset D. Backups and generalizations are denoted by blue and red arrows respectively.
4.1 Backup-Geenealizzatoo Cycle in Offline RL
Offline RL learns from static offline dataset with no access online interaction over the course of policy learning. To gain a better understanding of the intricacies of offline deep RL, for the first, we introduce a view called Seccku--eeneralization Cycle. This view, as depicted in Fig. T fosters an understanding of typical offline value function learning via two key components: Backup and Generalization
Typical offline deep RL algorithms employ various forms of backups to learn the Q-function network. Here we use C-) to denote a generic form of the target value such as Clipped Double Q-Learning [3]. is notable that only the values of state-action pairs in the offline dataset are directly updated.
Since the values of all state-action pairs that are not in the dataset will never be directly updated, changes to the values of such (s. a) D are solely indirectly instigated by the backups on (8, a) E DD i.e., through generalization. Note that the state-action pairs for the target values of backup as the learning of current policy I.
highly likely be absent from the dataset, i.e., (8', (ss))) D,
Hence, the typical offline learning process of value function can be regarded consequence the complex interplay between backup and generalization. This dynamic interplay forms cycle: (1) the backups on (s, a) E consistently influence the values of (s, a) & D (highly likely including (s', (s))))) (2) the consistently changing Q(s', , T(s') participates in the backups on (s, a) & D; the two kinds of dynamics iterate and twine during the learning process.
There are also some similar opinions mentioned in few recent works on offline RL 41 29)
We utilize the Backup-Generalizzn Generalization Cycle to distinctly separate and better analyze these two contributing factors. In the next subsection, we concentrate on the overgeneralization issue with formal characterization of value generalization among state-action pairs.
4.2
Overgeneralization in Offline RL
According to the Backup-Gennrrllizatton Cycle presented above, generalization plays significant role for (s, a) & 'D. contrast to backup, which is relatively explicit and controllable, generalization is implicit and intricate. In the following of this subsection, we analyze value generalization among state-action pairs during the offline RL learning process.
For starting case, we consider how the Q function update caused by typical Demferal- (TD) learning on single state-action pair (s, a) D (denoted as - o'), affects the Q-value of an arbitrary state-action pair (5, a). The post-update parameter ø' can be formalized as follows:
o' = (TQa(s, Qu a)) Qo),s,))
(I)
where learning rate is omitted for convenience. We further formalize the post-update Q-value of (s, á) by Taylor expansion at the pre-update parameter
(5, = Qa VoQo(5, ()) + (10' ((0'-w1()
(2)
Now, we can characterize the generalization from (s, a) to (8, a) by plugging Eq into Eq 2
+ -Q((s. + O (Ilo - +bl?)
(3)

---

Policy-Dalaset Generalization Inhibition
Over-Inhibition when Distributions Overtap
Policy-OOD Generalization Inhibition
Leaming
Generalization
Generalization
Over-Inhibiiton
0 0g)
(()))
(G))
+ ((Q)).
((x-)
((sc)
QG.a.))
((
CS (s)))
Behavioral Policy Distribution
Leaming Policy Distribution
Leamed OOD Policy Distribution
Figure 2: A conceptual illustration of Reining Generalization via Kernel Control.
where (3,3, s, a) = (oQa(5,)) (oooo) a) and is called Neural Tangent Kernel in 27 Eq elucidates the change the Q-value of any state-action pair (3, @) can be mainly characterized by the kernel (3, a, 5, a) and the TD-error (Qa a) (s a). Apparently, the kernel controls the extent generalization: for small values of (5, a, s, a), the generalization effect minor and the classic tabular learning (i.e., generalization) can be viewed as special case when KG and the higher-order term are 0; for large values of kgg (8, 8,00 a. s, a), the generalization effect is prominent and Q (8, a) be largely changed in either the same or the opposite direction of TD--rror. Intriguingly, Eq. 3 indicates that we can control the generalization by mainly adjusting the kernel (
(8, a, a).
It is widely deemed that the of offline RL is the overestimation of OOD data, i... (s. D. Recall the Generaliztaiiiiooo Cycle in Fig. The overestimation propagates directly via backups and further spreads via generalization. Most existing works on offline RL focus eliminating or suppressing overestimation by conservative value estimation [8] [0] or distribution constraint [4 [5],
leading to various forms of C(.). These works be categorized as overestimation-comme control backups. However, since the source overestimation is improper generalization, i.e., overgeneralization issue, argue that it inadequate to only cut the downstream while leave the upstream uncontrolled. Naturally, the generalization should also be carefully controlled as preposed complement to the overestimation-conro backups. Therefore, take further step and propose novel method to control the generalization the next subsection.
4.3 Reining Generalization via Kernel Control
Since the control of generalization (i.c., the upstream) is neglected in most previous offline RL works, we propose novel method to address the overgeneralization issue by controlling the kernel in Eq. 3. The key idea is two-stage generalization control: Volicy-Datased Generalization Inhibition and Policy-OOD Generalization Inhibition. Fig. 2 iillustrates the concept of the two stages and an intermediate scene between them, which are detailed below.
As illustrated in the left of Fig. [ the first stage of generalization control is inhibit the general- ization from behavior policy distribution (or dataset) to learning policy distribution, i.e., suppress Generalization
Q(s, π (s)) for (s. E Such kind of generalization is called extrapolation in the offline RL literature []. Extrapolation is main component of generalization as its conse- quence Q(s, x (s)) consistently participates the backups. Thus, overestimation caused by improper extrapolation severely handicaps the learning. For stable and effective learning process, we argue that cxtrapolation should be inhibited, especially at the early learning stage, where the learning policy usually largely different with the behavior policy. This can be achieved by encouraging the kernel to O, i.c., ming Vo a) VoQo(s {s))]- Consequently, training on (s, a) E D induces only a minor change to (o(s,T(s))
Q(s,a) a)
With effective suppression of overestimation achieved by the first stage of generalization inhibition. we further consider an intermediate scene where the learning policy evolves and resembles the behavioral policy as training proceeds, leading to a distribution overlap. this scene, the effect of the first stage could become over-inhibition. In another word, minş (= a) VaQa(S x(s)) for a As π (s)) could lead to sharp and bumpy Q-value landscape as shown the middle of Fig adversely impacts the robustness of Q function, further resulting in brittle policy. Hence, it 'ss unwise
5

---

to impose the Policy- Dattsst Generalization Inhibition all along the learning process. For some empirical evidence of this, we observe performance decay in some of our experiments Table Jand and 4 A natural remedy is to gradually remove the inhibition. However, it leaves value generalization uncontrolled once again, exposing the policy learning under the risk of collapse due to overestimation.
To address the dilemma above, we introduce the second stage, Policy-OOD Generalization Inhibition, as illustrated the right of Fig. - In this stage, we make use of an additional policy Food that outputs OOD actions. The OOD action here is defined as an action with a lower Q-value than the action selected by the learning policy for the same state. The actual effect of the second stage suppression of the generalization between the learning policy distribution and OOD policy distribution. Such a means of kernel control can be viewed as a special way of increasing the action gap [42]. It improves the robustness of value function learning by discouraging erroneous generalization on suboptimal actions. Formally, this is achieved by minş /V (s)) T V VoQo(s, Tood (s)]l-
Overall, our kernel control method consists of the two stages of generalization inhibition, which effectively controls the generalization over the entire course of learning, providing favorable upstream for better offline RL.
5 Representation Distinction in Offline RL
In this section, we devise practical algorithm to rein the generalization in offline RL following the design of two stages of generalization inhibition. A heuristic method to achieve smooth transition among them is also proposed. We term our algorithm of reining generalization Representation Distinction (RD) and apply them to both existing and designed backbone algorithms.
5.1 Practical Algorithm Design
We first focus on the phase depicted in the left of Fig 2 natural approach is directly minimize E (s, r(8), 8, a) across the dataset. However, due to the high dimensionality of Vo a),
s.a-/D
the direct computation of IE Ikol,, (s), s,a]] involves the computation and backpropagation s.a-D
through per-example gradient dot products, which is computationally prohibitive. To mitigate this, we adopt the method proposed by [33 and approximate A(Ø) with the contribution solely from the last layer parameters, i.e., E e,a~D
V Vo (=(, V TwQole (0(()))))))))) (s).)). Consequently, we derive the following
loss function:
C1
E S
(ss, n (s)) (s.
(4)
where ((s, a) signifies the representation of state-action pairs as mentioned in Section By minimizing L1, we encourage the learned Q-function to yield representations that are as orthogonal as possible between data from dataset and π. In this manner, the representations of (s, a) and (s, (s)) can be distinctly discerned.
To substantiate the efficacy of in- hibiting generalization between in-sample data and data derived from the learning policy n, we conduct rudimentary experiments by directly minimizing L, dur- ing the training of the stan- dard off-policy SAC algorithm 2 on both halfeheetah- and The evaluation results depicted in Fig demonstrate that enhancing the representation alone, devoid of explicit pessimism, can achieve commendable performance, substantially surpassing the baseline SAC. These outcomes underline the potency of inhibiting generalization to eliminate overestimation in offline setting.
hallt aall:bbeetaheeediurry
hwlicheel www.eeealmeddimmmmmmm... esspertioc?
80-
Normaturd Bettrm Neturns
60
Normallzed Natumns
600
40-
20
20-
0
250
500
250
560
750
1000
Trxining Steas (1w3)
Thaining Steps 11w31
Figure 3: Average results of SAC over 5 seeds with and w/o RD.

---

Further, we consider the phase in the right of Fig 2 To improve the robustness of the Q function by cireumventing the excessive generalization of the Q-function OOD data, we introduce a simple OOD data generator that creates data with lower Q-values compared to the data produced by current policy æ. Specifically, the OOD data generator Tood trained to generate OOD state-action pairs that satisfy the following condition: and (s Food are assumed to be positive for ease of presentation, and < a < 3 is the predefined hyperparameters. In practice, the optimization of Food implemented via the following margin ranking loss:
(s, a) < Qo(s, (s)) < B (s, a), where Q a)
COOD max (0, Qs (s)) 5 Qs a)) max (0, a * Qo(s, a) Qo(ss moad(s))) (5)
As the offline RL training progresses, we gradually shift our focus from distinguishing in in-sample data and data generated by to distinguishing data generated by and data generated by the OOD actor Tood in the representation space, i.e., L2 - gm1))
(s, Tona(s) ((8, (+))] In practice, a simple
heuristic approach is proposed to help balance L1 and L2:
LRD (1 w)
w
where w gradually increases according to w = tanh(
step sstep_mum mum MMM stsepeppmm frp__ummm___ refers to the current training step, and M is hyperparameter. During the update of Rood, the gradient towards Qa is detached.
We would like to emphasize that the OOD generator is not required to be meticulously trained, as we only expect the Q value of the generated OOD data to be appropriately lower than that of the data generated by the current policy. This is due to the fact that the OOD data is not employed to directly influence the policy's 's decision-makiin process. Instead, it serves to inhibit the excessive generalization of the Q function, thereby enhancing the robustness.
5.2 Applying RD in Offline RL Algorithms
Algorithm 1: Offline RL with RD (Psedudocode in a PyTorch-like style)
replay_buffer: save dataset
agent: offline RL agent
# itv: 00D actor and RD updating interval
Loading dataset into replay buffer (omitted)
7 for t in range (max_ www.ieeeannnnnepaaaeeeeee wwwpssaaaaaaaeeeeeeeeaaaaaaaaaea buffer) buffer)
# Update 0OD actor and RD according to Eq and respectively at intervals
10 If % itv== 0:
11
upgateuuddae__ooo
12
agent. update_RDaet
Update critic and actor following the offline RL algorithm design agent, (ptate_critie((
15
16
17
While the introducing of RD to the offline learning process offers benefits, it alone cannot boost the performance standard online RL algorithm to a cutting-edge level, as depicted Fig This is because merely reining generalization from the representation perspective does not necessarily prevent overestimation of the Q-value. Our objective is to construct RD as a universally applicable plug-in to enhance any offline RL algorithm. Thus, to evaluate RD's 's versatility as plug-in, we apply in both existing algorithms and two designed rudimentary backbone algorithms named SAC-N-Unc and TD3-N-Unc. By modifying the original SAC/TD3 algorithm to employ a critic ensemble of number and incorporate an uncertainty regularization term within the policy update process, we derive these backbone algorithms. It should be noted that these algorithms contribution of this paper as many similar uncertainty mncertainty-based methods [00 43 [2
not the original been proposed. Details of the backbone algorithms are provided in the appendix.
45 46 have
Note that RD algorichm-agnostic, meaning it can applied a wide range of actor-critic offline RL algorithms. Incorporating RD into offline RL algorithms is also straightforward, requiring only
7

---

the addition of two steps to the conventional policy updating scheme, as illustrated in Algorithm | in the appendix. In each training iteration, the OOD data generator is first trained to generate data with lower Q-values than the current policy. Then, the Q-function is updated by minimizing the loss function
LRD + Leritie. and the policy is updated based on Lucter- Here, represents weighted factor, and feiiti and Lactor refer to the optimization objectives of any offline RI. algorithm. psedudocode using RD offline RL is given Algorothm [ demonstrate the versatility of RD by integrating it into both backbone algorithms and existing offline RL algorithms.
In the experimental section, we
Experiments
We evaluate RD by applying it on TD3BC 77. CQL [8] SAC-N-Unc and TD3-N-Unc through series of experiments on D4RL [47] gym MuJoCo-v2 and Adroit-vl datasets, where the former is dataset commonly used in previous work for continuous control tasks, and the latter poses a significant challenge for most offline RL methods due to its sparse reward property. We compare method with BC and several model-free offline RL algorithms, including DT 48), TD3BC 77. CQL [8. IQL 44 EDAC (10), and Diffusion-QL (50]. We obtain the results of the baselines by re-running the official codes or directly extracting them from the original papers. In our experiments, all the re-run baselines and our algorithm
executed with five random seeds, and report the average normalized results of the final ten evaluations. We report the performance at IM gradient step for TD3BC and CQL MuJoCo tasks. For Adroit tasks,
and that at 3M gradient step for SAC-N-Unc and TD3-N-Unc we report results at 500K gradient step. Note that due to the space limitation, we abbreviate the names of the datasets from (RANDOM, MEDIUM, MEDIUM-REPLAY, MEDIUM-EXPERT, EXPERT) to (R, M, MR, ME, E} in all the tables.
6.1 Main Results
Table 1: Results of different algorithms and the ones equipped with RD
DATASET
TD3-K-UNC
TD3-N-UNC
+RD
SAC-N-UNC
SAC-N-UNC +RD
TD3BC
TD3BC +RD
CQL
CQL
+RD
0.5
RALPCHESTAR-MR
3.9
532
4..6.:..3
45.2
51.6
BAI RCURETAB-ME
9.2
101.1
0.4
9.
102.5 1.8
30.5+6.5
93.9
2.9
81.1+6.0
90.2
HOPPER-M
419.1305
103.0
45.7
1028
60.4140
61.0 2.6
63.016.1
74,9 7.1
18.5
0.8
1.4.7=09
20.2+2005
180.3
110.6
98.2
WALKERZD-X
69.2
97.6 3.4
24.21222
1.33
3..7
83.7
2.7
30.4
84.5 1.0
WALKERZD-MR
91.6
92.1 2.7
85.2=
86.9 3.1.1
82.1
84.8
1.4
79.2
94.4
90.6++45.00
1.2
113.1
116.4
110.2
110.1
109.7
113.0
Table 2: Average normalized scores of our methods and previous methods the D4RL benchmark.
SAC.N-UNC TD3-N-UNC
DATASET
BC
DT
TD3BC
CQI.
EDAC
DIFRUSION-QU
RALPCHEITAI-K
2.2
2.2
11.0
31.3
13.7
28.4
22.0
25.4
31.0
3.7
WALKER2D-R
1.3
2.2
1.7
5.4
5.9
16.6
5.5
21.2
21.7
FALFCHEETAR-M
42.6
42.6
48.0
47.1
47.4
65.9
51.5
65.9
66.8
HOPPER-M
52.9
67.6
60.4
65.0
66.3
101.6
96.6
102.8
103.0
WALKERID-M
75.3
74.0
82.7
80.4
78.3
92.5
87.3
92.3
97.6
HALPCHUETAH-IK
36.6
6
45.2
44.2
61.3
48.3
61.5
$7.7
HOPPER-MR
18.1
82.7
61.2
87.7
94.7
101.0
102.0
194.6
104.1
66.6
87.1
HALPCHOETAB-ME
55.2
86.8
90.5
81.1
86.7
106.3
97.2
102.5
101.1
52.5
107.6
105.4
93.9
110.7
112.3
110.6
110.7
WALKER2D-ME
107.5
110.2
109.7
109.6
114.7
111.2
118.8
ACHEETAAE
91.8
87.7
96.7
97.3
94.9
106.8
96.3
108.8
103.1
107.7
WALKER2D-E
108.7
108.3
110.2
109.3
109.7
115.1
109.5
112.3
111.2
MuJnCo
1252.6
1259.4
PEN HUMAN
25.8
73.9
-1.9
35.2
71.5
52.1
75.7
61.1
77.9
PEN CLONED
38.3
67.3
9.6
27.2
37.3
68.2
60.8
$3.0
65.5
ADROIT TOTAL
64.1
141.2
7.7
62.4
108.8
120.3
136.5
114.1
143.4
TOTAL
846.2
1113.8
1028.6
1 1106.8
1142.8
1363.7
1295.1
1366.7
1402.8

---

To demonstrate the applicability of RD, we integrate RD into the training regime of TD3BC, CQL. TD3-N-Une and SAC-N-Unc. As observed from Table both TD3BC and CQL exhibit improved overall performance when RD is applied. It is noteworthy as the core idea of CQL to increase the Q value of the data in the dataset and diminish the Q value of the data generated by a mixed policy comprising the learning policy and random policy, which incorporates the representation distinction of state-action pairs sourced from different distributions. Therefore in practice, we apply the insights of RRD to gradually transfer the original Q value restriction to the Q-value differentiation between the mixed policy and the designed OOD policy. As result, CQL'S performance is significantly improved. Moreover, the backbone algorithms themselves yield substantial advantages from the application of RD. Table 2 further reveals that backbone algorithms equipped with RD can achieve the best overall performance and attains state-of-the ahr performance on several datasets. Additional evidence supporting the efficacy of RD in enhancing convergence speed and its potential to reduce the quantity of Q ensembles is provided in the appendix.
6.2 Comparison and discussion with techniques that inhibit overestimation
Table 3: Average normalized scores of TD3-N-Une with RD or other variants, and DR3 and Layer
Norm that inhibit potential overestimation
DATASET
TD3-N-UKC
TDA-N-UNC
+PDD
TD1-N-IIN +PDDDAW
TD3-N-UNC +RDD
TD3-N-UNC POD
TRA-N-UNE +DR3
TD3-N-UNC +LAVERNORM
TD3-N-UNC RD
BALPCHEETAR-
66.8
66.1 1.9
6..6
6..9
63.1124
51.4
63.2
66.8 1.2
HOPPER-M
41.9 5.5
99.91.6.
100,7-622
17.4.3385 38.6
62.114.5
83.0-129.00
10:01.0:
WALKERZD-M
69.9 35.2
94.9
66.7 38.1
80.2231.00
94
92.1
65.5 = 20.7
97.6
BALECHEETAN-E
94.6 11.
96.3 L. 13.2
0229-122
9.59
93.7 13.0
100.0 4: 3.7
104.4 1.5
103.1 1: 0.6
HOPPER-E:
110.9 0.5
108.4-044 0.4
108.4=0.4
108.4
00.40000
108.2+0.7
108
0 0.5
15.4-428
108.8 + 0.3
WALKBR20-E
7.1
67.4 47.2
60.8 46.1
28.5 40.8
54.1
109.9 +: 0.4
111.7
111.2 : 0.7
We compare our method RD with the existing techniques that inhibit potential overestimation including DR3 [41] and Layer Norm [51]. In addition to the regularizer, all the other parameters are kept the same across different methods ensure fairness. As shown in Table 3 RD is more helpful in helping improving the backbone algorithm than DR3 and Layer Norm.
RD and DR3 differ mainly from two perspectives. From the angle of the framework derived from, DR3 is derived from the thcoretical characterizing implicit regularization in TD-Learning, which is a generalization of the implicit regularization from Supervised Learning 52 53 to TD-Learning in RL setting. In contrast, RD is derived from the proposed Backup-Gient Generalization framework. From the angle of regularization effect, DR3 is proposed to directly counter the implicit regularization of TD-Learnnng while RD is to suppress the generalization between in-sample data and OOD data. similar form, i.e., to minimize the NTK between
With some heuristies, DR3 regularizer arrives consecutive state-action pairs in backup, to RD regularizer. Apparently, DR3 regularizer can be a special case of RD regularizer in terms of the definition of out-of-sample actions. The regularization effect of DR3 is similar to the Policy-Datase Generalization Inhibition shown by Figure [ Such a regularization can induce over-inhibition of generalization when the distribution of current policy overlaps with the offline dataset as illustrated. We also show in the following that DR3 and Policy- Dataset Generalization Inhibition (i.c., PDD in the following part) achieve similar empirical results.
6.3 Ablation Studies and In-depth Analysis
We design ablation experiments to elucidate the significance of RD's components. We compare RD against several variants including, including Policy-Datase Distinction (PDD), Random-Dataset Distinction (RDD), Policy-OOD Distinction (POD), and Policy-Datasel Distinction with Dynamically Adjusted Weight (PDDDAW). PDD refers to the auxiliary representation loss calculated using data generated by the learning policy and that from the dataset. Likewise, RDD utilizes data generated by a random policy versus that from the dataset, while POD employs data produced by the learning policy against that by the OOD actor. As for PDDDAW, its PDD component is reweighted using the same heuristic w adjustment method as RD, but the POD component is persistently weighted by zero. We draw a comparison between TD3-N-Unc RD and these variants, as tabulated in Table 3 The results demonstrate that RD, fundamentally dynamic amalgamation of PDD and POD, attains superior overall performance, underscoring the indispensability of each RD component.

---

Noma lod houume
-M
100
530
1000
1950
2050
2503
1000
Tuining Stopa (1)2))
Figure 4: Performance using Figure 5: T-SNE visualization Figure 6: T-SNE visualization of representation via RD.
RD and PDD with same seed. of representation via PDD.
It is worth noting that using PDD can help backbone algorithms achieve relatively satisfactory performance. However, as depicted in Fig algorithm using PDD occasionally suffers performance degradation during the later stages of training, consequently impairing the overall efficacy of the approach. To elucidate this phenomenon, the final models trained via RD and PDD, as shown in 4 are saved and the quality of the representations obtained by these models compared. A total of 100 state state-action pairs are sampled from the halfcheetah-exper., halfcheetah-ment mediium,,mmmmmmm and halfchhettbb-random datasets, respectively, and then consolidated. Using t-Distributed Stochastic Neighbor Embedding (I-SNE), the distribution of the representations of these 300 state-action pairs derived from both models is charted. Each sample is marked with distinctive symbols (stars for random data, squares for medium data, and circles for expert data) to differentiate samples from diverse datasets. Moreover, varying colors are assigned the Q-value estimated by the proficiently trained Q- Q-value network acquired by RD, with brighter colors signifying higher Q-values. As depicted in Fig and 6 the RD-derived representation exhibits superior differentiation between state-action pairs with high and low Q-values, whereas the PDD-derived representation frequently misclassifies expert and medium data.
individual samples corresponding to
To better demonstrate that the representations learned by RD are superior, calculate which dataset the top five nearest samples to each sample belong to, and compute the mean all samples from the same dataset in Table 4 For instance, the EXPERT- EEXPERT metric of representation learned via RD indicates that 94% of the five nearest sam- ples to the expert samples are also expert samples. We can conclude that representation learned via RD can better help samples from the same dataset to cluster closer. More results are provided in the appendix.
Table 4: The proportion of the five sam- ples closest to each sample
REP PDD
REP VIA RD
EXPPRT-EXPERT EXPERT-MEDIUR
0.94
0.12
0.06
EXPERT RANDOM
0.01
0,00
M-XPERR
0.38
MEDIUM: MEDIUM
0.43
0.51
MEDIU M-RANDOM
0.19
0.11
Conclusions and Limitations
This paper analyze the important role of generalization in offline RL and presents a novel plug-in method, Representation Distinction (RD), to enhance the performance of offline RL algorithms by reining the generalization. By explicitly differentiating between in-sample and OOD state-action pairs, the generalization across data could be properly inhibited. Our extensive experiments demonstrate the efficacy of the RD method when applied to several offline RL algorithms, significantly enhancing their performance across various continuous control tasks D4RL datasets. In conclusion, our work contributes a valuable perspective to the field of offline RL by focusing on reining generalization from the representation perspective and providing a flexible solution that can be applied to variety of existing algorithms.
However, our work is not without limitations. For example, currently RD primarily focus on continuous control tasks, and its applicability to discrete control tasks yet to be explored. In addition, the transition between the two phases of RD is heuristic method, which could be improved by designing adaptive adjustment methods. Future research directions may include addressing the aforementioned limitation.
10

---

References
[1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv:2005.0164 fcs, stat], November 2020. arXiv: 2005.01643.
[2] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Off-po maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[3] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587- 1896. PMLR, 2018.
[4] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. In International conference muchine learning. pages 2052-2062. PMLR, 2019.
[5] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[6] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: /911.9111.. 1. 1361, 2019.
[7] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132 20145, 2021.
[8] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 19911999,,, 2020.
[9] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.0472 21474522022 2022.
[10] Gaon An, Seungyong Moon, Jang Hyun Kim, and Hyun Oh Song. learning with diversified g-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Uncertainty-basee based oftline reinforcement
[11] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-drrven offline reinforeement learning. arXiv preprint arXiv:2202. 11566, 2022.
[12] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. arXiv preprint arXiv:2266.028 02829, 2022.
[13] Xinyang Geng, Kevin Li, Abhishek Gupta, Aviral Kumar, and Sergey Levine. Effective offline rl needs going beyond pessimism: Representations and distributional shift. In Decision Awareness in Reinforcement Learning Workshop ICML 2022, 2022.
[14] Mengjiao Yang and Ofir Nachum. Representation matters: offline pretraining for sequential decision making. In International Conference on Machine Learning, pages 11784-11794, PMLR, 2021.
[15] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage--eiighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXi::1910.00177, 2019.
[16] Noah Y Siegel, Jost Tobias Springenberg. Felix Berkenkamp. Abbas Abdolmalcki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behav- ioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.083% 2020.
[17] Tlya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-leaming. arXiv preprint arXiv: 10.00: 16,0 2021.
[18] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719-1735. PMLR, 2021.
[19] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang. Wenzhe Li, Bin Liang. Chelsea Finn, and Chongjic Zhang. Latent-variable advantage-weighte policy optimization for offline rl. arXiv preprint arXiv:2203.089. 2022.
[20] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-basee offline policy optimization. Advances in Neural Information Processing Systems,
2020.

---

[21] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-21823, 2020.
[22] Arthur Argenson and Gabriel Dulac-Amold. arXiv:2008. 05556, 2020.
Model-based offline planning. arXiv preprint
[23] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offiine model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
[24] Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204.1258 2022.
[25] Kaiyang Guo, Yunfeng Shao, and Yanhui Geng. Model-based offline reinforcement learning with pessimism-modulater dynamics belief. arXiv preprint arXiv:2210.0669) 2022.
[26] Hamid Maei, Csaba Szepesvari, Shalabh Bhatnagar, Doina Precup, David Silver, and Richard S Sutton. Convergent cuuporal-differene learning with arbitrary smooth function approximation. Advances in neural information processing systems, 22, 2009.
[27] Joshua Achiam, Ethan Knight, and Pieter Abheel. Towards characterizing divergence in deep q-learning. arXiv preprint 1903vv:903,0000 2019.
[28] Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization temporal difference learning. International Conference on Machine Learning, pages 767-777. PMLR, 2020.
[29] Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning distribution correction. Advances Neural Information Processing Systems, 33:1 08118550-1855 18572, 2020.
[30] Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint 2rXiv:2204.095555 2022.
[31] Chenjun Xiao, Bo Dai, Jincheng Mei, Oscar A Ramirez, Ramki Gummadi, Chris Harris, and Dale Schuurmans. Understanding and leveraging overparameterization in recursive value estimation. In International Conference on Learning Representations, 2022.
[32] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit paaddrrrrraetiizzationn inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021.
[33] Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Value-based deep reinforeement learning requires explicit regularization. In International Conference on Learning Representations, 2022.
[34] Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski. Georg Ostrovski, Mehrdad Farajtabar, Matt Hoffman, Razvan Pascanu, and Amnad Doucet. An empirical study of implicit regularization in deep offline rl. arXiv preprint arXiv:2207.088 02099,, 2022.
[35] Wesley Chung. Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for nonlinear value function approximation. In International Conference Learning Representations, 2019.
[36] Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G Bellemare, and David Silver. The value-impro improvemptv path: Towards better representations for reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 7160-7168, 2021.
[37] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised representations for reinforcement learning. In International Conference Muchine Learning, volume 119, pages 5639-5650, 2020.
[38] Michael Laskin, Kimin Lec, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforce- ment learning with augmented data. In Advances in Neural Information Processing Systems, 2020.
[39] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, volume 139, pages 11920-1 1931, 2021.
[40] Amy Zhang. Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference Learning Representations, 2021.
12

---

[41] Aviral Kumar, Rishabh Agarwal, Tengyu Ma. Aaron Courville. George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Reprexentations.
[42] Marc G Bellemare, Georg Ostrovski, Arthur Guez, Philip Thomas, and Rémi Munos. Increasing the action gap: New operators for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[43] Alex Beeson and Giovanni Montana. Balancing policy constraint and ensemble size in uncertainty- offline reinforvement learning. arXiv preprint arXiv:2303.1473 147166 2023.
[44] Hongchang Zhang, Boyuan Wang. Yixiu Mao, Jianzhun Shao, Yuhang Jiang, Yi Xu, and Xiangyang Pessimistic policy iteration for offline reinforcement learning, 2023.
[45] Kamyar Ghasemipour, Shixiang (Shane) Gu, and Ofir Nachum. Why so pessimistic? estimating uncer- tainties for offline rl through ensembles, and why their independence matters. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh, editors. Advances in Neural Information Processing Systems, volume 35, pages 18267-18281 Curran Associates, Inc., 2022.
[46] Jordi Smit, Canmanie T Ponnambalam, Matthijs TJ Spaan, and Frans A Oliehock. Pebl: Pessimistic ensembles for offline deep reinforcement learning. In Robust and Reliable Autonomy in the Wild Workshop at the 30th International Joint Conference of Artificial Intelligence, 2021.
[47] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.022) 07219, 2020.
[48] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee. Aditya Grover. Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information potomastiopmoossiiiggg processing systems, 34:15084-15097, 2021.
[49] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforeement learning with implicit q-leamiin. In International Conference on Learning Representations, 2022.
[50] Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou, Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint wwXiv:2208.06/99 2022.
[51] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey arXiv:1607.06455 2016.
Hinton. Layer normalization. arXiv preprint
[52] Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhe unstein-ubleenbekk like process. In Conference on learning theory, pages 483-513. PMLR, 2020.
[53] Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34:27449-27461, 2021.
[54] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: sesearch-orientee deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as "Lounchpad",, 2022.
13


Reviews:
[{"review_id": "S9dqPnsBv3", "reviewer": "Reviewer_xZbD", "summary": "The paper proposes a flexible plug-in method called Representation Distinction (RD) to address the overgeneralization issue in offline Reinforcement Learning (RL) algorithms. The authors formalizes the process of generalization and investigates the potential to rein the generalization from the representation perspective to enhance offline RL, performing both Policy-Dataset and Policy OOD Generalization Inhibition with awareness of the two possible phases in the offline learning course. The proposed method explicitly differentiates between the representations of in-sample and out-of-distribution state-action pairs generated by the learning policy, which significantly improves the performance of the backbone and widely-used offline RL algorithms across various continuous control tasks on D4RL datasets.", "questions": "**Methodology**\n+ It is recommended to include more comparisons and conduct thorough analyses on how RD and other existing NTK-relevant approaches address generalization issues in offline RL. This would enhance the comprehensiveness and depth of the discussion.\n+ As far as I am concerned, RD via kernel control is doing value regularization implicitly on representation level. It would be beneficial to provide a clearer and more insightful explanation of the potential advantages of applying RD via kernel control instead of value regularization and policy constraint, particularly in the context of offline RL algorithms. \n\n**Evaluation**\n+ There appears to be an inconsistency between the results of SAC/TD3-N-Unc+RD in Table 1 and Table 2. It is important to address this discrepancy and provide clarification or additional information to ensure the accuracy and reliability of the reported findings.\n+ Considering that well-established techniques that inhibit potential overestimation induced by extrapolation already applied into reinforcement learning, such as Lipchitz Regularization with Spectral Norm [4] and Layer Norm [5], are flexible and widely applicable, it would be worth discussing and experimenting with the possibility of leveraging these existing and simpler methods to effectively constrain and smooth the critic approximation, thereby preventing overgeneralization.\n+ The comparison between curves obtained from a single seed in Figure 4 lacks experimental validity and may raise concerns about the reliability of the results. It is essential to address this limitation and consider conducting experiments with multiple seeds to improve the robustness and credibility of the presented findings.\n\n[4] Gogianu et al.  Spectral normalisation for deep reinforcement learning: An optimisation perspective. ICML 2021.\n\n[5] Ba et al. Layer normalization. Advances in NeurIPS 2016 Deep Learning Symposium, 2016.", "limitations": " The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ **Clarity**: the paper is well-structured and clear to follow.\n+ **Flexibility and Good Performance**: the authors propose a novel method called Representation Distinction (RD) to improve the performance of offline reinforcement learning algorithms, which explicitly differentiates between the representations of in-sample and out-of-distribution (OOD) state-action pairs generated by the learning policy, demonstrating the efficacy of the proposed approach by flexibly applying RD to specially-designed backbone algorithms and widely-used offline RL algorithms.\n+ **Insight**: the paper provides an insightful, nuanced and systematic formalization of the process of generalization in offline RL and then investigates the prevalent overgeneralization issue in offline RL.", "weaknesses": "+ **Missing Citations or Clear Indications**: some recent papers also utilize NTK to analyze the genralization ability of offline RL on similar considerations with RD [1, 2, 3]. It is advisable to provide more comparisons and at least analyses on the way different NTK-relevant approaches address generalization issues in offline RL.\n+ **Evaluation**: questions about experimental setups and results are listed in the next section.\n\n[1] Kumar et al. DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization. ICLR 2022.\n\n[2] Ghasemipour et al. Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters. NeurIPS 2022.\n\n[3] Li et al. When data geometry meets deep function: Generalizing offline reinforcement learning. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### analyses on how RD and other existing NTK-relevant approaches address generalization issues in offline RL\nSee the global rebuttal parts.\n\n\n### RD via kernel control is doing value regularization implicitly on representation level\n\nThe core idea of RD is to encourage the learned Q-function to yield representations that are as orthogonal as possible between data from dataset and $\\pi$ in the first stage and that between $\\pi$ and $\\pi_{ood}$  in the second stage.  Taking stage 1 for example,  in comparison to the value regularization on the q value **backup** process of $\\pi$ (part of) in CQL, the value of $\\pi$ using RD is regularized by cutting off the **generalization** from the learning on dataset. In extreme cases when $\\left|\\nabla_\\phi Q_\\phi(s, a)^{\\top} \\nabla_\\phi Q_\\phi(s, \\pi(s))\\right|=0$, the values of the unreliable actions of $\\pi$ will be kept the same as the initialized ones. Although achieving similar results that the values of untrusted actions are low, the mechanism behind RD is completely different from the regularization of the Q value as shown in the Backup-Generalization Cycle in Figure 1.\n\n### comparisons with DR3 and layer norm\nAccording to the suggestion, we perform experiments against TD3-N-Unc with DR3 and with layer norm. Below table demonstrates the results on six datasets. Note that in addition to the auxiliary loss, all the other  parameters are kept the same across different methods to ensure fairness. Overall, RD is more helpful than DR3 and Layer Norm.\n\n| Dataset               | TD3-N-Unc + RD | TD3-N-Unc + DR3 | TD3-N-Unc + Layer Norm |\n| --------------------- | -------------- | --------------- | ---------------------- |\n| halfcheetah-medium-v2 | **66.8$\\pm$1.2**   | 64.4 $\\pm$ 1.7  | 63.2 $\\pm$ 0.8         |\n| hopper-medium-v2      | **103.0$\\pm$0.8**  | **103.4 $\\pm$ 0.7** | 83.0 $\\pm$ 29.0        |\n| walker2d-medium-v2    |  **97.6$\\pm$3.4**   | 92.1 $\\pm$ 2.0  | 68.5 $\\pm$ 20.7        |\n| halfcheetah-expert-v2 | 103.1$\\pm$0.6  | 100.0 $\\pm$ 3.7 | **104.4 $\\pm$ 1.5**        |\n| hopper-expert-v2      | **108.8$\\pm$0.3**  | **108.0 $\\pm$ 0.5** | 88.4 $\\pm$ 42.8        |\n| walker2d-expert-v2    | **111.2$\\pm$0.7** | 109.9 $\\pm$ 0.4 | **111.7 $\\pm$ 0.4**        |\n\n### single seed in Figure 4 \n\nWe use the result of one seed in Figure 4 to try to depict the occasionally happened performance degradation during the later stages of training in a clear manner.  The subsequent T-SNE visualization is also based on the model trained obtained in Figure 4.  Average results of several seeds could blur the result.  In the appendix, we provide the curves of five seeds, which could demonstrate the reliability of the results. \n\n###  inconsistency between the results of SAC/TD3-N-Unc+RD in Table 1 and Table 2.\nThanks for pointing out it. We have revised the results carefully and update the results in the draft.\n"}, {"review_id": "49gynPW6Gp", "reviewer": "Reviewer_S7qm", "summary": "This paper explains the necessity of reining generalization in offline RL and proposed a new method Representation Distinction (RD) based on this principle. RD shows good performance on D4RL benchmarks. ", "questions": "- The experiment in Figure 3 is only conducted on halfcheetah environment, which is not very convincing as halfcheetah has some special features. Can the authors provide more results on other environments?\n- In section 6.2, the authors only demonstrated the representation of  PDD and RD. While I'm curious about the representation of the original algorithm.", "limitations": "The limitation of this work is that it is limited to continuous control tasks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper presented a view called Backup-Generalization Cycle, which explains that the generalization issue should be considered in the training phase.\n- This experiment shown in Figure 3 is interesting, which demonstrates overgeneralization is an important issue in offline RL and the method is able to mitigate it.\n- By applying RD to existing algorithms, their performance is improved significantly. ", "weaknesses": "- There is a gap between analysis and practical implementation. Therefore, it is difficult to identify whether the performance improvement comes from the overgeneralization issue. \n- The algorithm introduces some hyperparameters which may make it a little difficult to tune.\n- The difference between PDD and RD is not significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### gap between analysis and practical implementation\n\nThe conceptual illustration of reining generalization via kernel control in Section 4.3 including two stages: Policy-Dataset generalization inhibition and Policy-OOD generalization inhibition. In the practice algorithm design, we design loss  $\\mathcal{L}_1$ and $\\mathcal{L}_2$ for the two stages, respectively,  and design a simple heuristic approach to smoothly transit  from stage 1 to stage 2 by dynamically adjusting the weight of $\\mathcal{L}_1$ and $\\mathcal{L}_2$. Please let me know if you still find this correspondence is unclear. \n\n###  hyperparameters which may make it a little difficult to tune\n\n There are four hyperparameters used in RD, including: $\\alpha$, $\\beta$ to control the OOD data generator to create data with lower Q-values compared to the data produced by current policy; $M$ to control the balance between $\\mathcal{L}\\_{1}$ and  $\\mathcal{L}\\_{2}$; $\\epsilon$ to balance $\\mathcal{L}\\_\\text{RD}$ and  $\\mathcal{L}\\_\\text{critic}$ of any offline RL algorithm. Note that for all the datasets and algorithms used in the experiments, we set $\\alpha=0.6$, $\\beta=0.7$, $\\epsilon=0.1$. For $M$, we set it to 2e6 when applying RD to CQL. For other algorithms including TD3BC, SAC-N-Unc and TD3-N-Unc, $M$ is set to 1/10 of the total training steps.  The relatively general setting demonstrate the hyperparameters could be applied to different algorithms and datasets. Hyperparameters setting of RD are also provided in the appendix.  \n\n### difference between PDD and RD is not significant\n\nAccording to Table 3, the differences between PDD and RD on half of the datasets are relatively significant. On walker2d-expert dataset, the difference is quite large.  Although the differences between PDD and RD on some datasets are not clear, the differences between algorithm with PDD and the pure algorithm is quite significant, which demonstrates the effectiveness of PDD, which is an important component of RD a described in the main text of Section 6.2.\n\n### more results on other environments in Figure 3\n\nWe provide average results of SAC-2 (vanilla SAC) and SAC-2+RD of three seed after training 1M gradient steps on more dataset in the following table. It can be observed that RD is helpful in improving SAC-2 without explicit  pessimism.\n\n|  Dataset    | SAC-2 |  SAC-2 + RD |\n| -------- | ------- | ------- |\n| halfcheetah-random-v2  |     29.7       |   30.0   |\n| halfcheetah-medium-v2  |      38.2      |   68.5    |\n| halfcheetah-medium-replay-v2  |  0.8   |   49.2    |\n| halfcheetah-full-replay-v2 |   86.8     |   82.3    |\n| hopper-random-v2  |    9.9        |   15.2    |\n| hopper-medium-v2  |   0.8         |  2.1   |\n| hopper-medium-replay-v2  |  7.4   |   64.3    |\n| hopper-full-replay-v2 |   41.1     |    110.0   |\n| walker2d-random-v2  |     0.9       |   0.3    |\n| walker2d-medium-v2  |     -0.3       |   -0.2    |\n| walker2d-medium-replay-v2  |  -0.4   |  52.6     |\n| walker2d-full-replay-v2 |    27.9    |  97.3     |\n\n\n### the representation of the original algorithm\n\nWe expand Table 4 by adding corresponding statistics of the original algorithm on halfcheetah-expert-v2 in the following table. The performance of the original algorithm TD3-N-Unc of the same seed is 80.1, which is slightly lower than that of TD3-N-Unc with PDD ( 82.5).  It can be observed that both the representations learned by PDD and RD are better than that of the original algorithm. \n\n|  Dataset  |  Rep of ori | Rep via PDD |  Rep via RD |\n| -------- | ------- | ------- | ------- |\n| expert-expert | 0.83   | 0.87   | 0.94 |\n| expert-medium | 0.15   | 0.12   | 0.06  | \n| expert-random | 0.01   | 0.01   | 0.00  |\n| medium-expert | 0.38   | 0.38   | 0.38  |\n| medium-medium | 0.40   | 0.43   | 0.51 |\n| medium-random | 0.22   | 0.19   | 0.11   |  \n\n"}, {"review_id": "8xrIsRkE1y", "reviewer": "Reviewer_yLhs", "summary": "The authors investigated the effect of generalization in overestimation of critic values and devised an algorithm that can mitigate overgeneralization. The proposed method, Representation Distinction (RD), is orthogonal to most offline RL algorithms and thus be applied to most of them. Experimental results show that RD significantly enhances the performance of baseline algorithms.", "questions": "Table 3 reports the average normalized score of TD3-N-Unc on the Walker2D-Expert domain to be $4.1\\pm 7.1$\nand Table 1 reports the average normalized score of TD3-N-Unc on the Waler2D-Medium domain to be $69.9\\pm 35.2$. Why does TD3-N-Unc perform a lot better on the medium dataset?\n\n&nbsp;\n\n### Minor suggestions\n\nEq. (4) It would be easier for the readers to follow if the authors explicitly mention that $\\nabla_{\\mathbf{w}}Q_\\phi(s, a)=\\Phi(s, a)$.\n\nLine 315: gruadually $\\to$ gradually\n\nThere are duplicate references: [33] and [41] are both Kumar et al. (2022).", "limitations": "The authors have addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Originality\nThe paper introduces the Backup-Generalization framework to analyze the overestimation of critic values.\nThe authors devised a novel method of learning a suboptimal policy to sample OOD actions and use them in the later stage of training. They also proposed a simple and effective heuristic that allows a natural transition from PDD to POD as the training progresses.\n### Quality\nMost of the claims are technically sound. The authors have analyzed the performance of their method on popular offline RL benchmarks and conducted multiple interesting ablation studies.\n### Clarity\nThe paper is overall well-written and easy to understand.\n### Significance\nMost works on offline RL focus on the backup step. This work instead points out the role of generalization in overestimation of critic values, which seems like a promising research direction. Also, the proposed method can be applied to most existing offline RL algorithms to improve their performance.\n", "weaknesses": "1. Line 315~316 is difficult to understand. A pseudocode of CQL+RD will be helpful.\n2. A theoretical analysis of how RD differs from DR3 (Kumar et al. 2022) and why using OOD actions is better than just optimizing the DR3 auxiliary loss, $\\mathbb{E}_{(s, a, s', a')\\sim\\mathcal{D}}\\left[\\Phi(s', a')^\\intercal\\Phi(s, a)\\right]$, is missing from the paper. Adding this will improve the originality of the paper, as this work looks like a slight modification of DR3 at first glance. Also, the authors do not compare their algorithm with DR3 variants of CQL, TD3-BC, and SAC. Although I expect their performance to be on par with the PDD variants, I believe they should be included in the experiments section.\n3. TD3-N-Unc and SAC-N-Unc use an ensemble of N critics. It is unfair to compare their performance with other baselines that only use one critic.\n\n### References\nAviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Value-based deep reinforcement learning requires explicit regularization. In *International Conference on Learning Representations*, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### pseudocode of CQL+RD will be helpful.\n\nHere we provide the detailed illustration of modified CQL based on the insight of RD.  The core idea of CQL conservativeness is to increase the Q value of the data in the dataset $\\mathcal{D}$ and diminish the Q value of the data generated by a mixed policy $\\mu$ comprising the learning policy $\\hat{\\pi}\\_\\theta$ and random policy as following:\n$$\n\\begin{aligned}\n\\min\\_Q \\alpha\\left(\\mathbb{E}\\_{\\mathbf{s} \\sim \\mathcal{D}, \\mathbf{a} \\sim \\mu(\\mathbf{a} \\mid \\mathbf{s})}[Q(\\mathbf{s}, \\mathbf{a})]-\\mathbb{E}\\_{\\mathbf{s} \\sim \\mathcal{D}, \\mathbf{a} \\sim \\hat{\\pi}\\_\\theta(\\mathbf{a} \\mid \\mathbf{s})}[Q(\\mathbf{s}, \\mathbf{a})]\\right) \\\\\n\\end{aligned}\n$$\nThis formulation incorporates the representation distinction of state-action pairs sourced from different distributions. Therefore in practice, we apply the insights of RD to gradually transfer the original Q value restriction in the above equation to the Q-value differentiation between the learning policy and the designed OOD policy in the below equation:\n\n$$\n\\begin{aligned}\n\\min\\_Q \\alpha\\left(\\mathbb{E}\\_{\\mathbf{s} \\sim \\mathcal{D}, \\mathbf{a} \\sim \\hat{\\pi}\\_\\theta(\\mathbf{a} \\mid \\mathbf{s})}[Q(\\mathbf{s}, \\mathbf{a})]-\\mathbb{E}\\_{\\mathbf{s} \\sim \\mathcal{D}, \\mathbf{a} \\sim \\pi_\\text{ood}(\\mathbf{a} \\mid \\mathbf{s})}[Q(\\mathbf{s}, \\mathbf{a})]\\right) \n\\end{aligned}\n$$\n\n### this work looks like a slight modification of DR3 at first glance\n\nRD and DR3 differ at both the theoretical framework from which they are derived and the regularization effect in terms of out-of-sample data. \n\nDR3 is derived from the theoretical characterizing implicit regularization in TD-Learning, which is a generalization of the implicit regularization effect from Supervised Learning as studied in Blanc et al. (2020) and Damian et al. (2021) to TD-Learning in RL setting. In contrast, RD is derived from the Backup-Generalization framework proposed by us.\n\nFrom the angle of regularization effect, DR3 is proposed to directly counter the implicit regularization of TD-Learning while RD is to suppress the generalization between in-sample data and out-of-sample data. With some heuristics, DR3 regularizer arrives at a similar form, i.e., to minimize the NTK between consecutive state-action pairs in backup $(s,a)$ and $(s^{\\prime},a^{\\prime})$ (where $a^{\\prime} \\sim \\pi(\\cdot|s^{\\prime})$), to RD regularizer. Apparently, DR3 regularizer can be a special case of RD regularizer in terms of the definition of out-of-sample actions. The regularization effect of DR3 is similar to the Policy-Dataset Generalization Inhibition shown by Figure 2 in our paper. Such a regularization can induce over-inhibition of generalization when the distribution of current policy overlaps with the offline dataset as illustrated.\n\nWe appreciate the reviewer's inspiring comment and we will add more analysis on the differences between RD and DR3 in our paper as suggested. For the empircal comparison between DR3 and RD, we provide additional experimental results as follows.\n\n### comparisons with DR3\n\nAccording to the suggestion, we perform experiments against TD3-N-Unc with DR3 and with layer norm. Below table demonstrates the results on six datasets. Note that in addition to the auxiliary loss, all the other  parameters are kept the same across different methods to ensure fairness. Overall, RD is more helpful than DR3 and Layer Norm.\n\n| Dataset               | TD3-N-Unc + RD | TD3-N-Unc + DR3 | TD3-N-Unc + Layer Norm |\n| --------------------- | -------------- | --------------- | ---------------------- |\n| halfcheetah-medium-v2 | **66.8$\\pm$1.2**   | 64.4 $\\pm$ 1.7  | 63.2 $\\pm$ 0.8         |\n| hopper-medium-v2      | **103.0$\\pm$0.8**  | **103.4 $\\pm$ 0.7** | 83.0 $\\pm$ 29.0        |\n| walker2d-medium-v2    |  **97.6$\\pm$3.4**   | 92.1 $\\pm$ 2.0  | 68.5 $\\pm$ 20.7        |\n| halfcheetah-expert-v2 | 103.1$\\pm$0.6  | 100.0 $\\pm$ 3.7 | **104.4 $\\pm$ 1.5**        |\n| hopper-expert-v2      | **108.8$\\pm$0.3**  | **108.0 $\\pm$ 0.5** | 88.4 $\\pm$ 42.8        |\n| walker2d-expert-v2    | **111.2$\\pm$0.7** | 109.9 $\\pm$ 0.4 | **111.7 $\\pm$ 0.4**        |\n\n\n\n###  unfair to compare performance with other baselines that only use one critic.\n\nEnsemble is a way to implement pessimism, which is concurrent to other types of pessimism such as policy constraint and value penalization. Beside, as noted in the appendix F of Gaon,et al (2021), increasing the number of Q-networks in CQL is of little help or even harmful. Thus aligning the number of Q-networks in each baseline algorithm may be not helpful. In addition, even for ensemble-based baseline EDAC, SAC-N-Unc+RD and TD3-N-Unc+RD can achieve better performance.\n\n### Why does TD3-N-Unc perform a lot better on the medium dataset\n\nTable 3 is provided to elucidate the significance of RD’s components on different datasets. In order to achieve this, the hyperparameter settings of offline RL algorithm itself of each variant in Table 3 are exactly the same. This means that the results of the original TD3-N-Unc in Table 3 are not the optimal results it can achieve, so it is meaningless to compare the performance of the original TD3-N-Unc between different datasets. When we tune the hyperparameters of the original TD3-N-Unc, we can see in Table 6 of the appendix that the performance learned by TD3-N-Unc on the expert data set is better than that learned on the medium data. \n\n#### Reference\n* Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on learning theory, pp.483–513. PMLR, 2020.\n* Alex Damian, Tengyu Ma, and Jason Lee. Label noise sgd provably prefers flat global minimizers. arXiv preprint arXiv:2106.06530, 2021.\n* An, Gaon, et al. \"Uncertainty-based offline reinforcement learning with diversified q-ensemble.\" *Advances in neural information processing systems* 34 (2021): 7436-7447."}, {"review_id": "iEPbv2gmn6", "reviewer": "Reviewer_XJcM", "summary": "Authors consider overgeneralization in offline RL which is new and propose the modification which helps to solve the problem and  can be applied to any algorithm.", "questions": "How does you modification affect training runtime? Is the change significant or not?", "limitations": "Limitations are not clear without additional benchmarking or runtime information.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Proposed method boosted performance of different algorithms by a notable margin in most of the cases and it can be integrated in any algorithm while being simple. Good ablation study.", "weaknesses": "The problem epxeriments is that algorithms are tested using MuJoCo datasets and  2 pen datasets and  appeares not to be sufficient. For example, in CORL (https://arxiv.org/abs/2210.07105) benchmarks which you mention and  in original work (https://arxiv.org/abs/2110.01548), it is seen that EDAC performs great on the MuJoCo but fails on AntMaze (the recent update of CORL tested even more datasets and EDAC seems to fail on Adroit too). There are already some algorithms which are competitive or outperform EDAC on MuJoCo while working better on other datasets, see https://arxiv.org/abs/2206.02829 as an example. Some algorithms can performe not that good on MuJoCo but outperform others on different datasets, see concurrent work (https://arxiv.org/abs/2305.09836) or IQL in CORL.\n\nI suggest to run more experiments using AntMaze or all of the Adroit datasets to verify that your approach helps. We can't conclude that it helps without checking more datasets. Wihout it I'm not sure that paper should be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### run more experiments using AntMaze or all of the Adroit datasets\n\nAs AntMaze datasets are designed to investigate the stitching ability and the datasets can well cover the state-action space, our approach may not help significantly. Therefore, we perform experiments on Adroit datasets. We report average results across five seeds after training for 50 epochs on Adroit-expert datasets. On these datasets, RD significantly improve TD3-N-Unc's performance as shown in the following table. We also tried our best to tune algorithms on the remaining Adroit-cloned and Adroit-human datasets, however, it seems none of widely-used algorithms and backbone algorithms can obtain final models that have satisfying average performance, and our approach also failed to help improving the evaluation performance of final models. This phenomenon indicates RD cannot boost existing methods from the perspective of reining generalization on these datasets and more in-depth analysis of potential factors is required (e.g., limited data quantity), which is left for future works.\n\n| Dataset            | TD3-N-Unc       | TD3-N-Unc + RD  |\n| ------------------ | --------------- | --------------- |\n| pen-expert-v1      | 61.8 $\\pm$ 26.1 | **84.7 $\\pm$ 5.1**  |\n| door-expert-v1     | 11.1 $\\pm$ 3.2  | **75.8 $\\pm$ 29.4** |\n| relocate-expert-v1 | 1.9 $\\pm$ 2.9   | **12.8 $\\pm$ 9.0**  |\n| hammer-expert-v1   | 0.7 $\\pm$ 0.9   | **20.9 $\\pm$ 41.5** |\n\nNote that in addition to the auxiliary loss, all the other  parameters are kept the same across different methods to ensure fairness.\n\n### How does you modification affect training runtime\n\nWe counted the mean training time of one epoch (1000 gradient steps) of TD3-N-Unc and TD3-N-Unc + RD on all MuJoCo datasets,  the training time of TD3-N-Unc + RD is 1.3x that of TD3-N-Unc.  Although training time of one epoch is increased, we provide evidence in Table 6 (appendix C) of our paper, supporting the efficacy of RD in enhancing convergence speed and its potential to reduce the quantity of Q ensembles. As shown in Table 6, the performance at 1M and 3M steps of both backbone methods is improved by incorporating RD on a large portion of the datasets, demonstrating the effectiveness of RD in increasing convergence speed and final performance. Additionally, using RD with fewer Q ensembles can achieve similar or even better results than the backbone methods using more Q ensembles, indicating its potential in reducing computing resource consumption. "}]

Author rebuttal:

### analyses on how RD and other existing NTK-relevant approaches address generalization issues in offline RL
RD v.s. DR3 [1]

- RD and DR3 differ at both the theoretical framework from which they are derived and the regularization effect in terms of out-of-sample data. DR3 is derived from the theoretical characterizing implicit regularization in TD-Learning, which is a generalization of the implicit regularization effect from Supervised Learning as studied in [2] and [3] to TD-Learning in RL setting. In contrast, RD is derived from the Backup-Generalization framework proposed by us.

- From the angle of regularization effect, DR3 is proposed to directly counter the implicit regularization of TD-Learning while RD is to suppress the generalization between in-sample data and out-of-sample data. With some heuristics, DR3 regularizer arrives at a similar form, i.e., to minimize the NTK between consecutive state-action pairs in backup $(s,a)$ and $(s^{\prime},a^{\prime})$ (where $a^{\prime} \sim \pi(\cdot|s^{\prime})$), to RD regularizer. Apparently, DR3 regularizer can be a special case of RD regularizer in terms of the definition of out-of-sample actions. The regularization effect of DR3 is similar to the Policy-Dataset Generalization Inhibition shown by Figure 2 in our paper. Such a regularization can induce over-inhibition of generalization when the distribution of current policy overlaps with the offline dataset as illustrated.

RD v.s. MSG [4]
- MSG use NTK to characterize the difference of learning dynamics between Independent-taget ensemble and Shared-target ensemble for ensemble-based pessimistic offline RL methods, showing that commonly adopted Shared-target ensemble method could lead to a paradoxically optimistic estimate. The NTK analysis in MSG paper is not directly related to generalization issues in offline RL. Besides, MSG also has nothing to do with representation.
- One thing to notice is that MSG uses the general infinite-width NTK regime proposed by [5], while we follow the NTK notion especially in the context of RL proposed in [6]. Although the latter one is a derivative notion of the former one, they are often different in the settings and aims of analysis at least for RD and MSG here.

RD v.s. DOGE [7]

- DOGE uses NTK to characterize the difference of Q-function generalization for interpolated and extrapolated data, showing that Q-function generalizes better to interpolated state-action pairs. This then motivates the proposal of a new plug-in policy constraint that regularizes the learning policy to select actions in the approximated interpolated action space. Such a policy constraint can relieve the over-conservatism issue in prior policy constraint-based offline RL methods.
- In our paper, we use NTK to characterize the general generalization case within the Backup-Generalization framework. The major difference is that DOGE addresses the generalization issue by leveraging a less conservative policy constraint, while RD regularizes the generalization on the level of representation. To some degree, DOGE explicitly controls the policy space where  generalization issues are addressed; in contrast, RD can be viewed as realizing an implicit control of the policy space by the penultimate-layer representation. Interestingly, we think there is chance to make use the definition and approximation method of interpolated data and extrapolated data in DOGE paper to design new representation regularization schemes. We will consider this as a future direction.

We appreciate the reviewer's inspiring comment and we will add more analysis on the differences between RD and the related works above in our paper as suggested. 
#### Reference
[1] Kumar et al. DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization. ICLR 2022.

[2] Blanc et al. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on learning theory, pp.483–513. PMLR, 2020.

[3] Damian et al. Label noise sgd provably prefers flat global minimizers. arXiv preprint arXiv:2106.06530, 2021.

[4] Ghasemipour et al. Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters. NeurIPS 2022.

[5] Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1806.07572, 2018.

[6] Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint arXiv:1903.08894, 2019.

[7] Li et al. When data geometry meets deep function: Generalizing offline reinforcement learning. ICLR 2023.

Meta-review: All reviewers are agreed that this submission should be accepted. After having gone through the paper myself, I agree.

I would recommend the authors improve their manuscript based on the suggestions given by the reviewers, and include the new experiments run during the rebuttals.
Decision: Accept (poster)