SlowGuess/NeurIPS_2023_OCR huggingface dataset:
of explicit pessimism, can achieve commendable performance, substantially surpassing the baseline SAC. These outcomes underline the potency of inhibiting generalization to eliminate overestimation in offline setting.
Further, we consider the phase in the right of Fig.2. To improve the robustness of the Q function by circumventing the excessive generalization of the Q-function to OOD data, we introduce a simple OOD data generator that creates data with lower Q-values compared to the data produced by current policy  $\pi$ . Specifically, the OOD data generator  $\pi_{\mathrm{odd}}$  is trained to generate OOD state-action pairs that satisfy the following condition:  $\alpha * Q_{\phi}(s,a) < Q_{\phi}(s,\pi_{\mathrm{odd}}(s)) < \beta * Q_{\phi}(s,a)$ , where  $Q_{\phi}(s,a)$  and  $Q_{\phi}(s,\pi_{\mathrm{odd}}(s))$  are assumed to be positive for ease of presentation, and  $0 < \alpha < \beta < 1$  is the predefined hyperparameters. In practice, the optimization of  $\pi_{\mathrm{odd}}$  is implemented via the following margin ranking loss:
$$
\mathcal {L} _ {\mathrm {O O D}} = \max  \left(0, Q _ {\phi} (s, \pi_ {\mathrm {o o d}} (s)) - \beta * Q _ {\phi} (s, a)\right) + \max  \left(0, \alpha * Q _ {\phi} (s, a) - Q _ {\phi} (s, \pi_ {\mathrm {o o d}} (s))\right) \tag {5}
$$
As the offline RL training progresses, we gradually shift our focus from distinguishing in-sample data and data generated by  $\pi$  to distinguishing data generated by  $\pi$  and data generated by the OOD actor  $\pi_{\mathrm{odd}}$  in the representation space, i.e.,  $\mathcal{L}_2 = \underset {s\sim \mathcal{D}}{\mathbb{E}}|\Phi (s,\pi_{\mathrm{odd}}(s))^{\top}\Phi (s,\pi (s))|$ . In practice, a simple heuristic approach is proposed to help balance  $\mathcal{L}_1$  and  $\mathcal{L}_2$ :
$$
\mathcal {L} _ {\mathrm {R D}} = (1 - w) * \mathcal {L} _ {1} + w * \mathcal {L} _ {2} \tag {6}
$$
where  $w$  gradually increases according to  $w = \tanh\left(\frac{\text{step\_num}}{\mathbf{M}}\right)$ , step_num refers to the current training step, and  $\mathbf{M}$  is a hyperparameter. During the update of  $\pi_{\mathrm{odd}}$ , the gradient towards  $Q_{\phi}$  is detached.
We would like to emphasize that the OOD generator is not required to be meticulously trained, as we only expect the Q value of the generated OOD data to be appropriately lower than that of the data generated by the current policy. This is due to the fact that the OOD data is not employed to directly influence the policy's decision-making process. Instead, it serves to inhibit the excessive generalization of the Q function, thereby enhancing the robustness.
5.2 Applying RD in Offline RL Algorithms
Algorithm 1: Offline RL with RD (Pseudocode in a PyTorch-like style)
1 # replay_buffer: save dataset
2 # agent: offline RL agent
3 # itv: OOD actor and RD updating interval
4
5 # Loading dataset into replay buffer (omitted)
6 ...
7 for t in range(max_learning_steps):
8 agent.sample_data(replay_buffer)
9 # Update OOD actor and RD according to Eq.5 and 6 respectively at intervals
10 If t % itv == 0:
11 agent.update_oodActor()
12 agent.update_RD()
13 # Update critic and actor following the offline RL algorithm design
14 agent.update_critic()
15 agent.update Actor()
16 ...
17 ...
While the introducing of RD to the offline learning process offers benefits, it alone cannot boost the performance of a standard online RL algorithm to a cutting-edge level, as depicted in Fig.3. This is because merely reining generalization from the representation perspective does not necessarily prevent overestimation of the Q-value. Our objective is to construct RD as a universally applicable plug-in to enhance any offline RL algorithm. Thus, to evaluate RD's versatility as a plug-in, we apply it in both existing algorithms and two designed rudimentary backbone algorithms named SAC-N-Unc and TD3-N-Unc. By modifying the original SAC/TD3 algorithm to employ a critic ensemble of number N and incorporate an uncertainty regularization term within the policy update process, we derive these backbone algorithms. It should be noted that these algorithms are not the original contribution of this paper as many similar uncertainty-based methods [10, 43, 12, 44, 45, 46] have been proposed. Details of the backbone algorithms are provided in the appendix.
Note that RD is algorithm-agnostic, meaning it can be applied to a wide range of actor-critic offline RL algorithms. Incorporating RD into offline RL algorithms is also straightforward, requiring only
the addition of two steps to the conventional policy updating scheme, as illustrated in Algorithm 1 in the appendix. In each training iteration, the OOD data generator is first trained to generate data with lower Q-values than the current policy. Then, the Q-function is updated by minimizing the loss function  $\mathcal{L} = \epsilon * \mathcal{L}_{\mathrm{RD}} + \mathcal{L}_{\mathrm{critic}}$ , and the policy is updated based on  $\mathcal{L}_{\mathrm{actor}}$ . Here,  $\epsilon$  represents a weighted factor, and  $\mathcal{L}_{\mathrm{critic}}$  and  $\mathcal{L}_{\mathrm{actor}}$  refer to the optimization objectives of any offline RL algorithm. A pseudocode using RD in offline RL is given in Algorithm 1. In the experimental section, we demonstrate the versatility of RD by integrating it into both backbone algorithms and existing offline RL algorithms.
We evaluate RD by applying it on TD3BC [7], CQL [8], SAC-N-Unc and TD3-N-Unc through a series of experiments on D4RL [47] gym MuJoCo-v2 and Adroit-v1 datasets, where the former is a dataset commonly used in previous work for continuous control tasks, and the latter poses a significant challenge for most offline RL methods due to its sparse reward property. We compare our method with BC and several model-free offline RL algorithms, including DT [48], TD3BC [7], CQL [8], IQL [49], EDAC [10], and Diffusion-QL [50]. We obtain the results of the baselines by re-running the official codes or directly extracting them from the original papers. In our experiments, all the re-run baselines and our algorithm are executed with five random seeds, and we report the average normalized results of the final ten evaluations. We report the performance at 1M gradient step for TD3BC and CQL and that at 3M gradient step for SAC-N-Unc and TD3-N-Unc on MuJoCo tasks. For Adroit tasks, we report results at 500K gradient step. Note that due to the space limitation, we abbreviate the names of the datasets from {RANDOM, MEDIUM, MEDIUM-REPLAY, MEDIUM-EXPERT, EXPERT} to {R, M, MR, ME, E} in all the tables.
Table 1: Results of different algorithms and the ones equipped with RD
DATASET	TD3-N-UNC	TD3-N-UNC +RD	SAC-N-UNC	SAC-N-UNC +RD	TD3BC	TD3BC +RD	CQL	CQL +RD
HALFCHEETAH-M	66.8 ± 0.5	66.8 ± 1.2	65.9 ± 1.0	65.9 ± 1.9	48.0 ± 0.3	48.3 ± 0.5	47.1 ± 0.2	53.0 ± 0.5
HALFCHEETAH-MR	53.4 ± 3.9	57.7 ± 0.9	53.2 ± 5.4	61.5 ± 1.4	44.6 ± 0.3	44.6 ± 0.5	45.2 ± 0.6	51.6 ± 0.9
HALFCHEETAH-ME	97.7 ± 2.2	101.1 ± 0.4	99.4 ± 2.5	102.5 ± 1.8	90.5 ± 6.6	93.9 ± 2.9	81.1 ± 6.0	90.2 ± 5.8
HOPPER-M	41.9 ± 50.5	103.0 ± 0.8	45.7 ± 41.0	102.8 ± 0.2	60.4 ± 4.0	61.0 ± 2.6	65.0 ± 6.1	74.9 ± 7.1
HOPPER-MR	92.5 ± 18.1	104.1 ± 0.8	104.7 ± 0.9	104.6 ± 0.4	61.2 ± 20.5	72.1 ± 8.4	87.7 ± 14.4	100.3 ± 3.2
HOPPER-ME	100.3 ± 22.6	110.7 ± 0.6	110.9 ± 0.2	110.6 ± 0.3	105.4 ± 6.1	104.8 ± 2.8	93.9 ± 14.3	98.2 ± 9.7
WALKER2D-M	69.9 ± 35.2	97.6 ± 3.4	24.2 ± 28.2	92.3 ± 1.3	82.7 ± 5.5	83.7 ± 2.7	80.4 ± 3.5	84.5 ± 1.0
WALKER2D-MR	91.6 ± 2.7	92.1 ± 2.7	85.2 ± 2.7	86.9 ± 3.1	82.1 ± 2.5	84.8 ± 1.4	79.2 ± 5.0	94.4 ± 2.5
WALKER2D-ME	90.6 ± 45.0	118.8 ± 1.2	113.1 ± 9.6	116.4 ± 1.5	110.2 ± 0.5	110.1 ± 0.5	109.7 ± 0.5	113.0 ± 0.5
Table 2: Average normalized scores of our methods and previous methods on the D4RL benchmark.
DATASET	BC	DT	TD3BC	CQL	IQL	EDAC	DIFFUSION-QL	SAC-N-UNC+RD	TD3-N-UNC+RD
HALFCHEETAH-R	2.2	2.2	11.0	31.3	13.7	28.4	22.0	25.4	31.0
HOPPER-R	3.7	5.4	8.4	5.3	8.4	25.3	18.3	31.6	31.7
WALKER2D-R	1.3	2.2	1.7	5.4	5.9	16.6	5.5	21.2	21.7
HALFCHEETAH-M	42.6	42.6	48.0	47.1	47.4	65.9	51.5	65.9	66.8
HOPPER-M	52.9	67.6	60.4	65.0	66.3	101.6	96.6	102.8	103.0
WALKER2D-M	75.3	74.0	82.7	80.4	78.3	92.5	87.3	92.3	97.6
HALFCHEETAH-MR	36.6	36.6	44.6	45.2	44.2	61.3	48.3	61.5	57.7
HOPPER-MR	18.1	82.7	61.2	87.7	94.7	101.0	102.0	104.6	104.1
WALKER2D-MR	26.0	66.6	82.1	79.2	73.9	87.1	98.0	86.9	92.1
HALFCHEETAH-ME	55.2	86.8	90.5	81.1	86.7	106.3	97.2	102.5	101.1
HOPPER-ME	52.5	107.6	105.4	93.9	91.5	110.7	112.3	110.6	110.7
WALKER2D-ME	107.5	108.1	110.2	109.7	109.6	114.7	111.2	116.4	118.8
HALFCHEETAH-E	91.8	87.7	96.7	97.3	94.9	106.8	96.3	108.8	103.1
HOPPER-E	107.7	94.2	107.8	106.5	108.8	110.1	102.6	109.8	108.8
WALKER2D-E	108.7	108.3	110.2	109.3	109.7	115.1	109.5	112.3	111.2
MUJoCO TOTAL	782.1	972.6	1020.9	1044.4	1034.0	1243.4	1158.6	1252.6	1259.4
PEN-HUMAN	25.8	73.9	-1.9	35.2	71.5	52.1	75.7	61.1	77.9
PEN-CLONED	38.3	67.3	9.6	27.2	37.3	68.2	60.8	53.0	65.5
ADROIT TOTAL	64.1	141.2	7.7	62.4	108.8	120.3	136.5	114.1	143.4
TOTAL	846.2	1113.8	1028.6	1106.8	1142.8	1363.7	1295.1	1366.7	1402.8
To demonstrate the applicability of RD, we integrate RD into the training regime of TD3BC, CQL, TD3-N-Unc and SAC-N-Unc. As observed from Table 1, both TD3BC and CQL exhibit improved overall performance when RD is applied. It is noteworthy as the core idea of CQL to increase the Q value of the data in the dataset and diminish the Q value of the data generated by a mixed policy comprising the learning policy and random policy, which incorporates the representation distinction of state-action pairs sourced from different distributions. Therefore in practice, we apply the insights of RD to gradually transfer the original Q value restriction to the Q-value differentiation between the mixed policy and the designed OOD policy. As a result, CQL's performance is significantly improved. Moreover, the backbone algorithms themselves yield substantial advantages from the application of RD. Table 2 further reveals that backbone algorithms equipped with RD can achieve the best overall performance and attains state-of-the-art performance on several datasets. Additional evidence supporting the efficacy of RD in enhancing convergence speed and its potential to reduce the quantity of Q ensembles is provided in the appendix.
6.2 Comparison and discussion with techniques that inhibit overestimation
Table 3: Average normalized scores of TD3-N-Unc with RD or other variants, and DR3 and Layer Norm that inhibit potential overestimation
DATASET	TD3-N-UNC	TD3-N-UNC +PDD	TD3-N-UNC +PDDDAW	TD3-N-UNC +RDD	TD3-N-UNC POD	TD3-N-UNC +DR3	TD3-N-UNC +LAYERNORM	TD3-N-UNC RD
HALFCHEETAH-M	66.8 ± 0.5	66.1 ± 0.9	66.6 ± 0.7	65.9 ± 0.8	63.1 ± 2.4	64.4 ± 1.7	63.2 ± 0.8	66.8 ± 1.2
HOPPER-M	41.9 ± 50.5	99.9 ± 6.4	100.7 ± 6.2	77.4 ± 38.6	82.8 ± 40.5	103.4 ± 0.7	83.0 ± 29.0	103.0 ± 0.8
WALKER2D-M	69.9 ± 35.2	94.9 ± 1.7	66.7 ± 38.1	80.2 ± 31.0	94.2 ± 4.7	92.1 ± 2.0	68.5 ± 20.7	97.6 ± 3.4
HALFCHEETAH-E	94.6 ± 11.5	96.3 ± 13.2	102.9 ± 1.2	99.9 ± 4.5	93.7 ± 13.0	100.0 ± 3.7	104.4 ± 1.5	103.1 ± 0.6
HOPPER-E	110.9 ± 0.5	108.4 ± 0.4	108.4 ± 0.4	108.4 ± 0.4	108.2 ± 0.7	108.0 ± 0.5	88.4 ± 42.8	108.8 ± 0.3
WALKER2D-E	4.1 ± 7.1	67.4 ± 47.2	60.8 ± 46.1	28.5 ± 40.8	44.1 ± 54.0	109.9 ± 0.4	111.7 ± 0.4	111.2 ± 0.7
We compare our method RD with the existing techniques that inhibit potential overestimation including DR3 [41] and Layer Norm [51]. In addition to the regularizer, all the other parameters are kept the same across different methods to ensure fairness. As shown in Table 3, RD is more helpful in helping improving the backbone algorithm than DR3 and Layer Norm.
RD and DR3 differ mainly from two perspectives. From the angle of the framework derived from, DR3 is derived from the theoretical characterizing implicit regularization in TD-Learning, which is a generalization of the implicit regularization from Supervised Learning [52, 53] to TD-Learning in RL setting. In contrast, RD is derived from the proposed Backup-Generalization framework. From the angle of regularization effect, DR3 is proposed to directly counter the implicit regularization of TD-Learning while RD is to suppress the generalization between in-sample data and OOD data. With some heuristics, DR3 regularizer arrives at a similar form, i.e., to minimize the NTK between consecutive state-action pairs in backup, to RD regularizer. Apparently, DR3 regularizer can be a special case of RD regularizer in terms of the definition of out-of-sample actions. The regularization effect of DR3 is similar to the Policy-Dataset Generalization Inhibition shown by Figure 2. Such a regularization can induce over-inhibition of generalization when the distribution of current policy overlaps with the offline dataset as illustrated. We also show in the following that DR3 and Policy-Dataset Generalization Inhibition (i.e., PDD in the following part) achieve similar empirical results.
6.3 Ablation Studies and In-depth Analysis
We design ablation experiments to elucidate the significance of RD's components. We compare RD against several variants including, including Policy-Dataset Distinction (PDD), Random-Dataset Distinction (RDD), Policy-OOD Distinction (POD), and Policy-Dataset Distinction with Dynamically Adjusted Weight (PDDDAW). PDD refers to the auxiliary representation loss calculated using data generated by the learning policy and that from the dataset. Likewise, RDD utilizes data generated by a random policy versus that from the dataset, while POD employs data produced by the learning policy against that by the OOD actor. As for PDDDAW, its PDD component is reweighted using the same heuristic  $w$  adjustment method as RD, but the POD component is persistently weighted by zero. We draw a comparison between TD3-N-Unc + RD and these variants, as tabulated in Table 3. The results demonstrate that RD, fundamentally a dynamic amalgamation of PDD and POD, attains superior overall performance, underscoring the indispensability of each RD component.
Figure 4: Performance using RD and PDD with same seed.
Figure 5: T-SNE visualization of representation via PDD.
Figure 6: T-SNE visualization of representation via RD.
It is worth noting that using PDD can help backbone algorithms achieve relatively satisfactory performance. However, as depicted in Fig 4, algorithm using PDD occasionally suffers performance degradation during the later stages of training, consequently impairing the overall efficacy of the approach. To elucidate this phenomenon, the final models trained via RD and PDD, as shown in Fig 4, are saved and the quality of the representations obtained by these models are compared. A total of 100 state-action pairs are sampled from the halfcheetah-expert, halfcheetah-medium, and halfcheetah-random datasets, respectively, and then consolidated. Using t-Distributed Stochastic Neighbor Embedding (t-SNE), the distribution of the representations of these 300 state-action pairs derived from both models is charted. Each sample is marked with distinctive symbols (stars for random data, squares for medium data, and circles for expert data) to differentiate samples from diverse datasets. Moreover, varying colors are assigned to individual samples corresponding to the Q-value estimated by the proficiently trained Q-value network acquired by RD, with brighter colors signifying higher Q-values. As depicted in Fig 5 and 6, the RD-derived representation exhibits superior differentiation between state-action pairs with high and low Q-values, whereas the PDD-derived representation frequently misclassifies expert and medium data.
To better demonstrate that the representations learned by RD are superior, we calculate which dataset the top five nearest samples to each sample belong to, and compute the mean over all samples from the same dataset in Table 4. For instance, the EXPERT-EXPERT metric of representation learned via RD indicates that  $94\%$  of the five nearest samples to the expert samples are also expert samples. We can conclude that representation learned via RD can better help samples from the same dataset to cluster closer. More results are provided in the appendix.
Table 4: The proportion of the five samples closest to each sample
REP VIA PDD	REP VIA RD
EXPERT-EXPERT	0.87	0.94
EXPERT-MEDIUM	0.12	0.06
EXPERT-RANDOM	0.01	0.00
MEDIUM-EXPERT	0.38	0.38
MEDIUM-MEDIUM	0.43	0.51
MEDIUM-RANDOM	0.19	0.11
This paper analyzes the important role of generalization in offline RL and presents a novel plug-in method, Representation Distinction (RD), to enhance the performance of offline RL algorithms by reining the generalization. By explicitly differentiating between in-sample and OOD state-action pairs, the generalization across data could be properly inhibited. Our extensive experiments demonstrate the efficacy of the RD method when applied to several offline RL algorithms, significantly enhancing their performance across various continuous control tasks on D4RL datasets. In conclusion, our work contributes a valuable perspective to the field of offline RL by focusing on reining generalization from the representation perspective and providing a flexible solution that can be applied to a variety of existing algorithms.
However, our work is not without limitations. For example, currently RD primarily focus on continuous control tasks, and its applicability to discrete control tasks is yet to be explored. In addition, the transition between the two phases of RD is a heuristic method, which could be improved by designing adaptive adjustment methods. Future research directions may include addressing the aforementioned limitation.
[41] Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations.
[42] Marc G Bellemare, Georg Ostrovski, Arthur Guez, Philip Thomas, and Rémi Munos. Increasing the action gap: New operators for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[43] Alex Beeson and Giovanni Montana. Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning. arXiv preprint arXiv:2303.14716, 2023.
[44] Hongchang Zhang, Boyuan Wang, Yixiu Mao, Jianzhun Shao, Yuhang Jiang, Yi Xu, and Xiangyang Ji. Pessimistic policy iteration for offline reinforcement learning, 2023.
[45] Kamyar Ghasemipour, Shixiang (Shane) Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 18267-18281. Curran Associates, Inc., 2022.
[46] Jordi Smit, Canmanie T Ponnambalam, Matthijs TJ Spaan, and Frans A Oliehoek. Pebl: Pessimistic ensembles for offline deep reinforcement learning. In Robust and Reliable Autonomy in the Wild Workshop at the 30th International Joint Conference of Artificial Intelligence, 2021.
[47] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[48] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
[49] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
[50] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022.
[51] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[52] Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an Ornstein-uhlenbeck like process. In Conference on learning theory, pages 483–513. PMLR, 2020.
[53] Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34:27449-27461, 2021.
[54] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenikov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as a "Launchpad", 2022.