SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\max  _ {\psi} \mathbb {E} _ {s \sim \mathbb {P} _ {\mathcal {D}}, a \sim \pi_ {\psi} (\cdot | s), (s, a) \in \tilde {\mathcal {S}} \times \tilde {\mathcal {A}}} [ Q _ {\eta} (s, a) ], \tag {7}
$$
where  $\pi_{\psi}(\cdot |s)$  is the learning policy, and  $\tilde{S}\times \tilde{\mathcal{A}}$  is an estimate of the bounded safe area  $\mathcal{S}\times \mathcal{A}$  in Definition 3.1.
There are many approaches for specifying the safe area  $\tilde{S} \times \tilde{\mathcal{A}}$ . Kumar et al. [17] bounds the expected MMD distance between the learning policy and the behavior policy within a predetermined threshold, while Wu et al. [22] constrains the lower bound of the estimated behavior density to identify the safe region. In this work, since we can have a direct density estimation of  $L_{\theta}^{\pi_{\beta}}(s,a)$ , we can utilize it and specify the safe region as the region with not too small density. Specifically, we set  $\tilde{S} \times \tilde{\mathcal{A}} = \{(s,a) \in S \times \mathcal{A} : -\log L_{\theta}^{\pi_{\beta}}(s,a) < \epsilon\}$ , where  $\epsilon$  is the threshold<sup>3</sup>.
The critic training is following the traditional Bellman equation as shown in Eq.1. The entire algorithm of CPED is summarized in Algorithm 1. In CPED, the Flow-GAN model plays a crucial role in estimating the behavior density. The density function is obtained from the generator model and is iteratively updated along with the discriminator. We further use the TD3 framework to solve the RL problem. It is noted that during Actor updates, the density of sampled actions should be higher than the threshold to avoid out-of-distribution areas.
In this section, we provide theoretical analysis of our proposed method. In Section 4.1, we prove the convergence of GAN with the hybrid loss, and based on this result, we show the convergence of CPED in Section 4.2.
$^3$ Eq.7 is a constrained optimization problem, and we are indeed using Lagrangian techniques to solve the constraint problem. During the practical optimization, the constraint on  $\vec{S} \times \vec{\mathcal{A}} = \{(s,a) \in S \times \mathcal{A} : -\log L_{\theta}^{\pi_{\beta}}(s,a) < \epsilon\}$  in Eq.7 works as penalty terms in the Lagrange function (with Lagrangian multiplier  $\alpha$ ).
Algorithm 1 The CPED algorithm
Input: dataset  $\mathcal{D}$  target network update rate  $\kappa$  Lagrange multiplier  $\alpha$  training ratio of the generator and discrimination  $r$  mini-batch size  $N$  hyperparameters  $\lambda ,c$    
Initialize generator  $G_{\theta}$  discriminator  $D_{\phi}$  loss function in GAN  $\mathcal{L}_{\mathcal{G}}(\cdot)$  , behavior policy likelihood  $L_{\theta}^{\pi_{\beta}}(\cdot)$  Q networks  $\{Q_{\eta_1},Q_{\eta_2}\}$  actor  $\pi_{\psi}$  target networks  $\{Q_{\eta_1}',Q_{\eta_2'}\}$  , target actor  $\pi_{\psi '}$  with  $\psi^{\prime}\gets \psi ,\eta_{i}^{\prime}\gets \eta_{i},i = 1,2.$    
for  $i = 1$  to  $M$  do Sample mini-batch of transitions  $(s,a,r,s')\sim \mathcal{D}$    
Initial Training Flow-GAN:  $\phi \leftarrow \mathrm{argmax}_{\phi}\mathcal{L}_{\mathcal{G}}(G_{\theta},D_{\phi})$ $\theta \leftarrow \mathrm{argmin}_{\theta}\max_{\phi}\mathcal{L}_{\mathcal{G}}(G_{\theta},D_{\phi}) - [\lambda \mathbb{E}_{(s,a)\sim \mathcal{D}}\log (L_{\theta}^{\pi_{\beta}}(s,a))]$    
end for   
for  $i = 1$  to  $N$  do Sample mini-batch of transitions  $(s,a,r,s')\sim \mathcal{D}$    
Updating Flow-GAN:  $\phi \leftarrow \mathrm{argmax}_{\phi}\mathcal{L}_{\mathcal{G}}(G_{\theta},D_{\phi})$ $\theta \leftarrow \mathrm{argmin}_{\theta}\max_{\phi}\mathcal{L}_{\mathcal{G}}(G_{\theta},D_{\phi}) - [\lambda \mathbb{E}_{(s,a)\sim \bar{\mathcal{D}}}\log (L_{\theta}^{\pi_{\beta}}(s,a))]$    
Updating Q-function: Get action for the next state,  $\{\tilde{a}\sim \pi_{\psi '}(\cdot |s') + \varepsilon ,\varepsilon \sim clip(\mathcal{N}(0,\sigma), - c,c)\}$  Let  $y(s,a)\coloneqq \min (Q_{\eta_1'}(s',\tilde{a}),Q_{\eta_2'}(s',\tilde{a}))$ $\eta_i\gets \mathrm{argmin}_{\eta_i}(Q_{\eta_i}(s,a) - (r + \gamma y(s,a)))^2,i = 1,2$    
Updating Actor: Update  $\psi$  according to Eq.7, by using dual gradient descent with Lagrange multiplier  $\alpha$    
Update Target Networks:  $\psi^{\prime}\gets \kappa \psi +(1 - \kappa)\psi^{\prime};\eta_{i}^{\prime}\gets \kappa \eta_{i} + (1 - \kappa)\eta_{i}^{\prime},i = 1,2$    
end for
4.1 Convergence of GAN with the Hybrid Loss
In this subsection, we show that training GAN with the hybrid loss function Eq.4 can yield an accurate density estimator. For ease of presentation, assume we observed  $X_{j}\sim p^{*} = \frac{\mathrm{d}\mu^{*}}{\mathrm{d}\nu}, j = 1,\dots,n$ , where  $\mu^{*}$  is the underlying true probability measure (the probability measure of behavior policy), and  $\nu$  is the Lebesgue measure. Motivated by [24, 75, 76], we consider the Integral Probability Metric (IPM) defined as
$$
d _ {\mathcal {F} _ {d}} \left(\mu_ {1}, \mu_ {2}\right) := \sup  _ {f \in \mathcal {F} _ {d}} \mathbb {E} _ {X \sim \mu_ {1}} f (X) - \mathbb {E} _ {Y \sim \mu_ {2}} f (Y), \tag {8}
$$
where  $\mu_{1}$  and  $\mu_{2}$  are two probability measures, and  $\mathcal{F}_d$  is a discriminator class. Under IPM, the GAN framework with the hybrid loss Eq.4 solves the empirical problem (cf., Eq. (1.1) of [75] for the original GAN loss function)
$$
\mu_ {n} \in \underset {\mu \in \mathcal {Q} _ {g}} {\operatorname {a r g m i n}} \max  _ {f \in \mathcal {F} _ {d}} \int_ {\Omega} f \mathrm {d} \mu - \int_ {\Omega} f \mathrm {d} \tilde {\mu} _ {n} - \lambda \int_ {\Omega} \log p \mathrm {d} \hat {\mu} _ {n}, \tag {9}
$$
where  $\hat{\mu}_n = \frac{1}{n}\sum_{j=1}^n\delta_{X_j}$  is the empirical measure,  $\tilde{\mu}_n$  is a (regularized) density estimation,  $\mathcal{Q}_g$  is the generator class, and  $p = \frac{\mathrm{d}\mu}{\mathrm{d}\nu}$ . In the following (informal) theorem, we show that  $\mu_n$  is an accurate estimator of the underlying true distribution. A detailed version of Theorem 4.1 and its proof can be found in Appendix A.
Theorem 4.1 (Informal). Suppose the generator class  $\mathcal{Q}_g$  and the discriminator class  $\mathcal{F}_d$  are induced by some Sobolev spaces. Choose  $\lambda$  as a constant. Under certain conditions,
$$
d _ {\mathcal {F} _ {d}} \left(\mu^ {*}, \mu_ {n}\right) = O _ {\mathbb {P}} \left(n ^ {- 1 / 2}\right), \operatorname {K L} \left(\mu^ {*} \mid \mid \mu_ {n}\right) = O _ {\mathbb {P}} \left(n ^ {- 1 / 2}\right),
$$
where  $\mu^{*}$  is the true probability measure,  $\mu_{n}$  is as in Eq.9 with hybrid loss, and  $\mathrm{KL}(\mu^{*}||\mu_{n})$  is the KL divergence.
Theorem 4.1 states a fast convergence rate  $O_{\mathbb{P}}(n^{-1 / 2})$  of the IPM and KL divergence can be achieved. Therefore, applying the hybrid loss can provide not only a good generator, but also provide an accurate density estimate, which can be utilized in our CPED framework to control the safe region.
Remark 4.1. Given the universal approximation theorem [77, 78], the neural networks can approximate any smooth function. Thus, we consider Sobolev function classes instead of neural networks for the ease of mathematical treatment. If both the generator and discriminator classes are neural networks, one may adopt similar approach as in [75, 79] and then apply Theorem 4.1.
Remark 4.2. In [75], the convergence rate of  $\mathbb{E}d_{\mathcal{F}_d}(\mu^*,\mu_n)$  is considered, while we provide a sharper characterization of  $d_{\mathcal{F}_d}(\mu^*,\mu_n)$ , by showing the convergence is in probability with certain convergence rate.
4.2 Convergence of CPED
With an accurate estimate of the underlying density, in this subsection, we give a theoretical analysis of how CPED can find the optimal  $Q$ -function value over the product space of  $\mathcal{S} \times \mathcal{A}$ . The proof of Theorem 4.2 can be found in Appendix C.
Theorem 4.2. Suppose  $Q^{*}(s,a)$  is the optimal  $Q$ -value on product space  $\mathcal{S} \times \mathcal{A}$ . Let  $\hat{\pi}(\cdot|s) \coloneqq \operatorname{argmax}_{a} Q^{*}(s,a)$  be a learning policy. Let  $\hat{\pi}^{\Delta}(\cdot|s)$  be the probability-controlled learning policy using CPED, which is defined on the support of  $\pi_{\beta}$ . Suppose  $\hat{\pi}^{\Delta}(\cdot|s) > 0$  on  $\mathcal{A}$ . Then with probability tending to one,
$$
Q ^ {*} (s, a) = Q (s, \hat {\pi} ^ {\Delta} (\cdot | s)). \tag {10}
$$
Furthermore, when minimizing the Bellman residual to solve the optimal  $Q$ -function and training the policy by iteration, we have the following theorem, whose proof is provided in Appendix D.
Theorem 4.3. Suppose in step  $k + 1$ , the learned policy is defined as  $\pi_{k + 1} := \operatorname{argmax}_a Q^{\pi_k}(s, a), \forall s \in \mathcal{S}$ . Let  $V^{\pi_k}$  be the value function related to  $\pi_k$  in step  $k$ , defined as  $V^{\pi_k}(s) = \mathbb{E}_{a \sim \pi_k(\cdot | s)}[Q(s, a)]$ , and  $V^*$  be the optimal value function over the product space  $\mathcal{S} \times \mathcal{A}$ . Let  $\hat{\pi}_k^\Delta$  be the probability-controlled learning policy using CPED in step  $k$ , which is defined on the support of  $\pi_\beta$ . Then with probability tending to one,
$$
\left\| V ^ {\hat {\pi} _ {k + 1} ^ {\Delta}} - V ^ {*} \right\| _ {\infty} \leq \gamma^ {k + 1} \left\| V ^ {\hat {\pi} _ {0} ^ {\Delta}} - V ^ {*} \right\| _ {\infty}. \tag {11}
$$
Theorem 4.3 provides assurance that CPED can achieve the optimal value function with a linear convergence rate as the iteration number approaches infinity. Despite under offline settings, this rate is consistent with the standard rate in online RL [80]. When combined with Theorems 4.1 and 4.2, we can confidently conclude that CPED's effectiveness is theoretically guaranteed.
In the experimental section, we analyze the performance of CPED on several standard offline RL tasks [81]. We compare CPED with a wide range of offline RL methods and analyze the performance of CPED. Further implementation details are provided in Appendix E.2.
5.1 Performance Comparison on Standard Benchmarking Datasets for Offline RL
The proposed CPED algorithm is evaluated on the D4RL [81] Gym-MuJoCo and AntMaze tasks. Details of these tasks are shown in Appendix E.1. As a model-free method, CPED is compared with the following model-free baselines: behavioral cloning (BC) method, BCQ [19], DT [82], AWAC [44], BEAR [17], Onestep RL [83], TD3+BC [45], CQL [13], IQL [15], SPOT [22], and PLAS [47]. In the experiments, the performance of all alternatives are measured by the average normalized scores in an episode. The experimental results reported in this paper are either from the authors' original experiments or from our replications.
Tables 1 and 2 show the experimental results of Gym-MuJoCo and AntMaze tasks, respectively. In Gym-MuJoCo tasks, CPED outperforms the competitors in most cases. In particular, our method has achieved remarkable success on the halfcheetah and walker2d tasks, especially under the medium-quality dataset. For the medium-expert task, CPED performs slightly inferior to other tasks, which is due to the fact that the expert task needs tight control so that the learning policy stays close to the behavior policy. In the AntMaze tasks, CPED performs slightly worse than IQL but outperforms the other baselines, and it achieves better returns under "unmaze" and "medium-play" tasks.
Table 1: The performance of CPED and other competing methods on the various D4RL Gym-MuJoCo tasks. We report the performance of Average normalized scores and standard deviations over 5 random seeds. med = medium, r = replay, e = expert, hc = halfcheetah, wa = walker2d, ho=hopper
Dataset	BC	AWAC	DT	Onestep	TD3+BC	CQL	IQL	SPOT	CPED(Ours)
hc-med	42.6	43.5	42.6	48.4	48.3	44.0	47.4	58.4	61.8±1.6
ho-med	52.9	57.0	67.6	59.6	59.3	58.5	66.2	86.0	100.1±2.8
wa-med	75.3	72.4	74.0	81.8	83.7	72.5	78.3	86.4	90.2±1.7
hc-med-r	36.6	40.5	36.6	38.1	44.6	45.5	44.2	52.2	55.8±2.9
ho-med-r	18.1	37.2	82.7	97.5	60.9	95.0	94.7	100.2	98.1±2.1
wa-med-r	26.0	27.0	66.6	49.5	81.8	77.2	73.8	91.6	91.9±0.9
hc-med-e	55.2	42.8	86.8	93.4	90.7	91.6	86.7	86.9	85.4±10.9
ho-med-e	52.5	55.8	107.6	103.3	98.0	105.4	91.5	99.3	95.3±13.5
wa-med-e	107.5	74.5	108.1	113.0	110.1	108.8	109.6	112.0	113.04±1.4
Total	466.7	450.7	672.6	684.6	677.4	698.5	692.4	773.0	791.7±37.8
Table 2: The performance of CPED and other competing methods on the various AntMaze tasks. We report the performance of Average normalized scores and standard deviations over 5 random seeds. med = medium, d = diverse, p = play.
Dataset	BCQ	BEAR	BC	DT	TD3+BC	PLAS	CQL	IQL	SPOT	CPED(Ours)
umaze	78.9	73.0	49.2	54.2	73.0	62.0	82.6	89.6	93.5	96.8±2.6
umaze-d	55.0	61.0	41.8	41.2	47.0	45.4	10.2	65.6	40.7	55.6±2.2
med-p	0.0	0.0	0.4	0.0	0.0	31.4	59.0	76.4	74.7	85.1±3.4
med-d	0.0	8.0	0.2	0.0	0.2	20.6	46.6	72.8	79.1	72.1±2.9
large-p	6.7	0.0	0.0	0.0	0.0	2.2	16.4	42.0	35.3	34.9±5.3
large-d	2.2	0.0	0.0	0.0	0.0	3.0	3.2	46.0	36.3	32.3±7.4
Total	142.8	142.0	91.6	95.4	120.2	164.6	218.0	392.4	359.6	376.8±23.8
Figure 2: (a) Average performance of BEAR and CPED on halfcheetah-medium task averaged over 5 seeds. BEAR can reach a bottleneck very quickly. CPED remain increasing after reaching the bottleneck. (b) The time(epoch) varying constrain parameter  $\alpha$  used in Gym-MuJoCo task. (c) The time(epoch) varying constrain parameter  $\alpha$  used in AntMaze task
To further discuss the learning dynamics of the proposed methods, the learning curve of BEAR, SPOT, and CPED in halfcheetah-medium task is shown in Figure 2(a). It is noted that the average return of BEAR increases very fast and remains unchanged after a smooth decrease in subsequent training, indicating that policy exploration is limited. This is consistent with the conservative nature of policy control methods that prevent the learning policy from straying too far from behavior policy. The pattern is similar to SPOT's learning curve, which shows a rapid increase and then enters a plateau. In contrast, the average returns of CPED methods continue to rise after a sharp increase to a
certain extent, indicating that CPED keeps exploring the data and benefits from exploration. This also confirms Theorem 4.3 (see Section C) from the experimental perspective.
Time varying hyperparameter  $\alpha$  in policy control methods. When improving actor using Eq.7, a Lagrange hyperparameter  $\alpha$  is needed for this constrained optimization problem. The larger the  $\alpha$ , the more restricted actions taken by the learning policy. Previous methods either use an auto-learning  $\alpha$  [17, 14] or a constant  $\alpha$  [18, 22]. BRAC [18] find that auto-learning  $\alpha$  performs worse than the constant  $\alpha$ . In our experiment, we apply a time (epoch) varying  $\alpha$  scheme with piece-wise constant functions and find this scheme performs better than the constant setting. The rationale behind this time-varying scheme is that we employ strong constraints for the action taken at the early stage of policy training in case the model fails [8]. The constraint is further relaxed by decreasing  $\alpha$  when the policy starts to converge. The piece-wise function of  $\alpha$  for the Mujoco tasks and AntMaze tasks are plot in Figure 2(b)-(c), and the performance results of the Mujoco tasks are shown in Figure E.5 in in Appendix E.5. We can see that the Mujoco task significantly benefits from this time varying scheme and delivers better returns in most scenarios.
Likelihood threshold  $\epsilon$ . The hyperparameter  $\epsilon$  determines the tightness of our estimated safety boundaries. A larger  $\epsilon$  leads to more relaxed boundaries and versa vice. A commonly used setting for  $\epsilon$  is the quantile of behavior policy density. However, the quantile value is difficult to set when the dimension is large. In our experiment, CPED uses the mean likelihood of the training batch as  $\epsilon$ , which decreases the effect of the estimation error of the behavior policy. See Appendix E.5 for additional discussion and details.
In this article, we present CPED, a novel approach for policy control in offline RL. CPED leverages the Flow-GAN model to estimate the density of the behavior policy, enabling the identification of safe areas and avoiding the learning of policies that visit OOD points. We validate the effectiveness of CPED both theoretically and experimentally. Theoretical analysis demonstrates that CPED has a high probability of achieving the optimal policy and generating large returns. Experimental results on the Gym-MuJoCo and AntMaze tasks demonstrate that CPED surpasses state-of-the-art competitors, highlighting its superior performance.
While CPED shows great capacity in experiment studies, there are several potential directions for its extension.. Firstly, as a generic method, GAN has diverse variants and it is necessary to try out more powerful GAN structures for density estimation. Secondly, when the feasible region is identified, how to explore the region efficiently remains another important issue, and we leave it to future work. Finally, it is an interesting direction to examine the performance of CPED in more complex scenarios, including dataset generated by multiple behavior policies or multiple agents, etc.
We would like to thank AC and reviewers for their valuable comments on the manuscript. Wenjia Wang was supported by the Guangzhou-HKUST(GZ) Joint Funding Program (No. 2023A03J0019) and Guangzhou Municipal Science and Technology Project (No. 2023A03J0003)
[1] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. volume 5, 2018.
[2] Bruno Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, Boris Lesner, and Matthieu Geist. Approximate modified policy iteration and its application to the game of tetris. Journal of Machine Learning Research, 16, 2015. ISSN 15337928.
[3] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.
[80] Alekh Agarwal, Nan Jiang, and Sham M. Kakade. Reinforcement learning: Theory and algorithms. 2019.
[81] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[82] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
[83] David Brandonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 34:4933-4946, 2021.
[84] David L Donoho, Iain M Johnstone, Gérard Kerkyacharian, and Dominique Picard. Wavelet shrinkage: asymptopia? Journal of the Royal Statistical Society: Series B (Methodological), 57 (2):301-337, 1995.
[85] Yves Meyer. Wavelets and Operators: Volume 1. Number 37. Cambridge university press, 1992.
[86] Sara van de Geer. Empirical Processes in  $M$ -estimation. Cambridge University Press, 2000.
[87] Robert A Adams and John JF Fournier. Sobolev Spaces, volume 140. Academic press, 2003.