SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bingyi Kang* Xiao Ma* Chao Du Tianyu Pang Shuicheng Yan Sea AI Lab
{bingykang,yusufma555,duchao0726}@gmail.com {tianyupang,yansc}@sea.com
Offline reinforcement learning (RL) aims to learn optimal policies from offline datasets, where the parameterization of policies is crucial but often overlooked. Recently, Diffusion-QL [37] significantly boosts the performance of offline RL by representing a policy with a diffusion model, whose success relies on a parametrized Markov Chain with hundreds of steps for sampling. However, Diffusion-QL suffers from two critical limitations. 1) It is computationally inefficient to forward and backward through the whole Markov chain during training. 2) It is incompatible with maximum likelihood-based RL algorithms (e.g., policy gradient methods) as the likelihood of diffusion models is intractable. Therefore, we propose efficient diffusion policy (EDP) to overcome these two challenges. EDP approximately constructs actions from corrupted ones at training to avoid running the sampling chain. We conduct extensive experiments on the D4RL benchmark. The results show that EDP can reduce the diffusion policy training time from 5 days to 5 hours on gym-locomotion tasks. Moreover, we show that EDP is compatible with various offline RL algorithms (TD3, CRR, and IQL) and achieves new state-of-the-art on D4RL by large margins over previous methods. Our code is available at https://github.com/sail-sg/edp.
Offline reinforcement learning (RL) is much desired in real-world applications as it can extract knowledge from previous experiences, thus avoiding costly or risky online interactions. Extending online RL algorithms to the offline domain faces the distributional shift [9] problem. Existing methods mainly focus on addressing this issue by constraining a policy to stay close to the data-collecting policy [6, 39], making conservative updates for Q-networks [17, 15, 40], or combining these two strategies [21, 12]. However, Offline RL can also be viewed as a state-conditional generative modeling problem of actions, where the parameterization of the policy network is important but largely overlooked. Most offline RL works follow the convention of
Figure 1: Efficiency and Generality. D-QL is Diffusion-QL. Left: The training time on the locomotion tasks in D4RL. Right: the performance of EDP and previous SOTA on each domain in D4RL. EDP is trained with TD3 on locomotion and IQL on the other three domains. (Best viewed in color.)
parameterizing the policy as a diagonal Gaussian distribution with the learned mean and variance. This scheme might become inferior when the data distribution is complex, especially when offline
data are collected from various sources and present strong multi-modalities [32]. Therefore, more expressive models for the policy are strongly desired.
Recently, Diffusion-QL [37] made a successful attempt by replacing the diagonal Gaussian policy with a diffusion model, significantly boosting the performance of the TD3+BC [6] algorithm. Diffusion models [34, 10] have achieved the new state-of-the-art (SOTA) in image generation tasks [26, 5], demonstrating a superior ability to capture complex data distributions.
Despite the impressive improvement that Diffusion-QL has achieved, it has two critical drawbacks preventing it from practical applications. First, training a diffusion policy with offline RL is computationally inefficient. Consider a parameterized diffusion policy  $\pi_{\theta}(a|s)$ , Diffusion-QL optimizes it by maximizing the Q value  $Q(s, a_{\theta})$  of a state  $s$  given a policy-generated action  $a_{\theta} \sim \pi_{\theta}(a|s)$ . However, sampling from a diffusion model relies on a long parameterized Markov chain (e.g., 1,000 steps), whose forward inference and gradient backpropagation are unaffordably expensive. Second, diffusion policy is not a generic policy class as it is restricted to TD3-style algorithms. As computing the sample likelihood  $\pi_{\theta}(a|s)$  is intractable in diffusion models [35], diffusion policy is incompatible with a large family of policy gradient algorithms (e.g., V-Trace [22], AWR [28], IQL [15]), which require a tractable and differentiable log-likelihood  $\log \pi_{\theta}(a|s)$  for policy improvement.
In this work, we propose efficient diffusion policy (EDP) to address the above two limitations of diffusion policies. Specifically, we base EDP on the denoising diffusion probabilistic model (DDPM) [10], which learns a noise-prediction network to predict the noise used to corrupt an example. In the forward diffusion process, a corrupted sample follows a predefined Gaussian distribution when the clean example and timestep are given. In turn, given a corrupted sample and predicted noise, we can approximate its clean version by leveraging the reparametrization trick. Based on this observation, to avoid the tedious sampling process, we propose action approximation to build an action from a corrupted one, which can be easily constructed from the dataset. In this way, each training step only needs to pass through the noise-prediction network once, thus substantially reducing the training time. As experimented, by simply adding action approximation, we obtain  $2\mathbf{x}$  speed-up without performance loss. Moreover, we apply DPM-Solver [20], a faster ODE-based sampler, to further accelerate both the training and sampling process. Finally, to support likelihood-based RL algorithms, we leverage the evidence lower bound for the likelihood developed in DDPM and approximate the policy likelihood from a constructed Gaussian distribution with variance fixed and mean obtained from action approximation.
We evaluate the efficiency and generality of our method on the popular D4RL benchmarking, as shown in Fig. 1. We first benchmark the efficiency of EDP on gym-locomotion tasks. By replacing the diffusion policy in Diffusion-QL with our EDP, the training time of Diffusion-QL is reduced substantially from five days to five hours (compared to their official code). Meanwhile, we observe slight to clear performance improvements on different tasks as the improved efficiency enables training DDPM with more timesteps than before. Moreover, we plug EDP into three different offline algorithms (including TD3+BC, CRR, and IQL), and the results justify its superiority over standard diagonal Gaussian policies. As a result, EDP set up new state-of-the-art on all four domains in D4RL.
Offline RL Distributional shift between the learned and behavior policies is offline RL's biggest challenge. Existing research mitigates this problem by making modifications to policy evaluation [24, 17, 15, 21] or policy improvement [9, 39, 6, 33, 38, 42, 41]. For example, conservative Q-learning (CQL) [17] penalizes out-of-distribution actions for having higher Q-values, proving that this is equivalent to optimizing a lower bound of Q-values. Onestep RL [1] conducts policy evaluation on in-distribution data to avoid querying unseen actions. IQL [15] introduces expectile regression [14] to approximate dynamic programming with the Bellman optimality function. TD3+BC explicitly constrains the learned policy by adding a behavior cloning loss to mimic the behavior policy. Instead, CRR and AWR impose an implicit policy regularization by performing policy gradient-style policy updates. Despite their effectiveness, they ignore that the capacity of a policy representation plays a vital role in fitting the data distribution. This paper instead focuses on an orthogonal aspect (i.e., policy parameterization) that all the above methods can benefit. Another line of work tries to cast offline RL as a sequence-to-sequence model [3, 11], which is beyond the scope of this work.
Policy Parametrization Different RL algorithms may pose different requirements for parameterizing a policy distribution. There are mainly two categories of requirements: 1) The sampling process
is differentiable, such as the deterministic policy in DDPG [18] and TD3 [7]. 2) The log-likelihood of samples is tractable. For example, policy gradient methods [30, 31, 38, 28] optimize a policy based on maximum likelihood estimation (MLE). Therefore, most works represent policy with a diagonal Gaussian distribution with mean and variance parameterized with a multi-layer perceptron (MLP). On the other hand, BCQ [9] and BEAR [16] choose to model policy with a conditional variational autoencoder (CVAE). Recently, Diffusion-QL [37] introduced diffusion models into offline RL and demonstrated that diffusion models are superior at modeling complex action distributions than CVAE and diagonal Gaussian. However, it takes tens to hundreds more time to train a diffusion policy than a diagonal Gaussian one. Moreover, diffusion policy only satisfies the first requirement, which means many other offline RL algorithms can not use it, including the current SOTA IQL.
Our method is motivated to solve the about two limitations. We first propose a more efficient way to train diffusion policies, which reduces training time to the level of a Gaussian policy. Then, we generalize the diffusion policy to be compatible with MLE-based RL methods.
3.1 Offline Reinforcement Learning
A decision-making problem in reinforcement learning is usually represented by a Markov Decision Process (MDP):  $\mathcal{M} = \{\mathcal{S},\mathcal{A},P,R,\gamma \}$ .  $\mathcal{S}$  and  $\mathcal{A}$  are the state and action spaces respectively,  $P(s^{\prime}|s,a)$  measures the transition probability from state  $s$  to state  $s^{\prime}$  after taking action  $a$  while  $R(s,a,s^{\prime})$  gives the reward for the corresponding transition,  $\gamma \in [0,1)$  is the discount factor. A policy  $\pi (\pmb {a}|s)^2$  describes how an agent interacts with the environment. The optimal policy  $\pi^{*}(\pmb {a}|\pmb {s})$  is the one achieves maximal cumulative discounted returns:  $\pi^{*} = \arg \max_{\pi}[\sum_{t = 0}^{\infty}\gamma^{t}r(\pmb{s}_{t},\pmb{a}_{t})]$ . Reinforcement learning algorithms frequently rely on the definition of value function  $V(s) = \mathbb{E}_{\pi}[\sum_{t = 0}^{\infty}\gamma^{t}r(\pmb{s}_{t},\pmb{a}_{t})|s_{0} = s]$ , and action value (Q) function  $Q(s,\pmb {a}) = \mathbb{E}_{\pi}[\sum_{t = 0}^{\infty}\gamma^{t}r(\pmb{s}_{t},\pmb{a}_{t})|s_{0} = s,\pmb{a}_{0} = \pmb{a}]$ , which represents the expected cumulative discounted return of a policy  $\pi$  given the initial state  $s$  or state-action pair  $(s,a)$ .
In the offline RL setting, instead of learning from interactions with the environment, agents focus on learning an optimal policy from a previously collected dataset of transitions:  $\mathcal{D} = \{(s_t, a_t, s_{t+1}, r_t)\}$ . Offline RL algorithms for continuous control are usually based on an actor-critic framework that alternates between policy evaluation and policy improvement. During policy evaluation, a parameterized Q network  $Q_{\phi}(s, a)$  is optimized based on approximate dynamic programming to minimize the following temporal difference (TD) error  $L_{\mathrm{TD}}(\phi)$ :
$\mathbb{E}_{(s,a,s')\sim \mathcal{D}}\left[\left(r(s,a) + \gamma \max_{a'}Q_{\hat{\phi}}(s',a') - Q_{\phi}(s,a)\right)^2\right]$ , where  $Q_{\hat{\phi}}(s,a)$  denotes a target network. Then at the policy improvement step, knowledge in the Q network is distilled into the policy network in various ways. Offline RL methods address the distributional shift [9] problem induced by the offline dataset  $\mathcal{D}$  by either modifying the policy evaluation step to regularize Q learning or constraining the policy improvement directly. In the following, we will show that our diffusion policy design is compatible with any offline algorithms and can speed up policy evaluation and improvement.
Consider a real data distribution  $q(\pmb{x})$  and a sample  $\pmb{x}^0 \sim q(\pmb{x})$  drawn from it. The (forward) diffusion process fixed to a Markov chain gradually adds Gaussian noise to the sample in  $K$  steps, producing a sequence of noisy samples  $\pmb{x}^1, \dots, \pmb{x}^K$ . Note that we use superscript  $k$  to denote diffusion timestep to avoid conflicting with the RL timestep. The noise is controlled by a variance schedule  $\beta^1, \dots, \beta^K$ :
$$
q \left(\boldsymbol {x} ^ {k} \mid \boldsymbol {x} ^ {k - 1}\right) = \mathcal {N} \left(\boldsymbol {x} ^ {k}; \sqrt {1 - \beta^ {k}} \boldsymbol {x} ^ {k - 1}, \beta^ {k} \boldsymbol {I}\right), \quad q \left(\boldsymbol {x} ^ {1: K} \mid \boldsymbol {x} ^ {0}\right) = \prod_ {k = 1} ^ {K} q \left(\boldsymbol {x} ^ {k} \mid \boldsymbol {x} ^ {k - 1}\right). \tag {1}
$$
When  $K \to \infty$ ,  $\pmb{x}^K$  distributes as an isotropic Gaussian distribution. Diffusion models learn a conditional distribution  $p_{\theta}(\pmb{x}^{t-1}|\pmb{x}^t)$  and generate new samples by reversing the above process:
$$
p _ {\theta} \left(\boldsymbol {x} ^ {0: K}\right) = p \left(\boldsymbol {x} ^ {K}\right) \prod_ {k = 1} ^ {K} p _ {\theta} \left(\boldsymbol {x} ^ {k - 1} \mid \boldsymbol {x} ^ {k}\right), \quad p _ {\theta} \left(\boldsymbol {x} ^ {k - 1} \mid \boldsymbol {x} ^ {k}\right) = \mathcal {N} \left(\boldsymbol {x} ^ {k - 1}; \boldsymbol {\mu} _ {\theta} \left(\boldsymbol {x} ^ {k}, k\right), \boldsymbol {\Sigma} _ {\theta} \left(\boldsymbol {x} ^ {k}, k\right)\right), \tag {2}
$$
${}^{2}$  A policy could either be deterministic or stochastic,we use its stochastic form without loss of generality.
where  $p(\boldsymbol{x}^K) = \mathcal{N}(\boldsymbol{0},\boldsymbol{I})$  under the condition that  $\prod_{k=1}^{K}(1 - \beta^k) \approx 0$ . The training is performed by maximizing the evidence lower bound (ELBO):  $\mathbb{E}_{\boldsymbol{x}_0}[\log p_\theta(\boldsymbol{x}^0)] \geq \mathbb{E}_q\left[\log \frac{p_\theta(\boldsymbol{x}^{0:K})}{q(\boldsymbol{x}^{1:K}|\boldsymbol{x}^0)}\right]$ .
4 Efficient Diffusion Policy
In this section, we detail the design of our efficient diffusion policy (EDP). First, we formulate an RL policy with a diffusion model. Second, we present a novel algorithm that can train a diffusion policy efficiently, termed Reinforcement-Guided Diffusion Policy Learning (RGDPL). Then, we generalize the diffusion policy to work with arbitrary offline RL algorithms and compare our EDP with Diffusion-QL to highlight its superiority in efficiency and generality. Finally, we discuss several methods to sample from the diffusion policy during evaluation.
4.1 Diffusion Policy
Following [37], we use the reverse process of a conditional diffusion model as a parametric policy:
$$
\pi_ {\theta} (\boldsymbol {a} | \boldsymbol {s}) = p _ {\theta} \left(\boldsymbol {a} ^ {0: K} | \boldsymbol {s}\right) = p \left(\boldsymbol {a} ^ {K}\right) \prod_ {k = 1} ^ {K} p _ {\theta} \left(\boldsymbol {a} ^ {k - 1} | \boldsymbol {a} ^ {k}, \boldsymbol {s}\right), \tag {3}
$$
where  $\pmb{a}^{K}\sim \mathcal{N}(\pmb {0},\pmb {I})$  . We choose to parameterize  $\pi_{\theta}$  based on Denoising Diffusion Probabilistic Models (DDPM) [10], which sets  $\Sigma_{\theta}(a^{k},k;s) = \beta^{k}I$  to fixed time-dependent constants, and constructs the mean  $\mu_{\theta}$  from a noise prediction model as:  $\mu_{\theta}(\pmb{a}^{k},k;\pmb {s}) = \frac{1}{\sqrt{\alpha^{k}}}\left(\pmb{a}^{k} - \frac{\beta^{k}}{\sqrt{1 - \bar{\alpha}^{k}}}\pmb{\epsilon}_{\theta}(\pmb{a}^{k},k;\pmb {s})\right)$  , where  $\alpha^k = 1 - \beta^k$ $\bar{\alpha}^k = \prod_{s = 1}^k$  , and  $\pmb {\epsilon}_{\theta}$  is a parametric model.
To obtain an action from DDPM, we need to draw samples from  $K$  different Gaussian distributions sequentially, as illustrated in Eqn. (2)-(3). The sampling process can be reformulated as
$$
\boldsymbol {a} ^ {k - 1} = \frac {1}{\sqrt {\alpha^ {k}}} \left(\boldsymbol {a} ^ {k} - \frac {\beta^ {k}}{\sqrt {1 - \bar {\alpha} ^ {k}}} \boldsymbol {\epsilon} _ {\theta} \left(\boldsymbol {a} ^ {k}, k; \boldsymbol {s}\right)\right) + \sqrt {\beta^ {k}} \boldsymbol {\epsilon}, \tag {4}
$$
with the reparametrization trick, where  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ ,  $k$  is the reverse timestep from  $K$  to 0.
Similar to DDPM, plugging in the conditional Gaussian distributions, the ELBO in Sec. 3.2 can be simplified to the following training objective  $L_{\mathrm{diff}}(\theta)$ :
$$
\mathbb {E} _ {k, \epsilon , \left(\boldsymbol {a} ^ {0}, \boldsymbol {s}\right)} \left[ \left\| \boldsymbol {\epsilon} - \boldsymbol {\epsilon} _ {\theta} \left(\sqrt {\bar {\alpha} ^ {k}} \boldsymbol {a} ^ {0} + \sqrt {1 - \bar {\alpha} ^ {k}} \boldsymbol {\epsilon}, k; \boldsymbol {s}\right) \right\| ^ {2} \right], \tag {5}
$$
where  $k$  follows a uniform distribution over the discrete set  $\{1, \dots, K\}$ . It means the expectation is taken over all diffusion steps from clean action to pure noise. Moreover,  $\epsilon \in \mathcal{N}(\mathbf{0}, \mathbf{I})$ , and  $(\mathbf{a}^0, s) \in \mathcal{D}$  are state-action pairs drawn from the offline dataset. Given a dataset, we can easily and efficiently train a diffusion policy in a behavior-cloning manner as we only need to forward and backward through the network once each iteration. As shown in Diffusion-QL [37], diffusion policies can greatly boost the performance when trained with TD3-based Q learning. However, it still faces two main drawbacks that limit its real-world application: 1) It is inefficient in sampling and training; 2) It is not generalizable to other strong offline reinforcement learning algorithms.
4.2 Reinforcement-Guided Diffusion Policy Learning
To understand how a parametric policy  $\pi_{\theta}$  is trained with offline RL algorithms, we start with a typical Q-learning actor-critic framework for continuous control, which iterates between policy evaluation and policy improvement. Policy evaluation learns a Q network by minimizing the TD error  $L_{\mathrm{TD}}(\phi)$ :
$$
\mathbb {E} _ {\left(\boldsymbol {s}, \boldsymbol {a}, \boldsymbol {s} ^ {\prime}\right) \sim \mathcal {D}} \left[ \left(r (\boldsymbol {s}, \boldsymbol {a}) + \gamma Q _ {\hat {\phi}} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right) - Q _ {\phi} (\boldsymbol {s}, \boldsymbol {a})\right) ^ {2} \right], \tag {6}
$$
where the next action  $\pmb{a}^{\prime} \sim \pi_{\theta}(\cdot |\pmb{s}^{\prime})$ . The policy is optimized to maximize the expected Q values:
$$
\max  _ {\theta} \mathbb {E} _ {\boldsymbol {s} \sim \mathcal {D}, \boldsymbol {a} \sim \pi_ {\theta} (\boldsymbol {a} | \boldsymbol {s})} \left[ Q _ {\phi} (\boldsymbol {s}, \boldsymbol {a}) \right]. \tag {7}
$$
It is straightforward to optimize this objective when a Gaussian policy is used, but things get much more difficult when a diffusion policy is considered due to its complicated sampling process. Instead, we propose to view the offline RL problem from the perspective of generative modeling, where a diffusion policy can be easily learned in a supervised manner from a given dataset. However, unlike in computer vision, where the training data are usually perfect, offline RL datasets often contain suboptimal state-action pairs. Suppose we have a well-trained Q network  $Q_{\phi}$ , the question becomes how we can efficiently use  $Q_{\phi}$  to guide diffusion policy training procedure. We now show that this can be achieved without sampling actions from diffusion policies.
Let's revisit the forward diffusion process in Eqn. 1. A notable property of it is that the distribution of noisy action  $\pmb{a}^k$  at any step  $k$  can be written in closed form:  $q(\pmb{a}^k | \pmb{a}^0) = \mathcal{N}(\pmb{a}^k; \sqrt{\bar{\alpha}^k} \pmb{a}^0, (1 - \bar{\alpha}^k) \pmb{I})$ . Using the reparametrization trick, we are able to connect  $\pmb{a}^k$ ,  $\pmb{a}^0$  and  $\epsilon$  by:
$$
\boldsymbol {a} ^ {k} = \sqrt {\bar {\alpha} ^ {k}} \boldsymbol {a} ^ {0} + \sqrt {1 - \bar {\alpha} ^ {k}} \boldsymbol {\epsilon}, \quad \boldsymbol {\epsilon} \sim \mathcal {N} (\boldsymbol {0}, \boldsymbol {I}). \tag {8}
$$
Recall that our diffusion policy is parameterized to predict  $\epsilon$  with  $\epsilon_{\theta}(\pmb{a}^k, k; s)$ . By relacing  $\epsilon$  with  $\epsilon_{\theta}(\pmb{a}^k, k; s)$  and rearranging Eqn. (8), we obtain the approximated action:
$$
\hat {\boldsymbol {a}} ^ {0} = \frac {1}{\sqrt {\bar {\alpha} ^ {k}}} \boldsymbol {a} ^ {k} - \frac {\sqrt {1 - \bar {\alpha} ^ {k}}}{\sqrt {\bar {\alpha} ^ {k}}} \boldsymbol {\epsilon} _ {\theta} (\boldsymbol {a} ^ {k}, k; \boldsymbol {s}). \tag {9}
$$
In this way, instead of running the reverse diffusion process to sample an action  $\pmb{a}^0$ , we can cheaply construct  $\hat{\pmb{a}}^0$  from a state-action pair  $(s, \pmb{a})$  in the dataset by first corrupting the action  $\pmb{a}$  to  $\pmb{a}^k$  then performing one-step denoising to it. We will refer to this technique as action approximation in the following. Accordingly, the policy improvement for diffusion policies is modified as follows:
$$
L _ {\pi} (\theta) = - \mathbb {E} _ {\boldsymbol {s} \sim \mathcal {D}, \hat {\boldsymbol {a}} ^ {0}} \left[ Q _ {\phi} \left(\boldsymbol {s}, \hat {\boldsymbol {a}} ^ {0}\right) \right]. \tag {10}
$$
To improve the efficiency of policy evaluation, we propose to replace the DDPM sampling in Eqn. (4) with DPM-Solver [20], which is an ODE-based sampler. The algorithm is deferred to the appendix.
4.3 Generalization to Various RL algorithms
There are mainly two types of approaches to realize the objective in Eqn. 7 for policy improvement.
Direct policy optimization: It maximizes Q values and directly backpropagate the gradients from Q network to policy network, i.e.,  $\nabla_{\theta}L_{\pi}(\theta) = -\frac{\partial Q_{\phi}(s,a)}{\partial a}\frac{\partial a}{\partial\theta}$ . This is only applicable to cases where  $\frac{\partial a}{\partial\theta}$  is tractable, e.g., when a deterministic policy  $a = \pi_{\theta}(s)$  is used or when the sampling process can be reparameterized. Sample algorithms belonging to this category include TD3 [7], TD3+BC [6], and CQL [17]. One can easily verify that both the expensive DDPM sampling in Eqn. (4) and our efficient approximation in Eqn. (9) can be used for direct policy optimization.
Likelihood-based policy optimization: It tries to distill the knowledge from the Q network into the policy network indirectly by performing weighted regression or weighted maximum likelihood:
$$
\max  _ {\theta} \mathbb {E} _ {(\boldsymbol {s}, \boldsymbol {a}) \sim \mathcal {D}} \left[ f \left(Q _ {\phi} (\boldsymbol {s}, \boldsymbol {a})\right) \log \pi_ {\theta} (\boldsymbol {a} | \boldsymbol {s}) \right], \tag {11}
$$
where  $f(Q_{\phi}(\boldsymbol{s}, \boldsymbol{a}))$  is a monotonically increasing function that assigns a weight to each state-action pair in the dataset. This objective requires the log-likelihood of the policy to be tractable and differentiable. AWR [28], CRR [38], and IQL [15] fall into this category but each has a unique design in terms of the weighting function  $f$ . Since the likelihood of samples in Diffusion models is intractable, we propose the following two variants for realizing Eqn. 11.
First, instead of computing the likelihood, we turn to a lower bound for  $\log \pi_{\theta}(\pmb{a}|\pmb{s})$  introduced in DDPM [10]. By discarding the constant term that does not depend on  $\theta$ , we can have the objective:
$$
\mathbb {E} _ {k, \epsilon , (\boldsymbol {a}, \boldsymbol {s})} \left[ \frac {\beta^ {k} \cdot f \left(Q _ {\phi} (\boldsymbol {s} , \boldsymbol {a})\right)}{2 \alpha^ {k} \left(1 - \bar {\alpha} ^ {k - 1}\right)} \left\| \epsilon - \epsilon_ {\theta} \left(\boldsymbol {a} ^ {k}, k; \boldsymbol {s}\right) \right\| ^ {2} \right]. \tag {12}
$$
Second, instead of directly optimizing  $\log \pi_{\theta}(\pmb{a}|\pmb{s})$ , we propose to replace it with an approximated policy  $\hat{\pi}_{\theta}(\pmb{a}|\pmb{s}) \triangleq \mathcal{N}(\hat{\pmb{a}}^0, \pmb{I})$ , where  $\hat{\pmb{a}}^0$  is from Eqn. (9). Then, we get the following objective:
$$
\mathbb {E} _ {k, \epsilon , (\boldsymbol {a}, \boldsymbol {s})} \left[ f \left(Q _ {\phi} (\boldsymbol {s}, \boldsymbol {a})\right) \left\| \boldsymbol {a} - \hat {\boldsymbol {a}} ^ {0} \right\| ^ {2} \right]. \tag {13}
$$