SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiao Ma* Bingyi Kang*† Zhongwen Xu Min Lin Shuicheng Yan
Sea AI Lab
{yusufma555,bingykang}@gmail.com
The major challenge of offline RL is the distribution shift that appears when out-of-distribution actions are queried, which makes the policy improvement direction biased by extrapolation errors. Most existing methods address this problem by penalizing the policy or value for deviating from the behavior policy during policy improvement or evaluation. In this work, we propose a novel MISA framework to approach offline RL from the perspective of Mutual Information between States and Actions in the dataset by directly constraining the policy improvement direction. MISA constructs lower bounds of mutual information parameterized by the policy and Q-values. We show that optimizing this lower bound is equivalent to maximizing the likelihood of a one-step improved policy on the offline dataset. Hence, we constrain the policy improvement direction to lie in the data manifold. The resulting algorithm simultaneously augments the policy evaluation and improvement by adding mutual information regularizations. MISA is a general framework that unifies conservative Q-learning (CQL) and behavior regularization methods (e.g., TD3+BC) as special cases. We introduce 3 different variants of MISA, and empirically demonstrate that tighter mutual information lower bound gives better offline RL performance. In addition, our extensive experiments show MISA significantly outperforms a wide range of baselines on various tasks of the D4RL benchmark, e.g., achieving 742.9 total points on gym-locomotion tasks. Our code is attached and will be released upon publication.
Reinforcement learning (RL) has made remarkable achievements in solving sequential decision-making problems, ranging from game playing [28, 39, 5] to robot control [26, 20, 36]. However, its success heavily relies on 1) an environment to interact with for data collection and 2) an online algorithm to improve the agent based only on its own trial-and-error experiences. These make RL algorithms incapable in real-world safety-sensitive scenarios where interactions with the environment are dangerous or prohibitively expensive, such as in autonomous driving and robot manipulation with human autonomy [27, 25]. Therefore, offline RL is proposed to study the problem of learning decision-making agents from experiences that are previously collected from other agents when interacting with the environment is costly or not allowed.
Though much demanded, extending RL algorithms to offline datasets is challenged by the distributional shift between the data-collecting policy and the learning policy. Specifically, a typical RL algorithm alternates between evaluating the Q values of a policy and improving the policy to have better cumulative returns under the current value estimation. When it comes to the offline setting, policy improvement often involves querying out-of-distribution (OOD) state-action pairs that
* equal contribution, † corresponding author.
have never appeared in the dataset, for which the Q values are over-estimated due to extrapolation error of neural networks. As a result, the policy improvement direction is erroneously affected, eventually leading to a catastrophic explosion of value estimations as well as policy collapse after error accumulation. Existing methods [25, 42, 14, 44] tackle this problem by either forcing the learned policy to stay close to the behavior policy [16, 43, 14] or generating low value estimations for OOD actions [29, 25, 44]. Though these methods are effective at alleviating the distributional shift problem of the learning policy, the improved policy is unconstrained and might still deviate from the data distribution. A natural question thus arises: can we directly constrain the policy improvement direction to lie in the data manifold?
In this paper, we step back and consider the offline dataset from a new perspective, i.e., the Mutual Information between States and Actions (MISA). By viewing state and action as two random variables, the mutual information represents the reduction of uncertainty of actions given certain states,  $a.k.a.$ , information gain in information theory [31]. Therefore, mutual information is an appealing metric to sufficiently acquire knowledge from a dataset and characterize a behavior policy. We for the first time introduce it into offline RL as a regularization that directly constrains the policy improvement direction. Specifically, to allow practical optimizations of state-action mutual information estimation, we introduce the MISA lower bound of state-action pairs, which connects mutual information with RL by treating a parameterized policy as a variational distribution and the Q-values as the energy functions. We show that this lower bound can be interpreted as the likelihood of a non-parametric policy on the offline dataset, which represents the one-step improvement of the current policy based on the current value estimation. Maximizing MISA lower bound is equivalent to directly regularizing the policy improvement within the dataset manifold. However, the constructed lower bound involves integration over a self-normalized energy-based distribution, whose gradient estimation is intractable. To alleviate this dilemma, Markov Chain Monte Carlo (MCMC) estimation is adopted to produce an unbiased gradient estimation for MISA lower bound.
Theoretically, MISA is a general framework for offline RL that unifies several existing offline RL paradigms including behavior regularization and conservative learning. As examples, we show that TD3+BC [14] and CQL [25] are degenerated cases of MISA. Empirically, we verify that the tighter the mutual information lower bound, the better the final performance. We also demonstrate that MISA achieves significantly better performance on various environments of the D4RL [13] benchmark than a wide range of baselines, including CQL and IQL [24]. Additional visualizations are discussed to better understand the proposed method. Our code will be released upon publication.
Offline Reinforcement Learning The most critical challenge for extending an off-policy RL algorithm to an offline setup is the distribution shift between the behavior policy, i.e., the policy for data collection, and the learning policy [22]. To tackle this challenge, most of the offline RL algorithms consider a conservative learning framework. They either regularize the learning policy to stay close to the behavior policy [16, 43, 14, 38, 42, 45, 46], or force Q values to be low for OOD state-action pairs [29, 25, 44]. For example, TD3+BC [14] adds an additional behavior cloning (BC) signal along with the TD3 [15], which encourages the policy to stay in the data manifold; CQL [25], from the Q-value perspective, penalizes the OOD state-action pairs for generating high Q-value estimations and learns a lower bound of the true value function. However, their policy improvement direction is unconstrained and might deviate from the data distribution. On the other hand, SARSA-style updates [40] are considered to only query in-distribution state-action pairs [34, 24]. Nevertheless, without explicitly querying Bellman's optimality equation, they limit the policy from producing unseen actions. Our proposed MISA follows the conservative framework and directly regularizes the policy improvement direction to lie within the data manifold with mutual information, which more fully exploits the dataset information while learning a conservative policy. Different from the above discussed methods, a separate line works considering learning lower-bounded Q values by taking the minimum of multiple Q networks, e.g., SAC-n [2] uses  $50\mathrm{Q}$  networks for hopper tasks, or using more powerful policy representations [41, 21]. Such methods achieve strong performance on existing benchmarks. However, it greatly increases the sample complexity and we do not directly compare with them in this work.
Mutual Information Estimation. Mutual information is a fundamental quantity in information theory, statistics, and machine learning. However, direct computation of mutual information is
intractable as it involves computing a log partition function of a high dimensional variable. Thus, how to estimate the mutual information  $I(x,z)$  between random variables  $\mathcal{X}$  and  $\mathcal{Z}$ , accurately and efficiently, is a critical issue. One straightforward lower bound for mutual information estimation is Barber-Agakov bound [3], which introduces an additional variational distribution  $q(z\mid x)$  to approximate the unknown posterior  $p(z\mid x)$ . Instead of using an explicit "decoder"  $q(z\mid x)$ , we can use unnormalized distributions for the variational family  $q(z\mid x)$  [12, 4, 33], i.e., approximate the distribution as  $q(z\mid x) = \frac{p(z)e^{f(x,z)}}{\mathbb{E}_{p(z)}[e^{f(x,z)}]}$ , where  $f(x,z)$  is an arbitrary critic function. As an example, InfoNCE [33] has been widely used in representation learning literature [33, 18, 9]. To further improve the mutual information estimation, a combination of normalized and unnormalized variational distribution can be considered [8, 35]. Our MISA connects mutual information estimation with RL by parameterizing a tractable lower bound with a policy network as a variational distribution and the Q values as critics. In this way, MISA explicitly regularizes the policy improvement direction to lie in the data manifold and produces strong empirical performance.
Reinforcement Learning We consider a Markov Decision Process (MDP) denoted as a tuple  $\mathcal{M} = (\mathcal{S},\mathcal{A},p_0(s),p(s'|s,a),r(s,a),\gamma)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $p_0(s)$  is the initial state distribution,  $p(s'|s,a)$  is the transition function,  $r(s,a)$  is the reward function, and  $\gamma$  is the discount factor. The target of a learning agent is to find a policy  $\pi^{*}(a|s)$  that maximizes the accumulative reward by interacting with the environment
$$
\arg \max  _ {\pi} \mathbb {E} _ {\pi} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \mid s _ {0} \sim p _ {0} (s), a _ {t} \sim \pi (a \mid s _ {t}) \right]. \tag {1}
$$
Q-learning is a set of off-policy RL algorithms that utilize the optimal Bellman's optimality operator  $\mathcal{B}^{*}Q(s,a) = r(s,a) + \gamma \mathbb{E}_{s^{\prime}\sim p(s^{\prime}|s,a)}[\max_{a^{\prime}}Q(s^{\prime},a^{\prime})]$  to learn a Q function. Differently, Bellman's expectation operator  $\mathcal{B}^{\pi}Q(s,a) = r(s,a) + \gamma \mathbb{E}_{s^{\prime}\sim p(s^{\prime}|s,a);a^{\prime}\sim \pi (\cdot |s^{\prime})}[Q(s^{\prime},a^{\prime})]$  gives an actor-critic framework that alternates between policy evaluation and policy improvement. Consider a value network  $Q_{\phi}(s,a)$  parameterized by  $\phi$  and a policy network  $\pi_{\theta}(a|s)$  parameterized by  $\theta$ . Let  $\mu_{\pi}(s)$  denote the stationary distribution induced with policy  $\pi$ , which is also called occupancy measure [37]. Given the current policy, the policy evaluation aims to learn a Q network that can accurately predict its values minimizing  $\mathbb{E}_{\mu_{\pi_{\theta}}(s)\pi_{\theta}(a|s)}[(Q_{\phi}(s,a) - \mathcal{B}^{\pi_{\theta}}Q_{\phi}(s,a))^2]$ . Policy improvement focuses on learning the optimal policy that maximizes  $\mathbb{E}_{\mu_{\pi}(s)\pi (a|s)}[Q_{\phi}(s,a)]$ . In practical implementations, the Bellman operator is replaced with its sample-based version  $\hat{B}$ , and the expectation over  $\mu_{\pi}(s)\pi (a|s)$  is approximated by an online replay buffer or an offline dataset  $\mathcal{D}$ .
Nevertheless, as it is unavoidable to query the OOD actions when performing the maximization over actions, an inaccurate over-estimation of the Q value will be selected and the error will accumulate during the Bellman's update. Conservative RL methods, in turn, aim to perform "conservative" updates of the value / policy function during optimization by constraining the updates on only the in-distribution samples, which minimizes the negative impact of OOD actions.
KL Divergence Given two probability distributions  $p(x)$  and  $q(x)$  on the same probability space, the KL divergence (i.e., relative entropy) from  $q$  to  $p$  is given by  $D_{\mathrm{KL}}(p||q) = \mathbb{E}_{p(x)}\left[\log \frac{p(x)}{q(x)}\right] \geq 0$ . The minimum value is achieved when the two densities are identical. We consider two dual representations that result in tractable estimators for the KL divergence.
Lemma 3.1 (f-divergence representation [32]) The KL divergence admits the following lower bound:
$$
D _ {K L} (p \mid \mid q) \geq \sup  _ {T \in \mathcal {F}} \mathbb {E} _ {p (x)} [ T (x) ] - \mathbb {E} _ {q (x)} [ e ^ {T (x) - 1} ], \tag {2}
$$
where the supremum is taken over a function family  $\mathcal{F}$  satisfying the integrability constraints.
Lemma 3.2 (Donsker-Varadhan representation [30]) The KL divergence has the lower bound:
$$
D _ {K L} (p \mid | q) \geq \sup  _ {T \in \mathcal {F}} \mathbb {E} _ {p (x)} [ T (x) ] - \log (\mathbb {E} _ {q (x)} [ e ^ {T (x)} ]), \tag {3}
$$
The above two bounds are tight for large families  $\mathcal{F}$ .
4 Mutual Information Regularized Offline RL
We propose to tackle the offline RL problem from the perspective of mutual information estimation and develop a novel framework (MISA) by estimating the Mutual Information between States and Actions of a given offline dataset. MISA is a general framework that unifies multiple existing offline RL algorithms as special cases, including behavior cloning, TD3+BC [14], and CQL [25].
4.1 Mutual Information Regularization
Consider the state  $S$  and action  $A$  as two random variables. Let  $p_{(S,A)}(s,a)$  denote the joint distribution of state-action pairs, and  $p_S(s), p_A(a)$  be the marginal distributions. The subscripts are omitted in the following for simplicity. The mutual information between  $S$  and  $A$  is defined with:
$$
I (S; A) = \mathbb {E} _ {p (s, a)} \left[ \log \frac {p (s , a)}{p (s) p (a)} \right] = \mathbb {E} _ {p (s, a)} \left[ \log \frac {p (a \mid s)}{p (a)} \right] = H (A) - H (A \mid S), \tag {4}
$$
where  $H$  is Shannon entropy, and  $H(A|S)$  is conditional entropy of  $A$  given  $S$ . The higher mutual information between  $S$  and  $A$  means the lower uncertainty in  $A$  given the state  $S$ . This coincides with the observation that the actions selected by a well-performing agent are usually coupled with certain states. Therefore, given a joint distribution of state-action pairs induced by a (sub-optimal) behavior agent, it is natural to learn a policy that can recover the dependence between states and actions produced by the behavior agent. By regularizing the agent with  $I(S;A)$  estimation, we encourage the agent to 1) perform policy update within the dataset distribution and 2) avoid being over-conservative and make sufficient use of the dataset information.
Let  $\pi_{\beta}(a|s)$  represent a behavior policy and  $p_{\beta}(s,a)$  be the joint distribution of state-action pairs induced by  $\pi_{\beta}$ . Calculating the mutual information is often intractable as accessing  $p_{\beta}(s,a)$  is infeasible. Fortunately, in the problem of offline reinforcement learning, a dataset  $\mathcal{D} = \{(s_t,a_t,r_t,s_{t + 1})\}$  of transitions is given by drawing samples independently from  $p_{\beta}(s,a)$ . This dataset can thus be seen as a sample-based empirical joint distribution  $p_{\mathcal{D}}(s,a)$  for  $p_{\beta}$ . Let  $\mathcal{I}(\theta ,\phi)$  denote a mutual information lower bound that relies on parameterized functions with parameters  $\theta$  and  $\phi^2$ , which are usually the policy network and Q network in the context of RL. We defer the derivation of such bounds in Sec. 4.2. Based on the above motivation, we aim at learning a policy that can approximate the mutual information of the dataset while being optimized to get the best possible cumulative return. We focus on the actor-critic framework, and formulate the offline RL problem with mutual information regularization as follows:
$$
\min  _ {\phi} \mathbb {E} _ {s, a, s ^ {\prime} \sim \mathcal {D}} \left[ \frac {1}{2} \left(Q _ {\phi} (s, a) - \mathcal {B} ^ {\pi_ {\theta}} Q _ {\phi} (s, a)\right) ^ {2} \right] - \alpha_ {1} \hat {\mathcal {I}} _ {\mathcal {D}} (\theta , \phi), \quad \text {(P o l i c y E v a l u a t i o n)} \tag {5}
$$
$$
\max  _ {\theta} \mathbb {E} _ {s \sim \mathcal {D}, a \sim \pi_ {\theta} (a | s)} [ Q _ {\phi} (s, a) ] + \alpha_ {2} \hat {\mathcal {I}} _ {\mathcal {D}} (\theta , \phi), \quad \text {(P o l i c y I m p r o v e m e n t)} \tag {6}
$$
where  $\alpha_{1}$  and  $\alpha_{2}$  are the coefficients to balance RL objective and mutual information objective, and  $\hat{\mathcal{I}}_{\mathcal{D}}(\theta ,\phi)$  denotes the sample-based version of  $\mathcal{I}(\theta ,\phi)$  estimated from dataset  $\mathcal{D}$ .
4.2 State-Action Mutual Information Estimation
In this section, we develop practical solutions to approximate the mutual information  $I(S;A)$  from samples of the joint distribution. We use the learning policy  $\pi_{\theta}(a|s)$  as a variational variable and Eqn. 4 can be rewritten as:
$$
I (S; A) = \mathbb {E} _ {p (s, a)} \left[ \log \frac {\pi_ {\theta} (a | s) p (a | s)}{p (a) \pi_ {\theta} (a | s)} \right] = \mathbb {E} _ {p (s, a)} \left[ \log \frac {\pi_ {\theta} (a | s)}{p (a)} \right] + D _ {\mathrm {K L}} \left(p (s, a) \mid \mid p (s) \pi_ {\theta} (a | s)\right), \tag {7}
$$
where  $p(s)\pi_{\theta}(a|s)$  is an induced joint distribution. Let  $\mathcal{I}_{\mathrm{BA}} \triangleq \mathbb{E}_{p(s,a)}\left[\log \frac{\pi_{\theta}(a|s)}{p(a)}\right]$ . We have  $I(S;A) \geq \mathcal{I}_{\mathrm{BA}}$  as the KL divergence is always non-negative. This is exactly the Barber-Agakov (BA) lower bound developed by [3].
2Note some lower bounds might only have one parameterized function.
To obtain tighter bounds, we turn to KL dual representations of  $D_{\mathrm{KL}}$  ( $p(s,a) || p(s)\pi_{\theta}(a|s)$ ) in Eqn. 7. To this end, we choose  $\mathcal{F}$  to be a set of parameterized functions  $T_{\phi}: S \times A \to \mathbb{R}, \phi \in \Phi$ , which can be seen as an energy function.
With the  $f$ -divergence dual representation, we derive MISA- $f$  as
$$
\mathcal {I} _ {\mathrm {M I S A} - f} \triangleq \mathbb {E} _ {p (s, a)} \left[ \log \frac {\pi_ {\theta} (a | s)}{p (a)} \right] + \mathbb {E} _ {p (s, a)} \left[ T _ {\phi} (s, a) \right] - \mathbb {E} _ {p (s) \pi_ {\theta} (a | s)} \left[ e ^ {T _ {\phi} (s, a) - 1} \right]. \tag {8}
$$
The  $\mathcal{I}_{\mathrm{MISA - f}}$  bound is tight when  $p(a|s)\propto \pi_{\theta}(a|s)e^{T_{\phi}(s,a) - 1}$ . Similarly, using the DV representation in Theorem 3.2, we can have another bound  $\mathcal{I}_{\mathrm{MISA - DV}}\leq I(S;A)$ , as shown below:
$$
\mathcal {I} _ {\mathrm {M I S A - D V}} \triangleq \mathbb {E} _ {p (s, a)} \left[ \log \frac {\pi_ {\theta} (a | s)}{p (a)} \right] + \mathbb {E} _ {p (s, a)} \left[ T _ {\phi} (s, a) \right] - \log \mathbb {E} _ {p (s) \pi_ {\theta} (a | s)} \left[ e ^ {T _ {\phi} (s, a)} \right], \tag {9}
$$
which is tight when  $p(a|s) = \frac{1}{\mathcal{Z}} p(s)\pi_{\theta}(a|s)e^{T_{\phi}(s,a)}$ , where  $\mathcal{Z} = \mathbb{E}_{p(s)\pi_{\theta}(a|s)}\left[e^{T_{\phi}(s,a)}\right]$ .
We observe that the KL term in Eqn. 7 can be rewritten as:
$$
D _ {\mathrm {K L}} \left(p (s, a) | | p (s) \pi_ {\theta} (a | s)\right) = \mathbb {E} _ {p (s)} \left[ \mathbb {E} _ {p (a | s)} \left[ \log \frac {p (a | s)}{\pi_ {\theta} (a | s)} \right] \right] = \mathbb {E} _ {p (s)} \left[ D _ {\mathrm {K L}} (p (a | s) | | \pi_ {\theta} (a | s)) \right].
$$
Applying the DV representation of  $D_{\mathrm{KL}}(p(a|s)||\pi_{\theta}(a|s))$ , we can have a new lower bound  $\mathcal{I}_{\mathrm{MISA}}$ :
$$
\mathcal {I} _ {\mathrm {M I S A}} \triangleq \mathbb {E} _ {p (s, a)} \left[ \log \frac {\pi_ {\theta} (a | s)}{p (a)} \right] + \mathbb {E} _ {p (s, a)} \left[ T _ {\phi} (s, a) \right] - \mathbb {E} _ {p (s)} \log \mathbb {E} _ {\pi_ {\theta} (a | s)} \left[ e ^ {T _ {\phi} (s, a)} \right]. \tag {10}
$$
The bound is tight when  $p(a|s) = \frac{1}{\mathcal{Z}(s)}\pi_{\theta}(a|s)e^{T_{\phi}(s,a)}$ , where  $\mathcal{Z}(s) = \mathbb{E}_{\pi_{\theta}(a|s)}[e^{T_{\phi}(s,a)}]$ .
Theorem 4.1 Given the joint distribution of state  $s$  and action  $a$ , the lower bounds of mutual information  $I(S; A)$  defined in Eqn. 8-10 have the following relations:
$$
I (S; A) \geq \mathcal {I} _ {M I S A} \geq \mathcal {I} _ {M I S A - D V} \geq \mathcal {I} _ {M I S A - f}. \tag {11}
$$
The proof is deferred to the appendix due to space limit.
4.3 Integration with Offline Reinforcement Learning
We now describe how our MISA lower bound is integrated into the above framework (Eqn. 12-13) to give a practical offline RL algorithm. Theoretically,  $T_{\psi}(s,a)$  should be an arbitrary function optimized to accurately estimate the mutual information. As regularizing Q-functions is critical to offline RL, we propose to use a Q network  $Q_{\phi}(s,a)$  as the energy function  $T_{\phi}(s,a)$ , and use  $p_{\mathcal{D}}(s,a)$  as the joint distribution in Eqn. 10. Then we have the following objective to learn a Q-network during policy evaluation:
$$
J _ {Q} (\phi) = J _ {Q} ^ {\mathcal {B}} (\phi) - \gamma_ {1} \mathbb {E} _ {s, a \sim \mathcal {D}} \left[ Q _ {\phi} (s, a) \right] - \gamma_ {1} \mathbb {E} _ {s \sim \mathcal {D}} \left[ \log \mathbb {E} _ {\pi_ {\theta} (a | s)} \left[ e ^ {Q _ {\phi} (s, a)} \right] \right], \tag {12}
$$
where  $J_{Q}^{\mathcal{B}}(\phi) = \mathbb{E}_{s,a,s^{\prime}\sim \mathcal{D}}\left[\frac{1}{2}\left(Q_{\phi}(s,a) - \mathcal{B}^{\pi_{\theta}}Q_{\phi}(s,a)\right)^{2}\right]$  represents the TD error. For policy improvement, note that the entropy term  $H(a)$  in Eqn. 10 can be omitted as it is a constant given dataset  $\mathcal{D}$ . Thus, we have the below objective to maximize:
$$
J _ {\pi} (\theta) = \mathbb {E} _ {s \sim \mathcal {D}, a \sim \pi_ {\theta} (a | s)} \left[ Q _ {\phi} (s, a) \right] + \gamma_ {2} \mathbb {E} _ {s, a \sim \mathcal {D}} [ \log \pi_ {\theta} (a | s) ] - \gamma_ {2} \mathbb {E} _ {s \sim \mathcal {D}} \left[ \log \mathbb {E} _ {\pi_ {\theta} (a | s)} \left[ e ^ {Q _ {\phi} (s, a)} \right] \right]. \tag {13}
$$
The formulations for other regularizers (e.g.,  $\mathcal{I}_{\mathrm{MISA - DV}}$  and  $\mathcal{I}_{\mathrm{MISA - f}}$ ) can be derived similarly. A detailed description of the MISA algorithm for offline RL can be found in Algo. 1.
Intuitive Explanation on the Mutual Information Regularizer. By rearranging the terms in Eqn. 10, MISA can be written as:
$$
\mathcal {I} _ {\mathrm {M I S A}} = \mathbb {E} _ {s, a \sim \mathcal {D}} \left[ \log \frac {\pi_ {\theta} (a \mid s) e ^ {Q _ {\phi} (s , a)}}{\mathbb {E} _ {\pi_ {\theta} \left(a ^ {\prime} \mid s\right)} \left[ e ^ {Q _ {\phi} (s , a ^ {\prime})} \right]} \right], \tag {14}
$$
Input: Initialize Q network  $Q_{\phi}$ , policy network  $\pi_{\theta}$ , dataset  $\mathcal{D}$ , hyperparameters  $\alpha_{1}$  and  $\alpha_{2}$ . for  $t \in \{1, \dots, \text{MAX_STEP}\}$  do
Algorithm 1 Mutual Information Regularized Offline RL
Train the Q network by gradient descent with objective  $J_{Q}(\phi)$  in Eqn. 12:  $\phi := \phi - \eta_{Q}\nabla_{\phi}J_{Q}(\phi)$  Improve policy network by gradient ascent with object  $J_{\pi}(\theta)$  in Eqn. 13:  $\theta := \theta + \eta_{\pi}\nabla_{\theta}\mathbb{E}_{s\sim \mathcal{D},a\sim \pi_{\theta}(a|s)}[Q_{\phi}(s,a)] + \alpha_{2}\nabla_{\theta}I_{\mathrm{MISA}}$
Output: The well-trained  $\pi_{\theta}$