Title: LEARNING GAUSSIAN POLICIES FROM SMOOTHED ACTION VALUE FUNCTIONS Anonymous

Abstract: State-action value functions (i.e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning. We propose a new notion of action value defined by a Gaussian smoothed version of the expected Q-value. We show that such smoothed Q-values still satisfy a Bellman equation, making them learnable from experience sampled from an environment. Moreover, the gradients of expected reward with respect to the mean and covariance of a parameterized Gaussian policy can be recovered from the gradient and Hessian of the smoothed Q-value function. Based on these relationships we develop new algorithms for training a Gaussian policy directly from a learned smoothed Q-value approximator. Our approach is amenable to proximal optimization techniques by augmenting the objective with a penalty on KLdivergence from a previous policy. We find that the ability to learn both a mean and covariance during training allows this approach to achieve much better results on standard continuous control benchmarks.

1. INTRODUCTION:

Model-free reinforcement learning algorithms often alternate between two concurrent but interacting processes: (1) policy evaluation, where an action value function (i.e., a Q-value) is updated to obtain a better estimate of the return associated with taking a specific action, and (2) policy improvement, where the policy is updated aiming to maximize the current value function. In the past, different notions of Q-value have led to distinct but important families of RL methods. For example, SARSA (Rummery & Niranjan, 1994; Sutton & Barto, 1998; Van Seijen et al., 2009) uses the expected Q-value, defined as the expected return of following the current policy. Q-learning (Watkins, 1989 ) exploits a hard-max notion of Q-value, defined as the expected return of following an optimal policy. Soft Q-learning (Haarnoja et al., 2017) and PCL (Nachum et al., 2017a) both use a soft-max form of Q-value, defined as the future return of following an optimal entropy regularized policy. Clearly, the choice of Q-value function has a considerable effect on the resulting algorithm; for example, restricting the types of policies that can be expressed, and determining the type of exploration that can be naturally applied.

In this work we introduce a new notion of action value: the smoothed action value function Qπ . Unlike previous notions, which associate a value with a specific action at each state, the smoothed Qvalue associates a value with a specific distribution over actions. In particular, the smoothed Q-value of a state-action pair (s, a) is defined as the expected return of first taking an action sampled from a normal distribution N (a, Σ(s)), centered at a, then following actions sampled from the current policy thereafter. In this way, the smoothed Q-value can also be interpreted as a Gaussian-smoothed or noisy version of the expected Q-value.

We show that smoothed Q-values possess a number of interesting properties that make them attractive for use in RL algorithms. For one, the smoothed Q-values satisfy a single-step Bellman consistency, which allows bootstrapping to be used to train a function approximator. Secondly, for Gaussian policies, the standard optimization objective (expected return) can be expressed in terms of smoothed Q-values. Moreover, the gradient of this objective with respect to the mean and covariance of the Gaussian policy is equivalent to the gradient and the Hessian of the smoothed Q-value function, which allows one to derive updates to the policy parameters by having access to the derivatives of a sufficiently accurate smoothed Q-value function.

This observation leads us to propose an algorithm called Smoothie, which in the spirit of (Deep) Deterministic Policy Gradient (DDPG) (Silver et al., 2014; Lillicrap et al., 2016) , trains a policy using the derivatives of a trained (smoothed) Q-value function, thus avoiding the high-variance of stochastic updates used in standard policy gradient algorithms (Williams & Peng, 1991; Konda & Tsitsiklis, 2000) . Unlike DDPG, which is well-known to have poor exploratory behavior (Haarnoja et al., 2017) , the approach we develop is able to utilize a non-deterministic Gaussian policy parameterized by both a mean and a covariance, thus allowing the policy to be exploratory by default and alleviating the need for excessive hyperparameter tuning.

Furthermore, we show that Smoothie can be easily adapted to incorporate proximal policy optimization techniques by augmenting the objective with a penalty on KL-divergence from a previous version of the policy. The inclusion of a KL-penalty is not feasible in the standard DDPG algorithm, but we show that it is possible with our formulation, and it significantly improves stability and overall performance. On standard continuous control benchmarks, our results are competitive with or exceed state-of-the-art, especially for more difficult tasks in the low-data regime.

2. NOTATION & BACKGROUND:

We consider the standard model-free RL framework, where an agent interacts with a stochastic black-box environment by sequentially observing the state of the environment, emitting an action, and receiving a reward feedback; the goal is to find an agent that achieves maximal cumulative discounted reward. This problem can be expressed in terms of a Markov decision process (MDP) that consists of a state space S and an action space A, where at iteration t the agent encounters a state s t ∈ S and emits an action a t ∈ A, after which the environment returns a scalar reward r t ∼ R(s t , a t ) and places the agent in a new state s t+1 ∼ P (s t , a t ).

We model the behavior of the agent using a stochastic policy π that produces a distribution over feasible actions at each state s as π(a | s). The optimization objective (expected discounted return), as a function of the policy, can then be expressed in terms of the expected action value function Q π (s, a) by,

EQUATION

where ρ π (s) is the stationary distribution of the states under π, and Q π (s, a) is recursively defined using the Bellman equation,

EQUATION

where γ ∈ [0, 1] is the discount factor. For brevity, we will often suppress explicit denotation of the sampling distribution R over immediate rewards and the distribution P over state transitions.

The policy gradient theorem (Sutton et al., 2000) expresses the gradient of O ER (π θ ) w.r.t. θ, the tunable parameters of a policy π θ , as,

EQUATION

Many reinforcement learning algorithms, including policy gradient and actor-critic variants, trade off variance and bias when estimating the random variable inside the expectation in (4); for example, by attempting to estimate Q π (s, a) accurately using function approximation. In the simplest scenario, an unbiased estimate of Q π (s, a) is formed by accumulating discounted rewards from each state forward using a single Monte Carlo sample.

In this paper, we focus on multivariate Gaussian policies over continuous action spaces, A ≡ R da . We represent the observed state of the MDP as a d s -dimensional feature vector Φ(s) ∈ R ds , and parametrize the Gaussian policy by a mean and covariance function, respectively µ(s) : R ds → R da and Σ(s) : R ds → R da × R da . These map the observed state of the environment to a Gaussian distribution,

EQUATION

where v 2 A = v T Av. Below we develop new RL training methods for this family of parametric policies, but some of the ideas presented may generalize to other families of policies as well. We begin the formulation by reviewing some prior work on learning Gaussian policies.

2.1 DETERMINISTIC POLICY GRADIENT Silver et al. (2014) present a new formulation of the policy gradient, called the deterministic policy gradient, for the family of Gaussian policies in the limit where the policy covariance approaches zero. In such a scenario, the policy becomes deterministic because sampling from the policy always returns the Gaussian mean. The key observation of (Silver et al., 2014) is that under a deterministic policy π ≡ (µ, Σ → 0), one can estimate the expected future return from a state s as,

EQUATION

Then, one can express the gradient of the optimization objective (expected discounted return) for a parameterized π θ ≡ µ θ as,

∇ θ O ER (π θ ) = S ∇ θ Q π (s, µ θ (s))dρ π (s) = S ∂Q π (s, a) ∂a a=µ θ (s) ∇ θ µ θ (s)dρ π (s) . (7)

This can be thought of as a characterization of the policy gradient theorem for deterministic policies.

In the limit of Σ → 0, one can also re-express the Bellman equation ( 2) as,

EQUATION

Therefore, a value function approximator Q π w can be optimized by minimizing the Bellman error,

EQUATION

for transitions (s, a, r, s ) sampled from a dataset D of interactions of the agent with the environment. Algorithms like DDPG (Lillicrap et al., 2016 ) alternate between improving the value function by gradient descent on (9), and improving the policy based on (7).

In practice, to gain better sample efficiency, Degris et al. (2012) and Silver et al. (2014) replace the on-policy state distribution ρ π (s) in ( 7) with an off-policy distribution ρ β (s) based on a replay buffer. After this substitution, the policy gradient identity in (7) does not hold exactly, however, prior work finds that this works well in practice and improves sample efficiency. We also adopt a similar approximation in our method to make use of off-policy data.

3. SMOOTHED ACTION VALUE FUNCTIONS:

In this paper, we introduce smoothed action value functions, the gradients of which provide an effective signal for optimizing the parameters of a Gaussian policy. Our notion of smoothed Qvalues, denoted Qπ (s, a), differs from ordinary Q-values Q π (s, a) in that smoothed Q-values do not assume the first action of the agent is fully specified, but rather they assume that only the mean of the distribution of the first action is known. Hence, to compute Qπ (s, a), one has to perform an expectation of Q π (s, ã) for actions ã drawn in the vicinity of a. More formally, smoothed action values are defined as,

EQUATION

)

With this definition of Qπ , one can re-express the expected reward objective for a Gaussian policy π ≡ (µ, Σ) as,

EQUATION

The insight that differentiates this approach from prior work including Heess et al. (2015) ; Ciosek & Whiteson (2017) is that instead of learning a function approximator for Q π (s, a) and then drawing samples to approximate the expectation in (10) and its derivative, we directly learn a function approximator for Qπ (s, a).

The key observation that enables direct bootstrapping of smoothed Q-values, Qπ (s, a), is that their form allows a notion of Bellman consistency. First, note that for Gaussian policies π ≡ (µ, Σ) we have

EQUATION

Then, combining ( 10) and ( 12), one can derive the following one-step Bellman equation for smoothed Q-values,

EQUATION

where r and s are sampled from R(s, ã) and P (s, ã). Below, we elaborate on how one can make use of the derivatives of Qπ to learn µ and Σ, and how the Bellman equation in ( 13) enables direct optimization of Qπ .

3.1 POLICY IMPROVEMENT -OPTIMIZING (µ θ , Σ φ ) GIVEN Qπ

We parameterize a Gaussian policy π θ,φ ≡ (µ θ , Σ φ ) in terms of two sets of parameters θ and φ for the mean and the covariance. The gradient of the objective w.r.t. mean parameters follows from the policy gradient theorem and is almost identical to (7),

EQUATION

)

Estimating the derivative of the objective w.r.t. covariance parameters is not as straightforward, since Qπ is not a direct function of Σ. However, a key observation of this work is that the second derivative of Qπ w.r.t. actions is sufficient to exactly compute the derivative of Qπ w.r.t. Σ,

EQUATION

)

A proof of this identity is provided in the Appendix. The proof may be easily derived by expressing both sides of the equation using standard matrix calculus like

∂ ∂A |A| -1/2 = -1 2 |A| -1/2 A -1 and ∂ ∂A ||v|| 2 A -1 = -A -1 vv T A -1 .

Then, the full derivative w.r.t. φ takes the form,

∇ φ O ER (π θ,φ ) = 1 2 S ∂ 2 Qπ (s, a) ∂a 2 a=µ θ (s) ∇ φ Σ φ (s)dρ π (s). (16) 3.2 POLICY EVALUATION -OPTIMIZING Qπ w GIVEN (µ, Σ)

We can think of two ways to optimize Qπ w . The first approach leverages (10) to update Qπ based on expected Q-value function Q π . In such an approach, one trains a parameterized Q π w to approximate the standard expected Q-value function Q π using standard methods (see e.g., Rummery & Niranjan (1994); Sutton & Barto (1998) ; Van Seijen et al. (2009) ). Then, one fits Qπ w based on Q π w . In particular, given transitions (s, a, r, s ) sampled from interactions with the environment, one can train Q π w to minimize the Bellman error (Q π w (s, a)-r -γQ π w (s , a )) 2 where a ∼ N (µ(s ), Σ(s )). Then, Qπ w can be optimized to minimize the squared error ( Qπ w (s, a) -E ãQ π w (s, ã)) 2 where ã ∼ N (a, Σ(s)), using several samples. When the target values in these residuals are treated as fixed (i.e., using a target network), such a training procedure will achieve a fixed point when Qπ w (s, a) satisfies the recursion in the Bellman equation (10).

The second approach requires a single function approximator for Qπ w (s, a), resulting in a simpler implementation, and thus we use this approach in our experimental evaluation. Suppose one has access to a tuple (s, ã, r, s ) sampled from a replay buffer with knowledge of the sampling probability q(ã | s) (possibly unnormalized). Then assuming that this sampling distribution has a full support, we draw a phantom action a ∼ N (ã, Σ(s)) and optimize Qπ w (s, a) by minimizing a weighted Bellman error 1 q(ã|s) ( Qπ w (s, a) -r -γ Qπ w (s , µ(s )) 2 . For a specific pair of state and action (s, a) the expected value of the objective is,

EQUATION

)

Note that N (a|ã, Σ(s)) = N (ã|a, Σ(s)). Therefore, when the target value r + γ Qπ w (s , µ(s )) is treated as fixed (e.g., when using target networks) this training procedure reaches an optimum when Qπ w (s, a) satisfies the recursion in the Bellman equation ( 13).

In practice, we find that it is unnecessary to keep track of the probabilities q(ã | s), and assume the replay buffer provides a near-uniform distribution of actions conditioned on states. Other recent work has also benefited from ignoring or heavily damping importance weights (Munos et al., 2016; Wang et al., 2017; Schulman et al., 2017) . However, it is possible when interacting with the environment to save the probability of sampled actions along with their transitions, and thus have access to q(ã | s) ≈ N (ã | µ old (s), Σ old (s)).

3.3. PROXIMAL POLICY OPTIMIZATION:

Policy gradient algorithms are notoriously unstable, particularly in continuous control problems. Such instability has motivated the development of trust region methods that attempt to mitigate the issue by constraining each gradient step to lie within a trust region (Schulman et al., 2015) , or augmenting the expected reward objective with a penalty on KL-divergence from a previous policy (Nachum et al., 2017b; Schulman et al., 2017; Azar et al., 2012) . These stabilizing techniques have thus far not been applicable to algorithms like DDPG, since the policy is deterministic. The formulation we propose in this paper, however, is easily amenable to trust region optimization. Specifically, we may augment the objective (11) with a penalty

EQUATION

where π old ≡ (µ old , Σ old ) is a previous parameterization of the policy. The optimization is straightforward, since the KL-divergence of two Gaussians can be expressed analytically.

4. RELATED WORK:

This paper follows a long line of work that uses Q-value functions to stably learn a policy, which in the past has been used to either approximate expected (Rummery & Niranjan, 1994; Van Seijen et al., 2009; Gu et al., 2017) or optimal (Watkins, 1989; Silver et al., 2014; Nachum et al., 2017a; Haarnoja et al., 2017; Metz et al., 2017) future value.

Work that is most similar to what we present are methods that exploit gradient information from the Q-value function to train a policy. Deterministic policy gradient (Silver et al., 2014) is perhaps the best known of these. The method we propose can be interpreted as a generalization of the deterministic policy gradient. Indeed, if one takes the limit of the policy covariance Σ(s) as it goes to 0, the proposed Q-value function becomes the deterministic value function of DDPG, and the updates for training the Q-value approximator and the policy mean are identical.

Stochastic Value Gradient (SVG) (Heess et al., 2015) also trains stochastic policies using an update that is similar to DDPG (i.e., SVG(0) with replay). The key differences with our approach are that SVG does not provide an update for the covariance, and the mean update in SVG estimates the gradient with a noisy Monte Carlo sample, which we avoid by estimating the smoothed Q-value function. Although a covariance update could be derived using the same reparameterization trick as in the mean update, that would also require a noisy Monte Carlo estimate. Methods for updating the covariance along the gradient of expected reward are essential for applying the subsequent trust region and proximal policy techniques.

More recently, Ciosek & Whiteson (2017) introduced expected policy gradients (EPG), a generalization of DDPG that provides updates for the mean and covariance of a stochastic Gaussian policy using gradients of an estimated Q-value function. In that work, the expected Q-value used in standard policy gradient algorithms such as SARSA (Sutton & Barto, 1998; Rummery & Niranjan, 1994; Van Seijen et al., 2009) is estimated. The updates in EPG therefore require approximating an integral of the expected Q-value function. Our analogous process directly estimates an integral (via the smoothed Q-value function) and avoids approximate integrals, thereby making the updates simpler. Moreover, while Ciosek & Whiteson (2017) rely on a quadratic Taylor expansion of the estimated Q-value function, we instead rely on the strength of neural network function approximators to directly estimate the smoothed Q-value function.

The novel training scheme we propose for learning the covariance of a Gaussian policy relies on properties of Gaussian integrals (Bonnet, 1964; Price, 1958) . Similar identities have been used in the past to derive updates for variational auto-encoders (Kingma & Welling, 2014) and Gaussian back-propagation (Rezende et al., 2014) .

Finally, the perspective presented in this paper, where Q-values represent the averaged return of a distribution of actions rather than a single action, is distinct from recent advances in distributional RL (Bellemare et al., 2017) . Those approaches focus on the distribution of returns of a single action, whereas we consider the single average return of a distribution of actions. Although we restrict our attention in this paper to Gaussian policies, an interesting topic for further investigation is to study the applicability of this new perspective to a wider class of policy distributions.

5. EXPERIMENTS:

We utilize the insights from Section 3 to introduce a new RL algorithm, Smoothie. Smoothie maintains a parameterized Qπ w trained via the procedure described in Section 3.2. It then uses the gradient and Hessian of this approximation to train a Gaussian policy µ θ , Σ φ using the updates stated in ( 14) and ( 16). See Algorithm 1 for a simplified pseudocode of our algorithm.

We perform a number of evaluations of Smoothie compared to DDPG. We choose DDPG as a baseline because it (1) utilizes gradient information of a Q-value approximator, much like our algorithm; and (2) is a standard algorithm well-known to have achieve good, sample-efficient performance on continuous control benchmarks.

5.1. A SYNTHETIC TASK:

To evaluate Smoothie we begin with a simple synthetic task which allows us to study its behavior in a restricted setting. We devised a simple single-action one-shot environment in which the reward function is a mixture of two Gaussians, one better than the other (see Figure 1 (Right)). We initialize the policy mean to be centered on the worse of the two Gaussians. We plot the learnable policy mean and standard deviation during training for Smoothie and DDPG in Figure 1 (Left). Smoothie learns both the mean and variance, while DDPG learns only the mean and the variance plotted is exploratory noise, whose scale is kept fixed during training.

As expected we observe that DDPG cannot escape the local optimum. At the beginning of training it exhibits some movement away from the local optimum (likely due to the initial noisy approximation given by Q π w ), it is unable to progress very far from the initial mean. Note that this is not an issue of exploration. The exploration scale is high enough that Q π w is aware of the better Gaussian. The issue is in the update for µ θ , which is only with regard to the derivative of Q π w at the current mean. On the other hand, we find Smoothie is successfully able to solve the task. This is because the smoothed reward function approximated by Qπ w has a derivative which clearly points µ θ towards the better Gaussian. We also observe that Smoothie is able to suitably adjust the covariance Σ φ during training. Initially, Σ φ decreases due to the concavity of the smoothed reward function. As a region of convexity is entered, it begins to increase, before again decreasing to near-zero as µ θ approaches the global optimum. The reward function for the synthetic task along with its Gaussian-smoothed version. We find that Smoothie can successfully escape the lower-reward local optimum. We also notice Smoothie increases and decreases its policy variance as the convexity/concavity of the smoothed reward function changes.

5.2. CONTINUOUS CONTROL:

We now turn our attention to standard continuous control benchmarks available on OpenAI Gym (Brockman et al., 2016) utilizing the MuJoCo environment (Todorov et al., 2012) .

Our implementations utilize feed forward neural networks for policy and Q-values. We parameterize the covariance Σ φ as a diagonal given by e φ . The exploration for DDPG is determined by an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930; Lillicrap et al., 2016) . Additional implementation details are provided in the Appendix. Each plot shows the average reward and standard deviation clipped at the min and max of six randomly seeded runs after choosing best hyperparameters. We see that Smoothie is competitive with DDPG even when DDPG uses a hyperparameter-tuned noise scale, and Smoothie learns the optimal noise scale (the covariance) during training. Moreoever, we observe significant advantages in terms of final reward performance, especially in the more difficult tasks like Hopper, Walker2d, and Humanoid. Across all tasks, TRPO is not sufficiently sampleefficient to provide a competitive baseline.

We compare the results of Smoothie and DDPG in Figure 2 . For each task we performed a hyperparameter search over actor learning rate, critic learning rate and reward scale, and plot the average of six runs for the best hyperparameters. For DDPG we extended the hyperparameter search to also consider the scale and damping of exploratory noise provided by the Ornstein-Uhlenbeck process. Smoothie, on the other hand, contains an additional hyperparameter to determine the weight on KL-penalty.

Despite DDPG having the advantage of its exploration decided by a hyperparameter search while Smoothie must learn its exploration without supervision, we find that Smoothie performs competitively or better across all tasks, exhibiting a slight advantage in Swimmer and Ant, while showing more dramatic improvements in Hopper, Walker2d, and Humanoid. The improvement is especially dramatic for Hopper, where the average reward is doubled. We also highlight the results for Humanoid, which as far as we know, are the best published results for a method that only trains on the order of millions of environment steps. In contrast, TRPO, which to the best of our knowledge is the only other algorithm which can achieve better performance, requires on the order of tens of millions of environment steps to achieve comparable reward. This gives added evidence to the benefits of using a learnable covariance and not restricting a policy to be deterministic.

Empirically, we found the introduction of a KL-penalty to improve performance of Smoothie, especially on harder tasks. We present a comparison of results of Smoothie with and without the KL-penalty on the four harder tasks in Figure 3 . A KL-penalty to encourage stability is not possible in DDPG. Thus, our algorithm provides a much needed solution to the inherent instability in DDPG training. Figure 3 : Results of Smoothie with and without a KL-penalty. The x-axis is in millions of environment steps. We observe benefits of using a proximal policy optimization method, especially in Hopper and Humanoid, where the performance improvement is significant without sacrificing sample efficiency.

6. CONCLUSION:

We have presented a new Q-value function, Qπ , that is a Gaussian-smoothed version of the standard expected Q-value, Q π . The advantage of using Qπ over Q π is that its gradient and Hessian possess an intimate relationship with the gradient of expected reward with respect to mean and covariance of a Gaussian policy. The resulting algorithm, Smoothie, is able to successfully learn both mean and covariance during training, leading to performance that can match or surpass that of DDPG, especially when incorporating a penalty on divergence from a previous policy.

The success of Qπ is encouraging. Intuitively it may be argued that learning Qπ is more sensible than learning Q π . The smoothed Q-values by definition make the true reward surface smoother, thus possibly easier to learn; moreover the smoothed Q-values have a more direct relationship with the expected discounted return objective. We encourage future work to further investigate these claims as well as techniques to apply the underlying motivations for Qπ to other types of policies.

A PROOF OF EQUATION (15) We note that similar identities for Gaussian integrals exist in the literature (Price, 1958; Rezende et al., 2014) and point the reader to these works for further information.

The specific identity we state may be derived using standard matrix calculus. We make use of the fact that

EQUATION

and for symmetric A, ∂ ∂A

EQUATION

)

We omit s from Σ(s) in the following equations for succinctness. The LHS of ( 15) is 

A Q π (s, ã) ∂ ∂Σ N (ã|a, Σ)dã = A Q π (s, ã) exp - 1 2 ||ã -a|| 2 Σ -1 ∂ ∂Σ |2πΣ| -1/2 - 1 2 |2πΣ| -1/2 ∂ ∂Σ ||ã -a|| 2 Σ -1 dã = 1 2 A Q π (

:



Figures and Tables:

Figure1: Left: The learnable policy mean and standard deviation during training for Smoothie and DDPG on a simple one-shot synthetic task. The standard deviation for DDPG is the exploratory noise kept constant during training. Right: The reward function for the synthetic task along with its Gaussian-smoothed version. We find that Smoothie can successfully escape the lower-reward local optimum. We also notice Smoothie increases and decreases its policy variance as the convexity/concavity of the smoothed reward function changes.

Figure2: Results of Smoothie, DDPG, and TRPO on continuous control benchmarks. The x-axis is in millions of environment steps. Each plot shows the average reward and standard deviation clipped at the min and max of six randomly seeded runs after choosing best hyperparameters. We see that Smoothie is competitive with DDPG even when DDPG uses a hyperparameter-tuned noise scale, and Smoothie learns the optimal noise scale (the covariance) during training. Moreoever, we observe significant advantages in terms of final reward performance, especially in the more difficult tasks like Hopper, Walker2d, and Humanoid. Across all tasks, TRPO is not sufficiently sampleefficient to provide a competitive baseline.

s, ã)N (ã|a, Σ) -Σ -1 + Σ -1 (ã -a)(ã -a) T Σ -1 dã. (21) Meanwhile, towards tackling the RHS of (15) we note that∂ Qπ (s, a) ∂a = A Q π (s, ã)N (ã|a, Σ)Σ -1 (ã -a)dã. s, ã)N (ã|a, Σ)(Σ -1 (ã -a)(ã -a) T Σ -1 -Σ -1 )dã.(24)B COMPATIBLE FUNCTION APPROXIMATIONA function approximator Qπ w of Qπ should be sufficiently accurate so that updates for µ θ , Σ φ are not affected by substituting Qπ w is compatible with respect to µ θ if1. ∇ a Qπ w (s, a) a=µ θ (s) = ∇ θ µ θ (s) T w, 2. ∇ w S ∇ a Qπ w (s, a) a=µ θ (s) -∇ a Qπ (s, a) a=µ θ (s)2 dρ π (s) = 0 (i.e., w minimizes the expected squared error of the gradients).Additionally, Qπw is compatible with respect toΣ φ if 1. ∇ 2 a Qπ w (s, a) a=µ θ (s) = ∇ φ Σ φ (s) T w, 2. ∇ w S ∇ 2 a Qπ w (s, a) a=µ θ (s) -∇ 2 a Qπ (s, a) a=µ θ (s)2 dρ π (s) = 0 (i.e., w minimizes the expected squared error of the Hessians).One possible parameterization of Qπw may be achieved by taking w = [w 0 , w 1 , w 2 ] and parameterizing Qπ w (s, a) = V w0 (s) + (a -µ θ (s)) T ∇ θ µ θ (s) T w 1 + (a -µ θ (s)) T ∇ φ Σ φ (s) T w 2 (a -µ θ (s)). (25)

Peer review (iclr2018):
 This paper explores the idea of using policy gradients to learn a stochastic policy on complex control problems. The central idea is to frame learning in terms of a new kind of Q-value that attempts to smooth out Q-values by framing them in terms of expectations over Gaussian policies. To be honest, I didn*t really "get" this paper. * As far I understand, all of the original work policy gradients involved stochastic policies. Many are/were Gaussian. * All Q-value estimators are designed to marginalize out the randomness in these stochastic policies. * As far as I can tell, this is equivalent to a slightly different formulation, where the agent emits a deterministic action (.mu,.Sigma) and the environment samples an action from that distribution. In other words, it seems that if we just draw the box a bit differently, the environment soaks up the nondeterminism, instead of needing to define a new type of Q-value. Ultimately, I couldn*t discern /why/ this was a significant advance for RL, or even a meaningful new perspective on classic ideas. I thought the little 2-mode MOG was a nice example of the premise of the model. While I may or may not have understood the core technical contribution, I think the experiments can be critiqued: they didn*t really seem to work out. Figures 2&3 are unconvincing - the differences do not appear to be statistically significant. Also, I was disappointed to see that the authors only compared to DDPG; they could have at least compared to TRPO, which they mention. They dismiss it by saying that it takes 10 times as long, but gets a better answer - to which I respond, "Very well, run your algorithm 10x longer and see where you end up!" I think we need to see a more compelling demonstration of why this is a useful idea before it*s ready to be published. The idea of penalizing a policy based on KL-divergence from a reference policy was explored at length by Bert Kappen*s work on KL-MDPs. Perhaps you should cite that?