Year: 2018

Abstract:
We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then converts the optimal non-parameterized policy to a parameterized policy, from which it draws new samples. The methodology is general in that it applies to both discrete and continuous action spaces, and can handle a wide variety of proximity constraints for the non-parameterized optimization problem. We show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. The SPU implementation is much simpler than TRPO. In terms of sample efficiency, our extensive experiments show SPU outperforms TRPO in Mujoco simulated robotic tasks and outperforms PPO in Atari video game tasks.

heading: 1 INTRODUCTION
text: The policy gradient problem in deep reinforcement learning (DRL) can be defined as seeking a parameterized policy with high expected reward. An issue with policy gradient methods is poor sample efficiency (Kakade, 2003; Schulman et al., 2015a; Wang et al., 2016b; Wu et al., 2017; Schulman et al., 2017). In algorithms such as REINFORCE (Williams, 1992), new samples are needed for every gradient step. When generating samples is expensive (such as robotic environments), sample efficiency is of central concern. The sample efficiency of an algorithm is defined to be the number of calls to the environment required to attain a specified performance level (Kakade, 2003).
Thus, given the current policy and a fixed number of trajectories (samples) generated, the goal of the sample efficiency problem is to construct a new policy with the highest performance improvement possible. To do so, it is desirable to limit the search to policies that are close to the original policy πθk (Kakade, 2002; Schulman et al., 2015a; Wu et al., 2017; Achiam et al., 2017; Schulman et al., 2017; Tangkaratt et al., 2018). Intuitively, if the candidate new policy πθ is far from the original policy πθk , it may not perform better than the original policy because too much emphasis is being placed on the relatively small batch of new data generated by πθk , and not enough emphasis is being placed on the relatively large amount of data and effort previously used to construct πθk .
This guideline of limiting the search to nearby policies seems reasonable in principle, but requires a distance η(πθ, πθk) between the current policy πθk and the candidate new policy πθ, and then attempt to solve the constrained optimization problem:
maximize θ Ĵ(πθ | πθk , new data) (1)
subject to η(πθ, πθk) ≤ δ (2)
where Ĵ(πθ | πθk , new data) is an estimate of J(πθ), the performance of policy πθ, based on the previous policy πθk and the batch of fresh data generated by πθk . The objective (1) attempts to
maximize the performance of the updated policy, and the constraint (2) ensures that the updated policy is not too far from the policy πθk that was used to generate the data. Several recent papers (Kakade, 2002; Schulman et al., 2015a; 2017; Tangkaratt et al., 2018) belong to the framework (1)-(2).
Our work also strikes the right balance between performance and simplicity. The implementation is only slightly more involved than PPO (Schulman et al., 2017). Simplicity in RL algorithms has its own merits. This is especially useful when RL algorithms are used to solve problems outside of traditional RL testbeds, which is becoming a trend (Zoph & Le, 2016; Mingxing Tan, 2018).
We propose a new methodology, called Supervised Policy Update (SPU), for this sample efficiency problem. The methodology is general in that it applies to both discrete and continuous action spaces, and can address a wide variety of constraint types for (2). Starting with data generated by the current policy, SPU optimizes over a proximal policy space to find an optimal non-parameterized policy. It then solves a supervised regression problem to convert the non-parameterized policy to a parameterized policy, from which it draws new samples. We develop a general methodology for finding an optimal policy in the non-parameterized policy space, and then illustrate the methodology for three different definitions of proximity. We also show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. While SPU is substantially simpler than NPG/TRPO in terms of mathematics and implementation, our extensive experiments show that SPU is more sample efficient than TRPO in Mujoco simulated robotic tasks and PPO in Atari video game tasks.
Off-policy RL algorithms generally achieve better sample efficiency than on-policy algorithms (Haarnoja et al., 2018). However, the performance of an on-policy algorithm can usually be substantially improved by incorporating off-policy training (Mnih et al. (2015), Wang et al. (2016a)). Our paper focuses on igniting interests in separating finding the optimal policy into a two-step process: finding the optimal non-parameterized policy, and then parameterizing this optimal policy. We also wanted to deeply understand the on-policy case before adding off-policy training. We thus compare with algorithms operating under the same algorithmic constraints, one of which is being on-policy. We leave the extension to off-policy to future work. We do not claim state-of-the-art results.

heading: 2 PRELIMINARIES
text: We consider a Markov Decision Process (MDP) with state space S, action space A, and reward function r(s, a), s ∈ S , a ∈ A. Let π = {π(a|s) : s ∈ S, a ∈ A} denote a policy, let Π be the set of all policies, and let the expected discounted reward be:
J(π) , E τ∼π [ ∞∑ t=0 γtr(st, at) ] (3)
where γ ∈ (0, 1) is a discount factor and τ = (s0, a0, s1, . . . ) is a sample trajectory. Let Aπ(s, a) be the advantage function for policy π (Levine, 2017). Deep reinforcement learning considers a set of parameterized policies ΠDL = {πθ|θ ∈ Θ} ⊂ Π, where each policy is parameterized by a neural network called the policy network. In this paper, we will consider optimizing over the parameterized policies in ΠDL as well as over the non-parameterized policies in Π. For concreteness, we assume that the state and action spaces are finite. However, our methodology also applies to continuous state and action spaces, as shown in the Appendix.
One popular approach to maximizing J(πθ) over ΠDL is to apply stochastic gradient ascent. The gradient of J(πθ) evaluated at a specific θ = θk can be shown to be (Williams, 1992):
∇θJ(πθk) = Eτ∼πθk [ ∞∑ t=0 γt∇θ log πθk(at|st)Aπθk (st, at) ] . (4)
We can approximate (4) by sampling N trajectories of length T from πθk :
∇θJ(πθk) ≈ 1
N N∑ i=1 T−1∑ t=0 ∇θ log πθk(ait|sit)Aπθk (st, at) , gk (5)
Additionally, define dπ(s) , (1 − γ) ∑∞ t=0 γ
tPπ(st = s) for the the future state probability distribution for policy π, and denote π(·|s) for the probability distribution over the action space A
when in state s and using policy π. Further denote DKL(π ‖ πθk)[s] for the KL divergence from π(·|s) to πθk(·|s), and denote the following as the “aggregated KL divergence”.
D̄KL(π ‖ πθk) = E s∼dπθk [DKL(π ‖ πθk)[s]] (6)

heading: 2.1 SURROGATE OBJECTIVES FOR THE SAMPLE EFFICIENCY PROBLEM
text: For the sample efficiency problem, the objective J(πθ) is typically approximated using samples generated from πθk (Schulman et al., 2015a; Achiam et al., 2017; Schulman et al., 2017). Two different approaches are typically used to approximate J(πθ)− J(πθk). We can make a first order approximation of J(πθ) around θk (Kakade, 2002; Peters & Schaal, 2008a;b; Schulman et al., 2015a):
J(πθ)− J(πθk) ≈ (θ − θk)T∇θJ(πθk) ≈ (θ − θk)T gk (7) where gk is the sample estimate (5). The second approach is to approximate the state distribution dπ(s) with dπθk (s) (Achiam et al., 2017; Schulman et al., 2017; Achiam, 2017):
J(π)− J(πθk) ≈ Lπθk (π) , 1 1− γ E s∼dπθk E
a∼πθk (·|s) [ π(a|s) πθk(a|s) Aπθk (s, a) ] (8)
There is a well-known bound for the approximation (8) (Kakade & Langford, 2002; Achiam et al., 2017). Furthermore, the approximation Lπθk (πθ) matches J(πθ) − J(πθk) to the first order with respect to the parameter θ (Achiam et al., 2017).

heading: 3 RELATED WORK
text: Natural gradient (Amari, 1998) was first introduced to policy gradient by Kakade (Kakade, 2002) and then in (Peters & Schaal, 2008a;b; Achiam, 2017; Schulman et al., 2015a). referred to collectively here as NPG/TRPO. Algorithmically, NPG/TRPO finds the gradient update by solving the sample efficiency problem (1)-(2) with η(πθ, πθk) = D̄KL(πθ ‖ πθk), i.e., use the aggregate KL-divergence for the policy proximity constraint (2). NPG/TRPO addresses this problem in the parameter space θ ∈ Θ. First, it approximates J(πθ) with the first-order approximation (7) and D̄KL(πθ ‖ πθk) using a similar second-order method. Second, it uses samples from πθk to form estimates of these two approximations. Third, using these estimates (which are functions of θ), it solves for the optimal θ∗. The optimal θ∗ is a function of gk and of hk, the sample average of the Hessian evaluated at θk. TRPO also limits the magnitude of the update to ensure D̄KL(πθ ‖ πθk) ≤ δ (i.e., ensuring the sampled estimate of the aggregated KL constraint is met without the second-order approximation).
SPU takes a very different approach by first (i) posing and solving the optimization problem in the non-parameterized policy space, and then (ii) solving a supervised regression problem to find a parameterized policy that is near the optimal non-parameterized policy. A recent paper, Guided Actor Critic (GAC), independently proposed a similar decomposition (Tangkaratt et al., 2018). However, GAC is much more restricted in that it considers only one specific constraint criterion (aggregated reverse-KL divergence) and applies only to continuous action spaces. Furthermore, GAC incurs significantly higher computational complexity, e.g. at every update, it minimizes the dual function to obtain the dual variables using SLSQP. MPO also independently propose a similar decomposition (Abbas Abdolmaleki, 2018). MPO uses much more complex machinery, namely, Expectation Maximization to address the DRL problem. However, MPO has only demonstrates preliminary results on problems with discrete actions whereas our approach naturally applies to problems with either discrete or continuous actions. In both GAC and MPO, working in the non-parameterized space is a by-product of applying the main ideas in those papers to DRL. Our paper demonstrates that the decomposition alone is a general and useful technique for solving constrained policy optimization.
Clipped-PPO (Schulman et al., 2017) takes a very different approach to TRPO. At each iteration, PPO makes many gradient steps while only using the data from πθk . Without the clipping, PPO is the approximation (8). The clipping is analogous to the constraint (2) in that it has the goal of keeping πθ close to πθk . Indeed, the clipping keeps πθ(at|st) from becoming neither much larger than (1 + )πθk(at|st) nor much smaller than (1 − )πθk(at|st). Thus, although the clipped PPO objective does not squarely fit into the optimization framework (1)-(2), it is quite similar in spirit. We note that the PPO paper considers adding the KL penalty to the objective function, whose gradient
is similar to ours. However, this form of gradient was demonstrated to be inferior to Clipped-PPO. To the best of our knowledge, it is only until our work that such form of gradient is demonstrated to outperform Clipped-PPO.
Actor-Critic using Kronecker-Factored Trust Region (ACKTR) (Wu et al., 2017) proposed using Kronecker-factored approximation curvature (K-FAC) to update both the policy gradient and critic terms, giving a more computationally efficient method of calculating the natural gradients. ACER (Wang et al., 2016a) exploits past episodes, linearizes the KL divergence constraint, and maintains an average policy network to enforce the KL divergence constraint. In future work, it would of interest to extend the SPU methodology to handle past episodes. In contrast to bounding the KL divergence on the action distribution as we have done in this work, Relative Entropy Policy Search considers bounding the joint distribution of state and action and was only demonstrated to work for small problems (Jan Peters, 2010).

heading: 4 SPU FRAMEWORK
text: The SPU methodology has two steps. In the first step, for a given constraint criterion η(π, πθk) ≤ δ, we find the optimal solution to the non-parameterized problem:
maximize π∈Π Lπθk (π) (9)
subject to η(π, πθk) ≤ δ (10)
Note that π is not restricted to the set of parameterized policies ΠDL. As commonly done, we approximate the objective function (8). However, unlike PPO/TRPO, we are not approximating the constraint (2). We will show below the optimal solution π∗ for the non-parameterized problem (9)-(10) can be determined nearly in closed form for many natural constraint criteria η(π, πθk) ≤ δ. In the second step, we attempt to find a policy πθ in the parameterized space ΠDL that is close to the target policy π∗. Concretely, to advance from θk to θk+1, we perform the following steps:
(i) We first sample N trajectories using policy πθk , giving sample data (si, ai, Ai), i = 1, ..,m. Here Ai is an estimate of the advantage value Aπθk (si, ai). (For simplicity, we index the samples with i rather than with (i, t) corresponding to the tth sample in the ith trajectory.)
(ii) For each si, we define the target distribution π∗ to be the optimal solution to the constrained optimization problem (9)-(10) for a specific constraint η.
(iii) We then fit the policy network πθ to the target distributions π∗(·|si), i = 1, ..,m. Specifically, to find θk+1, we minimize the following supervised loss function:
L(θ) = 1
N m∑ i=1 DKL(πθ ‖ π∗)[si] (11)
For this step, we initialize with the weights for πθk . We minimize the loss function L(θ) with stochastic gradient descent methods. The resulting θ becomes our θk+1.

heading: 5 SPU APPLIED TO SPECIFIC PROXIMITY CRITERIA
text: To illustrate the SPU methodology, for three different but natural types of proximity constraints, we solve the corresponding non-parameterized optimization problem and derive the resulting gradient for the SPU supervised learning problem. We also demonstrate that different constraints lead to very different but intuitive forms of the gradient update.

heading: 5.1 FORWARD AGGREGATE AND DISAGGREGATE KL CONSTRAINTS
text: We first consider constraint criteria of the form:
maximize π∈Π ∑ s dπθk (s) E a∼π(·|s) [Aπθk (s, a)] (12)
subject to ∑ s dπθk (s)DKL(π ‖ πθk)[s] ≤ δ (13)
DKL(π ‖ πθk)[s] ≤ for all s (14)
Note that this problem is equivalent to minimizing Lπθk (π) subject to the constraints (13) and (14). We refer to (13) as the "aggregated KL constraint" and to (14) as the "disaggregated KL constraint". These two constraints taken together restrict π from deviating too much from πθk . We shall refer to (12)-(14) as the forward-KL non-parameterized optimization problem.
Note that this problem without the disaggregated constraints is analogous to the TRPO problem. The TRPO paper actually prefers enforcing the disaggregated constraint to enforcing the aggregated constraints. However, for mathematical conveniences, they worked with the aggregated constraints: "While it is motivated by the theory, this problem is impractical to solve due to the large number of constraints. Instead, we can use a heuristic approximation which considers the average KL divergence" (Schulman et al., 2015a). The SPU framework allows us to solve the optimization problem with the disaggregated constraints exactly. Experimentally, we compared against TRPO in a controlled experimental setting, e.g. using the same advantage estimation scheme, etc. Since we clearly outperform TRPO, we argue that SPU’s two-process procedure has significant potentials.
For each λ > 0, define: πλ(a|s) = πθk(a|s) Zλ(s) eA πθk (s,a)/λ where Zλ(s) is the normalization term. Note that πλ(a|s) is a function of λ. Further, for each s, let λs be such that DKL(πλs ‖ πθk)[s] = . Also let Γλ = {s : DKL(πλ ‖ πθk)[s] ≤ }.
Theorem 1 The optimal solution to the problem (12)-(14) is given by: π̃λ(a|s) = { πλ(a|s) s ∈ Γλ πλs(a|s) s /∈ Γλ (15)
where λ is chosen so that ∑ s d πθk (s)DKL(π̃ λ ‖ πθk)[s] = δ (Proof in subsection A.1).
Equation (15) provides the structure of the optimal non-parameterized policy. As part of the SPU framework, we then seek a parameterized policy πθ that is close to π̃λ(a|s), that is, minimizes the loss function (11). For each sampled state si, a straightforward calculation shows (Appendix B):
∇θDKL(πθ ‖ π̃λ)[si] = ∇θDKL(πθ ‖ πθk)[si]− 1
λ̃si E a∼πθk (·|si) [∇θ log πθ(a|si)Aπθk (si, a)]
(16)
where ∼ λsi = λ for si ∈ Γλ and λ̃si = λsi for si /∈ Γλ. We estimate the expectation in (16) with the sampled action ai and approximate Aπθk (si, ai) as Ai (obtained from the critic network), giving:
∇θDKL(πθ ‖ π̃λ)[si] ≈ ∇θDKL(πθ ‖ πθk)[si]− 1 ∼ λsi ∇θπθ(ai|si) πθk(ai|si) Ai (17)
To simplify the algorithm, we slightly modify (17). We replace the hyper-parameter δ with the hyper-parameter λ and tune λ rather than δ. Further, we set ∼ λsi = λ for all si in (17) and introduce per-state acceptance to enforce the disaggregated constraints, giving the approximate gradient:
∇θDKL(πθ ‖ π̃λ) ≈ 1
m m∑ i=1 [∇θDKL(πθ ‖ πθk)[si]− 1 λ ∇θπθ(ai|si) πθk(ai|si) Ai]1DKL(πθ‖πθk )[si]≤ (18)
We make the approximation that the disaggregated constraints are only enforced on the states in the sampled trajectories. We use (18) as our gradient for supervised training of the policy
network. The equation (18) has an intuitive interpretation: the gradient represents a trade-off
between the approximate performance of πθ (as captured by 1
λ ∇θπθ(ai|si) πθk(ai|si) Ai) and how far πθ
diverges from πθk (as captured by∇θDKL(πθ ‖ πθk)[si]). For the stopping criterion, we train until 1
m
∑ iDKL(πθ ‖ πθk)[si] ≈ δ.

heading: 5.2 BACKWARD KL CONSTRAINT
text: In a similar manner, we can derive the structure of the optimal policy when using the reverse KL-divergence as the constraint. For simplicity, we provide the result for when there are only disaggregated constraints. We seek to find the non-parameterized optimal policy by solving:
maximize π∈Π ∑ s dπθk (s) E a∼π(·|s) [Aπθk (s, a)] (19)
DKL(π ‖ πθk)[s] ≤ for all s (20)
Theorem 2 The optimal solution to the problem (19)-(20) is given by:
π∗(a|s) = πθk(a|s) λ(s)
λ′(s)−Aπθk (s, a) (21)
where λ(s) > 0 and λ′(s) > maxaAπθk (s, a) (Proof in subsection A.2).
Note that the structure of the optimal policy with the backward KL constraint is quite different from that with the forward KL constraint. A straight forward calculation shows (Appendix B):
∇θDKL(πθ ‖ π∗)[s] = ∇θDKL(πθ ‖ πθk)[s]− Ea∼πθk [ ∇θπθ(a|s) πθk(a|s) log ( 1 λ′(s)−Aπθk (s, a) )] (22)
The equation (22) has an intuitive interpretation. It increases the probability of action a if Aπθk (s, a) > λ′(s) − 1 and decreases the probability of action a if Aπθk (s, a) < λ′(s) − 1. (22) also tries to keep πθ close to πθk by minimizing their KL divergence.

heading: 5.3 L∞ CONSTRAINT
text: In this section we show how a PPO-like objective can be formulated in the context of SPU. Recall from Section 3 that the the clipping in PPO can be seen as an attempt at keeping πθ(ai|si) from becoming neither much larger than (1 + )πθk(ai|si) nor much smaller than (1− )πθk(ai|si) for i = 1, . . . ,m. In this subsection, we consider the constraint function
η(π, πθk) = max i=1,...,m |π(ai|si)− πθk(ai|si)| πθk(ai|si)
(23)
which leads us to the following optimization problem:
maximize π(a1|s1),...,π(am|sm) m∑ i=1 Aπθk (si, ai) π(ai|si) πk(ai|si)
(24)
subject to ∣∣∣∣π(ai|si)− πθk(ai|si)πθk(ai|si) ∣∣∣∣ ≤ i = 1, . . . ,m (25) m∑ i=1 ( π(ai|si)− πθk(ai|si) πθk(ai|si) )2 ≤ δ (26)
Note that here we are using a variation of the SPU methodology described in Section 4 since here we first create estimates of the expectations in the objective and constraints and then solve the optimization problem (rather than first solve the optimization problem and then take samples as done for Theorems 1 and 2). Note that we have also included an aggregated constraint (26) in addition to the PPO-like constraint (25), which further ensures that the updated policy is close to πθk .
Theorem 3 The optimal solution to the optimization problem (24-26) is given by:
π∗(ai|si) = { πθk(ai|si) min{1 + λAi, 1 + } Ai ≥ 0 πθk(ai|si) max{1 + λAi, 1− } Ai < 0
(27)
for some λ > 0 where Ai , Aπθk (si, ai) (Proof in subsection A.3).
To simplify the algorithm, we treat λ as a hyper-parameter rather than δ. After solving for π∗, we seek a parameterized policy πθ that is close to π∗ by minimizing their mean square error over sampled states and actions, i.e. by updating θ in the negative direction of ∇θ ∑ i(πθ(ai|si) − π∗(ai|si))2. This loss is used for supervised training instead of the KL because we take estimates before forming the optimization problem. Thus, the optimal values for the decision variables do not completely characterize a distribution. We refer to this approach as SPU with the L∞ constraint.
Although we consider three classes of proximity constraint, there may be yet another class that leads to even better performance. The methodology allows researchers to explore other proximity constraints in the future.

heading: 6 EXPERIMENTAL RESULTS
text: Extensive experimental results demonstrate SPU outperforms recent state-of-the-art methods for environments with continuous or discrete action spaces. We provide ablation studies to show the importance of the different algorithmic components, and a sensitivity analysis to show that SPU’s performance is relatively insensitive to hyper-parameter choices. There are two definitions we use to conclude A is more sample efficient than B: (i) A takes fewer environment interactions to achieve a pre-defined performance threshold (Kakade, 2003); (ii) the averaged final performance of A is higher than that of B given the same number environment interactions (Schulman et al., 2017). Implementation details are provided in Appendix D.

heading: 6.1 RESULTS ON MUJOCO
text: The Mujoco (Todorov et al., 2012) simulated robotics environments provided by OpenAI gym (Brockman et al., 2016) have become a popular benchmark for control problems with continuous action spaces. In terms of final performance averaged over all available ten Mujoco environments and ten different seeds in each, SPU with L∞ constraint (Section 5.3) and SPU with forward KL constraints (Section 5.1) outperform TRPO by 6% and 27% respectively. Since the forward-KL approach is our best performing approach, we focus subsequent analysis on it and hereafter refer to it as SPU. SPU also outperforms PPO by 17%. Figure 1 illustrates the performance of SPU versus TRPO, PPO.
To ensure that SPU is not only better than TRPO in terms of performance gain early during training, we further retrain both policies for 3 million timesteps. Again here, SPU outperforms TRPO by 28%. Figure 3 in the Appendix illustrates the performance for each environment. Code for the Mujoco experiments is at https://github.com/quanvuong/Supervised_Policy_Update.

heading: 6.2 ABLATION STUDIES FOR MUJOCO
text: The indicator variable in (18) enforces the disaggregated constraint. We refer to it as per-state acceptance. Removing this component is equivalent to removing the indicator variable. We refer to using ∑ iDKL(πθ ‖ πθk)[si] to determine the number of training epochs as dynamic stopping. Without this component, the number of training epochs is a hyper-parameter. We also tried removing ∇θDKL(πθ ‖ πθk)[si] from the gradient update step in (18). Table 1 illustrates the contribution of the different components of SPU to the overall performance. The third row shows that the term ∇θDKL(πθ ‖ πθk)[si] makes a crucially important contribution to SPU. Furthermore, per-state acceptance and dynamic stopping are both also important for obtaining high performance, with the former playing a more central role. When a component is removed, the hyper-parameters are retuned to ensure that the best possible performance is obtained with the alternative (simpler) algorithm.

heading: 6.3 SENSITIVITY ANALYSIS ON MUJOCO
text: To demonstrate the practicality of SPU, we show that its high performance is insensitive to hyperparameter choice. One way to show this is as follows: for each SPU hyper-parameter, select a reasonably large interval, randomly sample the value of the hyper parameter from this interval, and then compare SPU (using the randomly chosen hyper-parameter values) with TRPO. We sampled 100 SPU hyper-parameter vectors (each vector including δ, , λ), and for each one determined the relative performance with respect to TRPO. First, we found that for all 100 random hyper-parameter value samples, SPU performed better than TRPO. 75% and 50% of the samples outperformed TRPO by at least 18% and 21% respectively. The full CDF is given in Figure 4 in the Appendix. We can conclude that SPU’s superior performance is largely insensitive to hyper-parameter values.

heading: 6.4 RESULTS ON ATARI
text: (Rajeswaran et al., 2017; Mania et al., 2018) demonstrates that neural networks are not needed to obtain high performance in many Mujoco environments. To conclusively evaluate SPU, we compare it against PPO on the Arcade Learning Environments (Bellemare et al., 2012) exposed through OpenAI gym (Brockman et al., 2016). Using the same network architecture and hyper-parameters, we learn to play 60 Atari games from raw pixels and rewards. This is highly challenging because of the diversity in the games and the high dimensionality of the observations.
Here, we compare SPU against PPO because PPO outperforms TRPO by 9% in Mujoco. Averaged over 60 Atari environments and 20 seeds, SPU is 55% better than PPO in terms of averaged final performance. Figure 2 provides a high-level overview of the result. The dots in the shaded area represent environments where their performances are roughly similar. The dots to the right of the shaded area represent environment where SPU is more sample efficient than PPO. We can draw two conclusions: (i) In 36 environments, SPU and PPO perform roughly the same ; SPU clearly outperforms PPO in 15 environments while PPO clearly outperforms SPU in 9; (ii) In those 15+9 environments, the extent to which SPU outperforms PPO is much larger than the extent to which PPO outperforms SPU. Figure 5, Figure 6 and Figure 7 in the Appendix illustrate the performance of SPU vs PPO throughout training. SPU’s high performance in both the Mujoco and Atari domains demonstrates its high performance and generality.

heading: 7 ACKNOWLEDGEMENTS
text: We would like to acknowledge the extremely helpful support by the NYU Shanghai High Performance Computing Administrator Zhiguo Qi. We also are grateful to OpenAI for open-sourcing their baselines codes.

heading: Appendices

heading: A PROOFS FOR NON-PARAMETERIZED OPTIMIZATION PROBLEMS

heading: A.1 FORWARD KL AGGREGATED AND DISAGGREGATED CONSTRAINTS
text: We first show that (12)-(14) is a convex optimization. To this end, first note that the objective (12) is a linear function of the decision variables π = {π(a|s) :́ s ∈ S, a ∈ A}. The LHS of (14) can be rewritten as: ∑ a∈A π(a|s) log π(a|s)− ∑ a∈A π(a|s) log πθk(a|s). The second term is a linear function of π. The first term is a convex function since the second derivative of each summand is always positive. The LHS of (14) is thus a convex function. By extension, the LHS of (13) is also a convex function since it is a nonnegative weighted sum of convex functions. The problem (12)-(14) is thus a convex optimization problem. According to Slater’s constraint qualification, strong duality holds since πθk is a feasible solution to (12)-(14) where the inequality holds strictly.
We can therefore solve (12)-(14) by solving the related Lagrangian problem. For a fixed λ consider:
maximize π∈Π ∑ s dπθk (s){ E a∼π(·|s) [Aπθk (s, a)]− λDKL(π ‖ πθk)[s]} (28)
subject to DKL(π ‖ πθk)[s] ≤ for all s (29)
The above problem decomposes into separate problems, one for each state s:
maximize π(·|s) E a∼π(·|s) [Aπθk (s, a)]− λDKL(π ‖ πθk)[s] (30)
subject to DKL(π ‖ πθk)[s] ≤ (31)
Further consider the unconstrained problem (30) without the constraint (31):
maximize π(·|s) K∑ a=1 π(a|s) [ Aπθk (s, a)− λ log ( π(a|s) πθk(a|s) )] (32)
subject to K∑ a=1 π(a|s) = 1 (33)
π(a|s) ≥ 0, a = 1, . . . ,K (34)
A simple Lagrange-multiplier argument shows that the opimal solution to (32)-(34) is given by:
πλ(a|s) = πθk(a|s) Zλ(s) eA πθk (s,a)/λ
where Zλ(s) is defined so that πλ(·|s) is a valid distribution. Now returning to the decomposed constrained problem (30)-(31), there are two cases to consider. The first case is when DKL(πλ ‖ πθk)[s] ≤ . In this case, the optimal solution to (30)-(31) is πλ(a|s). The second case is when DKL(π
λ ‖ πθk)[s] > . In this case the optimal is πλ(a|s) with λ replaced with λs, where λs is the solution to DKL(πλ ‖ πθk)[s] = . Thus, an optimal solution to (30)-(31) is given by:
π̃λ(a|s) =  πθk(a|s) Z(s) eA πθk (s,a)/λ s ∈ Γλ
πθk(a|s) Z(s) eA πθk (s,a)/λs s /∈ Γλ
(35)
where Γλ = {s : DKL(πλ ‖ πθk)[s] ≤ }. To find the Lagrange multiplier λ, we can then do a line search to find the λ that satisfies:∑
s
dπθk (s)DKL(π̃ λ ‖ πθk)[s] = δ (36)

heading: A.2 BACKWARD KL CONSTRAINT
text: The problem (19)-(20) decomposes into separate problems, one for each state s ∈ S:
maximize π(·|s) E a∼πθk (·|s) [ π(a|s) πθk(a|s) Aπθk (s, a) ] (37)
subject to E a∼πθk (·|s)
[ log
πθk(a|s) π(a|s)
] ≤ (38)
After some algebra, we see that above optimization problem is equivalent to:
maximize π(·|s) K∑ a=1 Aπθk (s, a)π(a|s) (39)
subject to − K∑ a=1 πθk(a|s) log π(a|s) ≤ ′ (40)
K∑ a=1 π(a|s) = 1 (41)
π(a|s) ≥ 0, a = 1, . . . ,K (42) where ′ = + entropy(πθk). (39)-(42) is a convex optimization problem with Slater’s condition holding. Strong duality thus holds for the problem (39)-(42). Applying standard Lagrange multiplier arguments, it is easily seen that the solution to (39)-(42) is
π∗(a|s) = πθk(a|s) λ(s)
λ′(s)−Aπθk (s, a) where λ(s) and λ′(s) are constants chosen such that the disaggregegated KL constraint is binding and the sum of the probabilities equals 1. It is easily seen λ(s) > 0 and λ′(s) > maxaAπθk (s, a)

heading: A.3 L∞ CONSTRAINT
text: The problem (24-26) is equivalent to:
maximize π(a1|s1),...,π(am|sm) m∑ i=1 Aπθk (si, ai) π(ai|si) πk(ai|si)
(43)
subject to 1− ≤ π(ai|si) πθk(ai|si) ≤ 1 + i = 1, . . . ,m (44)
m∑ i=1 ( π(ai|si)− πθk(ai|si) πθk(ai|si) )2 ≤ δ (45)
This problem is clearly convex. πθk(ai|si), i = 1, . . . ,m is a feasible solution where the inequality constraint holds strictly. Strong duality thus holds according to Slater’s constraint qualification. To solve (43)-(45), we can therefore solve the related Lagrangian problem for fixed λ:
maximize π(a1|s1),...,π(am|sm) m∑ i=1
[ Aπθk (si, ai)
π(ai|si) πk(ai|si)
− λ ( π(ai|si)− πθk(ai|si)
πθk(ai|si)
)2] (46)
subject to 1− ≤ π(ai|si) πθk(ai|si) ≤ 1 + i = 1, . . . ,m (47)
which is separable and decomposes into m separate problems, one for each si:
maximize π(ai|si) Aπθk (si, ai) π(ai|si) πk(ai|si)
− λ ( π(ai|si)− πθk(ai|si)
πθk(ai|si)
)2 (48)
subject to 1− ≤ π(ai|si) πθk(ai|si) ≤ 1 + (49)
The solution to the unconstrained problem (48) without the constraint (49) is: π∗(ai|si) = πθk(ai|si) ( 1 + Aπθk (si, ai)
2λ ) Now consider the constrained problem (48)-(49). IfAπθk (si, ai) ≥ 0 and π∗(ai|si) > πθk(ai|si)(1+ ), the optimal solution is πθk(ai|si)(1 + ). Similarly, If Aπθk (si, ai) < 0 and π∗(ai|si) < πθk(ai|si)(1− ), the optimal solution is πθk(ai|si)(1− ). Rearranging the terms gives Theorem 3. To obtain λ, we can perform a line search over λ so that the constraint (45) is binding.

heading: B DERIVATIONS THE GRADIENT OF LOSS FUNCTION FOR SPU
text: Let CE stands for CrossEntropy.

heading: B.1 FORWARD-KL
text: CE(πθ||π̃λ̃s)[s] = − ∑ a πθ(a|s) log π̃λ̃s(a|s)(Expanding the definition of cross entropy)
= − ∑ a πθ(a|s) log ( πθk(a|s) Zλ̃s(s) eA πθk (s,a)/λ̃s ) (Expanding the definition of π̃λ̃s )
= − ∑ a πθ(a|s) log ( πθk(a|s) Zλ̃s(s) ) − ∑ a πθ(a|s) Aπθk (s, a) λ̃s (Log of product is sum of log)
= − ∑ a πθ(a|s) log πθk(a|s) + ∑ a πθ(a|s) logZλ̃s(s)− 1 λ̃s ∑ a πθk(a|s) πθ(a|s) πθk(a|s) Aπθk (s, a)
= CE(πθ||πθk)[s] + logZλ̃s(s)− 1
λ̃s E a∼πθk (.|s) [ πθ(a|s) πθk(a|s) Aπθk (s, a) ] ⇒ ∇θCE(πθ||π̃λ̃s)[s] = ∇θCE(πθ||πθk)[s]− 1
λ̃s E a∼πθk (.|s) [ ∇θπθ(a|s) πθk(a|s) Aπθk (s, a) ] (Taking gradient on both sides)
⇒ ∇θDKL(πθ ‖ π̃λ̃s)[s] = ∇θDKL(πθ ‖ πθk)[s]− 1
λ̃s E a∼πθk (.|s) [ ∇θπθ(a|s) πθk(a|s) Aπθk (s, a) ] (Adding the gradient of the entropy on both sides and collapse the sum of gradients of cross entropy and entropy into the gradient of the KL)

heading: B.2 REVERSE-KL
text: CE(πθ||π∗)[s] = − ∑ a πθ(a|s) log π∗(a|s)(Expanding the definition of cross entropy)
= − ∑ a πθ(a|s) log ( πθk(a|s)
λ(s)
λ′(s)−Aπθk (s, a)
) (Expanding the definition of π∗)
= − ∑ a πθ(a|s) log πθk(a|s)− ∑ a πθ(a|s) log λ(s) + ∑ a πθ(a|s) log(λ′(s)−Aπθk (s, a))
= CE(πθ||πθk)[s]− λ(s) + Ea∼πθk [ πθ(a|s) πθk(a|s) log(λ′(s)−Aπθk (s, a)) ]
= CE(πθ||πθk)[s]− λ(s)− Ea∼πθk [ πθ(a|s) πθk(a|s) log 1 λ′(s)−Aπθk (s, a) ] ⇒ ∇θCE(πθ||π∗)[s] = ∇θCE(πθ||πθk)[s]− Ea∼πθk [ ∇θπθ(a|s) πθk(a|s) log 1 λ′(s)−Aπθk (s, a) ] (Taking gradient on both sides)
⇒ ∇θDKL(πθ ‖ π∗)[s] = ∇θDKL(πθ ‖ πθk)[s]− Ea∼πθk [ ∇θπθ(a|s) πθk(a|s) log 1 λ′(s)−Aπθk (s, a) ] (Adding the gradient of the entropy on both sides and collapse the sum of gradients of cross entropy and entropy into the gradient of the KL)

heading: C EXTENSION TO CONTINUOUS STATE AND ACTION SPACES
text: The methodology developed in the body of this paper also applies to continuous state and action spaces. In this section, we outline the modifications that are necessary for the continuous case.
We first modify the definition of dπ(s) by replacing Pπ(st = s) with ddsPπ(st ≤ s) so that d π(s) becomes a density function over the state space. With this modification, the definition of D̄KL(π ‖ πk) and the approximation (8) are unchanged. The SPU framework described in Section 4 is also unchanged.
Consider now the non-parameterized optimization problem with aggregate and disaggregate constraints (12-14), but with continuous state and action space:
maximize π∈Π
∫ dπθk (s) E
a∼π(·|s) [Aπθk (s, a)]ds (50) subject to ∫ dπθk (s)DKL(π ‖ πθk)[s]ds ≤ δ (51)
DKL(π ‖ πθk)[s] ≤ for all s (52)
Theorem 1 holds although its proof needs to be slightly modified as follows. It is straightforward to show that (50-52) remains a convex optimization problem. We can therefore solve (50-52) by solving the Lagrangian (28-29) with the sum replaced with an integral. This problem again decomposes with separate problems for each s ∈ S giving exactly the same equations (30-31). The proof then proceeds as in the remainder of the proof of Theorem 1.
Theorem 2 and 3 are also unchanged for continuous action spaces. Their proofs require slight modifications, as in the proof of Theorem 1.
D IMPLEMENTATION DETAILS AND HYPERPARAMETERS

heading: D.1 MUJOCO
text: As in (Schulman et al., 2017), for Mujoco environments, the policy is parameterized by a fullyconnected feed-forward neural network with two hidden layers, each with 64 units and tanh nonlinearities. The policy outputs the mean of a Gaussian distribution with state-independent variable standard deviations, following (Schulman et al., 2015a; Duan et al., 2016). The action dimensions are assumed to be independent. The probability of an action is given by the multivariate Gaussian probability distribution function. The baseline used in the advantage value calculation is parameterized by a similarly sized neural network, trained to minimize the MSE between the sampled states TD−λ returns and the their predicted values. For both the policy and baseline network, SPU and TRPO use the same architecture. To calculate the advantage values, we use Generalized Advantage Estimation (Schulman et al., 2015b). States are normalized by dividing the running mean and dividing by the running standard deviation before being fed to any neural networks. The advantage values are normalized by dividing the batch mean and dividing by the batch standard deviation before being used for policy update. The TRPO result is obtained by running the TRPO implementation provided by OpenAI (Dhariwal et al., 2017), commit 3cc7df060800a45890908045b79821a13c4babdb. At every iteration, SPU collects 2048 samples before updating the policy and the baseline network. For both networks, gradient descent is performed using Adam (Kingma & Ba, 2014) with step size 0.0003, minibatch size of 64. The step size is linearly annealed to 0 over the course of training. γ and λ for GAE (Schulman et al., 2015b) are set to 0.99 and 0.95 respectively. For SPU, δ, , λ and the maximum number of epochs per iteration are set to 0.05/1.2, 0.05, 1.3 and 30 respectively. Training is performed for 1 million timesteps for both SPU and PPO. In the sensitivity analysis, the ranges of values for the hyper-parameters δ, , λ and maximum number of epochs are [0.05, 0.07], [0.01, 0.07], [1.0, 1.2] and [5, 30] respectively.

heading: D.2 ATARI
text: Unless otherwise mentioned, the hyper-parameter values are the same as in subsection D.1. The policy is parameterized by a convolutional neural network with the same architecture as described in Mnih et al. (2015). The output of the network is passed through a relu, linear and softmax layer in that order to give the action distribution. The output of the network is also passed through a different linear layer to give the baseline value. States are normalized by dividing by 255 before being fed into any network. The TRPO result is obtained by running the PPO implementation provided by OpenAI (Dhariwal et al., 2017), commit 3cc7df060800a45890908045b79821a13c4babdb. 8 different processes run in parallel to collect timesteps. At every iteration, each process collects 256 samples before updating the policy and the baseline network. Each process calculates its own update to the network’s parameters and the updates are averaged over all processes before being used to update the network’s parameters. Gradient descent is performed using Adam (Kingma & Ba, 2014) with step size 0.0001. In each process, random number generators are initialized with a different seed according to the formula process_seed = experiment_seed+ 10000 ∗ process_rank. Training is performed for 10 million timesteps for both SPU and PPO. For SPU, δ, , λ and the maximum number of epochs per iteration are set to 0.02, δ/1.3, 1.1 and 9 respectively.

heading: E ALGORITHMIC DESCRIPTION FOR SPU
text: Algorithm 1 Algorithmic description of forward-KL non-parameterized SPU Require: A neural net πθ that parameterizes the policy. Require: A neural net Vφ that approximates V πθ . Require: General hyperparameters: γ, β (advantage estimation using GAE), α (learning rate), N
(number of trajectory per iteration), T (size of each trajectory), M (size of training minibatch). Require: Algorithm-specific hyperparameters: δ (aggregated KL constraint), (disaggregated con-
straint), λ, ζ (max number of epoch). 1: for k = 1, 2, . . . do 2: under policy πθk , sample N trajectories, each of size T (sit, ait, rit, si(t+1)), i =
1, . . . , N, t = 1, . . . , T 3: Using any advantage value estimation scheme, estimate Ait, i = 1, . . . , N, t = 1, . . . , T 4: θ ← θk 5: φ← φk 6: for ζ epochs do 7: Sample M samples from the N trajectories, giving {s1, a1, A1, . . . , sM , aM , AM} 8: L(φ) = 1
M ∑ m (V targ(sm)− Vφ(sm))2
9: φ← φ− α∇φL(φ) 10: L(θ) = 1M ∑ m [ ∇θDKL(πθ ‖ πθk)[sm]− 1 λ ∇θπθ(am|sm) πθk(am|sm) Am ] 1DKL(πθ‖πθk )[sm]≤ 11: θ ← θ − αL(θ) 12: if 1
m
∑ mDKL(π ‖ πθk)[sm] > δ then
13: Break out of for loop

heading: 14: θk+1 ← θ

heading: 15: φk+1 ← φ

heading: F EXPERIMENTAL RESULTS

heading: F.1 RESULTS ON MUJOCO FOR 3 MILLION TIMESTEPS
text: TRPO and SPU were trained for 1 million timesteps to obtain the results in section 6. To ensure that SPU is not only better than TRPO in terms of performance gain early during training, we further retrain both policies for 3 million timesteps. Again here, SPU outperforms TRPO by 28%. Figure 3 illustrates the performance on each environment.

heading: F.2 SENSITIVITY ANALYSIS CDF FOR MUJOCO
text: When values for SPU hyper-parameter are randomly sampled as is explained in subsection 6.3, the percentage improvement of SPU over TRPO becomes a random variable. Figure 4 illustrates the CDF of this random variable.

heading: F.3 ATARI RESULTS
text: Figure 5, Figure 6 and Figure 7 illustrate the performance of SPU vs PPO throughout training in 60 Atari games.

Review summaries:
The authors formulate policy optimization as a two step iterative procedure: 1) solving a constrained optimization problem in the non-parameterized policy space, 2) using supervised regression to project this onto a parameterized policy. This approach generally applies to both continuous and discrete action spaces and can handle a variety of constraints. Their primary claims is that this approach improves sample-efficiency over TRPO on Mujoco tasks and over PPO on Atari games.