Title: BOOSTING TRUST REGION POLICY OPTIMIZATION BY NORMALIZING FLOWS POLICY

Abstract: We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraint, normalizing flows policy can generate samples far from the 'center' of the previous policy iterate, which potentially enables better exploration and helps avoid bad local optima. We show that normalizing flows policy significantly improves upon factorized Gaussian policy baseline, with both TRPO and ACKTR, especially on tasks with complex dynamics such as Humanoid.

1. INTRODUCTION:

In on-policy optimization, vanilla policy gradient algorithms suffer from occasional updates with large step size, which lead to collecting bad samples that the policy cannot recover from (Schulman et al., 2015) . Motivated to overcome such instability, Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) constraints the KL divergence between consecutive policies to achieve much more stable updates. However, with factorized Gaussian policy, such KL divergence constraint can put a very stringent restriction on the new policy, making it either hard to bypass locally optimal solutions or slow down the learning process.

Can we improve the learning process of trust region policy search by using a more expressive policy class? Intuitively, a more expressive policy class has more capacity to represent complex distributions and the KL constraint may not impose very strict restriction on the sample space. Though prior works (Haarnoja et al., 2017; 2018b; a) have proposed to use implicit generative models as policies, their focus is on off-policy learning. In this work, we show how normalizing flows can be combined with on-policy learning and boost the performance of trust region policy optimization.

The structure of our paper is as follows. In Section 2 and 3, we provide backgrounds on TRPO and related work. In Section 4, we introduce normalizing flows for control and analyze why KL constraint may not impose a constraint on the sampled action space. On illustrative examples, we show that normalizing flows policy can learn policies with correlated actions and multi-modal policies, which allows for potentially more efficient exploration. In Section 5, we show by comprehensive experiment results that normalizing flows significantly outperforms baseline policy classes when combined with trust region policy search algorithms.

2. BACKGROUND 2.1 MARKOV DECISION PROCESS:

In the standard formulation of Markov Decision Process (MDP), at time step t ≥ 0, an agent is in state s t ∈ S, takes an action a t ∈ A, receives an instant reward r t = r(s t , a t ) ∈ R and transitions to a next state s t+1 ∼ p(•|s t , a t ) ∈ S. Let π : S → P (A) be a policy, where P (A) is a set of distribution over the action space A. The discounted cumulative reward under policy π is

EQUATION

where γ ∈ [0, 1) is a discount factor. The objective of RL is to search for a policy π that achieves the maximum cumulative reward π * = arg max π J(π). For convenience, we define action value function Q π (s, a) = E π J(π)|s 0 = s, a 0 = a and value function V π (s) = E π J(π)|s 0 = s, a 0 ∼ π(•|s 0 ) . We also define the advantage function A π (s, a) = Q π (s, a) -V π (s).

2.2. POLICY OPTIMIZATION:

One way to search for π * is through direct policy search within a given policy class π θ , θ ∈ Θ where Θ is the parameter space for the policy parameter. We can update the paramter θ with policy gradient ascent, by computing

∇ θ J(π θ ) = E π θ ∞ t=0 A π θ (s t , a t )∇ θ log π θ (a t |s t )

, then updating θ new ← θ + α∇ θ J(π θ ) for some learning rate α > 0. Alternatively, the update can be formulated as trust region optimization problem

EQUATION

for some > 0. If we do a linear approximation of the objective in ( 2), E π θ π θnew (at|st)

π θ (at|st) A π θ (s t , a t ) ≈ E π θ A π θ (s t , a t ) + ∇ θ J(π θ ) T (θ new -θ),

we recover the gradient update by properly choosing .

2.3. TRUST REGION POLICY OPTIMIZATION:

Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) applies information theoretic constraints instead of Euclidean constraints on θ new and θ to better capture the geometry on the parameter space induced by the policy. In particular, consider the following trust region formulation

EQUATION

where E s • is w.r.t. the state visit distribution induced by π θ . The trust region enforced by the KL divergence entails that the update according to (3) optimizes a lower bound of J(π θ ), so as to avoid accidentally taking large steps that irreversibly degrade the policy performance during training as in vanilla policy gradient (2) (Schulman et al., 2015) . For a practical algorithm, the trust region constraint is approximated by a second order expansion

E s KL[π θ (•|s)||π θnew (•|s)] ≈ (θ new -θ) T Ĥ(θ new -θ) ≤ where Ĥ = ∂ 2 ∂θ 2 E π θ KL[π θ (•|s)||π θnew (•|s)

] is the expected Fisher information matrix. If we also linearly approximate the objective, the trust region formulation turns into a quadratic programming

EQUATION

The optimal solution to (4) is ∝ Ĥ-1 ∇ θ J(π θ ). In cases where π θ is parameterized by a neural network with a large number of parameters, Ĥ-1 is formidable to compute. Instead, (Schulman et al., 2015) proposes to approximate Ĥ-1 ∇ θ J(π θ ) by conjugate gradient (CG) descent (Wright & Nocedal, 1999) since it only requires relatively cheap Hessian-vector products. Given the approximated gradient direction ĝ ≈ Ĥ-1 ∇ θ J(π θ ) obtained from CG, the KL constraint is enforced by setting ∆θ = ĝT Ĥ ĝ ĝ. Finally a line search is carried out to determine a scaler s by enforcing the exact KL constraint E π θ KL[π θ+s∆θ ||π θ ] ≤ and finally θ new ← θ + s∆θ.

ACKTR ACKTR (Wu et al.) proposes to replace the above CG descent of TRPO by Kroneckerfactored approximation (Martens & Grosse, 2015) when computing the inverse of Fisher information matrix Ĥ-1 . This approximation is more stable than CG descent and yields performance gain over conventional TRPO.

3. RELATED WORK:

Most on-policy optimization algorithms are based on policy gradient theorem for function approximation (Sutton et al., 2000) . Vanilla policy gradient algorithms are typically more stable than off-policy learning due to their optimization based formulation, but can suffer from instability as a result of occasionally large step sizes. In policy based algorithms, updating policies with very large step sizes can be catastrophic to the learning process since the policy will collect bad samples and potentially never recover (Schulman et al., 2015) . Natural policy gradient (Kakade, 2002) applies natural gradient for the policy updates, which accounts for the information geometry induced by the policy and makes the update more stable. More recently, Trust region policy optimization (Schulman et al., 2015) derives a tractable trust region policy search algorithm based on the lower bound formulation of (Kakade & Langford, 2002) and achieves promising results on simulated locomotion tasks. The trust region is approximated by the Fisher information matrix, whose inverse is further approximated by conjugate gradient iterations (Wright & Nocedal, 1999) . To further improve the scalability and numerical performance of TRPO, ACKTR (Wu et al.) applies Kronecker-factored approximation (Martens & Grosse, 2015) to invert the Fisher information matrix. Orthogonal to prior works, we aim to improve TRPO with a more expressive policy representation, and we show significant improvements on both TRPO and ACKTR. We limit our attention to (Dinh et al., 2014) while other normalizing flows architectures might provide additional benefits (Kingma & Dhariwal, 2018) .

A number of recent prior works have proposed to boost RL algorithms with expressive policy classes.

For off-policy learning, Soft Q-learning (SQL) (Haarnoja et al., 2017) takes an implicit generative model as the policy and trains the policy by Stein variational gradient (Liu & Wang, 2016) . Similarly, (Tang & Agrawal, 2018) applies an implicit policy along with a discriminator to compute entropy regularized gradient for the implicit distribution. Latent space policy (Haarnoja et al., 2018a) applies normalizing flows as the policy and displays promising results on hierarchical tasks. Soft Actor Critic (SAC) applies a mixture of Gaussian as the policy. So far, expressive policy classes have shown improvement over baselines in the domain of off-policy learning. However, it is not clear whether such benefits come from an enriched policy class or a novel algorithmic procedure. In this work, we fix the trust region search algorithms and study the net effect of expressive policy classes.

By definition, normalizing flows stacks layers of invertible transformations to map a source noise into target samples (Dinh et al., 2016; Rezende & Mohamed, 2015) . Through invertible transformations, normalizing flows retains tractable probability densities while being very expressive. Normalizing flows is widely applied in probabilistic generative modeling, such as variational inference (Rezende & Mohamed, 2015) . Previous works have proposed to represent policies using normalizing flows in the context of off-policy learning (Haarnoja et al., 2018a) . Complement to prior works, we show that normalizing flows can significantly boost the performance of on-policy optimization.

4. NORMALIZING FLOWS POLICY FOR ON-POLICY OPTIMIZATION:



4.1. NORMALIZING FLOWS FOR CONTROL:

We construct a stochastic policy with normalizing flows. Normalizing flows (Rezende & Mohamed, 2015; Dinh et al., 2016) have been applied in variational inference and probabilistic modeling to represent complex distributions. In general, consider transforming a source noise ∼ ρ 0 (•) by a series of invertible nonlinear functions

EQUATION

Let Σ i be the inverse of the Jacobian matrix of g θ (•), then the log density of x is computed by change of variables formula,

EQUATION

For a general invertible transformation g θi (•), computing det(Σ i ) is expensive. We follow the architecture of (Dinh et al., 2014) to ensure that det(Σ i ) is computed in linear time. To combine state information, we embed state s by another neural network L θs (•) with parameter θ s and output a state vector L θs (s) with the same dimension as . We can then insert the state vector between any two layers of (5) to make the distribution conditional on state s. In our implementation, we insert the state vector after the first transformation (we detail our architecture design in the Appendix B).

EQUATION

Though the additive form of L θs (s) and g θ1 ( ) may in theory limit the capacity of the model, in experiments below we show that the resulting policy is still very expressive. For simplicity, we denote the above transformation (7) as a = f θ (s, ) with parameter θ = {θ s , θ i , 1 ≤ i ≤ K}. It is obvious that the transformation a = f θ (s, ) is still invertible between a and , which is critical for computing log π θ (a|s) according to (6). Such representations build complex policy distributions with explicit probability density π θ (•|s), and hence entail training using score function gradient estimators.

In on-policy optimizations, it is necessary to compute gradients of the entropy ∇ θ H π θ (•|s) , either for computing Hessian vector product (Schulman et al., 2015) or for entropy regularization (Schulman et al., 2015; 2017; Mnih et al., 2016) . For normalizing flows there is no analytic form for entropy, we use samples to estimate entropy by re-parameterization,

H π θ (•|s) = E a∼π θ (•|s) -log π θ (a|s) = E ∼ρ0(•) -log π θ (f θ (s, )|s) .

The gradient of the entropy can be easily computed by a pathwise gradient and easily implemented using back-propagation

∇ θ H π θ (•|s) = E ∼ρ0(•) -∇ θ log π θ (f θ (s, )|s) . 4.2 NORMALIZING FLOWS POLICY VS. GAUSSIAN POLICY UNDER KL CONSTRAINT

We analyze the properties of normalizing flows policy vs. Gaussian policy under the KL constraints of trust region policy search. As a low dimensional toy example, assume we have a factorized Gaussian in R 2 with zero mean and diagonal covariance I • σ 2 where σ 2 = 0.1 2 . Let πo be the empirical distribution formed by samples drawn from this Gaussian. We can define a KL ball centered on πo as all distributions such that a KL constraint is satisfied

B(π o , ) = {π : KL[π o ||π] ≤ }.

We study a typical normalizing flows distribution and factorized Gaussian distribution on the boundary of such a KL ball (such that KL[π o ||π] = ). We find such distributions by randomly initializing the distribution parameters then running gradient updates until KL[π o ||π]} ≈ . In Figure 1 (a) we show the log probability contour of such a factorized Gaussian vs. normalizing flows, and in (b) we show their samples (blue are samples from the distributions on the boundary of the KL ball and red are samples to generate πo ). As seen from both the contour and the sample plot, though both distributions have infinite support, normalizing flows distribution has much larger variance than the factorized Gaussian, which also leads to a much larger effective support, even though both satisfy the KL constraint to the origin distribution KL[π o ||π] = .

In Figure 1 (c), we show the samples drawn from factorized Gaussian and normalizing flows distribution with fixed levels of entropy H. To obtain distributions with fixed entropy, we randomly initialize the distribution parameters with H[π] as the entropy then obtain the desired level of entropy by minimizing (H[π] -H) 2 until convergence. As seen from the sample plot (red for Gaussian and blue for normalizing flows), under the same entropy level, normalizing flows distribution has significantly larger spread than factorized Gaussian.

As analyzed above, under similar entropy level and KL constraint, normalizing flows tends to have a probability density function that decays at a much slower rate than Gaussian from the 'center', which produces a much wider effective support on the sample space. In practice, this usually produces much better exploration and helps the agent to bypass bad locally optimal solutions. For a factorized Gaussian distribution, enforcing a KL constraint does not allow the new distribution to generate samples that are too far from the 'center' of the old distribution. On the other hand, for a normalizing flows distribution, the KL constraint does not hinder the new distribution to have a very distinct support from the reference distribution (as suggested in Figure 1 ), hence allowing for more efficient exploration.

4.3. EXPRESSIVENESS OF NORMALIZING FLOWS POLICY:

We illustrate two potential strengths of the normalizing flows policy: learning correlated actions and learning multi-modal policy. First consider a 2D bandit problem where the action a ∈ [-1, 1] 2 and r(a) = -a T Σ -1 a for some positive semidefinite matrix Σ. In the context of conventional RL objective J(π), the optimal policy is deterministic π * = [0, 0] T . However, in maximum entropy RL (Haarnoja et al., 2017; 2018b) where the objective is J(π) + cH π , the optimal policy is π * ent ∝ exp( r(a) c ), a Gaussian with Σ c as the covariance matrix (red curves show the density contours). In Figure 2 (a), we show the samples generated by various trained policies to see whether they manage to learn the correlations between actions in the maximum entropy policy π * ent . We find that factorized Gaussian cannot capture the correlations due to the factorized distribution. Though Gaussian mixtures models (GMM) with K ≥ 2 components are more expressive than factorized Gaussian, all the modes seem to collapse to the same location and suffer the same issue as factorized Gaussian. On the other hand, normalizing flows policy is much more flexible and can fairly accurately capture the correlation structure of π * ent . To illustrate multi-modality, consider again a 2D bandit problem (Figure 2 

(b)) with reward r(a) = max i∈I {(a -µ i ) T Λ - i (a -µ i )}

where Λ i , i ∈ I are diagonal matrices and µ i , i ∈ I are modes of the reward landscape. In our example we set |I| = 2 two modes and the reward contours are plotted as red curves. Notice that GMM with varying K can still easily collapse to one of the two modes while the normalizing flows policy generates samples that cover both modes.

To summarize the above two cases, since the maximum entropy objective J(π) + cH[π] = -KL[π||π * ent ], the policy search problem is equivalent to a variational inference problem where the variational distribution is π and the target distribution is π * ent ∝ exp( r(a) c ). Since normalizing flows policy is a more expressive class of distribution than GMM and factorized Gaussian, we also expect the approximation to the target distribution to be much better (Rezende & Mohamed, 2015) .

The properties of normalizing flows illustrated in Section 4.2 and Section 4.3 potentially allow for better exploration during training, and help bypass bad locally optimal solutions. For a more realistic example, we illustrate such benefits with the locomotion task of Ant robot (Brockman et al., 2016) . In Figure 2 (c) we show the robot's 2D center-of-mass trajectories generated by normalizing flows policy (red) vs. Gaussian policy (blue) after training for 2 • 10 6 time steps. We observe that the trajectories by normalizing flows policy are much more widespread, while trajectories of Gaussian policy are quite concentrated at the initial position (the origin [0.0, 0.0]). Behaviorally, Gaussian policy gets the robot to jump forward quickly, which achieves high immediate rewards but terminates the episode prematurely (due to a termination condition of the task). On the other hand, normalizing flows policy bypasses such locally optimal behavior by getting the robot to move forward in a fairly slow but steady manner, even occasionally move in the opposite direction to what reward function specifies (Details in the Appendix E).

5. EXPERIMENTS:

In experiments we aim to address the following questions: (1) Do normalizing flows policies outperform simple policies (e.g. factorized Gaussian baseline) with trust region search algorithms on benchmark tasks? (2) How sensitive are normalizing flows policies to hyper-parameters compared to Gaussian policies? To address (1), we compare normalizing flows policy against factorized Gaussian and mixture of Gaussians policies on OpenAI gym MuJoCo (Brockman et al., 2016; Todorov, 2008) , rllab (Duan et al., 2016) and Roboschool Humanoid (Schulman et al., 2017) locomotion tasks as illustrated in Figure 8 . We show that normalizing flows policy can fairly uniformly outperform policies with simple distributions, especially on tasks with highly complex dynamics. We show results for both TRPO and ACKTR. To address (2), we randomly sample hyper-parameters for both normalizing flows policy and Gaussian policy, and compare their quantiles.

Implementation Details. As we aim to study the net effect of an expressive policy on trust region policy search, we make minimal modification to the original TRPO/ACKTR algorithms during implementations. For both algorithms, the policy entropy H π θ (•|s) is analytically computed when π θ is factorized Gaussian, and is estimated by samples when π θ is GMM for K ≥ 2 or normalizing flows. The KL divergence is approximated by samples instead of analytically computed in a similar way. We leave all hyper-parameter settings in the Appendix A.

5.1. LOCOMOTION BENCHMARKS:

TRPO In Figure 3 we show the results on benchmark control problems from MuJoCo. We compare four policy classes under TRPO: factorized Gaussian (blue curves), GMM with K = 2 (yellow curves), GMM with K = 5 (green curves) and normalizing flows (red curves). For GMM, each cluster has the same probability weight and each cluster is a factorized Gaussian with independent parameters. We train each policy for a fixed number of time steps and report both mean and standard deviation of the performance averaged across 5 random seeds. We find that though GMM policies with K ≥ 2 outperform factorized Gaussian on relatively hard tasks such as Ant and HalfCheetah, they suffer from less stable learning for Humanoid tasks. However, normalizing flows consistently outperforms GMM and factorized Gaussian policies on a wide range of tasks, especially tasks with highly complex dynamics such as Humanoid.

In Table 1 , we compare with recently proposed policy classes which aim at bounding the support of policy distributions, such as Beta distribution (Chou et al., 2017) and Gaussian+tanh to bound the distribution mean. We evaluate the policies on benchmark tasks with relatively complex dynamics and show mean ± std rewards and we highlight the top two policies. We find that normalizinng flows policy consistently performs well across all complex tasks, while the performance of other policies is not as uniformly good. Also we find that bounding the final outputs by applying tanh at the final layer for normalizing flows does not perform as well, we omit the results here.

To further illustrate the strength of normalizing flows policy on Humanoid tasks, we evaluate normalizing flows vs. factorized Gaussian on Roboschool Humanoid tasks shown in Figure 4 . We observe that ever since the early stage of learning (steps ≤ 10 7 ) normalizing flows policy (red curves) already outperforms Gaussian (blue curves) by a large margin. In (a)(b), Gaussian is stuck in a locally optimal gait and cannot progress, while normalizing flows can bypass such locally optimal gaits and makes consistent improvement. Each curve is averaged over 5 random seeds and shows mean ± std performance. Each curve corresponds to a different policy representation (Red: Normalizing flows (labelled as implicit), Green: GMM K = 5, Yellow: GMM K = 2, Blue: Gaussian). Vertical axis is the cumulative rewards and horizontal axis is the number of time steps. 1 : A comparison of various policy classes on complex benchmark tasks. For each task, we show the cumulative rewards (mean ± std) after training for 10 7 steps across 5 seeds (for Humanoid (L) it is 7 • 10 6 steps). For each task, the top two results are highlighted in bold font. ACKTR We also evaluate different policy classes combined with ACKTR (Wu et al.) . In Figure 5 , we compare factorized Gaussian (red curves) against normalizing flows (blue curves) on a suite of MuJoCo and Roboschool control tasks. We train each policy for a fixed number of time steps and report both mean and standard deviation of the performance averaged across 3 random seeds.

Though ACKTR + normalizing flows does not uniformly outperform Gaussian on all tasks, we find that for tasks with relatively complex dynamics (e.g. Ant and Humanoid), normalizing flows policy achieves significant performance gains. We find that the effect of an expressive policy class is fairly orthogonal to the additional training stability introduced ACKTR over TRPO and the combined algorithm achieves even better performance. 

5.2. SENSITIVITY TO HYPER-PARAMETERS AND ABLATION STUDY:

We evaluate the policy classes' sensitivities to hyper-parameters in Figure 6 , where we compare Gaussian vs. normalizing flows. Recall that is the constant for KL constraint. For each policy, we uniformly random sample log 10 ∈ [-3.0, -2.0] and one of five random seeds, and train policies with TRPO for a fixed number of time steps. The final performance (cumulative rewards) is recorded and Figure 6 in Appendix C shows the quantile plots of final rewards across multiple tasks. We see that normalizing flows policy is generally much more robust to such hyper-parameters, importantly to . We speculate that such additional robustness partially stems from the fact that for normalizing flows policy, the KL constraint does not pose very stringent restriction on the sampled action space, which allows the system to efficiently explore even when is small.

We carry out a small ablation study that addresses how hyper-parameters inherent to normalizing flows can impact the results. Recall that normalizing flows for control (Section 3) consists of K transformations, with the first transformation embedding the state s into a vector L θs (s). Here we implement L θs (s) as a two-layer neural networks with l 1 hidden units per layer. We evaluate on the policy performance as we vary K ∈ {2, 4, 6} and l 1 ∈ {3, 5, 7}. We find that the performance of normalizing flows policies are fairly robust to such hyper-parameters (see Appendix C).

6. CONCLUSION:

We propose normalizing flows as a novel on-policy architecture to boost the performance of trust region policy search. In particular, we observe that normalizing flows policy can generate samples away from the old policy while enforcing the KL constraint, which entails potentially better exploration. We evaluate performance of normalizing flows policy combined with trust region algorithms (TRPO, ACKTR) and show that they outperform factorized Gaussian and GMM policies. We propose that such policy classes be used as baselines for future benchmarking. 

) for 1 ≤ j ≤ m -d.

For each layer, we can permute the input x before apply the simple transformation (8) so as to couple different components across layers. Such coupling entails a complex transformation when we stack multiple layers of (8). To define a policy, we need to incorporate state information. We propose to preprocess the state s ∈ R n by a neural network L θs (•) with parameter θ s , to get a state vector L θs (s) ∈ R m . Then combine the state vector into (8) as follows,

EQUATION

It is obvious that x ↔ y is still bijective regardless of the form of L θs (•) and the Jacobien matrix is easy to compute accordingly.

In locomotion benchmark experiments, we implement s, t both as 4-layers neural networks with l 1 = 3 units per hidden layer. We stack K = 4 transformations: we implement (9) to inject state information only after the first transformation, and the rest is conventional coupling as in (8). L θs (s) is implemented as a feedforward neural network with 2 hidden layers each with 64 hidden units.

Value function critic is implemented as a feedforward neural network with 2 hidden layers each with 64 hidden units with rectified-linear between hidden layers.

C SENSITIVITY TO HYPER-PARAMETERS AND ABLATION STUDY:

In Figure 7 , we show the ablation study of normalizing flows policy. We evaluate how the training curves change as we change the hyper-parameters of normalizing flows: number of transformation K ∈ {2, 4, 6} and number of hidden units l 1 ∈ {3, 5, 7} in the embedding function L θs (•). We find that the performance of normalizing flows policy is fairly robust to changes in K and l 1 . When K varies, l 1 is set to 3 by default. When l 1 varies, K is set to 4 by default. (Brockman et al., 2016; Todorov, 2008) , rllab (top line) (Duan et al., 2016) and Roboschool (bottom line) (Schulman et al., 2017) .

E REWARD STRUCTURE OF ANT LOCOMOTION TASK:

For Ant locomotion task (Brockman et al., 2016) , the state space S ⊂ R 116 and action space A ⊂ R 8 . The state space consists of all the joint positions and joint velocities of the Ant robot, while the action space consists of the torques applied to joints. The reward function at time t is r t ∝ v x where v x is the center-of-mass velocity of along the x-axis. In practice the reward function also includes terms that discourage large torques and encourages the Ant robot to stay alive (as defined by not flipping itself over).

Intuitively, the reward function encourages the robot to move along x-axis as fast as possible. This is reflected in Figure 2 (c) as the trajectories (red) generated by the normalizing flows policy is spreading along the x-axis. Occasionally, the robot also moves in the opposite direction.

:



Figures and Tables:

Figure 1: Analyzing normalizing flows vs. Gaussian: (a)(b) Consider a 2D Gaussian distribution with zero mean and factorized variance σ 2 = 0.1 2 . Samples from the Gaussian form an empirical distribution πo (red dots in (b)) and define the KL ball B(πo, ) = {π : KL[πo||π] ≤ } centered at πo. Find a typical normalizing flows distribution and a Gaussian distribution at the boundary of B(π, 0.01) such that the constraint is tight. (a) Contour of log probability of a normalizing flows distribution (right) vs. Gaussian distribution (left); (b) Samples (blue dots) generated from normalizing flows distribution (right) and Gaussian distribution (left). (c) Samples generated from a normalizing flows policy (blue) vs. Gaussian policy (red) with the same entropy H, left panel is H = -1.0 and right panel H = -4.0.

Figure 2: Expressiveness of normalizing flows policy: (a) Bandit problem with reward r(a) = -a T Σ -1 a. The maximum entropy optimal policy is a Gaussian distribution with Σ as its covariance (red contours). normalizing flows policy (blue) can capture such covariance while Gaussian cannot (green). (b) Bandit problem with multimodal reward (red contours the reward landscape). normalizing flows policy can capture the multimodality (blue) while Gaussian cannot (green). (c) Trajectories of Ant robot. The trajectories of Gaussian policy center at the initial position (the origin [0.0, 0.0]), while trajectories of normalizing flows policy are much more widespread.

Sim. Humanoid (L) (j) Humanoid (L) (k) Humanoid (l) HumanoidStandup

Figure 3: MuJoCo Benchmark: learning curves on MuJoCo locomotion tasks. Tasks with (L) are from rllab.

Figure 4: Roboschool Humanoid Benchmark : learning curves on Roboschool Humanoid locomotion tasks.Each curve corresponds to a separate random seed and we show two seeds per policy class. Each color corresponds to a different policy representation (Red: Normalizing flows (labelled as implicit), Blue: Gaussian). Vertical axis is the cumulative rewards and horizontal axis is the number of time steps.

Figure 5: MuJoCo and Roboschool Benchmarks : learning curves on locomotion tasks for ACKTR. Each curve is averaged over 3 random seeds and shows mean ± std performance. Each curve corresponds to a different policy representation (Red: Normalizing flows (labelled as implicit), Blue: Gaussian). Vertical axis is the cumulative rewards and horizontal axis is the number of time steps. Tasks with (R) are from Roboschool.

Figure 6: Sensitivity to Hyper-parameters: quantile plots of policies' performance on MuJoCo benchmark tasks under various hyper-parameter settings. For each plot, we randomly generate 30 hyper-parameters for the policy and train for a fixed number of time steps. Reacher for 10 6 steps, Hopper and HalfCheetah for 2 • 10 6 steps and SimpleHumanoid for ≈ 5 • 10 6 steps. normalizing flows policy is in general more robust to Gaussian policy.

Figure 7: Sensitivity to normalizing flows Hyper-parameters: training curves of normalizing flows policy under different hyper-parameter settings (number of hidden units l1 and number of transformation K, on Reacher and Hopper task. Each training curve is averaged over 5 random seeds and we show the mean ± std performance. Vertical axis is the cumulative rewards and horizontal axis is the number of time steps.)

Humanoid Standup 137955 ± 9238 133558 ± 9238 111497 ± 15031 142568 ± 9296

Table content: <table><tr><td>Tasks</td><td>Gaussian</td><td>Gaussian+tanh</td><td>Beta</td><td>NF</td></tr><tr><td>Ant</td><td>-76 ± 14</td><td>-89 ± 13</td><td>2362 ± 305</td><td>1982 ± 407</td></tr><tr><td>HalfCheetah</td><td>1576 ± 782</td><td>386 ± 78</td><td>1643 ± 819</td><td>2900 ± 554</td></tr><tr><td>Humanoid</td><td>3560 ± 288</td><td>6350 ± 486</td><td>3199 ± 2222</td><td>5222 ± 2436</td></tr><tr><td>Humanoid (L)</td><td>64.7 ± 7.6</td><td>38.2 ± 2.3</td><td>37.8 ± 3.4</td><td>87.2 ± 19.6</td></tr><tr><td>Sim. Humanoid (L)</td><td>6.5 ± 0.2</td><td>4.4 ± 0.1</td><td>4.2 ± 0.2</td><td>8.0 ± 1.8</td></tr><tr><td>Table</td><td/><td/><td/><td/></tr></table>

where t(•), s(•) are two arbitrary functions t, s : R d → R m-d . It can be shown that such transformation entails a simple Jacobien matrix | ∂y ∂x T | = exp( )] j ) where [s(x 1:d )] j refers to the jth component of s(x 1:d

Table content: <table><tr><td>m-d j=1 [s(x 1:d</td></tr></table>

Peer review (iclr2019):
 The authors in this work present an approach to policy optimization that relies on an alternative policy formulation based on normalizing flows. This is a relatively simple modification (this is no criticism) that essentially uses the same TRPO algorithm as previous approaches, but a different mechanism for generating the distribution over actions. The crux of the authors’ approach is detailed in equations (6) and (7), although it could have been useful to see more of the discussion of the architecture from appendix B in the actual text of the paper. The authors then go on to analyze the properties and expressiveness of the resulting properties and show that it is more capable of capturing complex interactions than a simple Gaussian. It was somewhat unclear, however, in section 4.2 what the exact form of the policies being compared are. Is this a simple example with only the parameters of the Gaussian, or was the Gaussian parameterized by a multi-layer model? Further, one thing I would also have liked to see the authors question more is, for the problems they attack, whether this expressiveness is more useful “during exploration” or for the ultimate performance of the final policy. The authors, finally, show that this approach is able to out-perform the alternative Gaussian policy. Ultimately this approach seems to be a simple modification (or replacement) of the standard policy formulation, and one that seems to lead to good performance gains.