Year: 2018

Abstract:
We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraint, normalizing flows policy can generate samples far from the ’center’ of the previous policy iterate, which potentially enables better exploration and helps avoid bad local optima. We show that normalizing flows policy significantly improves upon factorized Gaussian policy baseline, with both TRPO and ACKTR, especially on tasks with complex dynamics such as Humanoid.

heading: 1 INTRODUCTION
text: In on-policy optimization, vanilla policy gradient algorithms suffer from occasional updates with large step size, which lead to collecting bad samples that the policy cannot recover from (Schulman et al., 2015). Motivated to overcome such instability, Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) constraints the KL divergence between consecutive policies to achieve much more stable updates. However, with factorized Gaussian policy, such KL divergence constraint can put a very stringent restriction on the new policy, making it either hard to bypass locally optimal solutions or slow down the learning process.
Can we improve the learning process of trust region policy search by using a more expressive policy class? Intuitively, a more expressive policy class has more capacity to represent complex distributions and the KL constraint may not impose very strict restriction on the sample space. Though prior works (Haarnoja et al., 2017; 2018b;a) have proposed to use implicit generative models as policies, their focus is on off-policy learning. In this work, we show how normalizing flows can be combined with on-policy learning and boost the performance of trust region policy optimization.
The structure of our paper is as follows. In Section 2 and 3, we provide backgrounds on TRPO and related work. In Section 4, we introduce normalizing flows for control and analyze why KL constraint may not impose a constraint on the sampled action space. On illustrative examples, we show that normalizing flows policy can learn policies with correlated actions and multi-modal policies, which allows for potentially more efficient exploration. In Section 5, we show by comprehensive experiment results that normalizing flows significantly outperforms baseline policy classes when combined with trust region policy search algorithms.

heading: 2 BACKGROUND

heading: 2.1 MARKOV DECISION PROCESS
text: In the standard formulation of Markov Decision Process (MDP), at time step t ≥ 0, an agent is in state st ∈ S, takes an action at ∈ A, receives an instant reward rt = r(st, at) ∈ R and transitions to a next state st+1 ∼ p(·|st, at) ∈ S. Let π : S 7→ P (A) be a policy, where P (A) is a set of distribution over the action space A. The discounted cumulative reward under policy π is
J(π) = Eπ [ ∞∑ t=0 γtrt ] , (1)
where γ ∈ [0, 1) is a discount factor. The objective of RL is to search for a policy π that achieves the maximum cumulative reward π∗ = arg maxπ J(π). For convenience, we define action value function
Qπ(s, a) = Eπ [ J(π)|s0 = s, a0 = a ] and value function V π(s) = Eπ [ J(π)|s0 = s, a0 ∼ π(·|s0) ] . We also define the advantage function Aπ(s, a) = Qπ(s, a)− V π(s).

heading: 2.2 POLICY OPTIMIZATION
text: One way to search for π∗ is through direct policy search within a given policy class πθ, θ ∈ Θ where Θ is the parameter space for the policy parameter. We can update the paramter θ with policy gradient ascent, by computing ∇θJ(πθ) = Eπθ [∑∞ t=0A πθ (st, at)∇θ log πθ(at|st) ] , then updating θnew ← θ + α∇θJ(πθ) for some learning rate α > 0. Alternatively, the update can be formulated as trust region optimization problem
max θnew Eπθ [πθnew(at|st) πθ(at|st) Aπθ (st, at) ] ,
||θnew − θ||2 ≤ , (2) for some > 0. If we do a linear approximation of the objective in (2), Eπθ [πθnew (at|st) πθ(at|st) A πθ (st, at) ] ≈
Eπθ [ Aπθ (st, at) ] +∇θJ(πθ)T (θnew − θ), we recover the gradient update by properly choosing .

heading: 2.3 TRUST REGION POLICY OPTIMIZATION
text: Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) applies information theoretic constraints instead of Euclidean constraints on θnew and θ to better capture the geometry on the parameter space induced by the policy. In particular, consider the following trust region formulation
max θnew Eπθ [πθnew(at|st) πθ(at|st) Aπθ (st, at) ] ,
Es [ KL[πθ(·|s)||πθnew(·|s)] ] ≤ , (3)
where Es [ · ]
is w.r.t. the state visit distribution induced by πθ. The trust region enforced by the KL divergence entails that the update according to (3) optimizes a lower bound of J(πθ), so as to avoid accidentally taking large steps that irreversibly degrade the policy performance during training as in vanilla policy gradient (2) (Schulman et al., 2015). For a practical algorithm, the trust region constraint is approximated by a second order expansion Es [ KL[πθ(·|s)||πθnew(·|s)] ] ≈ (θnew − θ)T Ĥ(θnew − θ) ≤ where Ĥ = ∂ 2 ∂θ2Eπθ [ KL[πθ(·|s)||πθnew(·|s)] ] is the expected Fisher information matrix. If we also linearly approximate the objective, the trust region formulation turns into a quadratic programming
max θnew ∇θJ(πθ)T (θnew − θ),
(θnew − θ)T Ĥ(θnew − θ) ≤ . (4)
The optimal solution to (4) is ∝ Ĥ−1∇θJ(πθ). In cases where πθ is parameterized by a neural network with a large number of parameters, Ĥ−1 is formidable to compute. Instead, (Schulman et al., 2015) proposes to approximate Ĥ−1∇θJ(πθ) by conjugate gradient (CG) descent (Wright & Nocedal, 1999) since it only requires relatively cheap Hessian-vector products. Given the approximated gradient direction ĝ ≈ Ĥ−1∇θJ(πθ) obtained from CG, the KL constraint is enforced by setting ∆θ = √
ĝT Ĥĝ ĝ. Finally a line search is carried out to determine a scaler s by enforcing the exact KL constraint Eπθ [ KL[πθ+s∆θ||πθ] ] ≤ and finally θnew ← θ + s∆θ.
ACKTR ACKTR (Wu et al.) proposes to replace the above CG descent of TRPO by Kroneckerfactored approximation (Martens & Grosse, 2015) when computing the inverse of Fisher information matrix Ĥ−1. This approximation is more stable than CG descent and yields performance gain over conventional TRPO.

heading: 3 RELATED WORK
text: Most on-policy optimization algorithms are based on policy gradient theorem for function approximation (Sutton et al., 2000). Vanilla policy gradient algorithms are typically more stable than off-policy
learning due to their optimization based formulation, but can suffer from instability as a result of occasionally large step sizes. In policy based algorithms, updating policies with very large step sizes can be catastrophic to the learning process since the policy will collect bad samples and potentially never recover (Schulman et al., 2015). Natural policy gradient (Kakade, 2002) applies natural gradient for the policy updates, which accounts for the information geometry induced by the policy and makes the update more stable. More recently, Trust region policy optimization (Schulman et al., 2015) derives a tractable trust region policy search algorithm based on the lower bound formulation of (Kakade & Langford, 2002) and achieves promising results on simulated locomotion tasks. The trust region is approximated by the Fisher information matrix, whose inverse is further approximated by conjugate gradient iterations (Wright & Nocedal, 1999). To further improve the scalability and numerical performance of TRPO, ACKTR (Wu et al.) applies Kronecker-factored approximation (Martens & Grosse, 2015) to invert the Fisher information matrix. Orthogonal to prior works, we aim to improve TRPO with a more expressive policy representation, and we show significant improvements on both TRPO and ACKTR. We limit our attention to (Dinh et al., 2014) while other normalizing flows architectures might provide additional benefits (Kingma & Dhariwal, 2018).
A number of recent prior works have proposed to boost RL algorithms with expressive policy classes. For off-policy learning, Soft Q-learning (SQL) (Haarnoja et al., 2017) takes an implicit generative model as the policy and trains the policy by Stein variational gradient (Liu & Wang, 2016). Similarly, (Tang & Agrawal, 2018) applies an implicit policy along with a discriminator to compute entropy regularized gradient for the implicit distribution. Latent space policy (Haarnoja et al., 2018a) applies normalizing flows as the policy and displays promising results on hierarchical tasks. Soft Actor Critic (SAC) applies a mixture of Gaussian as the policy. So far, expressive policy classes have shown improvement over baselines in the domain of off-policy learning. However, it is not clear whether such benefits come from an enriched policy class or a novel algorithmic procedure. In this work, we fix the trust region search algorithms and study the net effect of expressive policy classes.
By definition, normalizing flows stacks layers of invertible transformations to map a source noise into target samples (Dinh et al., 2016; Rezende & Mohamed, 2015). Through invertible transformations, normalizing flows retains tractable probability densities while being very expressive. Normalizing flows is widely applied in probabilistic generative modeling, such as variational inference (Rezende & Mohamed, 2015). Previous works have proposed to represent policies using normalizing flows in the context of off-policy learning (Haarnoja et al., 2018a). Complement to prior works, we show that normalizing flows can significantly boost the performance of on-policy optimization.

heading: 4 NORMALIZING FLOWS POLICY FOR ON-POLICY OPTIMIZATION

heading: 4.1 NORMALIZING FLOWS FOR CONTROL
text: We construct a stochastic policy with normalizing flows. Normalizing flows (Rezende & Mohamed, 2015; Dinh et al., 2016) have been applied in variational inference and probabilistic modeling to represent complex distributions. In general, consider transforming a source noise ∼ ρ0(·) by a series of invertible nonlinear functions gθi(·), 1 ≤ i ≤ K each with parameter θi, to output a target sample x,
x = gθK ◦ gθK−1 ◦ ... ◦ gθ2 ◦ gθ1( ). (5) Let Σi be the inverse of the Jacobian matrix of gθ(·), then the log density of x is computed by change of variables formula,
log p(x) = log p( ) + K∑ i=1 log det(Σi). (6)
For a general invertible transformation gθi(·), computing det(Σi) is expensive. We follow the architecture of (Dinh et al., 2014) to ensure that det(Σi) is computed in linear time. To combine state information, we embed state s by another neural network Lθs(·) with parameter θs and output a state vector Lθs(s) with the same dimension as . We can then insert the state vector between any two layers of (5) to make the distribution conditional on state s. In our implementation, we insert the state vector after the first transformation (we detail our architecture design in the Appendix B).
a = gθK ◦ gθK−1 ◦ ... ◦ gθ2 ◦ (Lθs(s) + gθ1( )). (7)
Though the additive form of Lθs(s) and gθ1( ) may in theory limit the capacity of the model, in experiments below we show that the resulting policy is still very expressive. For simplicity, we denote the above transformation (7) as a = fθ(s, ) with parameter θ = {θs, θi, 1 ≤ i ≤ K}. It is obvious that the transformation a = fθ(s, ) is still invertible between a and , which is critical for computing log πθ(a|s) according to (6). Such representations build complex policy distributions with explicit probability density πθ(·|s), and hence entail training using score function gradient estimators.
In on-policy optimizations, it is necessary to compute gradients of the entropy ∇θH [ πθ(·|s) ] , either for computing Hessian vector product (Schulman et al., 2015) or for entropy regularization (Schulman et al., 2015; 2017; Mnih et al., 2016). For normalizing flows there is no analytic form for entropy, we use samples to estimate entropy by re-parameterization, H [ πθ(·|s) ] =
Ea∼πθ(·|s) [ − log πθ(a|s) ] = E ∼ρ0(·) [ − log πθ(fθ(s, )|s) ] . The gradient of the entropy can be easily computed by a pathwise gradient and easily implemented using back-propagation ∇θH [ πθ(·|s) ] = E ∼ρ0(·) [ −∇θ log πθ(fθ(s, )|s) ] .

heading: 4.2 NORMALIZING FLOWS POLICY VS. GAUSSIAN POLICY UNDER KL CONSTRAINT
text: We analyze the properties of normalizing flows policy vs. Gaussian policy under the KL constraints of trust region policy search. As a low dimensional toy example, assume we have a factorized Gaussian in R2 with zero mean and diagonal covariance I · σ2 where σ2 = 0.12. Let π̂o be the empirical distribution formed by samples drawn from this Gaussian. We can define a KL ball centered on π̂o as all distributions such that a KL constraint is satisfied B(π̂o, ) = {π : KL[π̂o||π] ≤ }. We study a typical normalizing flows distribution and factorized Gaussian distribution on the boundary of such a KL ball (such that KL[π̂o||π] = ). We find such distributions by randomly initializing the distribution parameters then running gradient updates until KL[π̂o||π]} ≈ . In Figure 1 (a) we show the log probability contour of such a factorized Gaussian vs. normalizing flows, and in (b) we show their samples (blue are samples from the distributions on the boundary of the KL ball and red are samples to generate π̂o). As seen from both the contour and the sample plot, though both distributions have infinite support, normalizing flows distribution has much larger variance than the factorized Gaussian, which also leads to a much larger effective support, even though both satisfy the KL constraint to the origin distribution KL[π̂o||π] = . In Figure 1 (c), we show the samples drawn from factorized Gaussian and normalizing flows distribution with fixed levels of entropy H . To obtain distributions with fixed entropy, we randomly initialize the distribution parameters with H[π] as the entropy then obtain the desired level of entropy by minimizing (H[π] − H)2 until convergence. As seen from the sample plot (red for Gaussian and blue for normalizing flows), under the same entropy level, normalizing flows distribution has significantly larger spread than factorized Gaussian.
As analyzed above, under similar entropy level and KL constraint, normalizing flows tends to have a probability density function that decays at a much slower rate than Gaussian from the ’center’, which produces a much wider effective support on the sample space. In practice, this usually produces much better exploration and helps the agent to bypass bad locally optimal solutions. For a factorized Gaussian distribution, enforcing a KL constraint does not allow the new distribution to generate samples that are too far from the ’center’ of the old distribution. On the other hand, for a normalizing flows distribution, the KL constraint does not hinder the new distribution to have a very distinct support from the reference distribution (as suggested in Figure 1), hence allowing for more efficient exploration.

heading: 4.3 EXPRESSIVENESS OF NORMALIZING FLOWS POLICY
text: We illustrate two potential strengths of the normalizing flows policy: learning correlated actions and learning multi-modal policy. First consider a 2D bandit problem where the action a ∈ [−1, 1]2 and r(a) = −aTΣ−1a for some positive semidefinite matrix Σ. In the context of conventional RL objective J(π), the optimal policy is deterministic π∗ = [0, 0]T . However, in maximum entropy RL (Haarnoja et al., 2017; 2018b) where the objective is J(π) + cH [ π ] , the optimal policy is
π∗ent ∝ exp( r(a) c ), a Gaussian with Σ c as the covariance matrix (red curves show the density contours). In Figure 2 (a), we show the samples generated by various trained policies to see whether they manage to learn the correlations between actions in the maximum entropy policy π∗ent. We find that factorized
Gaussian cannot capture the correlations due to the factorized distribution. Though Gaussian mixtures models (GMM) with K ≥ 2 components are more expressive than factorized Gaussian, all the modes seem to collapse to the same location and suffer the same issue as factorized Gaussian. On the other hand, normalizing flows policy is much more flexible and can fairly accurately capture the correlation structure of π∗ent.
To illustrate multi-modality, consider again a 2D bandit problem (Figure 2 (b)) with reward r(a) = maxi∈I{(a− µi)TΛ−i (a− µi)} where Λi, i ∈ I are diagonal matrices and µi, i ∈ I are modes of the reward landscape. In our example we set |I| = 2 two modes and the reward contours are plotted as red curves. Notice that GMM with varying K can still easily collapse to one of the two modes while the normalizing flows policy generates samples that cover both modes.
To summarize the above two cases, since the maximum entropy objective J(π) + cH[π] = −KL[π||π∗ent], the policy search problem is equivalent to a variational inference problem where the variational distribution is π and the target distribution is π∗ent ∝ exp( r(a) c ). Since normalizing flows policy is a more expressive class of distribution than GMM and factorized Gaussian, we also expect the approximation to the target distribution to be much better (Rezende & Mohamed, 2015).
The properties of normalizing flows illustrated in Section 4.2 and Section 4.3 potentially allow for better exploration during training, and help bypass bad locally optimal solutions. For a more realistic example, we illustrate such benefits with the locomotion task of Ant robot (Brockman et al., 2016). In Figure 2 (c) we show the robot’s 2D center-of-mass trajectories generated by normalizing flows policy (red) vs. Gaussian policy (blue) after training for 2 · 106 time steps. We observe that the trajectories by normalizing flows policy are much more widespread, while trajectories of Gaussian policy are quite concentrated at the initial position (the origin [0.0, 0.0]). Behaviorally, Gaussian policy gets the robot to jump forward quickly, which achieves high immediate rewards but terminates the episode prematurely (due to a termination condition of the task). On the other hand, normalizing flows policy bypasses such locally optimal behavior by getting the robot to move forward in a fairly slow but steady manner, even occasionally move in the opposite direction to what reward function specifies (Details in the Appendix E).

heading: 5 EXPERIMENTS
text: In experiments we aim to address the following questions: (1) Do normalizing flows policies outperform simple policies (e.g. factorized Gaussian baseline) with trust region search algorithms on benchmark tasks? (2) How sensitive are normalizing flows policies to hyper-parameters compared to Gaussian policies? To address (1), we compare normalizing flows policy against factorized Gaussian and mixture of Gaussians policies on OpenAI gym MuJoCo (Brockman et al., 2016; Todorov, 2008), rllab (Duan et al., 2016) and Roboschool Humanoid (Schulman et al., 2017) locomotion tasks as
illustrated in Figure 8. We show that normalizing flows policy can fairly uniformly outperform policies with simple distributions, especially on tasks with highly complex dynamics. We show results for both TRPO and ACKTR. To address (2), we randomly sample hyper-parameters for both normalizing flows policy and Gaussian policy, and compare their quantiles.
Implementation Details. As we aim to study the net effect of an expressive policy on trust region policy search, we make minimal modification to the original TRPO/ACKTR algorithms during implementations. For both algorithms, the policy entropy H [ πθ(·|s) ] is analytically computed when πθ is factorized Gaussian, and is estimated by samples when πθ is GMM for K ≥ 2 or normalizing flows. The KL divergence is approximated by samples instead of analytically computed in a similar way. We leave all hyper-parameter settings in the Appendix A.

heading: 5.1 LOCOMOTION BENCHMARKS
text: TRPO In Figure 3 we show the results on benchmark control problems from MuJoCo. We compare four policy classes under TRPO: factorized Gaussian (blue curves), GMM with K = 2 (yellow curves), GMM with K = 5 (green curves) and normalizing flows (red curves). For GMM, each cluster has the same probability weight and each cluster is a factorized Gaussian with independent parameters. We train each policy for a fixed number of time steps and report both mean and standard deviation of the performance averaged across 5 random seeds. We find that though GMM policies with K ≥ 2 outperform factorized Gaussian on relatively hard tasks such as Ant and HalfCheetah, they suffer from less stable learning for Humanoid tasks. However, normalizing flows consistently outperforms GMM and factorized Gaussian policies on a wide range of tasks, especially tasks with highly complex dynamics such as Humanoid.
In Table 1, we compare with recently proposed policy classes which aim at bounding the support of policy distributions, such as Beta distribution (Chou et al., 2017) and Gaussian+tanh to bound the distribution mean. We evaluate the policies on benchmark tasks with relatively complex dynamics and show mean± std rewards and we highlight the top two policies. We find that normalizinng flows policy consistently performs well across all complex tasks, while the performance of other policies is not as uniformly good. Also we find that bounding the final outputs by applying tanh at the final layer for normalizing flows does not perform as well, we omit the results here.
To further illustrate the strength of normalizing flows policy on Humanoid tasks, we evaluate normalizing flows vs. factorized Gaussian on Roboschool Humanoid tasks shown in Figure 4. We observe that ever since the early stage of learning (steps ≤ 107) normalizing flows policy (red curves) already outperforms Gaussian (blue curves) by a large margin. In (a)(b), Gaussian is stuck in a locally optimal gait and cannot progress, while normalizing flows can bypass such locally optimal gaits and makes consistent improvement.
ACKTR We also evaluate different policy classes combined with ACKTR (Wu et al.). In Figure 5, we compare factorized Gaussian (red curves) against normalizing flows (blue curves) on a suite of MuJoCo and Roboschool control tasks. We train each policy for a fixed number of time steps and report both mean and standard deviation of the performance averaged across 3 random seeds. Though ACKTR + normalizing flows does not uniformly outperform Gaussian on all tasks, we find that for tasks with relatively complex dynamics (e.g. Ant and Humanoid), normalizing flows policy achieves significant performance gains. We find that the effect of an expressive policy class is fairly orthogonal to the additional training stability introduced by ACKTR over TRPO and the combined algorithm achieves even better performance.

heading: 5.2 SENSITIVITY TO HYPER-PARAMETERS AND ABLATION STUDY
text: We evaluate the policy classes’ sensitivities to hyper-parameters in Figure 6, where we compare Gaussian vs. normalizing flows. Recall that is the constant for KL constraint. For each policy, we uniformly random sample log10 ∈ [−3.0,−2.0] and one of five random seeds, and train policies with TRPO for a fixed number of time steps. The final performance (cumulative rewards) is recorded and Figure 6 in Appendix C shows the quantile plots of final rewards across multiple tasks. We see that normalizing flows policy is generally much more robust to such hyper-parameters, importantly to . We speculate that such additional robustness partially stems from the fact that for normalizing flows policy, the KL constraint does not pose very stringent restriction on the sampled action space, which allows the system to efficiently explore even when is small.
We carry out a small ablation study that addresses how hyper-parameters inherent to normalizing flows can impact the results. Recall that normalizing flows for control (Section 3) consists of K transformations, with the first transformation embedding the state s into a vector Lθs(s). Here we implement Lθs(s) as a two-layer neural networks with l1 hidden units per layer. We evaluate on the policy performance as we vary K ∈ {2, 4, 6} and l1 ∈ {3, 5, 7}. We find that the performance of normalizing flows policies are fairly robust to such hyper-parameters (see Appendix C).

heading: 6 CONCLUSION
text: We propose normalizing flows as a novel on-policy architecture to boost the performance of trust region policy search. In particular, we observe that normalizing flows policy can generate samples away from the old policy while enforcing the KL constraint, which entails potentially better exploration. We evaluate performance of normalizing flows policy combined with trust region algorithms (TRPO, ACKTR) and show that they outperform factorized Gaussian and GMM policies. We propose that such policy classes be used as baselines for future benchmarking.

heading: A HYPER-PARAMETERS
text: Implementations. All implementations of algorithms (TRPO and ACKTR) are based on OpenAI baselines (Dhariwal et al., 2017). We implement our own GMM policy and normalizing flows policy. Environments are based on OpenAI gym (Brockman et al., 2016), rllab (Duan et al., 2016) and Roboschool (Schulman et al., 2017).
Common Interface to the Algorithms. We remark that various policy classes have exactly the same interface to TRPO and ACKTR. In particular, TRPO and ACKTR only requires the computation of log πθ(a|s) (and its derivative). Different policy classes only differ in how they parameterize πθ(a|s) and can be easily plugged into the algorithmic procedure originally designed for Gaussian (Dhariwal et al., 2017).
factorized Gaussian Policy. A factorized Gaussian policy has the form πθ(·|s) = N(µθ(s),Σ), where Σ is a diagonal matrix with Σii = σ2i . We use the default hyper-parameters in baselines for factorized Gaussian policy. The mean µθ(s) parameterized by a two-layer neural network with 64 hidden units per layer and tanh activation function. The standard deviation σ2i is each a single variable shared across all states.
factorized Gaussian+tanh Policy. The architecture is the same as above but the final layer is added a tanh transformation to ensure that the mean µθ(s) ∈ [−1, 1]. GMM Policy. A GMM policy has the form πθ(·|s) = ∑K i=1 piN(µ (i) θ (s),Σi), where the cluster weight pi = 1K is fixed and µ (i) θ (s),Σi are Gaussian parameters for the ith cluster. Each Gaussian has the same parameterization as the factorized Gaussian above.
Beta Policy. A Beta distribution policy has the form π(αθ(s), βθ(s)) where αθ(s) and βθ(s) are shape/rate parameters parameterized by two-layer neural network fθ(s) with a softplus at the end, i.e. αθ(s) = log(exp(fθ(s)) + 1) + 1, following (Chou et al., 2017). Actions sampled from this distribution have a strictly finite support. We notice that this parameterization introduces potential instability during optimization: for example, when we want to converge on policies that sample actions at the boundary, we require αθ(s)→∞ or βθ(s)→∞, which might be very unstable. We also observe such instability in practice.
normalizing flows Policy. A normalizing flows policy has a generative form: the sample a ∼ πθ(·|s) can be generated via a = fθ(s, ) with ∼ ρ0(·). The detailed architecture of fθ(s, ) is in the appendix below.
Others. Value functions are all parameterized as two-layer neural networks with 64 hidden units per layer and tanh activation function. Trust region sizes are enforced via a constraint parameter , where ∈ {0.01, 0.001} for TRPO and ∈ {0.02, 0.002} for ACKTR. All other hyper-parameters are default parameters from the baselines implementations.

heading: B NORMALIZING FLOWS POLICY ARCHITECTURE
text: We design the neural network architecture following the idea of (Dinh et al., 2014; 2016). Recall that normalizing flows (Rezende & Mohamed, 2015) consists of layers of transformation as follows ,
x = gθK ◦ gθK−1 ◦ ... ◦ gθ2 ◦ gθ1( ), where each gθi(·) is an invertible transformation. We focus on how to design each atomic transformation gθi(·). We overload the notations and let x, y be the input/output of a generic layer gθ(·), y = gθ(x). We design a generic transformation gθ(·) as follows. Let xI be the components of x corresponding to subset indices I ⊂ {1, 2...m}. Then we propose as in (Dinh et al., 2016),
y1:d = x1:d
yd+1:m = xd+1:m exp(s(x1:d)) + t(x1:d), (8)
where t(·), s(·) are two arbitrary functions t, s : Rd 7→ Rm−d. It can be shown that such transformation entails a simple Jacobien matrix | ∂y
∂xT | = exp( ∑m−d j=1 [s(x1:d)]j) where [s(x1:d)]j refers to
the jth component of s(x1:d) for 1 ≤ j ≤ m− d. For each layer, we can permute the input x before apply the simple transformation (8) so as to couple different components across layers. Such coupling entails a complex transformation when we stack multiple layers of (8). To define a policy, we need to incorporate state information. We propose to preprocess the state s ∈ Rn by a neural network Lθs(·) with parameter θs, to get a state vector Lθs(s) ∈ Rm. Then combine the state vector into (8) as follows,
z1:d = x1:d
zd+1:m = xd+1:m exp(s(x1:d)) + t(x1:d) y = z + Lθs(s). (9)
It is obvious that x↔ y is still bijective regardless of the form of Lθs(·) and the Jacobien matrix is easy to compute accordingly.
In locomotion benchmark experiments, we implement s, t both as 4-layers neural networks with l1 = 3 units per hidden layer. We stack K = 4 transformations: we implement (9) to inject state information only after the first transformation, and the rest is conventional coupling as in (8). Lθs(s) is implemented as a feedforward neural network with 2 hidden layers each with 64 hidden units. Value function critic is implemented as a feedforward neural network with 2 hidden layers each with 64 hidden units with rectified-linear between hidden layers.

heading: C SENSITIVITY TO HYPER-PARAMETERS AND ABLATION STUDY
text: In Figure 7, we show the ablation study of normalizing flows policy. We evaluate how the training curves change as we change the hyper-parameters of normalizing flows: number of transformation K ∈ {2, 4, 6} and number of hidden units l1 ∈ {3, 5, 7} in the embedding function Lθs(·). We find that the performance of normalizing flows policy is fairly robust to changes in K and l1. When K varies, l1 is set to 3 by default. When l1 varies, K is set to 4 by default.
D ILLUSTRATION OF LOCOMOTION TASKS

heading: E REWARD STRUCTURE OF ANT LOCOMOTION TASK
text: For Ant locomotion task (Brockman et al., 2016), the state space S ⊂ R116 and action spaceA ⊂ R8. The state space consists of all the joint positions and joint velocities of the Ant robot, while the action space consists of the torques applied to joints. The reward function at time t is rt ∝ vx where vx is the center-of-mass velocity of along the x-axis. In practice the reward function also includes terms that discourage large torques and encourages the Ant robot to stay alive (as defined by not flipping itself over).
Intuitively, the reward function encourages the robot to move along x-axis as fast as possible. This is reflected in Figure 2 (c) as the trajectories (red) generated by the normalizing flows policy is spreading along the x-axis. Occasionally, the robot also moves in the opposite direction.

Review summaries:
The authors in this work present an approach to policy optimization that relies on an alternative policy formulation based on normalizing flows. This is a relatively simple modification (this is no criticism) that essentially uses the same TRPO algorithm as previous approaches, but a different mechanism for generating the distribution over actions. The crux of the authors’ approach is detailed in equations (6) and (7), although it could have been useful to see more of the discussion of the architecture from appendix B in the actual text of the paper.