Hypothesis Driven Coordinate Ascent for Reinforcement Learning

1 INTRODUCTION . In recent years , Reinforcement Learning ( RL ) techniques have contributed to striking advancements in artificial intelligence . From mastering complicated games like Go ( Silver et al. , 2017 ) to autonomous helicopter flight ( Ng et al. , 2006 ) , RL agents continue to make impressive strides . RL is primarily concerned with discovering a policy that encodes a mapping from states to actions that optimizes rewards within a given environment . This work proposes a method for learning policies in stochastic environments where agent performance is evaluated via a black box function . Much of RL focuses on Markov Decision Processes ( MDPs ) , where an agent ’ s actions are chosen to maximize the expected value of a reward function over time ( Russell & Norvig , 2019 ) . Due to the inherent variability within real-world problem domains , the outcome of these actions depends on the action taken and noise within the environment . MDPs treat time as a discrete quantity , where actions are chosen at each time step based on a learned policy that leads the agent to a new state . Different states correspond to specific rewards , which are used to refine the policy parameters . Typically , parameter updates are taken in the direction that maximizes the expected reward and often involve calculating ( or approximating ) gradients . The function of interest is opaque for black box optimization ( BBO ) ; only point-wise evaluations are possible and the true gradient is inaccessible . As a result , gradient-based techniques must approximate the true gradient through function sampling and calculating mean and variance estimates . Calculating reliable estimates can be difficult in practice and is made especially challenging by the stochastic nature of MDPs . While various gradient-based BBO methods have found success ( Mania et al. , 2018 ; Salimans et al. , 2017 ; Bibi et al. , 2020 ) , they may converge to local optima and can be slow to converge for complex domains . As an alternative to gradient approximations , we propose Hypothesis-test Driven Coordinate Ascent ( HDCA ) , which leverages the use of derivative-free optimization ( DFO ) techniques operating on a black box function . Our first major insight is to leverage coordinate ascent to decompose the large search space of the original policy optimization problem into a sequence of sub-problems with smaller search spaces ( Wright , 2015 ) . By considering simple enough sub-problems , we are able to optimize each iterate using a basic random search with uniform sampling from a defined search range . This process involves generating batches of candidate solutions at each iteration and updating the policy to the best performing candidate . While there are numerous methods to explore the high-dimensional policy space , other more informed search policies , such as Importance Sampling ( Bibi et al. , 2020 ) , involve significant computational overhead as well as additional reliance on approximated gradients . Uniform sampling avoids these complications and allows us to take full advantage of modern advances in parallel and distributed computing . While sufficiently exploring the perturbation space may require generating several candidate solutions , each model evaluation may be done entirely in parallel of the other candidates . Therefore , with appropriate computational resources , our method can converge in fewer iterations than other current methods . Our second major insight is to account for the stochastic nature of our optimization problem . That is , the cumulative reward is a stochastic quantity whose distribution depends on the stochastic policy and environment . Thus , one must estimate the expected cumulative reward with finite samples of rollouts . This leads to variance and uncertainty about true expectations of cumulative rewards for candidate solutions . The stochastic nature ( and finite sample estimation ) of the cumulative reward criterion has been largely ignored by previous BBO approaches , where the mean over rollouts is ( incorrectly ) taken as the ground truth criterion . In practice this may result in updates that happen to perform better by chance , rather than due to policy improvement . Here , we develop a principled approach that accounts for stochasticity with one-sided , two-sample hypothesis tests to update policy parameters based on statistical significance . Lastly , we perform several experiments and ablations to empirically study HDCA . When evaluated on a sample of RL control problems , we found HDCA to consistently converge to a solution in fewer iterations than other comparable methods . Given HDCA ’ s highly scalable nature , the reduced number of iterations required to find a solution translates to quicker training time given adequate computational resources . 2 METHODS . 2.1 MARKOV DECISION PROCESS . An MDP describes an environment where an agent acts within a fully observable , stochastic domain to maximize its expected cumulative discounted reward ( Russell & Norvig , 2019 ) . The environment can be described as a set of states , S , along with a set of actions , A , that can be taken to transition between states . The transition function p ( s′|s , a ) describes the probability that taking action a in state s leads to state s′ , and the reward function R ( s , a ) describes the reward received for taking action a in state s. A solution to an MDP is represented as a policy πθ , encoded by parameters θ , that maps states to a distribution over actions in a way that maximizes accumulated rewards . This paper focuses on environments where the reward function acts as a black box . 2.2 COORDINATE ASCENT . Our first insight is to use coordinate ascent to decompose the problem of optimizing the large policy parameter space into a sequence of smaller , more feasible , sub-problems . Coordinate ascent maximizes a function f ( θ ) by successively maximizing sub-problems with respect to varying a single coordinate at a time . Similarly , block coordinate ascent iterates through blocks of coordinates and updates a subset of multiple coordinates sequentially ( Tseng , 2001 ) ( see Algorithm 1 ) . Choose θ0 ∈ Rd ; Set i← 0 ; repeat Given θi , choose a block of coordinates b in the set { 1 , ... , d } and compute new iterate θi+1 that satisfies ; θi+1b ∈ arg max θb f ( θp , θb ) ( 1 ) θi+1p = θ i p , ∀p /∈ b ( 2 ) i← i+ 1 ; until termination test satisfied ; Algorithm 1 : Block Coordinate Ascent We approximate coordinate ascent optimization to maximize policy parameters θ without gradients . To do so , we numerically optimize the sub-problem for a block of coordinates , rather than analyt- ically solving it . If the dimension of the block of coordinates being updated is small , even simple numerical optimization techniques yield good estimates of optimal updates . We propose approximate block coordinate ascent ( ABCA ) , which simply performs a random search over the coordinates of the block that is being maximized ( see Algorithm 2 ) . The primary benefit of utilizing ABCA is that optimization occurs without the use of gradient approximation , which can be challenging even for deterministic functions . When applied to reinforcement learning , the difficulties are exacerbated by the stochastic dynamics of many real-world RL problems , which lead to noisy function evaluations and rough approximations of the gradient . When combined with a potentially non-convex reward function , following an inaccurate gradient can lead to sub-optimal solutions and slow convergence . ABCA permits us to avoid gradient estimation altogether and instead make updates using simple global numerical techniques that are feasible for the smaller sub-problems . We can see from Algorithm 2 that the update rule simply takes the policy corresponding to the highest empirical average cumulative reward within each batch of perturbed θ values for a block of coordinates . This differs from typical gradient estimation , where the update rule is implemented as a weighted sum across the batch of perturbed coordinates . For example , in the popular Evolutionary Strategies approach discussed in Sec . 3 , updates are computed via an average ( Eq . 12 ) over perturbations . Due to the concentration of empirical means , this results in diminishing returns as one increases the number of perturbations per update ( iteration ) . In contrast , ABCA considers a maximum over perturbations , and thus is able to see greater returns as we increase the number of perturbations considered per update . As one may compute trajectories , t ( j ) m , and cumulative rewards , R ( t ( j ) m ) , in parallel , the better utility of perturbations implies that ABCA is able to take better advantage of the benefits of distributed computing compared to gradient estimation techniques . Input : Initial policy parameters θ0 ∈ Rd ; Hyperparameters : Search range s , number of noise samples n , Number of coordinates per block c , number of rollouts in mean calculation M ; Set i← 0 ; repeat Let coordinate block b consist of c randomly selected coordinates from the set { 1 , ... , d } ; Sample j , k ∼U ( s ) for j = 1 , ... , n , and k ∈ b ; Let θ̂ ( j ) be the jth candidate optimizer ( perturbation ) over block b with values : ∀q ∈ { 1 , . . . , d } \ b , θ̂ ( j ) q ← θiq , ∀k ∈ b , θ̂ ( j ) k ← θ i k + j , k ( 3 ) Using the policy samples ai ∼ πθ̂ ( j ) ( si ) and environment samples si+1 ∼ p ( si+1 | si , ai ) , generate M rollouts t ( j ) m , and compute respective cumulative rewards R ( t ( j ) m ) , ∀m ∈ { 1 , . . . , M } : t ( j ) m ← { ( s ( j ) m,0 , a ( j ) m,0 ) , ( s ( j ) m,1 , a ( j ) m,1 ) , . . . } R ( t ( j ) m ) = ∑ ( s , a ) ∈t ( j ) m R ( s , a ) ( 4 ) Update θ based on the empirical average cumulative rewards R̄ ( θ̂ ( j ) ) ← 1M ∑M m=1 R ( t ( j ) m ) : j← arg max j R̄ ( θ̂ ( j ) ) , θi+1 ← θ̂ ( j ) ( 5 ) i← i+ 1 ; until termination test satisfied ; Algorithm 2 : Approximate Block Coordinate Ascent ( ABCA ) 2.3 HYPOTHESIS TESTING . Our second insight is to note that , unlike deterministic optimization , the stochastic nature of our optimization problem ( through the environment dynamics and stochastic policy ) requires a careful consideration of variance over expected rewards when comparing different perturbations in ABCA iterates . That is , as we are enumerating random policies , it may be due to chance , rather than an improvement in parameters , that the average reward for a candidate optimizer perturbation is greater . Below we propose to account for the stochasticity of expected reward approximations using hypothesis testing . Given the variance in the sequence of states encountered during policy rollouts , the cumulative reward can be highly variable , and thus selecting the policy corresponding to the empirical maximum reward may not lead to improved overall performance . To generate a more accurate assessment of a given model ’ s expected reward , we make multiple function evaluations for each perturbed θ̂ ( j ) ( see Eq . 4 ) and compute corresponding mean and variance estimates from the set of policy rollouts : R̄ ( θ̂ ( j ) ) ← 1 M M∑ m=1 R ( t ( j ) m ) , σ 2 R ( θ̂ ( j ) ) ← 1 M M∑ m=1 ( R ( t ( j ) m ) − R̄ ( θ̂ ( j ) ) ) 2 . ( 6 ) We are ultimately interested in determining if there is a perturbed model that is statistically demonstrated to yield a higher expected reward than our current model , thus a one-sided , two-sample t-test is best suited for this task ( NIST/SEMATECH , 2003 ) . Since our policy perturbations at any given iteration are selected from a uniform distribution , it is likely that some of the perturbed models will lead to worse overall performance . Additionally , since function evaluations represent independent environment rollouts , the data in our two-sample t-test is unpaired . Operating under the likelihood that the variances of the two samples are not equivalent , the test statistic , T , for the two-sample t-test is calculated as : T = ( µ1 − µ2 ) / √ σ21/N1 + σ 2 2/N2 where µi , σi , and Ni correspond to the ith sample ’ s mean , standard deviation , and sample size , respectively ( Cressie & Whitford , 1986 ) . In our method , we seek to compare the rewards of candidate policy parameters θ̂ ( j ) to the rewards of the current iterate θi ; thus , test statistics are derived from the respective means and variances for reward : Tj = R̄ ( θ̂ ( j ) ) − R̄ ( θi ) √ σ2 R ( θ̂ ( j ) ) /M + σ2R ( θi ) /M ( 7 ) The null hypothesis H0 for this test is that the expected performance of the current policy is greater than or equal to the perturbed policy that is being compared . Since we are only interested in models with higher expected rewards , we reject the null hypothesis if Tj > t1−α , νj , ( 8 ) where t corresponds to the critical value of the t distribution with significance threshold α and νj degrees of freedom , calculated as νj = ( σ2 R ( θ̂ ( j ) ) /M + σ2R ( θi ) /M ) 2 ( σ2 R ( θ̂ ( j ) ) /M ) 2/ ( M − 1 ) + ( σ2R ( θi ) /M ) 2/ ( M − 1 ) . ( 9 ) Finally , the update step concludes by updating the model to the perturbed θ̂ with the highest degree of statistical evidence to outperform the previous iteration ’ s model , if such a model is found ( or passes through the previous iterate , otherwise ) . Thus , HDCA considers a test-based update , which replaces the update in ABCA ( Eq . 5 ) with : j← arg max j Tj − t1−α , νj , θi+1 ← θ̂ ( j ) if Tj − t1−α , νj > 0 , else θi . ( 10 ) That is , HDCA implements ABCA with an update that is based on statistical significance for better policy updates ( see appendix for full statement of HDCA algorithm ) . It is important to note that the two-sample t-test has a parameter for the significance level α that is likely to be problem specific . In the realm of hypothesis testing , this parameter controls whether or not the null hypothesis gets rejected . When applied to our optimization scheme , this parameter determines if a policy demonstrates enough statistical evidence to be considered a more optimal policy than the previous iteration . Therefore , this parameter can be used to control the degree of policy exploration , with a larger α indicating a higher affinity to update the policy to a perturbed value .

Summary: The paper proposes a derivative-free optimization algorithm which is applicable to stochastic black-box functions. (The target application area is reinforcement learning, but the algorithm is more broadly applicable.) The core elements of the algorithm are 1. Block coordinate ascent: Only a few parameters are optimized at each step, while the others are held fixed. The maximization is performed by random search over the parameters being optimized, so the function evaluations can be performed in parallel. 2. Hypothesis testing: Because of the high variability of Monte Carlo policy evaluation, a one-sided two-sample hypothesis test is performed to determine whether or not the candidate policy is in fact better than the current policy, and the policy is only updated if this is the case.  The proposed algorithm, hypothesis-driven coordinate ascent (HDCA), is compared to previous derivative-free optimizers proposed in the RL literature, where it is shown to require fewer iterations to converge on most tasks.