Introduction to Augmented Random Search. | by Ziad SALLOUM | Towards Data Science

Update: The best way of learning and practicing Reinforcement Learning is by going to http://rl-lab.com
This article is based on a paper published on March, 2018 by Horia Mania, Aurelia Guy, and Benjamin Recht from the University of California, Berkeley.
The authors assert that they have built an algorithm that is at least 15 times more efficient than the fastest competing model-free methods on MuJoCo locomotion benchmarks.
They baptised the algorithm as Augmented Random Search or ARS for short.
As in every RL problem, the target is to find a policy to maximize the expected reward that the agent might get when following this policy in the given environment.
The solution proposed by the paper is to enhance an existing algorithm called Basic Random Search.
The idea of Basic Random Search is to pick a pramaterized policy πθ, shock (or perturb) the parameters θ by applying +νδ and -νδ (where ν < 1 is a constant noise and δ is a random number generate from a normal distribution).
Then apply the actions based on π(θ+νδ) and π(θ-νδ) then collect the rewards r(θ+νδ) and r(θ-νδ) resulting from those actions.
Now that we have the rewards of the perturbed θ, compute the average Δ = 1/N * Σ[r(θ+νδ) - r(θ-νδ)]δ for all the δ and we update the parameters θ using Δ and a learning rate α.θj+1 = θj + α.Δ
The ARS is an improved version of BRS, it contains a three axis of enhancements that makes it more performant.
As iterations go on, the difference between r(θ+νδ) and r(θ-νδ) can vary significantly, with the learning rate α fixed, the update θj+1 = θj + α.Δ might oscillate considerably. For example if α = 0.01 and Δ = 10 then α.Δ will be 0.1, but if Δ becomes 1000, α.Δ becomes 10. This kind of brutal variations hurts the update. Remember that our goal is to make θ converge towards values that maximize rewards.To avert this type of variations we divide α.Δ by σr (Standard Deviation of the collected rewards).
The normalization of states ensures that policies put equal weight on the different components of the states. For example suppose that a state component takes values in the range [90, 100] while another state component takes values in the range [−1, 1]. Then, the first state component will dominate the computation, while the second won’t have any effect.To gain an intuition, consider a simple average, suppose C1 = 91 and C2 = 1, the average will be (C1 + C2) / 2 = 92 / 2 = 46. Now suppose that C2 dropped sharply to the minimum, C2 = -1. The average will be (91–1)/2 = 45 .Note that it barely moved with respect to C2 dramatic drop.Now let’s use normalization. For C1 = 91, NC1 = (91–90)/(100–90) = 0.1,for C2 = 1, NC2 = (1 - (-1))/(1-(-1)) = 2/2 =1. The normalized average will be (0.1 + 1)/2 = 0.55. Now if C2 drops to -1, NC2 = (-1-(-1))/2 = 0 and the normalized average becomes (0.1 + 0)/2 = 0.05. As you can see the average was greatly impacted by the sharp variation of C2.
The normalization technique used in ARS consists of subtracting the current observed average of the state, from the state input and divide it by the standard deviation of the state:(state_input - state_observed_average)/state_std
It would be useful to remember that our goal is to maximize the collected rewards. However we are computing the average reward in each iteration, meaning that in each iteration we comput 2N episodes each one following π(θ+νδ) and π(θ-νδ), then we average the collected rewards r(θ+νδ) and r(θ-νδ) for all the 2N episodes.This present some pitfalls because if some of the rewards are small compared to the others, they will push the average down.One way remedy to this issue is to sort the rewards in the decreasing order based on the key max(r(θ+νδ), r(θ-νδ)). Then use only the top b rewards (and their respective perturbation δ) in the computation of the average reward.Note that when b = N, the algorithm will be the same as the one without this enhancement.For example suppose we have the following perturbations and their respective rewards as listed in the table below.
We sort the table according to the decreasing order according to the key max([r(δi), r(-δi)]), the result is shown in the following table:
We assume that b = 3, then we consider these figures in the average computation:[δ9, r(δ9), r(-δ9)], [δ5, r(δ5), r(-δ5)], [δ1, r(δ1), r(-δ1)]
Finally the ARS algorithm becomes:
In simpler terms it becomes like the following:
Let ν a positive constant < 1Let α be the learning rateLet N the number of perturbationsLet θ a (p x n) matrix representing the parameters of the policy πLet δi a (p x n) matrix representing the ith perturbation1. While end condition not satisfied do:2. Generate N perturbations δ from a normal distribution3. Normalize πi+ = (θ+νδi)Tx and πi- = (θ-νδi)Tx for i = 1 to N4. Generate 2N episodes and their 2N rewards using πi+ and πi- and collect the rewards ri+ and ri-5. Sort all δ by max(ri+, ri-)6. Update θ = θ + (α/(b*σr)) Σ(ri+ - ri-)δi (where i = 1 to b)7. End While
The following is the standard ARS implementation that can be easily found on the internet.
Running the code will yield different stages of learning.
After the first few iterations
After the 100'th iteration
After the 300'th iteration