Policy-Based Methods. Hill Climbing algorithm | by Jordi TORRES.AI | Towards Data Science

This is a new post devoted to Policy-Based Methods, in the “Deep Reinforcement Learning Explained” series. Here we will introduce a class of algorithms that allow us to approximate the policy function, π, instead of the values functions (V, or Q). Remember that we defined policy as the entity that tells us what to do in every state. That is, instead of training a network that outputs action values, we will train a network to output (the probability of) actions, as we advanced with one example in Post 6 .
The central topic in Value Iteration and Q-learning, introduced in previous Posts, is the value of the state (denoted by V) or value of the state-action (denoted by Q). Remember that Value is defined as the discounted total reward that we can gather from a state or by issuing a particular action from the state. If the value is known, the decision on every step becomes simple and obvious: greedily in terms of value, and that guarantees a good total reward at the end of the episode. To obtain these values, we have used the Bellman equation, which expresses the value on the current step via the values on the next step (it makes a prediction from a prediction).
Reinforcement learning is ultimately about learning an optimal policy, denoted by π*, from interacting with the Environment. So far, we’ve been learning at value-based methods, where we first find an estimate of the optimal action-value function q* from which we obtain the optimal policy π*.
For small state spaces, like the Frozen-Lake example introduced in Post 1, this optimal value function q* can be represented in a table, the Q-table, with one row for each state and one column for each action. At each time step, for a given state, we only need to pull its corresponding row from the table, and the optimal action is just the action with the largest value entry.
But what about environments with much larger state spaces, like the Pong Environment introduced in the previous Posts? There’s a vast number of possible states, and that would make the table way too big to be useful in practice. So, we presented how to represent the optimal action-value function q* with a neural network. In this case, the neural network is fed with the Environment state as input, and it returns as output the value of each possible action.
But it is important to note that in both cases, whether we used a table or a neural network, we had to first estimate the optimal action-value function before we could tackle the optimal policy π*. Then, an interesting question arises: can we directly find the optimal policy without first having to deal with a value function? The answer is yes, and the class of algorithms to accomplish this are known as policy-based methods.
With value-based methods, the Agent uses its experience with the Environment to maintain an estimate of the optimal action-value function. The optimal policy is then obtained from the optimal action-value function estimate (e.g., using e-greedy).
Instead, Policy-based methods directly learn the optimal policy from the interactions with the environment, without having to maintain a separate value function estimate.
An example of a policy-based method, was already introduced at the beginning of this series when the cross-entropy method was presented in Post 6 . We introduced that a policy, denoted by π(a|s), says which action the Agent should take for every state observed. In practice, the policy is usually represented as a probability distribution over actions (that the Agent can take at a given state) with the number of classes equal to the number of actions we can carry out. We refer to it as a stochastic policy because it returns a probability distribution over actions rather than returning a single deterministic action.
Policy-based methods offer a few advantages over value-prediction methods like DQN presented in the previous three Posts. One is that, as we already discussed, we no longer have to worry about devising an action-selection strategy like ε-greedy policy; instead, we directly sample actions from the policy. And this is important; remember that we wasted a lot of time fixing up methods to improve the stability of training our DQN. For instance, we had to use experience replay and target networks, and there are several other methods in the academic literature that helps. A policy network tends to simplify some of that complexity.
In Deep Reinforcement Learning, it is common to represent the policy with a neural network (as we did for the first time in Post 6). Let ́s consider the Cart-Pole balancing problem from Post 12 of this series as an example to introduce how we can represent a policy with a neural network.
Remember that a cart is positioned on a frictionless track along the horizontal axis in this example, and a pole is anchored to the top of the cart. The objective is to keep the pole from falling over by moving the cart either left or right, and without falling off the track.
The system is controlled by applying a force of +1 (left) or -1 (right) to the cart. The pendulum starts upright, and the goal is to prevent it from falling over. A reward of +1 is provided for every time-step that the pole remains upright, including the episode’s final step. The episode ends when the pole is more than 15 degrees from the vertical, or the cart moves more than 2.4 units from the center.
The observation space for this Environment at each time point is an array of 4 numbers. At every time step, you can observe its position, velocity, angle, and angular velocity. These are the observable states of this world. You can look up what each of these numbers represents in this document. Notice the minimum (-Inf) and maximum (Inf) values for both Cart Velocity and the Pole Velocity at Tip. Since the entry in the array corresponding to each of these indices can be any real number, that means the state space is infinite!
At any state, the cart only has two possible actions: move to the left or move to the right. In other words, the state-space of the Cart-Pole has four dimensions of continuous values, and the action-space has one dimension of two discrete values.
We can construct a neural network that approximates the policy that takes a state as input. In this example, the output layer will have two nodes that return, respectively, the probability for each action. In general, if the Environment has discrete action space, as in this example, the output layer has a node for each possible action and contains the probability that the Agent should select each possible action.
The way to use the network is that the Agent feeds the current Environment state, and then the Agent samples from the probabilities of actions (left or right in this case) to select its next action.
Then, the objective is to determine appropriate values for the network weights represented by θ (Theta). θ encodes the policy that for each state that we pass into the network, it returns action probabilities where the optimal action is most likely to be selected. The actions chosen influences the Rewards obtained that are used to get the return.
Remember that the Agent’s goal is always to maximize expected return. In our case, let’s denote the expected return as J. The main idea is that it is possible to write the expected return J as a function of θ. Later we will see how we can express this relationship, J(θ), in a more “mathematical” way to find the values for the weights that maximize the expected return.
In the previous section, we have seen how a neural network can represent a policy. The weights in this neural network are initially set to random values. Then, the Agent updates the weights as it interacts with the Environment. This section will give an overview of approaches that we can take towards optimizing these weights, derivative-free methods, also known as zero-order methods.
Derivative-free methods directly search in parameter space for the vector of weights that maximizes the returns obtained by a policy; by evaluating only some positions of the parameter space, without derivatives that compute the gradients. Let’s explain the most straightforward algorithm in this category that will help us to understand later how policy gradient methods work, the Hill Climbing.
Hill Climbing is an iterative algorithm that can be used to find the weights θ for an optimal policy. It is a relatively simple algorithm that the Agent can use to gradually improve the weights θ in its policy network while interacting with the Environment.
As the name indicates, intuitively, we can visualize that the algorithm draws up a strategy to reach the highest point of a hill, where θ indicates the coordinates of where we are at a given moment and G indicates the altitude at which we are at that point:
This visual example represents a function of two parameters, but the same idea extends to more than two parameters. The algorithm begins with an initial guess for the value of θ (random set of weights). We collect a single episode with the policy that corresponds to those weights θ and then record the return G.
This return is an estimate of what the surface looks like at that value of Theta. It will not be a perfect estimate because the return we just collected is unlikely to be equal to the expected return. This is because due to randomness in the Environment (and the policy, if it is stochastic), it is highly likely that if we collect a second episode with the same values for θ, we’ll likely get a different value for the return G. But in practice, even though the (sampled) return is not a perfect estimate for the expected return estimates, it often turns out to be good enough.
At each iteration, we slightly perturb the values (add a little bit of random noise) of the current best estimate for the weights θ​, to yield a new set of candidate weights we can try. These new weights are then used to collect an episode. To see how good those new weights are, we’ll use the policy that they give us to again interact with the Environment for an episode and add up the return.
In up the new weights, give us more return than our current best estimate, we focus our attention on that new value, and then we just repeat by iteratively proposing new policies in the hope that they outperform the existing policy. In the event that they don’t well, we go back to our last best guess for the optimal policy and iterate until we end up with the optimal policy.
Now that we have an intuitive understanding of how the hill climbing algorithm should work, we can summarize it in the following pseudocode:
Initialize policy π with random weights θInitialize θbest (our best guess for the weights θ)Initialize Gbest​ (our highest return G we have gotten so far)Collect a single episode with θbest, and record the return GIf G​ > Gbest then θbest​ ← θ and Gbest​ ← GAdd a little bit of random noise to θbest​, to get a new set of weights θRepeat steps 4–6 until Environment solved.
Initialize policy π with random weights θ
Initialize θbest (our best guess for the weights θ)
Initialize Gbest​ (our highest return G we have gotten so far)
Collect a single episode with θbest, and record the return G
If G​ > Gbest then θbest​ ← θ and Gbest​ ← G
Add a little bit of random noise to θbest​, to get a new set of weights θ
Repeat steps 4–6 until Environment solved.
In our example, we assumed a surface with only one maximum in which the algorithm Hill-climbers are well-suited. Note that it’s not guaranteed to always yield the weights of the optimal policy on a surface with more than one local maxima. This is because if the algorithm begins in a poor location, it may converge to the lower maximum.
This section will explore an implementation of Hill-Climbing applied to the Cartpole Environment based in the previous pseudocode. The neural network model here is so simple, uses only the simplest matrix of shape[4x2] (state_space x action_space), that does not use tensors (no PyTorch required nor even GPU).
The code presented in this section can be found on GitHub (and can be run as a Colab google notebook using this link).
From this post, since it repeats many of the things that we have been using, we will not go into describing the code in detail; I think this is quite self-explanatory.
As always, we will start by importing the required packages and create the Environment:
import gymimport numpy as npfrom collections import dequeimport matplotlib.pyplot as pltenv = gym.make('CartPole-v0')
The policy π (and its initialization with random weights θ) can be coded as:
class Policy():    def __init__(self, s_size=4, a_size=2):        # 1. Initialize policy π with random weights        self.θ = 1e-4*np.random.rand(s_size, a_size)              def forward(self, state):        x = np.dot(state, self.θ)        return np.exp(x)/sum(np.exp(x))        def act(self, state):        probs = self.forward(state)        action = np.argmax(probs) # deterministic policy        return action
To visualize the training’s effect, we plot the weights θ before and after the training and render how the Agent apply the policy:
def watch_agent():  env = gym.make('CartPole-v0')  state = env.reset()  rewards = []  img = plt.imshow(env.render(mode='rgb_array'))  for t in range(2000):    action = policy.act(state)    img.set_data(env.render(mode='rgb_array'))     plt.axis('off')    display.display(plt.gcf())    display.clear_output(wait=True)    state, reward, done, _ = env.step(action)    rewards.append(reward)    if done:        print("Reward:", sum([r for r in rewards]))        break   env.close()policy = Policy()print ("Policy weights θ before train:\n", policy.θ)watch_agent()Policy weights θ before train: [[6.30558674e-06 2.13219853e-05] [2.32801200e-05 5.86359967e-05] [1.33454380e-05 6.69857175e-05] [9.39527443e-05 6.65193884e-05]]Reward: 9.0
The following code defines the function that trains the Agent:
def hill_climbing(n_episodes=10000, gamma=1.0, noise=1e-2):    """Implementation of hill climbing.         Params    ======        n_episodes (int): maximum number of training episodes        gamma (float): discount rate        noise(float): standard deviation of additive noise    """    scores_deque = deque(maxlen=100)    scores = []        #2. Initialize θbest    Gbest = -np.Inf    #3. Initialize Gbest    θbest = policy.θ    for i_episode in range(1, n_episodes+1):        rewards = []        state = env.reset()        while True:        #4.Collect a single episode with θ,and record the return G            action = policy.act(state)            state, reward, done, _ = env.step(action)            rewards.append(reward)            if done:                break         scores_deque.append(sum(rewards))        scores.append(sum(rewards))        discounts = [gamma**i for i in range(len(rewards)+1)]        G = sum([a*b for a,b in zip(discounts, rewards)])        if G >= Gbest: # 5. If G>Gbest then θbest←θ & Gbest←G            Gbest = G            θbest = policy.θ                #6. Add a little bit of random noise to θbes        policy.θ = θbest + noise * np.random.rand(*policy.θ.shape)        if i_episode % 10 == 0:            print('Episode {}\tAverage Score:                   {:.2f}'.format(i_episode,                     np.mean(scores_deque)))        # 7. Repeat steps 4-6 until Environment solved.            if np.mean(scores_deque)>=env.spec.reward_threshold:            print('Environment solved in {:d} episodes!\tAverage                        Score: {:.2f}'.format(i_episode,                    np.mean(scores_deque)))            policy.θ = θbest            break            return scores
The code does not require too much explanation as it is quite explicit, and it is annotated with the corresponding pseudocode steps. Maybe note some details. For instance, the algorithm seeks to maximize the cumulative discounted reward, and it looks in Python as follows:
discounts = [gamma**i for i in range(len(rewards)+1)]G = sum([a*b for a,b in zip(discounts, rewards)])
Remember that Hill Climbing is a simple gradient-free algorithm (i.e., we do not use the gradient ascent/gradient descent methods). We try to climb to the top of the curve by only changing the arguments of the target function G, the weight matrix θ determining the neural network that underlies in our mode, using a certain noise:
policy.θ = θbest + noise * np.random.rand(*policy.θ.shape)
As in some previous examples, we try to exceed a certain threshold to consider the Environment solved. For Cartpole-v0, this threshold score is 195, indicated by env.spec.reward_threshold. In the example that we used to write this post, we only needed 215 episodes to solve the Environment:
scores = hill_climbing(gamma=0.9)Episode 10	Average Score: 59.50Episode 20	Average Score: 95.45Episode 30	Average Score: 122.37Episode 40	Average Score: 134.60Episode 50	Average Score: 145.60Episode 60	Average Score: 149.38Episode 70	Average Score: 154.33Episode 80	Average Score: 160.04Episode 90	Average Score: 163.56Episode 100	Average Score: 166.87Episode 110	Average Score: 174.70Episode 120	Average Score: 168.54Episode 130	Average Score: 170.92Episode 140	Average Score: 173.79Episode 150	Average Score: 174.83Episode 160	Average Score: 178.00Episode 170	Average Score: 179.60Episode 180	Average Score: 179.58Episode 190	Average Score: 180.41Episode 200	Average Score: 180.74Episode 210	Average Score: 186.96Environment solved in 215 episodes!	Average Score: 195.65
With the following code, we can plot the scores obtained in each episode during training:
fig = plt.figure()plt.plot(np.arange(1, len(scores)+1), scores)plt.ylabel('Score')plt.xlabel('Episode #')plt.show()
Now we plot the weights θ again, after the training and also how the Agent applies this policy and seems smarter:
print ("Policy weights θ after train:\n", policy.θ)watch_agent()Policy weights θ after train: [[0.83126272 0.83426041] [0.83710884 0.86015151] [0.84691878 0.89171965] [0.80911446 0.87010399]]Reward: 200.0
Although in this example, we have coded a deterministic policy for simplicity, Policy-based methods can learn either stochastic or deterministic policies, and they can be used to solve Environments with either finite or continuous action spaces.
Hill Climbing algorithm does not need to be differentiable or even continuous, but because it is taking random steps, this may not result in the most efficient path up the hill. There are in the literature many improvements to this approach: adaptive noise scaling, Steepest ascent hill climbing, random restarts, simulated annealing, evolution strategies, or cross-entropy method (presented in Post 6 ) .
However, the usual solutions to this problem consider Policy Gradient Methods that estimate an optimal policy’s weights through gradient ascent. Policy gradient methods are a subclass of policy-based methods that we will present in the next post.
In this post, we introduced the concept of Policy-Based Methods. There are several reasons why we consider policy-based methods instead of value-based methods that seem to work so well as we saw in the previous post. Primarily because Policy-based methods directly get to the problem at hand (estimating the optimal policy) without having to store additional data, i.e., the action values that may not be useful. A further advantage over value-based methods, Policy-based methods are well-suited for continuous action spaces. As we will see in future posts, unlike value-based methods, policy-based methods can learn true stochastic policies.
See you in the next post!
by UPC Barcelona Tech and Barcelona Supercomputing Center
A relaxed introductory series that gradually and with a practical approach introduces the reader to this exciting technology that is the real enabler of the latest disruptive advances in the field of Artificial Intelligence.
I started to write this series in May, during the period of lockdown in Barcelona. Honestly, writing these posts in my spare time helped me to #StayAtHome because of the lockdown. Thank you for reading this publication in those days; it justifies the effort I made.
Disclaimers — These posts were written during this period of lockdown in Barcelona as a personal distraction and dissemination of scientific knowledge, in case it could be of help to someone, but without the purpose of being an academic reference document in the DRL area. If the reader needs a more rigorous document, the last post in the series offers an extensive list of academic resources and books that the reader can consult. The author is aware that this series of posts may contain some errors and suffers from a revision of the English text to improve it if the purpose were an academic document. But although the author would like to improve the content in quantity and quality, his professional commitments do not leave him free time to do so. However, the author agrees to refine all those errors that readers can report as soon as he can.