Solving a Reinforcement Learning Problem Using Cross-Entropy Method | by Jordi TORRES.AI | Towards Data Science

After a parenthesis of three posts introducing basics in Deep Learning and Pytorch, in this post we put the focus back to merge Reinforcement Learning and Deep Learning.
In a previous posts we advanced that an Agent make decisions to solve complex decision-making problems under uncertainty. For this purpouse the Agent employs a policy π, as a strategy to determine the next action a based on the current state s. From this post onwards we will explore different methods to obtain a policy that allows an Agent to make decisions.
In this post we will start with Cross-Entropy method that will help to the reader to warm-up in merging Deep Learning and Reinforcement Learning. It is an evolutionary algorithm for parameterized policy optimization that John Schulman claims works “embarrassingly well” on complex RL problems.
Cross-Entropy is considered an Evolutionary Algorithm: Some individuals are sampled from a population, and only the “elite” ones govern future generations’ characteristics.
Essentially, what the Cross-Entropy method does is take a bunch of inputs, see the outputs produced, choose the inputs that have led to the best outputs and tune the Agent till we are satisfied with the outputs we see.
Remember that a policy, denoted by π(a|s), says which action the Agent should take for every state observed. In this post, we will consider that the core of our Agent will be a neural network that produces the policy.
We refer to the methods that solve this type of problem as Policy-Based methods that train the neural network that produces the policy. In future posts, we will go in more deeply about this type of method.
In practice, the policy is usually represented as a probability distribution over actions (that the Agent can take at a given state), which makes it very similar to a classification problem presented before (in the Deep Learning post), with the amount of classes being equal to the amount of actions we can carry out. In our case the output of our neural network is an action vector that represents a probability distribution as shown visually in the following Figure:
In this case we refer to it as a stochastic policy, because it returns a probability distribution over actions rather than returning a deterministic single action.
We want a policy, a probability distribution, and we initialize it at random. Then we improve our policy by playing a few episodes and then adjusting our policy (parameter of the neural network) in a way that it is more efficient. Then repeat this process in order to our policy gradually gets better. This is the base of the Cross-Entropy method.
Since we will consider a neural network as the heart of this first Agent, we need to find some way to obtain data that we can assimilate as a training dataset, which includes input data and their respective labels.
The way we’re going to do this is considering the problem as a supervised learning problem where observed states are considered the features (input data) and the actions constitute the labels.
During the agent’s lifetime, its experience is presented as episodes. Every episode is a sequence of observations of states that the Agent has got from the Environment, actions it has issued, and rewards for these actions. The core of the Cross-Entropy method is to throw away bad episodes and train on better ones, so how we find the better ones?. Imagine that our Agent has played several such episodes. For every episode, we can calculate the return (total reward) that the Agent has claimed. Remember that an Agent tries to accumulate as much total reward as possible by interacting with the Environment.
Again, for simplicity we will use the Frozen-Lake example. To understand what’s going on, we need to look deeper at the reward structure of the Frozen-Lake Environment. We get the reward of 1.0 only when we reach the goal, and this reward says nothing about how good each episode was. Was it quick and efficient? or did we make many rounds on the lake before we randomly stepped into the final cell? We don’t know; it’s just 1.0 reward and that’s it.
Let’s imagine that we already have the Agent programmed and we use it to create 4 episodes, that we can visualize with the .render() method:
Note that due to randomness in the Environment and the way that the Agent selects actions to take, the episodes have different lenght and also shown different rewards. Obviously an episode that has a reward of 1.0 is better than one that has a reward of 0.0. What about episodes that end with the same reward?
It is clear that we can consider some episodes “better” than others, e.g. the third is shortest that the second. For this, the discount factor gamma is very helpful. We can use γ = 0,9. In this case, the discounted return will be equal to the reward r (1.0 or 0.0) obtained at the end of the episode at time step t by gamma to t.
Let’s illustrate these four episodes with a diagram where each cell represents the Agent’s step, a transition, in the episode, and the corresponding discounted return:
We can see that the discounted return for shorter episodes will be higher than for longer ones.
The core of the Cross-Entropy method is simple. Basically, it generates batches of episodes, throw away bad episodes in a batch to train the neural network of the Agent on better ones. To decide which ones to throw away, we used the 70th percentile in our example, which means that we only kept the 30% that did better than 70% of the others.
As a result, the neural network learns to repeat actions that lead to the results of neural network decisions becoming better and better as we use new batches of elite episodes. The Agent must be trained until a certain mean average reward for the batch of episodes threshold is reached.
So, a pseudocode of the method can be described by the following steps:
0. Initialize the Agent neural network model
Create a batch of episodes playing in the Environment using our current Agent model.Calculate the expected return for every episode and decide on a return boundary using a percentile of all rewards.Throw away all episodes with a return below the return boundary.Train the neural network of the Agent using episode steps, that means the transitions <s,a,r>) from the remaining “elite” episodes, using the state s as the input and issued actions a as the label.Repeat from step 1 until we become satisfied with the mean average Reward for the batch of episodes.
Create a batch of episodes playing in the Environment using our current Agent model.
Calculate the expected return for every episode and decide on a return boundary using a percentile of all rewards.
Throw away all episodes with a return below the return boundary.
Train the neural network of the Agent using episode steps, that means the transitions <s,a,r>) from the remaining “elite” episodes, using the state s as the input and issued actions a as the label.
Repeat from step 1 until we become satisfied with the mean average Reward for the batch of episodes.
A variant of the method, which we will discuss in the next post is that we can keep “elite” episodes for a longer time. I mean that the default version of the algorithm samples episodes from the Environment, train on the best ones, and threw them away. However, when the number of succesful episodes is small, the “elite” episodes can be maintained longer, keeping them for several iterations to train on them.
The Environment is the source of data from which we are going to create the dataset that will be used to train the neural network of our Agent.
The Agent will start from a random policy, where the probability of all actions is uniform, and while training, the Agent will hopefully learn from data obtained from the Environment to optimize its policy toward reaching the optimal policy.
The data that comes from the Environment are episode steps that should be expressed with tuples of the form <s,a,r> (state, action and reward) which are obtained in each time step as indicated in the following scheme:
The entire code of this post can be found on GitHub and can be run as a Colab google notebook using this link.
As we have done in other posts in this series, the code presented in this post has been inspired by the code of Maxim Lapan who has written an excellent practical book on the subject.
Let’s code it. We must first import several packages:
import numpy as npimport torchimport torch.nn as nnimport gymimport gym.spaces
We will start by using the not slippery FrozenLake Environment (in the next post we will discuss more about the slippery version):
env = gym.make(‘FrozenLake-v0’, is_slippery=False)
Our state space is discrete, which means that it’s just a number from zero to fifteen inclusive (our current position in the grid). The action space is also discrete, from zero to three.
Our neural network expects a vector of numbers. To get this, we can apply the traditional onehot encoding of discrete inputs (presented in this previous post), which means that the input to our network will have 16 numbers with zero everywhere except the index that we will encode. To ease the code, we can use the ObservationWrapper class from Gym and implement our OneHotWrapper class:
class OneHotWrapper(gym.ObservationWrapper):def __init__(self, env):   super(OneHotWrapper, self).__init__(env)   self.observation_space = gym.spaces.Box(0.0, 1.0,               (env.observation_space.n, ), dtype=np.float32)def observation(self, observation):    r = np.copy(self.observation_space.low)    r[observation] = 1.0    return renv = OneHotWrapper(env)
As a summary, we have in env a not slippery Frozen-Lake Environment that we will use for obtain episodes in order to obtain data to train our Agent.
We have already advanced that our Agent is based on a neural network. Let’s see how to code this neural network and how it is used to perform the selection of actions that an Agent does.
Our model’s core is a one-hidden-layer neural network with 32 neurons using a Sigmoid activation function. There is nothing special about our neural network. We start with an arbitrary number of layers and number of neurons.
obs_size = env.observation_space.shape[0]n_actions = env.action_space.nHIDDEN_SIZE = 32net= nn.Sequential(     nn.Linear(obs_size, HIDDEN_SIZE),     nn.Sigmoid(),     nn.Linear(HIDDEN_SIZE, n_actions))
The neural network takes a single observation from the Environment as an input vector and outputs a number for every action we can perform, a probability distribution over actions. A straightforward way to proceed would be to include softmax nonlinearity after the last layer. However, remember from a previous post that we try to avoid apply softmax to increase the numerical stability of the training process. Rather than calculating softmax and then calculating Cross-Entropy loss, in this example we use the PyTorch class nn.CrossEntropyLoss, which combines both softmax and Cross-Entropy in a single, more numerically stable expression. CrossEntropyLoss requires raw, unnormalized values from the neural network (also called logits).
Other “hyperparameters” as Loss function and the Optimizer are also set for this example using the following code:
objective = nn.CrossEntropyLoss()optimizer = optim.SGD(params=net.parameters(), lr=0.001)
This abstraction makes our Agent very simple: it needs to pass an observed state that receives from the Environment to the neural network model and perform random sampling using the probability distribution to get an action to carry out:
   sm = nn.Softmax(dim=1)   def select_action(state):1:      state_t = torch.FloatTensor([state])2:      act_probs_t = sm(net(state_t))3:      act_probs = act_probs_t.data.numpy()[0]4:      action = np.random.choice(len(act_probs), p=act_probs)        return action
Let’s explain in detail this code:
Line 1: This functions requires that a first step transform the state to a tensor to ingest it to our neural network. At every iteration, we convert our current observation (Numpy array of 16 positions) to a PyTorch tensor and pass it to the model to obtain action probabilities. Remember that our neural network model needs tensors as a input data (to clarify the code we use the suffix _t to indicate that the variable is a tensor).
Line 2: A consequence of using nn.CrossEntropyLoss we need to remember to apply softmax every time we need to get probabilities from our neural network output.
Line 3: We need to convert the output tensor (remember that the model and softmax function return tensors) into a NumPy array. This array will have the same 2D structure as the input, with the batch dimension on axis 0, so we need to get the first batch element to obtain a 1D vector of action probabilities.
Line 4: With the probability distribution of actions, we can use it to obtain the actual action for the current step by sampling this distribution using NumPy’s function random.choice().
In the next figure we show a screenshot of the training loop indicating the general steps of the Cross-Entropy algorithm :
In order not to make this post too long, and specially because this method is introduced as a warm-up in the DRL coding, we leave for the next post the detailed explanation of this algorithm. And in this way we are able to go directly to test the model and become familiar with it before entering with details of the training process.
For now I simply propose to run the code of this loop and see the results. Just to mention that we considered a good result to have a reward in 80% of the cases. Remember that entire code of this post can be found on GitHub.
What remains now is to see if the Agent really makes good decisions. To check this, we can create a new Environment ( test_env), and check if our Agent is able to reach the Goal cell after being trained. The following code will do that:
test_env = OneHotWrapper(gym.make(‘FrozenLake-v0’,            is_slippery=False))state= test_env.reset()test_env.render()is_done = Falsewhile not is_done:   action = select_action(state)   new_state, reward, is_done, _ = test_env.step(action)   test_env.render()   state = new_stateprint(“reward = “, reward)
The code is a simple loop that interact with the Environment with the .step()method until the episode finishes (is_done). Thanks to the .render() method, we can see the behavior of the Agent more visually:
If we try it several times we will see that it does it well enough.
In the next post we will describe in detail the training loop (which we have skipped in this post) as well as see how we can improve the learning of the Agent taking into account a better neural network (with more neurons or different activation functions). Also we will consider the variant of the method that keeps “elite” episodes for several iterations of the training process. See you in the following post.
The entire code of this post can be found on GitHub and can be run as a Colab google notebook using this link.
by UPC Barcelona Tech and Barcelona Supercomputing Center
A relaxed introductory series that gradually and with a practical approach introduces the reader to this exciting technology that is the real enabler of the latest disruptive advances in the field of Artificial Intelligence.
I started to write this series in May, during the period of lockdown in Barcelona. Honestly, writing these posts in my spare time helped me to #StayAtHome because of the lockdown. Thank you for reading this publication in those days; it justifies the effort I made.
Disclaimers — These posts were written during this period of lockdown in Barcelona as a personal distraction and dissemination of scientific knowledge, in case it could be of help to someone, but without the purpose of being an academic reference document in the DRL area. If the reader needs a more rigorous document, the last post in the series offers an extensive list of academic resources and books that the reader can consult. The author is aware that this series of posts may contain some errors and suffers from a revision of the English text to improve it if the purpose were an academic document. But although the author would like to improve the content in quantity and quality, his professional commitments do not leave him free time to do so. However, the author agrees to refine all those errors that readers can report as soon as he can.