Value Iteration for Q-function. and THE END of this introductory series... | by Jordi TORRES.AI | Towards Data Science

In the previous post, we presented how to implement the Value Iteration method for computing the state value, V-function, by solving the Frozen-Lake Environment. In this post, we will review the Q-function and present the Value Iteration method that learns the values of the actions to create a policy.
As we will see later in this post, Q-values are much more convenient in practice, as for the Agent, it’s much simpler to make decisions about actions based on Q-values than on V-values.
Spanish version of this publication
medium.com
The Value Iteration method can be used for learning the V-values or the Q-function. That is, store in a table the state value or the action-value. Here we will present how to use Value Iteration method for computing the Q-value instead of the V-value presented in the previous post.
Based on the previous Value Iteration method implementation for V-function, in the case of action values, only minor modifications to the preceding code are required. The most obvious change is to our value table. In the previous case, we kept the value of the state, so the key in the dictionary was just a state. Now we need to store values of the Q-function, which has two parameters: state and action, so the key in the value table is now a composite.
Regarding the update, in the post about the Bellman Equation, we show that we the optimal value of the action-state can be defined as:
Next, we will solve the Frozen-Lake environment with Q-function.
The entire code of this post can be found on GitHub and can be run as a Colab google notebook using this link.
Then, we will present in the code that makes up the Value Iteration method with Q-function, the main differences versus the V-function version.
The central data which will keep our tables and the functions, that we will be using in the training loop, are the same as the previous example of V-function. The main change is into thevalues table, a dictionary that now maps a state-action pair into the calculated value of this action.
In the previous case, we kept the value of the state, so the key in the dictionary was just a state. Now we need to store values of the Q-function, which has two parameters: state and action, so the key in the value table is now a composite. This implies that another difference is in the calc_action_value function. We just don’t need it anymore, as our action values are stored in the value table.
Finally, the most important change in the code is in the Agent’s value_iteration() method. Before, it was just a wrapper around the calc_action_value() call, which did the job of Bellman approximation. Now, as this function has gone and been replaced by a value table, we need to do this approximation in the value_iteration() method.
Let’s look at the code. As it’s almost the same from the previous implementation, I will jump directly to the main differences and the reader can go for the details in the GitHub code. Let’s start with the main function value_iteration():
def value_iteration_for_Q(self):    for state in range(self.env.observation_space.n):        for action in range(self.env.action_space.n):            action_value = 0.0            target_counts = self.transits[(state, action)]            total = sum(target_counts.values())            for tgt_state, count in target_counts.items():                key = (state, action, tgt_state)                reward = self.rewards[key]                best_action = self.select_action(tgt_state)                val = reward + GAMMA * \                      self.values[(tgt_state, best_action)]                action_value += (count / total) * val           self.values[(state, action)] = action_value
As the reader can notice, the code is equivalent to the code of calc_action_value in the previous implementation. The idea is that for a given state and action, it needs to calculate the action value using the information we collected by the function play_n_random_steps that plays N random steps from the Environment, populating the reward and transits tables with random experiences.
However, in the previous implementation, we had the V-function stored in the value table, so we just took it from this table. We can’t do this anymore, so we have to call the select_action method, which will choose for us the best action with the largest Q-value, and then we take this Q-value as the value of the target state.
In fact, this method is implemented differently, as it no longer calls the calc_action_valu method but we just iterate over the actions and look up their values in our values table.
def select_action(self, state):    best_action, best_value = None, None    for action in range(self.env.action_space.n):        action_value = self.values[(state, action)]        if best_value is None or best_value < action_value:           best_value = action_value           best_action = action    return best_action
As you can notice, the code for the learning loop is just the same as in the previous post. And also the code to test the client and plot the results in TensorBoard:
Finally, the reader can, as before, test with the FrozenLake8x8 environment or test other hyperparameters.
Acknowledgments: The code presented in this post has been inspired by the code of Maxim Lapan who has written an excellent practical book on the subject.
Q-values are much more convenient in practice, as for the agent, it’s much simpler to make decisions about actions based on Q-values than on V-values. In the case of Q-values, to choose the action based on the state, the agent just needs to calculate Q-value for all available actions using the current state and choose the action with the largest Q-value.
To do the same using values of the states, V-value, the agent needs to know not only values but also probabilities for transitions. In practice, we rarely know them in advance, so the agent needs to estimate transition probabilities for every action and state pair.
In the Value Iteration method for V-function, this dependency on probability adds an extra burden for the agent. That said, it is important to know this method because they are an essential part of advanced methods.
See you in the next post!
by UPC Barcelona Tech and Barcelona Supercomputing Center
A relaxed introductory series that gradually and with a practical approach introduces the reader to this exciting technology that is the real enabler of the latest disruptive advances in the field of Artificial Intelligence.
I started to write this series in May, during the period of lockdown in Barcelona. Honestly, writing these posts in my spare time helped me to #StayAtHome because of the lockdown. Thank you for reading this publication in those days; it justifies the effort I made.
Disclaimers — These posts were written during this period of lockdown in Barcelona as a personal distraction and dissemination of scientific knowledge, in case it could be of help to someone, but without the purpose of being an academic reference document in the DRL area. If the reader needs a more rigorous document, the last post in the series offers an extensive list of academic resources and books that the reader can consult. The author is aware that this series of posts may contain some errors and suffers from a revision of the English text to improve it if the purpose were an academic document. But although the author would like to improve the content in quantity and quality, his professional commitments do not leave him free time to do so. However, the author agrees to refine all those errors that readers can report as soon as he can.