Reinforcement Learning: Introducing Deep Q* Networks  Part 6
Source: tai_blog

You may have heard of Project Q*  a leaked idea from OpenAI in the year 2023 that is rumoured to represent a major breakthrough in the research for Artificial General Intelligence (AGI). While nobody knows what the project entails  I stumbled across an idea that is inspired by the name Q-star  by combining my previous knowledge in Q-Learning and my current foray into search algorithms  in particular the A* Search algorithm.   While I do not claim to have understood the meaning behind Project Q* (in fact  far from it)  this article reports a new model  which I will henceforth call the Deep Q* Networks  that has demonstrated a significant upgrade in efficiency to the vanilla Deep Q-Networks that is widely used in the field of Reinforcement Learning. This article represents a continuation (Part 6) of the series of explorations in Reinforcement Learning from scratch  and one can find the introductions of Q-Learning and Deep Q-Networks in the previous articles in the series here:   Reinforcement Learning: SARSA and Q-Learning  Part 3Introducing the Temporal Difference family of iterative techniques to solve the Markov Decision Processpub.towardsai.net   Reinforcement Learning: Function Approximation and Deep Q-Networks  Part 4Reinforcement Learning with continuous state spaces and gradient descent techniquespub.towardsai.net   1. The Analogy from A* Search AlgorithmOf note  the Deep Q-Networks applies the epsilon-greedy approach in training  which specifies a certain probability where the actions are executed completely at random so that the agent can explore the action-state space sufficiently.   Comparing this approach with the Search literature  this uniform randomness approach may be analogous to the Dijkstras or Breadth-First Search algorithm  where we trace a path from the starting point radially  in perhaps random directions  until the destination point is reached.   An upgrade to Dijkstras algorithm is the A* Search algorithm  which adds a heuristic map  which acts as a cost gradient  to guide the expansion of nodes most efficiently towards the goal. We can use a simple grid below as an illustration of the differences between the A* Search and Dijkstras algorithm. Note that in the simple grid  the 1 represents walls  while 0 represents possible paths.   # Dijkstra's Algorithm grid = [[S  1  0  0  0  0]  [0  1  0  0  0  0]  [0  1  0  0  0  0]  [0  1  0  0  0  0]  [0  0  0  0  0  G]] best_policy = [['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['>'  '>'  '>'  '>'  '>'  '*']] # -1 below represents the nodes that were not explored expansion_steps = [[0  -1  -1  -1  -1  -1]  [1  -1  12  -1  -1  -1]  [2  -1  9  13  -1  -1]  [3  -1  7  10  14  -1]  [4  5  6  8  11  15]]] # A* Algorithm heuristic_map = [[9  8  7  6  5  4]  [8  7  6  5  4  3]  [7  6  5  4  3  2]  [6  5  4  3  2  1]  [5  4  3  2  1  0]] best_policy = [['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['v'  ' '  ' '  ' '  ' '  ' ']  ['>'  '>'  '>'  '>'  '>'  '*']] expansion_steps = [[0  -1  -1  -1  -1  -1]  [1  -1  -1  -1  -1  -1]  [2  -1  -1  -1  -1  -1]  [3  -1  -1  -1  -1  -1]  [4  5  6  7  8  9]]From the short illustration above  we see that the A* algorithm took the most direct and efficient path to find the goal  while Dijkstras algorithm blindly expanded into the open space. This is because at each expansion step  the A* is guided by the value of the heuristic map  and the expansion always prioritizes expanding into the cell with the lowest heuristic value.   In the A* algorithm  the search becomes much faster but this also depends on the heuristic map that we craft  which must depend on our knowledge of the possible directions of the destination point relative to the start node.   2. Moving from A* Search to Q* LearningIn Deep Reinforcement Learning or Machine Learning in general  there is also a problem with the generalization of learning  which perhaps hinders the path towards Artificial General Intelligence (AGI). For instance  while a human being can logically differentiate amongst objects with perhaps few instructions  a supervised learning model may require thousands of training examples to be accurate enough in the differentiation.   In the context of Deep Reinforcement Learning  hundreds and thousands of episodes of training may need to be expended before the agent arrives at a good set of policy solutions. This poses another issue when we need to deploy an agent  perhaps a robot  which learns on the fly (after pretraining during simulation) in the real world  where the agent may break things if it erroneously executes a disastrous action.   To address this issue  we can port over the idea of a heuristic from A* Search to Q-Learning and Deep Q-Networks. This means that instead of relying on a blindly random exploration paradigm  we can alter our algorithm such that the exploration step is guided intelligently in the right direction  which is also in line with how humans naturally learn.   For instance  if a baby human knows that if he walks too fast  he may fall. If his goal is to walk steadily  naturally  he would not be suddenly jumping forward or attempting to run as his next experimentation. This is the logic behind an exploration paradigm guided by a reasonable heuristic.   To implement the Deep Q* Networks  I propose 3 main modifications to the DQN algorithm:   Normalize the Q-values from the Policy Network  which will then represent the probabilities of the respective actions by which the agent will act  based on the epsilon-greedy framework. This means that instead of taking complete random actions  the probability of the actions taken will be informed by the trained Q-values. This probability paradigm forms the heuristic that the agent will use to explore the action-state space.Allow human supervision in the early episodes will allow the good state-action pairs to be appended into the Replay Buffer. After the supervision ends  the independent agent will navigate the environment by itself and it will immediately experience failures because of the initial random weights. However  because of the good mix of state-action pairs in the Replay Buffer  the exploration heuristic is immediately enhanced  and the agent learns quickly.Using an Autoencoder architecture in both the Policy Network and Target Network has been observed to reduce overfitting and stabilize the training process. The Autoencoder applies an Unsupervised Learning approach to Supervised Learning allowing the network to better detect patterns and capture the overall trend. In a sense  this also mirrors how humans effectively learn  not only by memorizing knowledge through brute force (direct supervised learning) but also by self-organizing patterns in knowledge while they learn to understand and capture a bigger picture.With these above ideas in mind  let us now move on to transform the Deep Q-Networks into the Deep Q* Networks  and compare their distinct performances.   3. Deep Q* Networks  Modifications and ResultsSimilar to Part 4 of our Reinforcement Learning series  we will be using the Gymnasiums Lunar Lander environment  and I will encourage you to check it out (the link attached earlier) to better understand the requirements of the environment.   In short  the goal is to safely land the Lunar Lander on the Moons surface  as quickly as possible  without crashing. The environment is considered solved when the agent accumulates rewards of above 200 on average over 100 past episodes during training. After the training  the trained agent should also be evaluated over 100 episodes to validate its true performance.   A completely untrained Lunar Lander taking random actions probably crashes on every episode and looks like this below:   While a rigorously trained agent probably lands the Lunar Lander quite successfully and looks something like this:   Moving on we will now look at the modifications that I make. Instead of the TensorFlow Keras framework that I used in Part 4  in this article  we will explore the PyTorch implementation. In accordance with the 3 main modifications:   Normalize the Q-values from the Policy Networkclass DQNAgent: def __init__(self  input_dim  output_dim  gamma=0.99  lr=1e-3  tau=0.005): self.policy_network = DQN(input_dim  output_dim).float() self.target_network = DQN(input_dim  output_dim).float() self.target_network.load_state_dict(self.policy_network.state_dict()) self.lr = lr self.optimizer = optim.AdamW(self.policy_network.parameters()  lr=self.lr) self.gamma = gamma self.tau = tau def act(self  state  epsilon): state = torch.FloatTensor(state).unsqueeze(0) q_values = self.policy_network(state).detach() if np.random.rand() < epsilon: if np.random.rand() > epsilon: # Normalize Q-values to use as probabilities for exploration q_values -= q_values.min() # Shift Q-values to make them all positive q_values += 0.05 # Set a base probability value probs = q_values / q_values.sum() action = torch.multinomial(probs  1).item() else: action = np.random.randint(env.action_space.n) else: action = q_values.argmax().item() # Choose the best action based on Q-values return action # other class methods below Note that in the above implementation  we set a small Tau value of 0.005. This is critical to stabilize the steep learning curve that the Deep Q* Network will experience and the rewards will climb very fast. We also set a base Q-value of 0.05 such that the improbable actions would not get pushed to almost zero probability.   2. Allow human supervision in the early episodes   human = DQNAgent(env.observation_space.shape[0]  env.action_space.n  gamma=gamma  lr=lr  tau=tau) human.policy_network.load_state_dict(torch.load('dqn_agent_weights.pth')) for episode in range(episodes): state = env.reset() episode_reward = 0 done = False timestep = 0 while not done: # Use current epsilon in the act() method timestep += 1 if len(replay_buffer.buffer) < 10000: action = human.act(state  epsilon=0.5) else: action = agent.act(state  epsilon=epsilon) next_state  reward  done  _ = env.step(action) replay_buffer.add_to_buffer(state  action  reward  next_state  done) state = next_state episode_reward += reward if len(replay_buffer.buffer) > 10000: agent.train(replay_buffer  batch_size) # other codes belowWe used a perfectly trained agent in place of human supervision for our purpose. Of note  the loaded agent is trained with the (1) and (3) modifications  such that even when we set epsilon=0.5  it is taking probabilistic actions based on its trained Q-values. Hence  I observed that when epsilon  0.5  the agent still performs reasonably well. For the supervision process  I epsilon=0.5 to add uncertainty and variety to the agents actions  and it is observed to improve the performance.   3. Using an Autoencoder architecture   class DQN(nn.Module): def __init__(self  input_dim  output_dim): super(DQN  self).__init__() self.fc = nn.Sequential( nn.Linear(input_dim  64)  nn.ReLU()  nn.Linear(64  24)  nn.ReLU()  nn.Linear(24  64)  nn.Linear(64  output_dim) ) def forward(self  x): return self.fc(x)In the above simple Autoencoder architecture  notice a sharp bottleneck before the network propagates to the final outputs.   When the above modifications were applied  the Deep Q* Networks model is shown to converge much more quickly  stably and consistently compared with the vanilla DQN. In addition  the validation episodes from the Deep Q* Networks significantly outperform the vanilla DQN  with all episodes scoring above 200 rewards.   I also observe that modifications (1) and (2) contribute more critically to the efficiency gain  while modification (3) acts as a secondary advantage. Without either one of (1) and (2)  the speed of convergence quickly falls.   I illustrate the comparisons between the performances of Deep Q* Networks and the vanilla DQN below:   4. ConclusionWith the experimental results above  there is enough confidence to think that the Deep Q* Networks significantly outperform the vanilla Deep Q-Networks and that the idea of the trained exploration heuristic holds effective promise in improving training efficiency and outcomes.   In addition  the Deep Q* Network framework may allow better convergence of complex and tricky Reinforcement Learning tasks and environments  and this remains to be seen in future experimentations.   The problem of generalized learning and quick convergence in Deep Learning is an important field of research and may hold the key to Artificial General Intelligence (AGI). When we progress further in this field  hopefully  one day we may more confidently allow online learning for real-time robot agents  which would be much less likely to commit critical errors that endanger their environment and humans.   Finally  if you are interested in Deep Q-Networks extended to multiple agents  remember to check out the previous article in the series on Multi-Agent cooperation with DQN  which represents another fascinating field in Reinforcement Learning:   Reinforcement Learning: Multi-Agent Cooperation with MADQN Part 5Multi-agent reinforcement learning with 3 MADQN frameworks on the ma-gyms Switch4 environmentpub.towardsai.net   Congratulations on reaching the end of this research article! In Part 7 of this Reinforcement Learning series  we will be introducing the Policy Gradient methods  so stay tuned!   Thanks for reading! If you have enjoyed the content  pop by my other articles on Medium and follow me on LinkedIn.   Support me!  If you are not subscribed to Medium  and like my content  do consider supporting me by joining Medium via my referral link.   Join Medium with my referral link  Tan Pengshi AlvinAs a Medium member  a portion of your membership fee goes to writers you read  and you get full access to every storytanpengshi.medium.com