Asynchronous Advantage Actor Critic (A3C) algorithm - GeeksforGeeks

25 Feb, 2022
The Asynchronous Advantage Actor Critic (A3C) algorithm is one of the newest algorithms to be developed under the field of Deep Reinforcement Learning Algorithms. This algorithm was developed by Google’s DeepMind which is the Artificial Intelligence division of Google. This algorithm was first mentioned in 2016 in a research paper appropriately named Asynchronous Methods for Deep Learning. 
Decoding the different parts of the algorithm’s name:-  
Asynchronous: Unlike other popular Deep Reinforcement Learning algorithms like Deep Q-Learning which uses a single agent and a single environment, This algorithm uses multiple agents with each agent having its own network parameters and a copy of the environment. This agents interact with their respective environments Asynchronously, learning with each interaction. Each agent is controlled by a global network. As each agent gains more knowledge, it contributes to the total knowledge of the global network. The presence of a global network allows each agent to have more diversified training data. This setup mimics the real-life environment in which humans live as each human gains knowledge from the experiences of some other human thus allowing the whole “global network” to be better.
Actor-Critic: Unlike some simpler techniques which are based on either Value-Iteration methods or Policy-Gradient methods, the A3C algorithm combines the best parts of both the methods ie the algorithm predicts both the value function V(s) as well as the optimal policy function . The learning agent uses the value of the Value function (Critic) to update the optimal policy function (Actor). Note that here the policy function means the probabilistic distribution of the action space. To be exact, the learning agent determines the conditional probability P(a|s ;) ie the parameterized probability that the agent chooses the action a when in state s.
Advantage: Typically in the implementation of Policy Gradient, the value of Discounted Returns() to tell the agent which of it’s actions were rewarding and which ones were penalized. By using the value of Advantage instead, the agent also learns how much better the rewards were than it’s expectation. This gives a new-found insight to the agent into the environment and thus the learning process is better. The advantage metric is given by the following expression:- 
Advantage: A = Q(s, a) – V(s) 
The following pseudo-code is referred from the research paper linked above.  
Define global shared parameter vectors [Tex]and[/Tex][Tex] [/Tex]

[Tex]\theta '[/Tex][Tex]\theta _{v}'[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]T<T_{max}[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]d\theta = 0[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]d\theta _{v} = 0[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]\theta ' = \theta[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]\theta '_{v} = \theta _{v}[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]t_{start} = t[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]s = s_{t}[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]s_{t}[/Tex][Tex]t-t_{start} < t_{max}[/Tex][Tex]a_{t}[/Tex][Tex]\pi (a_{t}|s;\theta )[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]r_{t}[/Tex][Tex]s_{t+1}[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]s_{t}[/Tex][Tex]V(s_{t}, \theta _{v}')[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]t_{start}[/Tex][Tex]r_{i} + \gamma R[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]d\theta = d\theta + \Delta _{\theta '}log(\pi (a_{i}|s{i};\theta ')(R-V(s_{i};\theta _{v}')))[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]d\theta _{v}= d\theta _{v} + \frac{\partial ((R-V(s_{i};\theta _{v}'))^{2})}{\partial \theta _{v}'}[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]\theta = \theta + d\theta[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
[Tex]\theta _{v}= \theta + d\theta _{v}[/Tex]*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty
Where, – Maximum number of iterations – change in global parameter vector – Total Reward – Policy function – Value function – discount factor Advantages:  This algorithm is faster and more robust than the standard Reinforcement Learning Algorithms.It performs better than the other Reinforcement learning techniques because of the diversification of knowledge as explained above.It can be used on discrete as well as continuous action spaces. My Personal Notes
arrow_drop_upSave
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]T<T_{max}[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]d\theta = 0[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]d\theta _{v} = 0[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]\theta ' = \theta[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]\theta '_{v} = \theta _{v}[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]t_{start} = t[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]s = s_{t}[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]s_{t}[/Tex][Tex]t-t_{start} < t_{max}[/Tex][Tex]a_{t}[/Tex][Tex]\pi (a_{t}|s;\theta )[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]r_{t}[/Tex][Tex]s_{t+1}[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]s_{t}[/Tex][Tex]V(s_{t}, \theta _{v}')[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]t_{start}[/Tex][Tex]r_{i} + \gamma R[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]d\theta = d\theta + \Delta _{\theta '}log(\pi (a_{i}|s{i};\theta ')(R-V(s_{i};\theta _{v}')))[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]d\theta _{v}= d\theta _{v} + \frac{\partial ((R-V(s_{i};\theta _{v}'))^{2})}{\partial \theta _{v}'}[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]\theta = \theta + d\theta[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

[Tex]\theta _{v}= \theta + d\theta _{v}[/Tex]
*** QuickLaTeX cannot compile formula:
 

*** Error message:
Error: Nothing to show, formula is empty

Where, 
– Maximum number of iterations 
– change in global parameter vector 
– Total Reward 
– Policy function 
– Value function 
– discount factor 
Advantages:  
This algorithm is faster and more robust than the standard Reinforcement Learning Algorithms.
It performs better than the other Reinforcement learning techniques because of the diversification of knowledge as explained above.
It can be used on discrete as well as continuous action spaces.
 
gabaa406
simranarora5sos
Artificial Intelligence
Machine Learning
Python
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Introduction to Recurrent Neural Network
Support Vector Machine Algorithm
Intuition of Adam Optimizer
CNN | Introduction to Pooling Layer
Convolutional Neural Network (CNN) in Machine Learning
Read JSON file using Python
Adding new column to existing DataFrame in Pandas
Python map() function
How to get column names in Pandas dataframe