# Reinforcement Learning in Game Industry--Review, Prospects and Challenges
## 3 Publication Analysis and Trends
### Deep Reinforcement Learning and Games
#### 3.3.5 Dueling Networks  
Dueling Network architecture is based on the idea that assessing the value of each possible action is not always essential. Knowing which energy should employ would be critical in certain situations, but the energy choice has the minimum impact on the outcome in many others. However, estimating the relevance of the states is necessary for bootstrapping methods.  
To obtain this discovery, Google DeepMind created the Dueling Network [14]. The lowest layers are convolutional, as they were in the DQNs [51] at the time. They employ two sequences (or streams) of ultimately linked layers instead of utilizing coherent layers followed by a single sequence of fully connected layers. The streams are set up to produce independent estimations of value and reward functions. Finally, the two streams are combined to provide only one \(Q\) mode output. There are Q values in the network output, one for each service. Because the Dueling Network's performance is a \(Q\) function, it may be trained using several techniques, including DDQN and SARSA [52]. It will also benefit from improvements to these algorithms, such as improved repetition memories and exploration strategies. The module that combines the two streams of completely connected layers to generate a Q rating necessitates meticulous design.  
It is worth noting that \(Q\) values indicate how profitable it is for the agent to be in state \(s\) and take action \(a\) (\(Q(s,a)\)) as shown in the function below:  
\[Q(s,a)=A(s,a)+V(s) \tag{20}\]  
where \(V(s)\) is the agent's value for being in state \(s\), and \(A(s,a)\) is the advantage of the agent choosing energy \(a\) in state \(s\) (how much better the outcomes would be if this action were chosen, over all the possible actions).  
There is a need to isolate the estimator of these two components using two new streams in conjunction with DQN:  
1. One that calculates the state value \(V(s)\);
2. One that will estimate the advantage of each action \(a\) in state \(s\), \(A(s,a)\).  
Finally, the two estimators are combined through an aggregation layer to obtain \(Q(s,a)\). Figure 7 portrays the network architecture. This distinction was chosen as all potential actions will lead to a negative outcome in some instances. Consequently, there is no need to compute the values of all potential actions in that case. As a result, we can only measure \(V(s)\), which is particularly useful in circumstances where all actions have little or no impact on the environment. The entire procedure corresponds to the following energy value factorization:\[q_{\theta}(s,a)=v_{n}(f_{\xi}(s),a)\cdot\frac{\sum(a^{\prime})a_{\psi}(f_{\xi}(s),a ^{\prime})}{N_{actions}} \tag{21}\]  
where \(\xi,n,\psi\) are the parameters of the convolutional decoder \(f_{\xi}\), the value stream \(v_{n}\), and the advantage stream \(a_{\psi}\), and \(\theta=\xi,n,\psi\) is the set.  
To summarize, DeepMind outperformed the competition in the majority of ATARI games using the new dueling Q-network architecture, with notably higher high scores (compared to old DQN) in Asterix (shown in Figure 8) and Atlantis (shown in Figure 9) games (in comparison with Double DQN). Except for Bowling, where the top scores attained decreased by 1.89%, dueling DQN was more successful in all other ATARI games. Of course, not all ATARI games benefited from this attempt to enhance training. Examples include Pinball (72.10% reduction) and Freeway (absolute reduction, i.e., 100%).  
In the games Atlantis (as illustrated in Figure 9), Tennis, and Space Invaders, dueling DQN outperformed ordinary DQN by more than 100%. It is crucial to remember that each game is unique. As a result, the earlier DQ learning algorithm outscored even the more powerful DQN architecture in certain games. Ms. Pacman is a classic example of a game in this setting.  
Figure 8: Atari game Asterix, in which dueling Q-network outperformed the double DQN by 1097%. Image taken using the OpenAI Gym Python package [47].  
Figure 7: Dueling Q-network architecture, DeepMind [14].