# Reducing Q-Value Estimation Bias via Mutual Estimation and Softmax Operation in MADRL
## 2 Research Background
### Deep Q-Learning Algorithms  
[PERSON] and others [6] were the first to integrate deep learning and traditional reinforcement learning algorithm, thus proposing the DQN algorithm. This pioneering piece of work marked the first end-to-end deep reinforcement learning algorithm using only Atari game pixel data and game scores for reinforcement learning state information. Finally, the trained network could output action-value functions or Q-values for each action. From this, the decision with the highest corresponding Q-value is selected. The entire structure of the DQN algorithm is shown in Figure 1.  
The DQN algorithm introduces the experience replay mechanism, which stores the sample tuple \((s,a,r,s^{\prime})\) of the agent at each time step into an experience replay pool. During reinforcement learning training, a batch of samples is sampled from the experience replay pool and the neural network is updated using gradient descent. The use of experience replay mechanism greatly improves the data utilization of reinforcement learning algorithms and to some extent reduces the correlation between training data and sequences. Another important contribution of the DQN algorithm is the concept of the target network, which achieves stable value estimation. The target value \(y\) for updating the Q network by computing the loss function is represented as:  
Figure 1: DQN Algorithm Flowchart. DQN uses deep neural networks to fit the Q function, selects the action that maximizes the Q value through a \(\varepsilon\)-greedy policy, interacts with the environment, and stores the obtained samples in replay memory. DQN continuously extracts samples from replay memory, performs gradient descent based on the loss function, and updates the parameters of the Q-network. The parameters of the Q-network are periodically copied to the target Q-network, which participates in the calculation of the loss function and makes Q-learning more stable. \[y=r+\gamma\max_{\hat{a}}Q_{\theta^{\prime}}\big{(}s^{\prime},\hat{a}\big{)} \tag{1}\]  
where \(Q_{\theta^{\prime}}\) represents the target network. \(\gamma\) is a discount factor (\(0<\gamma<1\)), used to give a preference for rewards reaped sooner in time. \(\hat{a}\) is the action that allows the Q function to take the maximum value. The structure of the target network is the same as the Q network, but its parameters are only copied from the Q network every certain number of iterations. The Q network updates its parameters in each round by minimizing the loss function defined in Equation (2):  
\[L(\theta)=E_{(s,\sigma,r,s^{\prime})}\Big{[}(y-Q_{\theta}(s,a))^{2}\Big{]} \tag{2}\]  
\(E\) represents mathematical expectation, which is calculated using Monte Carlo estimation method in practice, taking the average value of batch samples to estimate. The neural network is updated to make the Q network approximate the target value \(y\), which essentially applies the Bellman equation. The use of target network helps improve the stability of the algorithm, and experiments have shown that NPC controlled by DQN algorithm can reach the level of professional players in many Atari games. The DDPG [7] algorithm can be seen as a combination of Deterministic Policy Gradient (DPG) algorithm and deep neural networks, and it can also be seen as an extension of DQN algorithm in a continuous action space. It can solve the problem of the DQN algorithm's inability to directly apply to continuous action space, but there is still the problem of overestimation of Q values. The DDPG algorithm simultaneously establishes a Q network (critic) and a policy network (actor) [8]. The Q network is the same as the DQN algorithm, updated using a temporal difference method. The policy network utilizes the estimation of the Q network and is updated using policy gradient method. In the deep deterministic policy gradient algorithm, the policy network is a deterministic policy function represented as \(\pi_{\phi}(s)\), with the learning parameters represented as \(\phi\).