# Reducing Q-Value Estimation Bias via Mutual Estimation and Softmax Operation in MADRL
## 3 Multi-Agent Mutual Twin Delayed Deep Deterministic Policy Gradient (M2 ATD3)  
In multi-agent scenarios, single Q network algorithms such as Multi-Agent Deep Deterministic Policy Gradient (MADDPG) have problems of overestimation. However, twin Q network's MATD3 algorithm, while resolving the overestimation of Q value, inevitably leads to underestimation. The Multi-Agent Three Delayed Deep Deterministic Policy Gradient (MATHD3) algorithm is proposed to create a third Q network. The overestimated results from it, and the underestimated results from the minimization of the first two Q networks, are added together by adjusting the weight hyperparameter \(\alpha\), and an unbiased estimate of the actual Q value can be obtained. In multi-agent algorithms, we denote the  
Figure 2: Multi-agent reinforcement learning training modes: (**a**) DTDE, each agent i makes decisions based on its own observation information, and each agent is relatively independent, with no information sharing between agents; (**b**) CTCE, which receives observation information from all agents through a centralized policy function and outputs the joint probability distribution of all agent action spaces; (**c**) CTDE, during the execution process of agents, each agent inputs its own local observation information into the policy function and outputs the probability distribution of the action space. However, during the training process, the agent can obtain additional information from other agents. observation received at runtime by agent i as \(o_{i}\), and the full state information as s, from which the observations \(o_{i}\) are derived. The Q network update target \(y_{i}\) of the MATHD3 algorithm is as follows:  
\[y_{i}=r_{i}+\gamma\alpha\min_{j=1,2}Q_{i,j}^{\pi^{\prime}}\big{(}s^{\prime},a_{1 }^{\prime},\cdots,a_{n}^{\prime}\big{)}+\gamma(1-\alpha)Q_{i,3}^{\pi^{\prime}} \big{(}s^{\prime},a_{1}^{\prime},\cdots,a_{n}^{\prime}\big{)} \tag{13}\]  
where \(a_{i}^{\prime}\) is obtained by adding noise \(\epsilon\) to the output of the target policy network \(\pi_{\phi_{i}^{\prime}}\) of agent i based on its observation \(o_{i}^{\prime}\), that is: \(a_{i}^{\prime}=\pi_{\phi_{i}^{\prime}}\big{(}o_{i}^{\prime}\big{)}+\epsilon\). While the MATHD3 algorithm effectively reduces bias in Q value estimates, it requires introducing a new Q network, leading to an increase in model parameters and increased model training difficulty. Psychological studies show that humans exhibit overconfidence, typically estimating themselves beyond their actual abilities [11]. Introducing external evaluations can mitigate this phenomenon. [PERSON] suggests that people are unrealistically optimistic because they focus on factors that improve their own chances of achieving desirable outcomes and fail to realize that others may have just as many factors in their favor [12]. Inspired by this, we propose the Multi-Agent Mutual Twin Delayed Deep Deterministic Policy Gradient (M2 ATD3) algorithm. The training mode of M2 ATD3 algorithm is shown in Figure 3.  
Single Q networks, affected by noise, always overestimate Q values. Averaging the estimated Q values from each agent's Q network results in a slightly overestimated mutual Q value estimate. The mutual estimate of the Q value and the minimum value of two Q network estimates for agent i are weighted together to obtain the Q network update target \(y_{i}\) of the M2 ATD3 algorithm:  
\[ \begin{split} Q_{agg}^{\pi^{\prime}}\big{(}s^{\prime},a_{1}^{ \prime},\cdots,a_{n}^{\prime}\big{)}=\frac{1}{n}\sum_{i=1,\cdots,n}Q_{i,1}^{ \pi^{\prime}}\big{(}s^{\prime},a_{1}^{\prime},\cdots,a_{n}^{\prime}\big{)}\\ y_{i}=r_{i}+\gamma\alpha\min_{j=1,2}Q_{i,j}^{\pi^{\prime}}\big{(}s ^{\prime},a_{1}^{\prime},\cdots,a_{n}^{\prime}\big{)}+\gamma(1-\alpha)Q_{agg} ^{\pi^{\prime}}\big{(}s^{\prime},a_{1}^{\prime},\cdots,a_{n}^{\prime}\big{)} \end{split}  \tag{14}\]  
The mutual estimate of the Q value is overestimated, and the minimization of the twin Q network will lead to underestimation.