# Reducing Q-Value Estimation Bias via Mutual Estimation and Softmax Operation in MADRL
## 3 Multi-Agent Mutual Twin Delayed Deep Deterministic Policy Gradient (M2 ATD3)
By adjusting the weight hyperparameter \(\alpha\), in theory, an unbiased estimate of the true Q value of agent i can be obtained. The M2 ATD3 algorithm reduces the Q value estimation bias without needing to introduce  
Figure 3: The training mode of M2 ATD3. In this figure, rectangles represent networks, circles represent values, and dashed boxes represent sets or processes. Under the CTDE framework, each agent, equipped with Q networks of the same structure, can obtain shared information and make estimates of the current action-state value. The Q network of other agents can act as external critics, estimating the Q value of agent i and taking the average, denoted as the \(Q_{agg}\). Agent i obtains the update objective \(y_{i}\) of the Q network through weighted sum of \(\min\big{(}Q_{i,1},Q_{i,2}\big{)}\) and \(Q_{agg}\). a new Q network. Moreover, the operation of taking the average of Q value estimates can also reduce the variance of Q value estimate errors [13], making the training process more stable. We have maintained the delayed policy updates used in the TD3 algorithm. The TD3 algorithm proposes that the policy network should be updated at a lower frequency than the value network, to first minimize error before introducing a policy update. This method requires updating the policy and target networks only after a fixed number of updates \(d\) to the critic. The M2 ATD3 algorithm pseudo code can be found in Algorithm 1. ```
initialize Replay buffer \(D\) and network parameters for t = 1 to T do  Select action \(a_{i}\sim\pi_{i}(o_{i})+\epsilon\)  Execute action \(a=(a_{1},\cdots,a_{n})\) and get reward \(r_{i}\), transition to next state \(s^{\prime}\)  Store the state transition tuple \((s,a,r_{1},\cdots r_{N},s^{\prime})\) into \(D\) \(s\gets s^{\prime}\) for agent i = 1 to N do  A sample batch of size \(S\) is sampled from \(D\)\(\left(s^{b},a^{b},r^{b},s^{b}\right)\) \(Q^{\pi^{\prime}}_{avg}=\frac{1}{n}\sum_{i=1,\cdots,n}Q^{\pi^{\prime}}_{i,1} \left(s^{b},a_{1},\cdots,a_{n}\right)\Big{|}_{a_{k}=\pi^{\prime}_{i}\left(o^{ \phi}_{k}\right)+\epsilon}\) \(y^{b}=r_{i}^{b}+\gamma a\min_{j=1,2}Q^{\pi^{\prime}}_{i,j}\left(s^{b},a_{1}, \cdots,a_{n}\right)\Big{|}_{a_{k}=\pi^{\prime}_{i}\left(o^{\phi}_{k}\right)+ \epsilon}+\gamma(1-\alpha)Q^{\pi^{\prime}}_{avg}\)  Minimize the loss of two Q-functions, j = 1,2: \(L\left(\theta_{j}\right)=\frac{1}{S}\sum_{b}\left(Q^{\pi}_{i,j}\left(s^{b},a_{ 1}^{b},\cdots,a_{n}^{b}\right)-y^{b}\right)^{2}\) if\(t\) mod \(d\) = 0 (\(d\) is the policy delay update parameter) then  update \(\pi_{i}\): \(\
abla_{\phi_{i}}J\approx\frac{1}{S}\sum_{b}\
abla_{\phi^{\prime}}\
abla_{ \phi_{i}}\left(o_{i}^{b}\right)\
abla_{a_{i}}Q^{\pi}_{i,1}\left(s^{b},a_{1}^{ b},\cdots,\pi_{\phi_{i}}\left(o_{i}^{b}\right),\cdots,a_{n}^{b}\right)\)  Update target network: \(\theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}\) \(\phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime}\) where \(\tau\) is the proportion of target network soft updates endif endfor
```  
**Algorithm 1** Multi-Agent Mutual Twin Delayed Deep Deterministic policy gradient