# Reducing Q-Value Estimation Bias via Mutual Estimation and Softmax Operation in MADRL
## 4 Multi-Agent Softmax Twin Delayed Deep Deterministic Policy Gradient (MASTD3)
### Multi-Agent Softmax Operation  
The method of utilizing softmax operator to reduce overestimation of Q value can also be applied in multi-agent deep reinforcement learning algorithms. However, as the number of agents increases, the action space of the task will increase exponentially. It is impractical to simply apply the softmax operator to every possible combination of multi-agent joint actions, and the estimated Q value given by the Q network for the joint action, which is rarely sampled, is unreliable. [PERSON] and others proposed an approximate softmax operator in Regularized Softmax Deep Multi-Agent Q-Learning (RES) ([PERSON] et al., 2016). Firstly, find the joint action \(\hat{a}=\max_{b}Q_{tot}(s,a)\) that makes the joint Q value the greatest. For the agent i, fix the actions of other agents \(\hat{a}_{-i}\) and only change the action of the agent a to obtain the corresponding action subset \(A_{i}=\{(a_{i},\hat{a}_{-i})\mid a_{i}\in A\}\). The approximate action space \(\widehat{A}=A_{1}\cup\cdots\cup A_{n}\) is obtained by combining the action subsets of all agents. The approximate softmax operator used in the RES algorithm is to use this approximate action space rather than the complete action space for Q value softmax operation calculation. [PERSON] and others proved that the difference brought by the calculation using the approximate action space and the complete action space is bounded, and because the approximate softmax operator reduces the dependence on unreliable joint Q values, this method performs better than the softmax operator using the complete action space. Inspired by the Softmax Deep Double Deterministic Policy Gradients (SD3) ([PERSON] et al., 2016) and RES, this paper applies the softmax operator to multi-agent algorithms and proposes the MASTD3 algorithm. Firstly, the MASTD3 algorithm further reduces the action space used for Q value softmax operation calculation in the choice of action space. Suppose there are \(N\) agents in a multi-agent task, each agent action space is of size \(K\), the complete action space includes all permutations and combinations of the actions, and the softmax operation requires computing \(K\) to the power of \(N\,\text{Q}\) values. The approximate action space proposed in the RES algorithm, while traversing the action space of a single agent, will fix the optimal action of the other agents, and then merge each agent's sub action space as the approximate action space, the softmax operation needs to calculate \(K\) times \(N\,\text{Q}\) values. This article believes, within the structure of the MASTD3 algorithm, the Q network of the agent i can make a reliable estimate of the \((s,a)\mid_{a\in A_{i}}\) state-action pair, but the estimate of the \((s,a)\mid_{a\in A_{-i}}\) state-action pair formed through changes in the actions of other agents is not accurate. Therefore, in the simple softmax operator proposed in this article, when agent i performs the softmax operation calculation on the Q value, only the sub-action space \(A_{i}\) containing all of its own actions, while the actions of other agents are fixed, is considered, and each agent only needs to calculate Q values \(K\) times:  
\[\text{softmax}_{\tau,i}(Q_{i}(s,\cdot))=\sum_{a\in A_{i}}\frac{\exp(\tau Q(s,a)) }{\sum_{a^{\prime}\in A_{i}}\exp(\tau Q(s,a^{\prime}))}Q_{i}(s,a) \tag{19}\]  
where \(\tau\) is the temperature parameter in the softmax operation, controls the level of exploration versus exploitation in decision-making. The softmax operator will lead to underestimation of the Q value, and the high estimate Q value obtained by using the twin Q network maximization operation to perform the softmax operation can alleviate the impact of underestimation. Therefore, the Q network update target \(y_{i}\) of the MASTD3 algorithm is as follows:  
\[