# Reducing Q-Value Estimation Bias via Mutual Estimation and Softmax Operation in MADRL
## 2 Research Background
### Estimation Bias of Q-Learning Algorithms  
When an agent executes action \(a\) in state \(s\), the environment will transition to state \(s^{\prime}\) and return an immediate reward \(r\). In the standard deep Q-learning algorithm, we update the Q-function using Equation (1). When using neural networks and other tools as function approximators to handle complex problems, there can be errors in the estimation of Q values, which means:  
\[Q\big{(}s^{\prime},\hat{a}\big{)}=Q^{*}\big{(}s^{\prime},\hat{a}\big{)}+Y_{s^ {\prime}}^{\hat{a}} \tag{3}\]  
where \(Y_{s^{\prime}}^{\hat{a}}\) is a zero-mean noise term, \(Q^{*}\) is a Q function that can output the real Q value, or the optimal Q function. However, when using the maximum operation to select actions, there can be errors between the estimated Q values and the real Q values. The error is represented as \(Z_{S}\):  
\[Z_{S}\ \stackrel{{\text{def}}}{{=}}\ \gamma\Big{(}\max_{\hat{a}}Q \big{(}s^{\prime},\hat{a}\big{)}-\max_{\hat{a}}Q^{*}\big{(}s^{\prime},\hat{a} \big{)}\Big{)} \tag{4}\]  
Considering the noise term \(Y_{s^{\prime}}^{\hat{a}}\), some Q values may be underestimated, while others may be overestimated. The maximization operation always selects the maximum Q value for each state, which makes the algorithm overly sensitive to the high estimated Q values of the corresponding actions. In this case, the noise term causes \(\mathbb{E}[Z_{S}]>0\), resulting in the problem of overestimation of Q values. The TD3 [9] algorithm introduces clipped Double Q-Learning on the basis of the DDPG algorithm, by establishing two Q networks \(Q_{\theta_{1}},Q_{\theta_{2}}\) to estimate the value of the next state, and using the minimum estimated Q-value of two target networks to calculate the Bellman equation:\[y=r+\gamma\min_{i=1,2}Q_{\theta_{i}^{\prime}}\big{(}s^{\prime},\pi_{\phi}\big{(}s^ {\prime}\big{)}\big{)} \tag{5}\]  
where \(y\) is the update target for \(Q_{\theta_{1}}\) and \(Q_{\theta_{2}}\), and \(\pi_{\phi}\) is the policy network. Using clipped Double Q-Learning, the value estimation of target networks will not bring excessive estimation errors to the Q-Learning target. However, this update rule may cause underestimation. Unlike actions with overestimated Q values, actions with underestimated Q values will not be explicitly updated. Here is proof of the underestimation of the clipped Double Q-Learning, the two outputs values (\(Q_{1}\) and \(Q_{2}\)) of the Q network, and that \(Q_{i}^{*}\) is the real Q value. The errors brought by the fitting of the two networks can be defined as:  
\[Y^{i}=Q_{i}\big{(}s^{\prime},\pi_{\phi}\big{(}s^{\prime}\big{)}\big{)}-Q^{*} \big{(}s^{\prime},\pi_{\phi}\big{(}s^{\prime}\big{)}\big{)} \tag{6}\]  
Assuming errors \(Y^{1}\) and \(Y^{2}\) are two independent random variables with nontrivial zero-mean distributions, the minimization operation of the TD3 algorithm introduces an error. The error denoted as \(D\) is as follows:  
\[D=r+\gamma\min_{i=1,2}Q_{i}\big{(}s^{\prime},\pi_{\phi}\big{(}s^{\prime}\big{)} \big{)}-\big{(}r+\gamma Q^{*}\big{(}s^{\prime},\pi_{\phi}\big{(}s^{\prime} \big{)}\big{)}\big{)}=\gamma\min\Big{(}Y^{1},Y^{2}\Big{)} \tag{7}\]  
where \(r\) is the reward and \(\gamma\) is the discount factor. Let \(Z\) be minimum value between \(Y^{1}\), \(Y^{2}\), and we have:  
\[\mathrm{E}[Z]=\mathrm{E}[Y^{1}]+\mathrm{E}[Y^{2}]-\mathrm{E}[|Y^{1}-Y^{2}|]/2 \tag{8}\]  
Obviously, there is \(E(Y^{1})=0\), \(E(Y^{2})=0\) and \(\mathrm{E}[|Y^{1}-Y^{2}|]\leq 0\). Therefore, we can deduce that \(E(D)\leq 0\), which means that the clipped Double Q-Learning usually has underestimation. Specifically, in order to have a more intuitive perception of underestimation, we can assume that the error follows an independent and uniform distribution in the range \([-\epsilon,\epsilon]\).