# On the Optimum Linear Soft Fusion of Classifiers
## 3 The Improvement Factor
### Fusion of Equally Operated Classifiers (EOCs)  
Let us consider the case where all individual classifiers have equal variances, equal pairwise covariances, and equal bias. We call this situation \"equally operated classifiers\" (EOCs). Although, in practice, this may not strictly be the case, it is representative of what might be expected when all classifiers have a similar individual performance and similar dependence on each other. More formally, this corresponds to  
\[\mathbf{C}_{s/i}= \begin{pmatrix}v_{/i}&\rho_{i}v_{/i}&\rho_{i}v_{/i}\\ \rho_{i}v_{/i}&\ddots&\rho_{i}v_{/i}\\ \rho_{i}v^{\prime}&\rho_{i}v_{/i}&v_{D/i}\end{pmatrix} ;\ \mathbf{b}_{s/i}= \begin{bmatrix}b_{/i}\\ \vdots\\ b_{/i}\end{bmatrix} \quad i=0,1, \tag{10}\]where \(0\leq\rho_{i}<1\) is the covariance coefficient. For simplicity, we consider the same models for both classes, i.e., \(\
u_{/0}=\
u_{/1}=\
u\); \(\rho_{0}=\rho_{1}=\rho\); \(b_{/0}=b_{/1}=b\), so that \(\mathbf{C}_{s/0}=\mathbf{C}_{s/1}=\mathbf{C}_{s}\); \(\mathbf{b}_{s/0}=\mathbf{b}_{s/1}=\mathbf{b}_{s}\). Then, considering (5) and that \(P_{0}+P_{1}=1\), matrix \(\mathbf{M}_{s}\) can be written in the form:  
\[ \begin{split}&\mathbf{M}_{s}=\mathbf{C}_{s}+\mathbf{b}_{s} \mathbf{b}_{s}^{T}=\left(\begin{array}{ccc}\
u&\rho\
u&\rho\\ \rho\
u&\ddots&\rho\
u\\ \rho\
u&\rho\
u&\
u\end{array}\right)+\left(\begin{array}{ccc}b^{2}&b^{2}& b^{2}\\ b^{2}&\ddots&b^{2}\\ b^{2}&b^{2}&b^{2}\end{array}\right)=\\ &=(1-\rho)\
u\left(\begin{array}{ccc}1&0&0\\ 0&\ddots&0\\ 0&0&1\end{array}\right)+\rho\
u\left(\begin{array}{ccc}1&1&1\\ 1&\ddots&1\\ 1&1&1\end{array}\right)+b^{2}\left(\begin{array}{ccc}1&1&1\\ 1&\ddots&1\\ 1&1&1\end{array}\right)=(1-\rho)\
u\mathbf{I}+\left(\rho\
u+b^{2}\right) \mathbf{11}^{T}\end{split} . \tag{11}\]  
Let us define \(\mathbf{G}=(1-\rho)\
u\mathbf{I}\), \(\mathbf{H}=\left(\rho\
u+b^{2}\right)\mathbf{11}^{T}\), where the rank of \(\mathbf{H}\) is \(1\) and \(\mathbf{G}\) is nonsingular. We can apply [PERSON]'s lemma to compute the inverse:  
\[ \begin{split}&\mathbf{M}_{s}^{-1}=\left(\mathbf{G}+\mathbf{H} \right)^{-1}=\mathbf{G}^{-1}-\frac{1}{1+trace\left(\mathbf{HG}^{-1}\right)} \mathbf{G}^{-1}\mathbf{HG}^{-1}=\\ &=\frac{1}{(1-\rho)\
u}\mathbf{I}-\frac{1}{\left(1+\frac{D(\
u+b^{2})}{(1 -\rho)\
u}\right)}\frac{\rho\
u+b^{2}}{(1-\rho)^{2}\
u }\mathbf{11}^{T}=\frac{1}{(1-\rho)\
u}\left(\mathbf{I}-\frac{\rho\
u+b^{2}}{(1 -\rho)\
u+D(\rho\
u+b^{2})}\mathbf{11}^{T}\right)\.\end{split}  \tag{12}\]  
From (6) and (12), we can write:  
\[ \begin{split}& MSE_{f}=\frac{1}{\mathbf{1}^{T}\mathbf{M}_{s}^{ -1}}=\frac{1}{\mathbf{1}^{T}\frac{1}{(1-\rho)\
u}\left(1-\frac{\rho\
u+b^{2}}{( 1-\rho)\
u+D(\rho\
u+b^{2})}\mathbf{11}^{T}\right)\mathbf{1}}=\\ &=\frac{(1-\rho)\
u}{D-\frac{\rho\
u+b^{2}}{(1-\rho)\
u+D(\rho\
u+ b^{2})}D^{2}}=\frac{(1-\rho)\
u+b^{2}}{(1-\rho)\
u+b^{2}}=\\ &=\frac{(1-\rho)\
u/D}{\frac{(1-\rho)\
u+D(\rho\
u+b^{2})}{(1-\rho) \
u+D(\rho\
u+b^{2})}D}=\frac{(1-\rho)\
u+D\left(\rho\
u+b^{2}\right)}{D} \end{split} . \tag{13}\]  
Finally, considering that, in this case, \(MSE_{D}=\mathbf{M}_{s}(D,D)=\
u+b^{2}\), we may write the expression of the improvement factor as:  
\[I_{f}=\frac{MSE_{D}}{MSE_{f}}=D\frac{\
u+b^{2}}{(1-\rho)\
u+D(\rho\
u+b^{2})} =\frac{D\
u+Db^{2}}{(\rho(D-1)+1)\
u+Db^{2}}. \tag{14}\]  
Notice that \(\rho(D-1)+1\leq D\Rightarrow I_{f}\geq 1\), as it should be. For \(b\) and \(D\), the maximum reduction is achieved if the individual scores are uncorrelated  
\[\rho=0\Rightarrow I_{f}=\frac{D\
u+Db^{2}}{\
u+Db^{2}}, \tag{15}\]  
which in turn reaches its maximum value of \(I_{f}=D\), if \(b=0\). Thus, bias is a limiting factor for the reduction in the \(MSE\). Actually, we demonstrate below that, in EOCs, it is \(\mathbf{w}_{LMMSE}=\frac{1}{D}\mathbf{I}\), so from (3), the fused score, \(s_{f}\), has the same bias, \(b\), as the individual scores, \(s_{d}\)\(\ d=1\ldots D\). Therefore, the fusion improvement is essentially due to the reduction in the variance. Other interesting conclusions can be derived from (14).