# On the Optimum Linear Soft Fusion of Classifiers
## 3 The Improvement Factor
### Fusion of Non-Equally Operated Classifiers (NEOCs)  
Notice that, in the case of individual classifiers with disparate performances, it is reasonable to consider that the scores they provide are uncorrelated. This simplifies an analysis that would otherwise be intractable. Furthermore, as for the EOCs, we assume for simplicity that the covariance-bias model is the same for both classes; then:  
\[\mathbf{C}_{s}=\mathbf{C}_{s/0}=\mathbf{C}_{s/1}= \begin{pmatrix}\
u_{1}&0&0\\ 0&\ddots&0\\ 0&0&\
u_{D}\end{pmatrix} =\mathbf{\Lambda}_{s};\,\mathbf{b}_{s}=\mathbf{b}_{s/ 0}=\mathbf{b}_{s/1}= \begin{bmatrix}b_{1}\\ \vdots\\ b_{D}\end{bmatrix} , \tag{17}\]  
Let us use [PERSON]'s lemma again to calculate the inverse of \(\mathbf{M}_{s}=\mathbf{\Lambda}_{s}+\mathbf{b}_{s}\mathbf{b}_{s}^{T}\):  
\[\mathbf{M}_{s}^{-1}=\Big{(}\mathbf{\Lambda}_{s}+\mathbf{b}_{s}\mathbf{b}_{s}^ {T}\Big{)}^{-1}=\mathbf{\Lambda}_{s}^{-1}-\frac{1}{1+trace\Big{(}\mathbf{b}_ {s}\mathbf{b}_{s}^{T}\mathbf{\Lambda}_{s}^{-1}\Big{)}}\mathbf{\Lambda}_{s}^{- 1}\mathbf{b}_{s}\mathbf{b}_{s}^{T}\mathbf{\Lambda}_{s}^{-1}. \tag{18}\]  
Figure 2: (**a**) Improvement factor as a function of correlation; bias is assumed to be 0; (**b**) improvement factor as a function of bias; correlation is assumed to be 0. Then:  
\[ \begin{split} MSE_{f}=\frac{1}{\mathbf{1}^{T}\mathbf{M}_{s}^{-1} \mathbf{1}}=\frac{1}{\mathbf{1}^{T}\left(\mathbf{\Lambda}_{s}^{-1}-\frac{1}{1+ trace\left(\mathbf{b}_{s}\mathbf{b}_{s}^{T}\mathbf{\Lambda}_{s}^{-1} \right)}\mathbf{\Lambda}_{s}^{-1}\mathbf{b}_{s}\mathbf{b}_{s}^{T}\mathbf{ \Lambda}_{s}^{-1}\right)\mathbf{1}}=\\ =\frac{1}{\sum\limits_{d=1}^{D}\frac{1}{v_{d}}-\frac{1}{1+\sum \limits_{d=1}^{D}\frac{b_{d}^{2}}{v_{d}}}\left(\sum\limits_{d=1}^{D}\frac{b_{ d}}{v_{d}}\right)^{2}}\end{split} . \tag{19}\]  
And, finally:  
\[I_{f}=\frac{MSE_{D}}{MSE_{f}}=\left(v_{D}+b_{D}^{2}\right)\left(\sum\limits_{d =1}^{D}\frac{1}{v_{d}}-\frac{1}{1+\sum\limits_{d=1}^{D}\frac{b_{d}^{2}}{v_{d} }}\left(\sum\limits_{d=1}^{D}\frac{b_{d}}{v_{d}}\right)^{2}\right). \tag{20}\]  
This is a complicated expression due to the arbitrary combinations of bias and variance in each individual classifier. However, we can impose some simplification to gain insights into the expected improvement. Thus, we may consider that all the individual scores have the same bias, \(b_{1}=b_{2}=\ldots=b_{D}=b\). Then, we can write:  
\[ \begin{split} I_{f}=\frac{MSE_{D}}{MSE_{f}}=\left(v_{D}+b^{2} \right)\left(\sum\limits_{d=1}^{D}\frac{1}{v_{d}}-\frac{1}{1+b^{2}\sum\limits _{d=1}^{D}\frac{1}{v_{d}}}b^{2}\left(\sum\limits_{d=1}^{D}\frac{1}{v_{d}} \right)^{2}\right)=\left(v_{D}+b^{2}\right)\frac{\sum\limits_{d=1}^{D}\frac{1 }{v_{d}}}{1+b^{2}\sum\limits_{d=1}^{D}\frac{1}{v_{d}}}=\\ =\frac{v_{D}\sum\limits_{d=1}^{D}\frac{v_{D}}{v_{d}}+b^{2}\sum \limits_{d=1}^{D}\frac{v_{D}}{v_{d}}}{v_{D}+b^{2}\sum\limits_{d=1}^{D}\frac{v_ {D}}{v_{d}}}=\frac{Cv_{D}+Cb^{2}}{v_{D}+Cb^{2}}\end{split} , \tag{21}\]  
where, \(C=\sum\limits_{d=1}^{D}\frac{v_{D}}{v_{d}}\). Considering that \(0\leq\frac{v_{D}}{v_{d}}\leq 1\)\(d=1\ldots D-1\), \(1\leq C\leq D\). If \(v_{D}\ll v_{d}\), \(d=1\ldots D-1\), we have a very unbalanced scenario; then, \(C=1\Rightarrow I_{f}=1\), i.e., if the best classifier is \"much better\" than the rest, little improvement can be expected from the fusion. In the opposite case, we have a totally balanced situation if \(v_{D}=v_{d}=1\), \(d=1\ldots D-1\), and then \(C=D\). Moreover, \(I_{f}\) reaches its maximum \(\frac{Dv_{D}+Db^{2}}{v_{D}+Db^{2}}\), which consistently coincides with (15) (this is actually the EOC case with uncorrelated classifiers). In general, every classifier contributes \(\frac{v_{D}}{v_{d}}\) to increase \(I_{f}\). Let us define the unbalanced factor \(U=\frac{D-C}{D-1}\)\(D>1\) such that \(0\leq U\leq 1\), being \(U=0\) in the totally balanced case (\(C=D\)), and \(U=1\) for the very unbalanced case (\(C=1\)).