# On the Optimum Linear Soft Fusion of Classifiers
## 5 Real Data Experiments
In a practical situation, decorrelation must be obtained through appropriate training, for example, bagging-type techniques [34], in which the same classifier is trained with different subsets of instances obtained from an original set. The hard decisions of the differently \begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & \(\hat{b}_{d/0}\) & \(\hat{b}_{d/1}\) & \(\hat{\upsilon}_{d/0}\) & \(\hat{\upsilon}_{d/1}\) & \(M\hat{SE}_{d}\) & \(\hat{Acc}_{d}\) \\ \hline Nearest-mean & 0.1047 & 0.2562 & 0.0539 & 0.1120 & 0.1213 & 83.29\% \\ Linear Discr. & 0.1175 & 0.3901 & 0.0221 & 0.0618 & 0.1250 & 82.99\% \\ Gauss. Bayes & 0.0670 & 0.2916 & 0.0469 & 0.1691 & 0.1528 & 82.55\% \\ Naive Bayes & 0.0686 & 0.3011 & 0.0492 & 0.1826 & 0.1636 & 82.08\% \\ Logistic Regr. & 0.0401 & 0.3586 & 0.0371 & 0.2242 & 0.1957 & 79.99\% \\  & \(\hat{b}_{f/0}\) & \(\hat{b}_{f/1}\) & \(\hat{\upsilon}_{f/0}\) & \(\hat{\upsilon}_{f/1}\) & \(M\hat{SE}_{f}\) & \(\hat{Acc}_{f}\) \\ FUSION & 0.1264 & 0.3032 & 0.0408 & 0.0670 & **0.1078** & **84.23\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Estimates of the score statistics and accuracy of the classifiers. Correlated case. trained classifiers are combined to obtain a final decision. More generally, in classifier fusion, each classifier can be trained on different subsets of instances to induce some level of score uncorrelation. Another scenario where the scores can be uncorrelated is in late-multimodal fusion [9], where the input of each individual classifier corresponds to a different modality. We show in Table 3 the same type of information shown in Table 2. Notice that the statistics of the individual classifiers are quite similar to those in Table 2, however the post-fusion \(MSE\) has been significantly reduced (mainly because of the variance reduction), and the accuracy has improved by 7% with respect to the best classifier. Notice the low level of correlation (especially in class 0), as indicated by the correlation coefficient matrices included at the bottom of Table 3.  
\[\hat{\mathfrak{p}}_{0}=\left[ \begin{array}{cccc}1.00&0.08&0.01&0.03&0.04\\ 0.08&1.00&0.01&0.07\\ 0.01&0.01&1.00&0.03&0.02\\ 0.03&0.01&0.03&1.00&0.00\\ 0.04&0.07&0.02&0.00&1.00\\ \end{array} \right]\hat{\mathfrak{p}}_{1}=\left[ \begin{array}{cccc}1.00&0.12&0.16 &0.17&0.06\\ 0.12&1.00&0.12&0.15&0.13\\ 0.16&0.12&1.00&0.27&0.10\\ 0.17&0.15&0.27&1.00&0.08\\ 0.06&0.13&0.10&0.08&1.00\\ \end{array} \right]\]  
The comparative impact of uncorrelation can be better appreciated in Figure 6, where we show (solid red) the variation in the improvement factor in the case of uncorrelation, \(I_{fu}\), with the number of fused classifiers. Notice the significant increase in the improvement factor with respect to the correlated case (\(I_{f}\)). We also represent in Figure 6 (red dot) the improvement factor, \(I_{fNEOC}\), of the NEOC case computed from (21) for every \(D\), where \(\hat{\mathcal{C}}=\frac{D}{d=1}\frac{\hat{\mathfrak{p}}_{D}}{\hat{\mathfrak{p }}_{d}}\); \(\hat{\mathfrak{p}}_{d}\)\(d=1\ldots D\) as well as \(b^{2}\) are computed as in (31). \(I_{fNEOC}\) shows a similar evolution to \(I_{fu}\), but with some underestimation. Notice that \(\hat{\mathfrak{v}}_{D}\) corresponds to the actual variance of the Linear classifier, which is the minimum variance among all the 5 classifiers, as deduced from Table 2. However, the \(MSE_{D}\) value considered to compute \(I_{fu}\) is that  
Figure 6: Variation in the improvement factor with the number \(D\) of classifiers. \(I_{f}\), estimated improvement factor with correlated scores. \(I_{fEOC}\), improvement factor calculated with Equation (14) with EOC parameters estimated from (31). \(I_{fu}\), estimated improvement factor with uncorrelated scores. \(I_{fNEOC}\), improvement factor calculated with Equation (21), with EOC parameters estimated from (31).