# On the Optimum Linear Soft Fusion of Classifiers
## 5 Real Data Experiments
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **Score** & **Training** \\ \hline Gaussian Bayes & \(\frac{s_{d}=}{e^{-\frac{1}{2}((\kappa-\mathbf{\mu}_{1})^{\mathsf{T}}C_{1}^{- 1}(\kappa-\mathbf{\mu}_{1}))}}\) & (ML sample estimates) \\ Gaussian Naive Bayes & \(s_{d}=\frac{-\frac{1}{2\sigma_{1}^{2}}((\kappa-\mathbf{\mu}_{1})^{\mathsf{T}}( \kappa-\mathbf{\mu}_{1}))}{e^{-\frac{1}{2}((\kappa-\mathbf{\mu}_{1})^{\mathsf{ T}}(\kappa-\mathbf{\mu}_{1}))}+e^{-\frac{1}{2}((\kappa-\mathbf{\mu}_{0})^{\mathsf{T}}C_{1}^{- 1}(\kappa-\mathbf{\mu}_{0}))}}\) & (ML sample estimates) \\ Nearest Mean & \(s_{d}=\frac{e^{-\frac{1}{2}[(\kappa-\mathbf{\mu}_{1})^{\mathsf{T}}(\kappa- \mathbf{\mu}_{1})]}}{e^{-\frac{1}{2}[(\kappa-\mathbf{\mu}_{1})^{\mathsf{T}}( \kappa-\mathbf{\mu}_{1})]}+e^{-\frac{1}{2}[(\kappa-\mathbf{\mu}_{0})^{\mathsf{ T}}C_{1}^{-1}(\kappa-\mathbf{\mu}_{0}))}}\) & (ML sample estimates) \\ Linear Discriminant & \(s_{d}=\frac{\mathbf{w}_{1}^{\mathsf{T}}\mathbf{x}}{\mathbf{w}_{1}^{\mathsf{T}} \mathbf{x}+\mathbf{w}_{0}^{\mathsf{T}}\mathbf{x}}\) & (Respective solutions to two) overdetermined systems of equations) \\ Logistic Regression & \(s_{d}=\frac{1}{1+e^{-\mathbf{w}^{\mathsf{T}}\mathbf{x}}}\) & (Solutions to one overdetermined system of equations) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Score computation method and required training. bias, especially in class 1, is also an issue for obtaining greater improvements. In any case, these real data results are in concordance with the expectation of the theoretical analysis. With the intention of reinforcing this perception, we show in Figure 6 (solid blue) the variation in the improvement factor with the number of fused classifiers. Starting from \(D=1\) with the Nearest Mean, which is the best individual classifier according to Table 2, we progressively incorporate a new classifier following the order of Table 2. We can see that \(I_{f}\) increases with \(D\) as predicted by the theory, but shifts rather slowly due to the high correlation and large bias. We also represent in Figure 6 (blue dot) the improvement factor of the EOC case computed from (14) for every \(D\), where \(\
u,\ \rho\), and \(b^{2}\) are estimated as:  
\[\hat{\
u}=\frac{1}{D}\sum_{d=1}^{D}\hat{\sigma}_{d};\quad\hat{\rho}=\frac{1}{D }\sqrt{\|\hat{\rho}\|_{F}^{2}-D};\quad\hat{b}^{2}=\frac{1}{D}\left\|\hat{ \mathbf{b}}_{s}\hat{\mathbf{b}}_{s}^{T}\right\|_{F} \tag{31}\]  
where \(\hat{\sigma}_{d}=\frac{1}{2}(\hat{\sigma}_{d/0}+\hat{\sigma}_{d/1})\), \(\hat{\rho}=\frac{1}{2}(\hat{\rho}_{0}+\hat{\rho}_{1})\), and \(\hat{\mathbf{b}}_{s}=\frac{1}{2}\left(\hat{\mathbf{b}}_{s/0}+\hat{\mathbf{b} }_{s/1}\right)\) are obtained from the results of Table 2. Thus, the Frobenius norms, \(\left\|\cdot\right\|_{F}\), of the correlation coefficient and the bias (constant entries) matrices of the EOC model, respectively, equate the Frobenios norms of the corresponding (non-constant entries) real-data matrices. In essence, we define an EOC model that \"resembles\" as much as possible the actual real-data model. Figure 6 suggests that the EOC model, in spite of its simplicity, can provide a reasonably approximation of the actual \(I_{f}\) value. We conducted another experiment, where the scores of each classifier are much more uncorrelated than in the previous experiment. To simulate high decorrelation, the inputs to the different classifiers in the interval under test are randomly selected (sampling with replacement) from all the available feature vectors of the training set that belong to the true class of that interval. In this way, the inputs to the different classifiers in each test interval belong to the same class, but are generally different, producing a high level of decorrelation between the scores obtained from them. This procedure serves to simulate decorrelation, but is not practical since the class being tested must be known in advance.