# On the Optimum Linear Soft Fusion of Classifiers
## 2 The Optimal Linear Combiner  
Let us call \(0\leq s_{d}\leq 1\) to the score assigned by the individual classifier \(d\) to class \(c_{1}\). Considering \(s_{d}\) as a the posterior probability of \(c_{1}\), the score assigned to \(c_{0}\) should be \(1-s_{d}\). Soft fusion of classifiers consists of generating a score, \(s_{f}\), assigned to class \(c_{1}\) (and so \(1-s_{f}\) to class \(c_{0}\)) by combining the scores \(s_{d}\)\(d=1\ldots D\), respectively, provided by \(D\) individual classifiers, as indicated in Figure 1.  
We then make an estimation approach. The fused score, \(s_{f}\), can be interpreted as an estimate of the binary random variable, \(u\), from the \"observations\" \(s_{d}\)\(d=1\ldots D\). It is \(u=1\) when the true class is \(c_{1}\) and \(u=0\) when the true class is \(c_{0}\). Conditional to \(c_{1}\), \(s_{f}\) should have values close to \(1\), and conditional to \(c_{0}\), it should be close to \(0\). It is well-known that the optimal estimate of \(u\) minimizing the \(MSE\) is the conditional mean \(E[u/s_{1}\ldots s_{D}]\), which  
Figure 1: Soft fusion of classifiers. is, in general, a complex non-linear function, not expressible in close form. Therefore, we consider the optimal linear combination of scores. On the other hand, it is quite important to relate the \(MSE\) of the estimate \(s_{f}\) with the \(MSE\)'s of the individual estimates \(s_{d}\ d=1\ldots D\). This facilitates the analysis of the expected improvement due to the fusion, which is the essential objective of this work. Thus, given the set of coefficients, we can write  
\[s_{f}=\sum\limits_{d=1}^{D}w_{d}s_{d}, \tag{1}\]  
but, we can express the \(MSE\) of the estimate \(s_{f}\) as  
\[MSE_{f}=E\bigg{[}\Big{(}u-s_{f}\Big{)}^{2}\bigg{]}=P_{0}\cdot E \bigg{[}\Big{(}u-s_{f}\Big{)}^{2}/u=0\bigg{]}+P_{1}\cdot E\bigg{[}\Big{(}u-s_{f }\Big{)}^{2}/u=1\bigg{]}=\ \, \tag{2}\] \[=P_{0}\cdot\Big{(}\
u_{f/0}+b_{f/0}^{2}\Big{)}+P_{1}\cdot\Big{(} \
u_{f/1}+b_{f/1}^{2}\Big{)}\]  
where \(P_{i}\), \(\
u_{f/i}\), \(b_{f/i}\), respectively, are the prior, the conditional variance, and the conditional bias corresponding to class \(c_{i}\). Let us define the vectors \(\mathbf{s}=[s_{1}\ldots s_{D}]^{T}\ \mathbf{w}=[w_{1}\ldots w_{D}]^{T}\), so that \(s_{f}=\mathbf{w}^{T}\mathbf{s}\). We can compute the conditional variances and bias as follows:  
\[ \begin{array}{l}v_{f/i}=E\Big{[}s_{f/i}^{2}\Big{]}-E^{2}\Big{[}s_{f/i}\Big{]} =E\Big{[}\big{(}\mathbf{w}^{T}\mathbf{s}\big{)}^{2}/u=i\Big{]}-E^{2}\big{[} \mathbf{w}^{T}\mathbf{s}/u=i\big{]}=\\ =\mathbf{w}^{T}E\big{[}\mathbf{s}\mathbf{s}^{T}/u=i\big{]}\mathbf{w}-\mathbf{ w}^{T}E[\mathbf{s}/u=i]E^{T}[\mathbf{s}/u=i]\mathbf{w}=\mathbf{w}^{T} \big{(}\mathbf{R}_{s/i}-E[\mathbf{s}/u=i]E^{T}[\mathbf{s}/u=i]\big{)}\mathbf{w }=\\ =\mathbf{w}^{T}\mathbf{C}_{s/i}\mathbf{w}\end{array} , \tag{3}\] \[b_{f/i}=E\Big{[}s_{f/i}\Big{]}-i\sum\limits_{d=1}^{D}w_{d}E[s_{d/ i}]-i\sum\limits_{d=1}^{D}w_{d}=\sum\limits_{d=1}^{D}w_{d}b_{d/i}=\mathbf{w}^{T} \mathbf{b}_{s/i}\quad\mathbf{b}_{s/i}=[b_{1/i}\ldots b_{D/i}]^{T}\]  
where \(\mathbf{R}_{s/i}\) and \(\mathbf{C}_{s/i}\) are, respectively, the correlation and covariance matrices of the score vector conditional to \(u=i\), and where we assumed \(\sum\limits_{d=1}^{D}w_{d}=1\). The latter is a regularization condition to allow for a non-trivial solution in the following and avoid an anomalous increase in bias in the fused score.