# On the Optimum Linear Soft Fusion of Classifiers
## 2 The Optimal Linear Combiner
Considering (3) in (2):  
\[MSE_{f}=\Big{(}\mathbf{w}^{T}\mathbf{C}_{s/0}\mathbf{w}+\mathbf{w}^{T} \mathbf{b}_{s/0}\mathbf{b}_{s/0}^{T}\mathbf{w}\Big{)}P_{0}+\Big{(}\mathbf{w} ^{T}\mathbf{C}_{s/1}\mathbf{w}+\mathbf{w}^{T}\mathbf{b}_{s/1}\mathbf{b}_{s/1} ^{T}\mathbf{w}\Big{)}P_{1}=\mathbf{w}^{T}\mathbf{M}_{s}\mathbf{w}, \tag{4}\]  
where  
\[\mathbf{M}_{s}=\Big{(}\mathbf{C}_{s/0}+\mathbf{b}_{s/0}\mathbf{b}_{s/0}^{T} \Big{)}P_{0}+\Big{(}\mathbf{C}_{s/1}+\mathbf{b}_{s/1}\mathbf{b}_{s/1}^{T} \Big{)}P_{1}. \tag{5}\]  
We next obtain the coefficients that minimize \(MSE_{f}\) under the constraint \(\sum\limits_{d=1}^{D}w_{d}=\mathbf{w}^{T}\mathbf{1}=1\), where \(\mathbf{1}\) is defined as a \(D\)-dimensional vector of 1. Notice that \(w_{d}\geq 0\ \ d=1\ldots D\); this is a sufficient condition to ensure that \(0\leq s_{f}\leq 1\). We do not explicitly impose in the following \(w_{d}\geq 0\ \ d=1\ldots D\), since this would prevent reaching a close-form solution. Therefore, eventually, some fused scores can be out of the interval range (0,1). This is not critical as we finally must make a decision by comparing \(s_{f}\) with a threshold in the interval (0,1) so we can simply round down to 1, in case \(s_{f}>1\), and round up to 0, in case \(s_{f}<0\), thus not affecting the final decision. We used the method of Lagrange multipliers, i.e.,  
\[ \begin{array}{l}J=\mathbf{w}^{T}\mathbf{M}_{s}\mathbf{w}+\lambda\big{(}1- \mathbf{w}^{T}\mathbf{1}\big{)}\\ \frac{\delta I}{\delta\mathbf{w}}=2\mathbf{M}_{s}\mathbf{w}-\lambda\mathbf{1} =0\Rightarrow\mathbf{w}_{LLMSE}=\frac{\lambda}{2}\mathbf{M}_{s}^{-1}\mathbf{ 1}\\ \mathbf{w}_{LLMSE}^{T}\mathbf{1}=\frac{\lambda}{2}\mathbf{1}^{T}\mathbf{M}_{s}^{ -1}\mathbf{1}=1\Rightarrow\lambda=\frac{2}{\mathbf{1}^{T}\mathbf{M}_{s}^{-1} \mathbf{1}}\end{array} \right\}\Rightarrow\mathbf{w}_{LLMSE}=\frac{\mathbf{M}_{s }^{-1}\mathbf{1}}{\mathbf{1}^{T}\mathbf{M}_{s}^{-1}\mathbf{1}}\ \.\end{array} \tag{6}\]The corresponding linear least-mean-square-error (_LLMSE_) solution thus obtained is (to ease the notation we simply indicate \(MSE_{f}\) instead of \(LLMSE_{f}\))  
\[MSE_{f}=\mathbf{w}_{LLMSE}^{T}\mathbf{M}_{s}\mathbf{w}_{LLMSE}=\frac{\mathbf{1 }^{T}\mathbf{M}_{s}^{-1}}{\mathbf{1}^{T}\mathbf{M}_{s}^{-1}\mathbf{1}} \mathbf{M}_{s}\frac{\mathbf{M}_{s}^{-1}\mathbf{1}}{\mathbf{1}^{T}\mathbf{M}_{ s}^{-1}\mathbf{1}}=\frac{1}{\mathbf{1}^{T}\mathbf{M}_{s}^{-1}\mathbf{1}}. \tag{7}\]  
Notice that the _MSE_ corresponding to every individual classifier can be written similar to (2). They are the entries to the main diagonal of \(\mathbf{M}_{s}\), i.e.,  
\[MSE_{d}=E\Big{[}(u-s_{d})^{2}\Big{]}=\Big{(}v_{d/0}+b_{d/0}^{2}\Big{)}P_{0}+ \Big{(}v_{d/1}+b_{d/1}^{2}\Big{)}P_{1}=\mathbf{M}_{s}(d,d)\quad d=1\ldots D. \tag{8}\]  
So, in conclusion we derived the expression of the _LLMSE_ estimator \(\hat{\mathbf{\mu}}_{LLMSE}=\mathbf{w}_{LLMSE}^{T}\mathbf{s}=\frac{\mathbf{1 }^{T}\mathbf{M}_{s}^{-1}\mathbf{s}}{\mathbf{1}^{T}\mathbf{M}_{s}^{-1}\mathbf{ 1}}\) in such a way that \(\mathbf{w}^{T}\mathbf{1}=1\), and that the provided is expressed in terms of the matrix, \(\mathbf{M}_{s}\), whose main diagonal entries are the _MSEs_ provided by the individual classifiers. As we will see in the next section, all this will allow us to analyze the expected improvements due to the fusion of classifiers.