## 5 Classification
### Combining Independent Classifiers  
The information provided by all the \(p\) likelihoods \(f(\mathbf{z}_{x};\boldsymbol{\mu}_{j},\Sigma_{j})\) of the classes \(j=1...p\) becomes interesting when combining different classifiers, such as an audio-based classifier and radio receiver based classifiers. The motivation for this approach is that often there are specialized classifiers which operate on certain type of sensor data (e.g., audio or accelerometer). Different sensors have very different data rates and feature statistics, and often it is impossible to concatenate raw features to the same feature vectors to perform feature level fusion. Instead, if we have a classifier, the output of this classifier can be fed to another classifier as a feature vector. This makes it possible to roughly match the data rates of different sensors, as in the case of a radio receiver producing a set of data every one minute combined with the output of the audio classifier at the same rate. In addition, in this case the application software does not have access to the classifiers, but only obtains the result containing the likelihoods for all the possible classes, and the decision itself.  
Thus it is convenient to use these likelihoods as features for another classifier that is running on a higher level in the software. By doing this we would not need to know any details of the original classifier. First we need to investigate how the \(f_{\mathbf{z}_{j}}(\mathbf{z})\) itself is distributed, considering \(f\) as a deterministic function of a random variable \(\mathbf{z}\). First, assume that the new observation \(\mathbf{z}=\mathbf{z}_{x}\) comes from the population class \(j=1\). To simplify the computation we take the natural logarithm of the likelihood, yielding a constant term (for each class) \(\ln\frac{1}{\sqrt{|2\pi\Sigma_{1}|}}\) plus the term  
\[-\frac{(\mathbf{z}-\boldsymbol{\mu}_{1})^{T}\Sigma_{1}^{-1}(\mathbf{z}- \boldsymbol{\mu}_{1})}{2} \tag{8}\]  
As the actual class is correct (_i.e._, \(\mathbf{z}\) and \(\boldsymbol{\mu}_{1}\) belong to the same class), by Theorem A.85 in [18], the numerator in Equation (8) is chi-square distributed with \(q\) degrees of freedom.  
Then, we need to study the distribution of incorrect class (_i.e._, classifier parameters are wrong). The distributions of likelihoods for the incorrect classes is not as straightforward. We will have a quadratic form \((\mathbf{z}+\mathbf{a})^{T}C(\mathbf{z}+\mathbf{a})\), where the vector \(\mathbf{a}\) and the matrix \(C\) have nothing to do with the parameters of the distribution of \(\mathbf{z}\). From [19] we find that  
\[P\{(\mathbf{z}+\mathbf{a})^{T}C(\mathbf{z}+\mathbf{a})\leq t\}=\sum_{k=0}^{ \infty}c_{k}F(\cdot,t) \tag{9}\]  
where F is a chi-square cumulative distribution function (CDF). This formula can be used to obtain a numerical approximation of the CDF for the likelihoods in the case of incorrect class. The normally distributed features produce chi-squared likelihoods when the class model is correct. We also know the shape of the distribution (Equation (9)) in the case of an incorrect class. Thus, if we leave the assumption of normally distributed features aside, we can train a new classifier using directly the likelihoods of the classifier running on a lower level. Feature compression is clearly necessary when feature vectors with a non-normal degenerate distribution are added to the classifier. We used this approach to combine an existing audio-based environment classifier with the radio-based classifier developed in this study. In our existing audio classifier we could only touch the outputs of the classifier (_i.e._, in this case likelihoods). In addition, these likelihoods were not from the same classes that were are interested in the final classification. Thus, classical ensemble methods [20] were not valid in this case.