Context-based classification via data-dependent mixtures of logistic and hidden markov model classifiers

This research addresses the problems encountered when designing classifiers for classes that contain multiple subclasses whose characteristics are dependent on the context. It is sometimes the case that when the appropriate context is chosen, classification is relatively easy, whereas in the absence of contextual information, classification may be difficult. Therefore, this research focuses on simultaneous learning of context and classification. One area where such problems are encountered is landmine detection. The signals collected by radar and metal detector systems show a lot of variability depending on a number of factors. Some of these factors can be listed as follows: (i) landmines are diverse in terms of their physical properties (large, small, plastic, metal), (ii) they may be buried at various depths in the ground (shallow, deep), (iii) they may be covered with clutter (stones, bushes, bottles, etc.), and (iv) the signals collected from these mines in different environments show a huge variability based on temperature, humidity, and soil conditions. When all of these factors are combined, one mine may look more similar to a mine of a different type (or to metallic clutter) than to itself in a different environment. Moreover, the exact reason that caused this confusion may not be always clear nor available. In such cases, a group of data that looks similar based on some unknown factor defines a context, and machine learning algorithms are needed to find these contexts and to learn classifiers within each context for a mine versus non-mine decision to be made. The mixture of experts (ME) model is suitable for problems such as the landmine detection that require context-based classification. During training, ME simultaneously learns the contexts and expert classifiers within each context all within one probabilistic framework. In initial experts applying standard ME model to landmine detection, significantly increased classification rates were achieved providing insightful parameters that can be understood by a system operator. However, two problems are encountered with the ME model. The first problem is over-fitting, which means that the model could be learning noise, and it may not generalize to future data. To prevent over-fitting, the variational mixture of experts model for multi-class classification is developed, and the lower bound associated with the model is derived. The lower bound provides a measure of correctness, and is used as an excellent stopping criterion. The second problem is that the ME model is not suitable for time series or sequential data as it requires fixed-length features and it cannot represent the dependency between a sequence of observations. To overcome these issues, a novel model, mixture of hidden Markov model experts (MHMME) has been introduced. The MHMME model can divide time series data into meaningful contexts, and learn an expert hidden Markov model (HMM) for each of these contexts. Although motivated by landmine detection, MHMME is a general method. Our experiments on multiple real and synthetic datasets show that MHMME can perform better than ME and HMMs, and can do well in comparison to state-of-the-art models.