# An Agent-Ensemble for Thresholded Multi-Target Classification
## 3 Related Work  
There is a wide body of literature on classifier combiners. [PERSON] [7] breaks the literature down into two broad categories: classifier selection and classifier fusion. Of these two categories, classifier selection ensembles are the most closely related to the proposed ensemble of classifier agents. Classifier selectors are often based on the mixture of experts originally proposed by [PERSON] et al. [8]. The expert classifiers are trained to perform well in a region of the input space, and thus a selection combiner chooses from one of the experts to classify each test sample based on the location of the test feature vector. Mixtures of experts have found wide use in a variety of applications [9; 10; 11; 12]. The motivation for our agent-classifiers is similar to that for classifier experts; in each case, the constituent classifiers are specialized with respect to a particular subproblem. However, whereas experts are trained for a specific region of the input space, agents are tuned for a particular target-type. This difference makes a standard classifier selector inappropriate for agent combination. Meta-classification, also known as classifier stacking, is another method of combining classifiers whereby the set of score/label pairs \(\{(\mathbf{x}_{i},y_{i})\}\) is treated as a new set of classifier training data to which standard classifiers are applied [13; 14; 15]. We relate our proposed agent-combination method to meta-classification in Section 4.3. Several authors have proposed ensembles of classifiers that are similar in spirit to the agents that we propose. [PERSON] et al. proposed a mixture of exemplar-support vector machines (SVMs) for object classification in computer vision systems [4]. An exemplar-SVM is a linear SVM trained to distinguish one target training sample from clutter. This is an extreme version of what we refer to here as a classifier agent, where each target training sample serves as its own target-type. [PERSON] et al. proposed an ensemble of agents approach for computer malware classification [3]. They specifically evaluated the advantages this approach has over traditional binary classifiers including responsiveness to changes in the target class and the semantic information it can provide to the operator. Additionally, in previous work, we proposed a multi-agent combination for target classification and a combiner algorithm that we called the _any-combiner_ given as:  
\[\hat{y}(\mathbf{x})=\text{sign}\left(\max_{j\in\mathbb{I}_{\sim},L}\left(f(x_ {j})\right)-\alpha\right), \tag{1}\]  
where \(f(\cdot)\) is a normalizing function (we describe several examples in detail in Section 4.4). The any-combiner combination strategy, classifying a test sample as target if any of the constituent agents classifies it as target, was used as well in both [3; 4]. However, none of these previous studies evaluated when or if this approach is optimal, or what the optimal normalization for the individual agents may be. We extend the results of these previous works by proposing a new combination strategy that is similar to the any-combiner and by providing theory on when this approach is optimal for combining the output of the agents. A critical element of the any-combiner (1) and the new combiner that we propose in this paper is agent normalization. Biometrics researchers have studied classifier normalization extensively [16; 17; 18; 19]. The problem studied in these previous works differs from that studied here in that they assumed that at test time a user would claim an identity, and the system would only evaluate the classifier associated with that user. Score normalization allowed the use of a single threshold for any claimed identity. The Z-norm [17], F-norm [16], equal error rate (EER) norm, and model-specific log-likelihood ratio (MS-LLR) norm [18] have all been investigated in such systems. In contrast, we assume that a test sample could belong to any one of the pre-defined target-types or clutter, requiring evaluation and combination of multiple agent scores.