# An Agent-Ensemble for Thresholded Multi-Target Classification
## 5 Experiments
### Simulated Data  
We evaluate the effectiveness of the proposed combiner and the any-combiner with the different normalization strategies given in Section 4.4 vis-a-vis the conditionally non-discriminative condition defined in Section 4.1. We do this by simulating scores for agents that are distributed according to a multi-variate Gaussian distribution based on the label. We compare the following agent-combiners: 1. (_Prpsd. Gaussian WLR_) the proposed combiner given in (3) with a Gaussian model for the agent score distributions, 2. (_Joint Gaussian_) a jointly Gaussian generative meta-classifier that models the joint distribution of the agent scores conditioned on the label as \(p(\mathbf{x}|Y=j)=\mathcal{N}(\mathbf{x};\!\!^{-}_{j},\Sigma_{j})\), 3. (_Ind. Gaussian_) an independent Gaussian generative meta-classifier that models the joint distribution of the agent scores conditioned on the label as \(p(\mathbf{x}|Y=j)=\prod_{i=1}^{L}\mathcal{N}(x_{i};\mu_{i,j},\sigma_{i,j})\), 4. (_AC Gaussian WLR_) the any-combiner with weighted-likelihood normalization, 5. (_AC Z-norm_) the any-combiner with Z-normalization, 6. (_AC F-norm_) the any-combiner with F-normalization, and 7. (_AC EER-norm_) the any-combiner with EER-normalization where the EER rate is calculated assuming Gaussian densities as in [18]. Both the joint Gaussian and independent Gaussian combiners form a test statistic using the target vs. clutter likelihood ratio, that is,  
\[\Lambda=\frac{\sum_{j=1}^{L}p(\mathbf{x}|Y=j)p(Y=j|Y\geq 1)}{p(\mathbf{x}|Y=-1)}.\]  
We first evaluate the combiners when the conditionally non-discriminative assumption holds. We do this by simulating a ten-agent problem where the agent-outputs are independent and non-discriminative for targets for which they are not trained. We simulate a complex problem where the degree of difficulty for different targets varies significantly, a situation that has been shown to occur in practice [27]. Define \(\mu_{j}^{(i)}\) and \(\sigma_{j}^{(i)}\) as the mean and standard deviation of the \(j^{\text{th}}\) agent's output when \(Y=i\). We first draw agent-score distribution parameters for clutter data, drawing \(\mu_{j}^{(-1)}\) from a uniform \([-3\;-1]\) distribution and \(\sigma_{j}^{(-1)}\) from a uniform \([0.5\;1.25]\) distribution for each agent. Next,we make each agent \(j\) non-discriminative for targets other than \(j\) by setting \(\mu_{j}^{(i)}\) and \(\sigma_{j}^{(i)}\), \(\forall i\
eq j,i\geq 1\) equal to the clutter parameters for agent \(j\): \(\mu_{j}^{(-1)}\) and \(\sigma_{j}^{(-1)}\). Finally, we draw \(\mu_{j}^{(j)}\) from a uniform [13] distribution and \(\sigma_{j}^{(j)}\) from a uniform [0.5 1.25] distribution for each agent. Figure 2 plots ROC curves in the low-PFA region for the ten agents on their specific target-type for one draw of the agent-score parameters. We generate ten-thousand test exemplars and a varying number of combiner training samples in order to evaluate the effectiveness of the combiners when they must estimate their parameters using either few or many combiner training samples. We set class-priors to 0.5 for clutter and 0.05 for each target-type. Due to the uniformity among the target-type priors the weighting in the proposed weighted-likelihood ratio normalization is irrelevant and thus the results compare the performance of the normalizations without considering the weighting. We repeat the experiment fifty times, generating new agent-output distributions and data for each iteration. Table 1 gives the results averaged over the fifty experiments for each number of combiner training samples. The rightmost column gives an upper bound on the performance of each combiner when provided with the true parameters, and the joint Gaussian, independent Gaussian, and proposed method all perform the best when provided with the true parameters. However, when the combiners must estimate the parameters from data, the joint and independent Gaussian combiners perform worse in all cases than the proposed method.