# An Agent-Ensemble for Thresholded Multi-Target Classification
## 5 Experiments
### Simulated Data
The proposed method is the best or statistically significantly tied for the best with 95% confidence at all numbers of training samples. At low numbers of training samples, the any-combiner with Z-normalization is statistically significantly tied for the best with the proposed method. The results show that, given limited training data, the performance of the proposed combiner greatly exceeds that of the joint combiner as well as the combiner based on an independence assumption. We can also see that the weighted likelihood normalization outperforms the normalization methods given in Section 4.4. However, we note that the performance of the different normalization methods will depend on a variety of factors including the number of combiner training samples and how well the agent distributions match the assumptions used to derive the normalization methods. Regularization techniques for estimating the model parameters can also improve the performance when there are few training samples [18]. Figure 2: Agent ROC curves for the first simulation in Section 5.1. Each line shows a ROC curve for one of the 10 random agents. The ROC curves show that the simulation mimics a scenario where some targets are much harder to distinguish from clutter than others. We now compare performance when the agents are not conditionally non-discriminative. We simulate a problem with four target-types and four corresponding agents. We again use Gaussian conditional densities, \(p(\mathbf{x}|Y=j)=\mathcal{N}(\mathbf{x};\hat{\mathbf{\gamma}}_{j},\Sigma)\). We set the parameters of the conditional distributions to make agents one and two offer discriminative power on each other's targets. In order to do this, we set:  
\[\mu_{-1} =[-1,\ -1,\ -1,\ -1]^{\mathrm{T}}\,,\] \[\mu_{1} =[1,\ 1,\ -1,\ -1]^{\mathrm{T}}\,,\] \[\mu_{2} =[1,\ 1,\ -1,\ -1]^{\mathrm{T}}\,,\] \[\mu_{3} =[-1,\ -1,\ 1,\ -1]^{\mathrm{T}}\,,\] \[\mu_{4} =[-1,\ -1,\ -1,\ 1]^{\mathrm{T}}\,,\]  
and  
\[\Sigma= \begin{bmatrix}1&\alpha&0&0\\ \alpha&1&0&0\\ 0&0&1&0\\ 0&0&0&1\end{bmatrix} .\]  
Using these definitions we can determine the following conditional distributions:  
\[p(x_{2}|x_{1},Y=1) =\mathcal{N}\left(ax_{1}+(1-\alpha),\sqrt{1-\alpha^{2}}\right),\] \[p(x_{2}|x_{1},Y=-1) =\mathcal{N}\left(ax_{1}-(1-\alpha),\sqrt{1-\alpha^{2}}\right)\]  
and  
\[p(x_{1}|x_{2},Y=2) =\mathcal{N}\left(ax_{2}+(1-\alpha),\sqrt{1-\alpha^{2}}\right),\] \[p(x_{1}|x_{2},Y=-1) =\mathcal{N}\left(ax_{2}-(1-\alpha),\sqrt{1-\alpha^{2}}\right).\]  
We can control how closely the conditionally non-discriminative assumption holds by varying \(\alpha\) between 0 and 1. When \(\alpha=0\) agents one and two are independent and discriminative for one another's target-types; thus the conditionally non-discriminative assumption is poor. At \(\alpha=1\) the conditionally non-discriminative condition holds. Tables 2 and 3 give the results, averaged over fifty random draws of the data for \(\alpha\in\{0.95,0.8,0.6,0.4,0.1\}\) when the combiners are provided with 100 and 1000 training samples, respectively. Table 2 shows that with fewer training samples, the proposed combiner is the best or \begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **100** & **1000** & **2000** & **10,000** & **Model** \\  & **Samples** & **Samples** & **Samples** & **Samples** & **Params** \\ \hline Prpsd. Gaussian WLR & **0.813** & **0.855** & **0.856** & **0.855** & **0.856** \\ Joint Gaussian & 0.208 & 0.799 & 0.831 & 0.851 & **0.856** \\ Ind. Gaussian & 0.600 & 0.844 & 0.850 & 0.854 & **0.856** \\ AC Gaussian WLR & 0.812 & 0.854 & 0.855 & 0.854 & 0.855 \\ AC Z-norm & **0.809** & 0.829 & 0.832 & 0.831 & 0.832 \\ AC F-norm & 0.716 & 0.753 & 0.751 & 0.756 & 0.757 \\ AC EER-norm & 0.780 & 0.791 & 0.792 & 0.792 & 0.821 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Average partial area under the curve (PAUC) for the first simulation with conditionally non-discriminative agents.