# Machine Learning Subjective Opinions: An Application in Forensic Chemistry
## 4 Results
### Visualizing Opinions and Performance  
As shown in Figure 1, the visualization of binary subjective opinions is possible in a ternary plot. The calculated \(\omega_{x}^{LDA}\) values for the validation samples trained on 1000 \(x\) and 1000 \(\overline{x}\) samples are shown in Figure 2. Figure 2a shows the values for validation samples belonging to class \(\overline{x}\), and the plot Figure 2b shows the values for validation samples belonging to class \(x\). The plots allow an immediate visual assessment of \(\omega_{x}^{LDA}\) for any sample. Other general trends are observed from the ternary plots in Figure 2. The \(u_{x}^{LDA}\) value for some of the validation samples belonging to class \(x\) are larger than some of the class \(\overline{x}\). The three class \(x\) samples with the highest \(u_{x}^{LDA}\) values all contained moderate or strong levels of the same aromatic solvent that had been weathered to 75% loss of volume, and they all contained \"glue stick\" as one of the substrate components. The samples correspond to reference numbers 694, 695 and 696 in the Fire Debris Database [24]. In this case, the identity and composition of the samples with the highest predicted uncertainty do not assist in understanding shortcomings in the LDA ML model, but the information could be helpful to the forensic science interpretation of model performance. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**LDA** & & & \\ \hline
**Samples** & **AUC\({}_{s,m,w}\)** & **AUC\({}_{s,m}\)** & **AUC\({}_{s}\)** \\ \hline
100 & 0.768 (0.014) & 0.847 (0.013) & 0.892 (0.013) \\
200 & 0.795 (0.014) & 0.875 (0.013) & 0.912 (0.011) \\
1000 & 0.789 (0.014) & 0. It is also evident from the plots that the \(P(\omega_{x}^{LDA})\), generated from the projection of each point straight down onto the bottom axis (due to \(a_{x}^{LDA}=0.5\)) results in substantial overlap of the two classes. A quantitative analysis of the extent of class separation requires visualizing the \(P(\omega_{x}^{LDA})\) values on an ROC graph. The \(P(\omega_{x}^{LDA})\) values for the binary opinions were used to calculate scores to generate an ROC curve. The ROC curve resulting from the LDA ensemble trained on 2000 samples, including strong, moderate and weak IL contributions, \(ROC_{s,m,w}\) is shown in Figure 3. The corresponding \(AUC_{s,m,w}\) is 0.789, see Table 4.  
Figure 3: ROC diagram based on the LLR for the projected opinions for each of the 1117 experimental validation fire debris samples from Figure 2. The ROC curve is shown as a solid line. The ROC CH is shown as a blue dashed line, and the random guess curve is shown as the red dashed line. The ROC \(AUC_{s,m,w}\) is 0.789. Figure 2: Ternary diagrams showing LDA subjective opinions for 1117 experimental validation fire debris samples. (**a**) Samples containing no ILR and (**b**) samples containing ILR. Opinions based on 100 LDA models, each comprised of 2000 training samples (1000 containing ILR and 1000 containing only substrate pyrolysis components). The ternary plots and ROC curve for the RF model trained with 2000 samples in each of the 100 base-models of the ensemble are shown in Figures 4 and 5, respectively. The ternary plots demonstrate the lower uncertainties arising from the RF model, with the highest value of approximately 0.20-0.30 calculated for several of the class \(\overline{x}\) samples. The largest uncertainties for the \(x\) samples were approximately 0.20. Again, the \(P(\omega_{x}^{RF})\) values are obtained by projecting each point straight down to the base of the ternary plot (\(a_{x}^{RF}=0.5\)). The class separation of the LLR scores calculated from the \(P(\omega_{x}^{RF})\) values, Equation (9) produces an ROC plot, as shown in Figure 5, with an \(AUC_{s,m,w}\) of 0.837, as well as Table 4.  
Figure 4: Ternary diagrams showing RF subjective opinions for 1117 experimental validation fire debris samples. (**a**) Samples containing no ILR and (**b**) samples containing ILR.