# Machine Learning Subjective Opinions: An Application in Forensic Chemistry
## 4 Results
### Training Data Set Size Effects
\(u_{x}^{ML}\)** & **Wilcoxon \(p\)** \\ \hline
100 & 286 & 4.44 \(\times\) 10\({}^{-1}\) & 1 & NA \({}^{1}\) \\
200 & 66 & 2.18 \(\times\) 10\({}^{-1}\) & 1 & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\
1000 & 2 & 3.46 \(\times\) 10\({}^{-2}\) & 1 & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\
2000 & 0 & 1.68 \(\times\) 10\({}^{-2}\) & 8.36 \(\times\) 10\({}^{-1}\) & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\
20,000 & 0 & 1.55 \(\times\) 10\({}^{-3}\) & 1.42 \(\times\) 10\({}^{-1}\) & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\
60,000 & 0 & 4.97 \(\times\) 10\({}^{-4}\) & 5.60 \(\times\) 10\({}^{-2}\) & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\ \hline
**RF** & & & & \\ \hline
100 & 0 & 1.76 \(\times\) 10\({}^{-1}\) & 3.52 \(\times\) 10\({}^{-1}\) & NA\({}^{1}\) \\
200 & 0 & 1.51 \(\times\) 10\({}^{-1}\) & 3.06 \(\times\) 10\({}^{-1}\) & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\
1000 & 0 & 9.28 \(\times\) 10\({}^{-2}\) & 3.62 \(\times\) 10\({}^{-1}\) & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\
2000 & 0 & 8.08 \(\times\) 10\({}^{-2}\) & 2.89 \(\times\) 10\({}^{-1}\) & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\
20,000 & 0 & 2.57 \(\times\) 10\({}^{-2}\) & 1.01 \(\times\) 10\({}^{-1}\) & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\
60,000 & 0 & 1.39 \(\times\) 10\({}^{-2}\) & 1.20 \(\times\) 10\({}^{-1}\) & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\ \hline
**SVM** & & & & \\ \hline
100 & 0 & 2.62 \(\times\) 10\({}^{-1}\) & 7.44 \(\times\) 10\({}^{-1}\) & NA\({}^{1}\) \\
200 & 0 & 2.15 \(\times\) 10\({}^{-1}\) & 6.54 \(\times\) 10\({}^{-1}\) & 1.21 \(\times\) 10\({}^{-14}\) \\
1000 & 0 & 1.37 \(\times\) 10\({}^{-1}\) & 6.31 \(\times\) 10\({}^{-1}\) & \textless{}2.2 \(\times\) 10\({}^{-16}\) \\
2000 & 0 & 1.18 \(\times\) 10\({}^{-1}\) & 5.53 \(\times\) 10\({}^{-1}\) & 5.55 \(\times\) 10\({}^{-5}\) \\
20,000 & 0 & 4.92 \(\times\) 10\(In Table 4, the \"95% CI\" column gives the confidence interval for the AUC and the \"\(p\)-value\" column gives the statistic for the two-sided Delong test. The LDA ROC curves are not statistically improving for models exceeding 200 training samples. The RF ROC curves did not statistically improve for models exceeding 20,000 training samples. The SVM ROC curves continued to get statistically better for models up to 20,000 training samples. The SVM ML was not applied to models exceeding 20,000 training samples due to limited computational facilities. Table 5 shows the change in ROC AUC as the size of the training set increased and the ratio of IL/SUB was changed. For all three ML models, at all training data set sizes, the AUC is seen to increase as the IL/SUB ratio incorporated into the model goes from all samples in the second column from the left (i.e., \(AUC_{s,m,w}\)) to only the high IL/SUB ratio (i.e., \(AUC_{s}\)) in the rightmost column. The standard error for each AUC is given in parentheses. The increase in AUC at each training sample size, proceeding from \(AUC_{s,m,w}\) to \(AUC_{s,m}\), and from \(AUC_{s,m}\) to \(AUC_{s}\) is statistically significant based on a Delong test, with each \(p<2.2\times 10^{-16}\). This result demonstrates that both ML methods are producing better \(x-\overline{x}\) class separation at strong ILR contribution. Notably, at even the smallest training size, the RF model produces better class separation than the LDA model at the largest training size. At the largest training size, the RF model produces an \(AUC_{s,m,w}\) of 0.849, and the \(AUC_{s}\) of 0.962 indicates very good class  
All models are performing similarly to what would be expected from a human analyst (i.e., better ILR detection as the ILR relative contribution increases). The most significant differences in the models are the excessive uncertainty produced by LDA at very small training sizes, the higher overall uncertainty by SVM, and the long training times required by SVM. The optimal model studied in this research is the RF ensemble trained on 60,000 training samples.