# Machine Learning Subjective Opinions: An Application in Forensic Chemistry
## 4 Results
### Extending the Opinions to Classification Decisions and Evidentiary Value  
This approach is demonstrated for the RF base-method where each ensemble member is trained on 2000 in silico samples. An uncertainty opinion threshold was set at 0.10 for this example. The \(u_{x}^{ML}\) threshold is subjective and should be adjusted based on the amount of uncertainty the application can tolerate. By setting a threshold of 0.10, many of the 1117 validation samples are determined to be ineligible for a decision. The opinions of the 795 validation samples that are decision eligible are shown in Figure 7. The ROC curve that was generated from the decision eligible validation samples is shown in Figure 8. The ROC AUC for the curve in Figure 8 has decreased slightly from 0.837, see Table 3, to 0.832. The LLR decision threshold of 0.07 was obtained by setting the maximum false positive rate at 0.05, which corresponds to the Neyman-Pearson \(\alpha_{NP}\) value. Projecting vertically from a false positive rate of 0.05, the score at the point of intersection with the ROC curve is the decision threshold LLR of 0.07. The decision LLR corresponds to a \(P(\omega_{x}^{RF})\) of 0.54 in the ternary plots. Projecting from the decision threshold horizontally to the true positive rate axis gives an intersection of 0.55, which is equal to the quantity \(1-\beta_{NP}\), where \(\beta_{NP}\) is the Neyman-Pearson \(\beta\) value of 0.45. The significance of Figures 7 and 8 is that a previously unseen sample with a calculated \(u_{x}^{RF}\) from the same model may be classified as belonging to class \(x\) or class \(\overline{x}\) with known true and false positive rates. Samples with \(P(\omega_{x}^{RF})\geq 0.54\) should be classified into class \(x\), only if a decision is required. Otherwise the analyst should be reporting an opinion, which may include the strength of the evidence expressed as a likelihood ratio. Although a 5% false positive rate is often employed in hypothesis testing, it can be argued that a 5% false positive rate in forensic science could call into question the admissibility of the evidence. Obtaining a validated false positive rate in fire debris analysis is difficult due in part to the challenge of obtaining casework-realistic ground truth samples. [PERSON] at al. reported an 89% true positive rate for IL detection and a 7% false positive rate for tests limited to three classes of IL while using two-dimensional gas chromatography  
Figure 6: Ternary diagrams showing RF subjective opinions for 1117 experimental validation fire debris samples. (**a**) Samples containing no ILR and (**b**) samples containing ILR. Opinions based on 100 RF models, each comprised of 20,000 training samples (10,000 containing ILR and 10,000 containing only substrate pyrolysis components). with time-of-flight mass spectrometry [44]. When setting a decision threshold on an ROC curve, a lower false positive rate will correspond to a lower true positive rate. This trade-off becomes less problematic as the ROC AUC increases. Figure 8: ROC diagram based on the LLR for the projected opinions for each of the 795 validation fire debris samples having an \(u_{x}^{RF}\leq 0.10\) from Figure 7. The ROC curve is shown as a solid line and the ROC convex hull (CH) is shown as a blue dashed line on the upper and left edge of the ROC curve, and the random guess curve is shown as the red dashed line. The ROC \(AUC_{s,m,w}\) of 0.832, with a standard error of 0.014, is labeled on the graph, along with the decision threshold based on a Neyman–Pearson \(a_{NP}=0.05\) (solid black arrow pointing down), and corresponding \(\beta_{NP}=0.45\). The left-pointing solid black arrow points to \(1-\beta_{NP}\). The decision threshold, shown as an open circle on the ROC curve corresponds to an \(LLR=0.07\). The \(P(\omega_{x^{RF}})=0.54\) is the corresponding decision threshold on the ternary diagrams, see Equation (9). Figure 7: Ternary diagrams showing RF subjective opinions for 795 validation fire debris samples with calculated \(u_{x}^{RF}\leq 0.10\).