# Machine Learning Subjective Opinions: An Application in Forensic Chemistry
## 2 Background
### Decisions and Strength of Evidence from Opinions  
The ML opinion ensemble method outlined in Section 2.5 can be extended if the application requires a decision, or if the analyst would like to express the strength of the evidence as a likelihood ratio. The process of proceeding from a set of ML subjective opinions to an ROC curve is described in this section. The basic approach is to obtain the \(P(\omega_{x}^{ML})\) values from a set of validation data with known ground truth. The uncertainty lies in the opinion, whereas the \(P(\omega_{x}^{ML})\) values are probabilities and follow the rules of probability calculus. The \(P(\omega_{x}^{ML})\) values can be used to calculate likelihood ratios which serve as scores for generating an ROC curve. Uncertainty in the opinion does not imply uncertainty in the likelihood ratio which is consistent with [PERSON] et al. [32]. A first step, though not required, is to decide on a maximum allowed opinion uncertainty for samples that require a decision. All samples having \(u_{x}^{ML}\) greater than the cutoff would be considered too uncertain based on the ML to proceed with a decision, where to set the \(u_{x}^{ML}\) limit would depend on the application and organizational standards and protocols. Decisions in binary cases can be made based on a decision threshold which can be plotted on the ROC curve. An optimal decision threshold score allows control over the false positive rate while simultaneously setting a true positive rate [31]. Creating an ROC curve requires the analysis of known ground truth validation samples and the creation of a \"score\" which is proportional to the probability that the sample belongs to the positive class [33; 34]. In this work, the log-likelihood ratio (LLR), Equation (9), based on the projected opinion \(P(\omega_{x}^{ML})\) has been used as a score for creating an ROC curve. The higher the score, the more likely it is that the sample belongs to class \(x\). The likelihood ratio can also be determined from the slope of a tangent to the ROC curve [35] at any point on the curve, irrespective of the score definition. However, it is important that classifications and the probabilities they are based on are calibrated. One popular method of calibrating probabilities, based on isotonic regression, is the pooled adjacent violator (PAV) algorithm. The PAV-calibrated probabilities can be obtained directly from the slopes of covering segments of the ROC convex hull (CH) [36]. Once the ROC curve is created and the convex hull is calculated, the \(P(\omega_{x}^{ML})\) for any previously unseen sample (e.g., a casework sample) can be used to calculate the likelihood ratio and the slope of the covering segment of the ROC CH gives the calibrated strength of the evidence. The equation for the likelihood ratio is obtained by rearranging the odds form of the Bayes equation, Equation (7), and making the appropriate substitutions. In Equation (7), \(H_{x}\) is the proposition that ILR is present, \(H_{\overline{x}}\) is the proposition that ILR is _not_ present and \(E\) is the evidence, which in the current context is the set of normalized relative intensities of the ions from feature selection, see Table 1. The term \(P(H_{x}|E)/P(H_{\overline{x}}|E)\) is the posterior odds, \(P(E|H_{x})/P(E|H_{\overline{x}})\) is the likelihood ratio and \(P(x)/P(\overline{x})\) is the prior odds. \[\frac{P(H_{x}|E)}{P(H_{\overline{x}}|E)}=\frac{P(E|H_{x})}{P(E|H_{\overline{x} })}\frac{P(x)}{P(\overline{x})} \tag{7}\]The projected probability of the opinion, \(P(\omega_{x}^{\text{ML}})\), is calculated by Equation (5), see Section 2.5. Rearranging Equation (7) to isolate the likelihood ratio and substituting \(P(\omega_{x}^{\text{ML}})\) for \(P(H_{x}|E)\), \(1-P(\omega_{x}^{\text{ML}})\) for \(P(H_{\overline{x}}|E)\), \(a_{x}^{\text{ML}}\) for \(P(x)\) and \((1-a_{x}^{\text{ML}})\) for \(P(\overline{x})\), the expression for the likelihood ratio, Equation (8), is obtained. Taking the base-10 logarithm of Equation (8) gives the log-likelihood ratio (LLR), Equation (9).