## 3 Materials and Methods
### Uncertainty Estimation  
DLWP loss uses softmax of the functional form represented in Equation (5), providing point estimates for the sample class probabilities without the associated uncertainties.  
\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**PRECISION**} & \multicolumn{2}{c}{**RECALL**} & \multicolumn{2}{c}{**F1**} & \multicolumn{2}{c}{**CM**} & \multicolumn{2}{c}{**Total**} \\ \hline
**Method** & **Macro \(\uparrow\)** & **N \(\uparrow\)** & **P \(\uparrow\)** & **Macro \(\uparrow\)** & **N \(\uparrow\)** & **P \(\uparrow\)** & **Macro \(\uparrow\)** & **N \(\uparrow\)** & **P \(\uparrow\)** & **FP \(\downarrow\)** & **FN \(\downarrow\)** & **Cost \(\downarrow\)** \\ \hline GPR-based GAN & 0.80 & 99.0 & 0.60 & 0.91 & 0.98 & 0.84 & 0.84 & 0.99 & 0.70 & 207 & 59 & 31,570 \\ DLWP-RLPr-RLPr & 0.70 & 1.0 & 0.39 & 0.97 & 0.96 & 0.98 & 0.77 & 0.98 & 0.56 & 569 & 9 & 10,190 \\ DLWP-RLPr-ENPr & 0.70 & 1.0 & 0.40 & 0.97 & 0.96 & 0.98 & 0.77 & 0.98 & 0.56 & 558 & 9 & 10,080 \\ DLWP-RLPr-None & 0.67 & 1.0 & 0.35 & 0.97 & 0.96 & 0.98 & 0.75 & 0.98 & 0.51 & 690 & 6 & 9900 \\ \hline DLWP-None-ENPr & 0.84 & 1.0 & 0.69 & 0.93 & 0.99 & 0.88 & 0.88 & 0.99 & 0.77 & 147 & 46 & 24,470 \\ DLWP-None-DENPr & 0.84 & 1.0 & 0.67 & 0.94 & 0.99 & 0.89 & 0.88 & 0.99 & 0.77 & 160 & 43 & 23,100 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results for fault detection systems. We consider metrics Precision, Recall F1, and False Positives (FP) and False Negatives (FN). The total misclassification cost is featured in the last column.  
Uncertainty estimation generates uncertainty measures for the associated sample class probabilities, enabling us to develop threshold-based isolation criteria. In particular, a subset of the results, the model predictions with high uncertainties exceeding some preset threshold, are set aside as preliminary outputs and passed on to human experts for cross-checking (see Table 3). The threshold setting for the minimum acceptable level of uncertainty is decided on a case-by-case basis depending on the level of precaution expected from the generated models.  
For our implementation, we make use of the following uncertainty estimation methods applied to the generated class probabilities [80; 81; 82; 83; 84; 85]:  
* Entropy: \(\mathbb{H}(x)=\underset{x}{\arg\max}\Big{\{}-\sum_{i=1}^{C}P(y_{i}\mid x_{i}) \log P(y_{i}\mid x_{i})\Big{\}}\) for a given set of \(C\) point estimates from a model prediction, we use the Entropy [81] method, which computes the entropy of the prediction as a score. The higher the entropy score, the more uncertain the model's prediction.
* Jain's Fairness Index: \(\mathcal{J}(x)=\underset{x}{\arg\max}\Big{\{}\big{(}\sum_{i=1}^{C}P(y_{i}|x_{ i})\big{)}^{2}/C\sum_{i=1}^{C}P(y_{i}|x_{i})^{2}\Big{\}}\) for a given set of \(C\) point estimates from a model prediction, we use the [PERSON]'s fairness index [85] or score defined where \(P(y_{i}|x_{i})\in\mathbb{R}^{C}\). The result ranges from \(1/C\) representing the lowest to \(1\) as the highest fairness score. The higher the fairness score, the more uncertain the model's prediction.  
Uncertainty estimation enables the implementation of a data-informed call to action, only invoking human experts for feedback on a subset of the output results requiring further assessment. This way, we improve operational efficiency by maintaining the number of queries to the human experts at a manageable size while still improving overall system safety.