The Predictive Diversity Score (PDS) offers a novel approach to quantifying epistemic uncertainty by measuring the diversity of predictions among ensemble members. Theoretically, PDS can be justified through the lens of Bayesian uncertainty estimation, where the diversity in predictions reflects the model's uncertainty about the true class of an input sample. When models disagree significantly on their predictions, it indicates that the input may lie in a region of the input space that is underrepresented in the training data, thus suggesting a higher likelihood of being an out-of-distribution (OOD) sample.
In comparison to traditional uncertainty estimation methods, such as Bayesian Model Averaging (BMA), PDS directly captures the variability in predictions rather than averaging them, which can mask the underlying uncertainty. While BMA provides a measure of uncertainty based on the average prediction, it may not effectively highlight regions of high epistemic uncertainty, especially in cases where models are overconfident in their predictions. PDS, on the other hand, emphasizes the number of unique predictions, making it a more sensitive metric for detecting OOD samples.
Furthermore, PDS can be theoretically linked to concepts such as entropy and mutual information, providing a solid foundation for its use as an uncertainty metric. By quantifying the spread of predictions across classes, PDS can be shown to correlate with the model's confidence and the likelihood of encountering OOD samples, thus offering a robust alternative to existing uncertainty estimation approaches.