Title: Epistemic Uncertainty Estimation in Regression Ensemble Models with Pairwise Epistemic Estimators
Venue: NeurIPS 2025
Accepted: yes
Keywords: Uncertainty Quantification

Dynamics Modeling

Ensembles
Abstract: This work introduces a novel approach, Pairwise Epistemic Estimators (PairEpEsts), for epistemic uncertainty estimation in ensemble models for regression tasks using pairwise-distance estimators (PaiDEs). By utilizing the pairwise distances between model components, PaiDEs establish bounds on entropy. We leverage this capability to enhance the performance of Bayesian Active Learning by Disagreement (BALD). Notably, unlike sample-based Monte Carlo estimators, PairEpEsts can estimate epistemic uncertainty up to 100 times faster and demonstrate superior performance in higher dimensions. To validate our approach, we conducted a varied series of regression experiments on commonly used benchmarks: 1D sinusoidal data, *Pendulum*, *Hopper*, *Ant*, and *Humanoid*, demonstrating PairEpEsts’ advantage over baselines in high-dimensional regression active learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper utilizes Paired Distance Estimators to efficiently measure epistemic uncertainty in regression ensemble models, i.e. estimating the differential entropy of the mixture distribution.\nSpecifically, two practical implementations based on the Bhattacharyya distance and KL divergence are introduced.\nThey demonstrate the performance of this approach on two synthetic tasks and for active learning on real-world datasets for an ensemble of NFlow base models.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- Good motivation for why using PaiDEs is necessary for estimating epistemic uncertainty in regression ensemble models. While this is especially true for their utilized NF predictive model, this may also be a concern for usual PNEs.\n- While the theoretical results are following the steps of Kolchinsky and Tracey and are not novel per se, they are relevant and interesting for the uncertainty estimation community in machine learning.\n\n## Weaknesses\n\n- The experimental results are unfortunately not very encouraging. The synthetic experiments look reasonable, but the active learning results only look convincing on the Humanoid task. Also adding results for the full dataset would put the active learning results better into perspective.\n- Figure 3 only reports the difference between means. Having variances or CI would put the differences between PaiDEs and MC estimation better into perspective.\n- The runtime issue of MC estimation is a bit unclear to me. Is this a general issue or intrinsic to an ensemble of NFlows base models? Because I don't see how sampling would be so much slower when given the output distributions of members of the PNE. Also, I did not find a time comparison for PNEs.\n\nQUESTIONS:\n- How does PairEpEst-KL relate to the Expected Pairwise KL-divergence discussed in Malinin et al. 2021 and Schweighofer et al. 2023 as measure of epistemic uncertainty in the classification setting? Is it an extention to the regression setting or is there a fundamental difference between those quantities?\n\n- Given the multimodel output distribution of an ensemble of NFlows base, how was the point prediction for calculating the RMSE obtained?\n\n- Why not reporting the MC estimate in Figure 2, when this is done in Figure 8 in the appendix? \nAlso, why does the estimated uncertainty look different in the two figures?\n\n- Why are many different colors introduced for different significant levels in Table 1, when there are only two entries with p < 0.01 and no entries that are significant for p < 0.05 and p < 0.001?\n\n- Why was a uniform weighting of components, i.e. $\\pi_j = 1/M$ (line 198) chosen? Are there alternative ways to weight ensemble members that could be interesting?\n\n\nMalinin, A., & Gales, M. (2021). Uncertainty estimation in autoregressive structured prediction. \n\nSchweighofer, K., Aichberger, L., Ielanskyi, M., & Hochreiter, S. (2023). Introducing an improved information-theoretic measure of predictive uncertainty.\n\nLIMITATIONS:\nI don't agree that the stated limitation of introducing bias in the estimation of epistemic uncertainty (line 287) does not affect the relative ordering of points.\nThe bias is input dependent as can also be seen in Figure 8, thus might change the relative ordering of individual inputs for active learning, unless I missed some technical result on the monotonicity of the bias depending on $H(y|x)$?\nNevertheless, it could be that the statement is true in practice, but I would like to see an experiment looking at ranking correlations between PairEpEsts and the MC estimate with a very high number of samples (to rule out MC variance) on the ordering of inputs on one of the tasks to confirm this.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Pairwise Epistemic Estimators, a new method for estimating epistemic uncertainty in regression tasks. This approach leverages an ensemble of models derived from Normalizing Flows and uses pairwise distance estimations to quantify uncertainty. The authors provide theoretical grounding, demonstrating that PaiDEs establish bounds on entropy and enhance the performance of the BALD method. Notably, PaiDEs efficiently estimate uncertainty using only a few members in ensemble, making them computationally faster than Monte Carlo-based alternatives. The effectiveness of the method is demonstrated on the active learning task on two 1D and four multidimensional regression datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper is well-structured with clear clearly written, providing a comprehensive background on normalized flows and probabilistic network.\n2. The paper offers theoretical justification for how PaiDEs establish entropy bounds using various generalized distance functions and leverage these to enhance the BALD method.\n3. The authors validate the proposed epistemic uncertainty estimation method on simple 1D regression tasks, showing expected uncertainty behavior across different regions. More importantly, they demonstrate its effectiveness for active learning in multidimensional regression tasks.\n4. Additionally, the paper highlights the computational efficiency of the method, achieving competitive or superior performance to Monte Carlo-based approaches with a small ensemble.\n\n\n\n**Weaknesses:**\n1. The approach for ensuring diversity of the ensemble (lines 152–153) is not clearly explained. Specifically, how is the fixed dropout mask computed? Is a different mask generated for each ensemble member? Furthermore, it is unclear for what part of the ensemble randomization or bootstrapping is applied. \n2. The computational efficiency of the proposed approach appears to result from a smaller committee size, and even this small ensemble achieves competitive results. However, it remains unclear how varying the committee size impacts active learning performance. While Figure 5 presents computational time across different ensemble sizes, including active learning performance metrics for the same range would significantly strengthen the empirical analysis. \n3. The paper would benefit from comparisons with additional baselines. In particular, methods based on Monte Carlo [1]. Moreover, an adaptation of least confidence method, which commonly used in classification, could be applied, given that the model outputs a distribution. \n4. Minor issue: there is a typo in line 118 “wtighter” should be corrected to “tighter”.\n\n[1] https://arxiv.org/pdf/1806.09856\n\nQUESTIONS:\nGiven that the proposed method only requires receiving a probability distribution from each ensemble member, could it be applied to other domains, such as image classification using Bayesian Neural Networks?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an alternative to sample-based estimation of epistemic uncertainty via pairwise distance estimators. The paper derives estimators based on two measures, Bhattacharyya distance and Kullback-Leibler divergence. Across two 1d synthetic examples and high-dimensional active learning tasks, the paper shows that the proposed sample-free estimators yield improved performance at matched compute budgets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper correctly identifies the computational burden of MC in most settings. Even though in many active learning settings, sample efficiency is the main bottleneck, improving compute efficiency can allow for screening more options in very high-dimensional settings.\n* The experimental evaluation is solid, with a range of settings from low-dimensional to high-dimensional.\n\nWeaknesses:\n* The theoretical contribution beyond reference [35] appears limited. Nevertheless, the manuscript could benefit from a clearer outline of the novelty of the analysis.\n* I am missing a comparison of MC and the proposed methods in terms of compute cost (time). For example, in Figure 3, it would be useful to indicate the #samples for MC such that the overall compute cost in terms of time is matched between MC and PaiDEs.\n* While the experimental evaluation is solid, the conclusion around the benefits of sample-free estimation in high-dimensional tasks would be more significant if evaluated also on another (very) high-dimensional task beyond Humanoid. It would also be nice to see an evaluation on a common benchmark for active learning (e.g., [1]).\n\nTypos:\n* line 118: \"wtighter\"\n\n[1]: Navigating the pitfalls of active learning evaluation: A systematic framework for meaningful performance assessment (https://proceedings.neurips.cc/paper_files/paper/2023/file/1ed4723f12853cbd02aecb8160f5e0c9-Paper-Conference.pdf)\n\nQUESTIONS:\n* In Figure 2, it appears that also a count-based method could perform well, at an even lower cost. I think this could be worth mentioning.\n* In line 103, why does $H(\\theta \\mid y, x) \\geq 0$ hold, given that this is a differential entropy?\n* In Figure 5, for which #samples is MC evaluated here? For which #samples would MC have the same compute time as Bhatt/KL?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes a novel estimator for epistemic uncertainty estimation in ensemble models for regression tasks, which does not require Monte Carlo sampling. The estimator (Pairwise Epistemic Estimators) is based on Pairwise-Distance Estimators (PaiDEs) to obtain a lower/upper bound for the mutual information (or equivalently, the epistemic uncertainty). The PaiDEs require closed-form expressions for the divergences/distances between distributions, which motivates the authors to use Normalizing Flows (NFs) to model the distribution and reduce the target distributions' premetrics to the base distributions. The algorithm is validated by active learning experiments compared to other uncertainty-based baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The proposed method is sampling-free, which will boost the computation once we are in a high-dimensional space.\n\n2. The theoretical understanding of the algorithm is solid.\n\n3. The empirical evidence is impressive.\n\n4. The presentation is clear and easy to follow.\n\nWeaknesses:\n\n1. The method only applies when all premetrics are of closed-form expressions. This limits the applicability in many scenarios. Also, the numerical experiments are all conducted in Gaussian bases. It remains unclear how robust it is with skewed or heavy-tailed distributions.\n\n2. The estimator of the epistemic uncertainty is biased (e.g., either a lower bound or an upper bound in the two examples). It shall be fine if the uncertainty is only an intermediate product (e.g., uncertainty-based active learning), but if one wants the epistemic uncertainty estimator as the final output, it becomes a big problem. The paper also lacks an analysis of the potential bias.\n\n3. The active learning algorithm benchmark is restricted. Uncertainty-based query strategy is only one category of active learning algorithms; it remains unclear if they are SOTA in many scenarios. More comparisons (e.g., coverage/representation-based active learning algorithms) should be included. (update: solved)\n\n4. The theoretical contribution of this paper is limited. All the theory parts are from the previous PaiDE paper.\n\nQUESTIONS:\n1. Have you ever tried other active learning benchmark algorithms? (See Weaknesses point 3)\n\n2. How precise is your evaluation of epistemic uncertainty (as a final output rather than an intermediate product)?\n\n3. Typos: line 118 \"wtighter\" should be \"tighter\"?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}