Before revision: We introduce a new approach for capturing model uncertainty for neural networks (NNs) in regression , which we call Neural Optimization-based Model Uncertainty (NOMU).
After revision: We study methods for estimating model uncertainty for neural networks (NNs). To isolate the effect of model uncertainty, we focus on a noiseless setting with scarce training data. We introduce five important desiderata regarding model uncertainty that any method should satisfy. However, we find that established benchmarks often fail to reliably capture some of these desiderata, even those that are required by Bayesian theory. To address this, we introduce a new approach for capturing model uncertainty for neural networks (NNs) in regression , which we call Neural Optimization-based Model Uncertainty (NOMU).