Kati8/ICLR_2023_2025 huggingface dataset:
authors: Kevin Pasini
authors: Clement Arlotti
authors: Milad Leyli-abadi
authors: Marc Nabhan
authors: Johanna Baro
reviews: 
Comments: In this work, the authors propose a framework for measuring the effectiveness of modern uncertainty quantification methods (UQ) for the task of disentangled uncertainty quantification (dUQ), which attempts to factorize the uncertainty of a model into reducible epistemic uncertainty and irreducible aleatoric uncertainty. To this end, the authors propose the disentangled Epistemic indicator (dE-Ind), corresponding to a ratio of predicted aleatoric to epistemic uncertainty from their qQU model output, and novel epistemic injection schemes for training and inference. The authors demonstrate their framework across 5 different modern ML UQ models and several datasets.
Strengths: - The authors conduct a thorough analysis over recent methods in the field of uncertainty quantification in application to their problem setup.
- The problem of measuring epistemic uncertainty, with no ground truth, is an important question, and the authors initially positioned their work nicely in context with respect to related work.
Weaknesses: - There is a lack of clarity in the story of this paper, and what the authors consider their main contributions. In the introduction, they state that "In this article, we seek to bridge this gap by introducing a dUQ evaluation methodology providing a comprehensive benchmark of various typologies of UQ models." but later on, in section three, they state that "The experimental goal is to highlight an epistemic confidence gap in model predictions, between nominal and altered queries (i.e. affected by injections of epistemic uncertainty)." 
- The writing is very dense and the presentation makes the content more difficult to understand. In particular, Figures 1 and 3 have to much detail and are too difficult to follow in tandem with the text, which is itself not precise and can be quite confusing. Figure 2 is also unclear as to where we should be paying attention, or what insight to draw from it. For an example of a sentence I had difficulty understanding: "These two components are defined for and in a given modeling scope, resulting from the methodological choice of both observed features and model type, which depends on the predictive task’s characteristics".
- The paper has an entire section dedicated to training epistemic injections, but this is never mentioned before as a part of their contribution or utility. The beginning of the paper makes it seem like they offer an evaluation of modern methods, but this along with the meta-model seem to be proposing training procedures?
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: This paper considers the problem of evaluating epistemic uncertainties outputted by a Gaussian auto-regressive model. The authors proposed an injection scheme where the model is asked to make predictions on modified inputs, and the epistemic uncertainty can then be compared between the raw input and modified inputs.
Strengths: This paper examines an important question of evaluating epistemic uncertainty, which remains challenging in predictive ML. On the top level, modifying inputs and checking how the predicted uncertainties vary sounds intuitively promising and could lead to interesting future developments.
Weaknesses: However, on the negative side, this paper has critical issues with the soundingness of the approach and the overall clarity.

For the soundingness of the approach, one fundamental problem in this paper was that the proposed (dE-Ind) didn't get any theoretical justification. While intuitively, the change in the proportion of epistemic uncertainty might be an interesting metric to check, it doesn't solve the essential problem of there isn't a so-called ground truth with epistemic uncertainty. In fact, there has been recent work that indicates there might not be a proper loss for the epistemic uncertainty that can be evaluated with data (https://arxiv.org/abs/2301.12736). Furthermore, the injection schemes to compute the proposed metric also lack support. There isn't a clear discussion on how the proposed injection was the correct one to generate inputs that provide the suitable change of uncertainty. The description of methods is also quite handwavy, as some steps are just given without any background (i.e. use SHAP to modify input features).

On the clarity part, this paper doesn't have consistent mathematical notations and might cause some significant issues for the readers. The boldface x is used both with the time index as part of a time series and with the instance index as a training example. The function notation f and expectation notation E use the bold and non-bold font without any apparent difference. Additional marks like \bar, \hat, * are all directly given without a clear introduction.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: The authors propose a new framework for evaluating different approaches for UQ in regression models.  They propose two types of epistemic variability injections in order to quantify a confidence gap of a model.
Strengths: The authors address the important task on how to quantify the qulaity of UQ methods; I appreciate the introduction of 2 new dataset, a real one and a synthetic one, which helps towards a more realistic evaluation
Weaknesses: - The authors missed a while section of the literature,  on calibration for regression models, which covers a highly related task. In particular [1] demonstrates how calibration quantifies a meaningful tradeoff between sharpness and coverage
- I found the concepts of Inference step injection of epistemic variability injections and Training step injectionare fairly straight-forward concepts, which are presented in a convoluted manner - here the clarity of the paper could be improved substantially
- It's not clear how the choices of the type of "attack" in the inference step rejection influence the conclusion - surely the way outliers are sampled from the tail (how far) is relevant? Similarly, how altered is the altered dataset and hwo does this effect conclusions? To me the main contribution is the benchmark study and this would need a much more exhaustive and systematic set of experiments 


[1] Kuleshov, V., Fenner, N., & Ermon,  S. (2018). Accurate uncertainties for deep learning using  calibrated regression. In International conference on machine learning (pp. 2796-2804). PMLR.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 2 fair