Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed Data-IQ, a framework compatible with any machine learning (classification) models whose training is conducted in stages (iterations/epochs), to assess the quality of data samples in tabular format by categorizing them into Easy, Ambiguous, Hard groups based on the aleatoric uncertainty of the data samples. The paper demonstrated the utility of Data-IQ in guiding feature acquisition, comparing training datasets and improving model generalization through experiments and analysis on multiple real-world datasets.
strengths_and_weaknesses: **Originality:** the idea of using prediction outcomes to quantify the uncertainty of data samples is not new, e.g. in some active learning approaches the next queried data point would be the sample with predicted probability close to 0.5 as it is the most uncertain. The novelty of Data-IQ lies in that it quantifies the aleatoric uncertainty of data samples by evaluating the variance of the prediction outcome which intends to capture the inherent uncertainty of the data samples rather than the uncertainty with regard to the model parameters. Therefore, Data-IQ is capable of providing guidance to improve data quality in new directions with regard to the dataset itself, e.g. new feature acquisition.

**Quality:** the evaluation of the variance of the prediction outcome is straightforward, by noting that the classification outcome is a Bernoulli random variable, the variance is $p(x, \theta_e)(1 - p(x, \theta_e))$ in which $p(x, \theta_e)$ is the predicted probability using the model parameters $\theta_e$ at training epoch $e$. Then the aleatoric uncertainty is quantified as the average variance over all training epochs and categorization of samples is based on thresholds chosen by heuristics. The analysis of the categorization on real-world datasets is quite comprehensive, e.g. how to use Data-IQ to assess the quality of a new feature and how the Ambiguous samples will impact model generalization etc., but with regard to methodological perspectives, the technical novelty / contribution is not significant.

**Clarity:** the paper is well written and easy to follow. All the figures and tables are very clear.

**Significance:**  I think the research topic in this paper is of high practical value as the assessment of data quality (uncertainty) is very important for training high-quality machine learning models and obtaining reliable predictions on unseen data. However, as stated in 
**Quality**, the methodological / theoretical novelty of the method is not very high.  
questions: 1. The aleatoric uncertainty is quantified as the average variance over all training epochs. I am wondering how the distribution of the variance of a single data sample over the training epochs looks like? If the distribution of one training example is skewed, e.g. the variance is high at the beginning but it becomes much lower as the model parameters are refined during training, and the variance of another training example remains at a medium level during training, these two training examples might end up with similar average uncertainty but the training dynamics are very different. In this case, would the average variance still be a good metric to evaluate the uncertainty of an individual data sample?


2. The evaluation are mainly conducted on neural networks, which demonstrates that Data-IQ is robust to different parametrizations of the same model type. Would it be possible to compare the results across different model types? Without comparison across model types, e.g. neural networks v.s. gradient boosting, the claim that Data-IQ is robust to model variation is less convincing as the aleatoric uncertainty might still be pertinent to the model class (though not to different parameters under the same model type).
limitations: The authors have addressed the limitations and I think another potential limitation is how robust Data-IQ will be across different model types as stated in Q2 in **Questions**.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors make a strong case that population level metrics for model performance may not be representative of model performance on subgroups. The authors proposed `DataIQ` a framework to phenotype and subgroup patients based on model uncertainity on an individual level. The authors apply their proposed technique on multiple real world healthcare datasets and demonstrate ability of the model to 1) identify robust data 2) Data collection 3) Model deployment. 
strengths_and_weaknesses: Overall I like this paper, even though there are a few weaknesses. 

# Strengths

- The paper describes provides several reasons to model model uncertainty and conducts several ablation experiments to justify each use case. 
- The multiple ablation studies, including the change in estimated uncertainty with addition of features (Figure 6)  as well as domain shift (Figure 7) are welcome. 
- The comparison of output of the model along a with Group DRO objective to improve subgroup level performance adds to the completeness of an already exhaustive set of experiments.

# Weaknesses

- The way the paper proposes to estimate the variance around the model outcomes is by assuming the model parameters at the end of each epoch to be sampled from a distribution. While interesting such a choice is not characteristic of an empirical distribution in the strictest sense. In this case the model parameters are going to be highly correlated which would break the assumption of sampling from an IID distribution.
questions: - Have the authors considered other potential sources of incorporating randomness for the aleatoric uncertainty they are trying to capture over the distribution of $v$, for example perhaps the use of a bayesian prior on $v$ or dropout masks? 

limitations: As indicated above the distribution of $v$ is intimately tied to the model choice and the optimization approach. Even when the model class is the same, the hyper parameter choices for the optimizer like step size, momentum, might lead to vastly different distributions in the weights. 

Since the authors propose to use the estimated groups as 'protected groups', it would be welcome to see more extensive discussion of how these choices effect recovery of the estimated groups.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present an approach that given a model can identify 3 general sub-groups in the dataset that differentiates the model performance across the dataset. In particular they analyzed tabular data to find such sub-groups that are useful for meaningful data-driven AI and satisfies requirements from an MLOps perspective. They conducted experiments and demonstrated various aspects of the proposed method. Overall, their proposed method aims at informing reliable model usage. 
strengths_and_weaknesses: The are several interesting aspects of this paper

- the proposed method of identifying the sub-groups using the training performance of the model using the data uncertainty aspect is while simple, is intuitive and more importantly the presented analyses supports the importance of this formulation. 
- The identified 4 paradigms for a useful sub-group discovery is interesting, especially from a MLops perspective. The paper is masterfully presented that describes and motivates each of these paradigm and analyzes how the proposed method addresses each of these requirements. Especially, the  plug and play aspect of this method is very appealing. The results presented in the appendix comparing the classical GBT models is also interesting (perhaps could be included in the main paper). 
- The insights about the importance of the `Ambiguous` groups is perhaps the most interesting aspect of the paper. Pertinently the novel usage of this identified sub-group for aspects such as identifying the most important features to comparing the `value` of a dataset is interesting. 

While the paper itself is quite interesting, there are a few aspects to improve upon:
- the Figures are very low resolution and hinders readability. While acknowledging the space limitations, some of the sections/paragraph could be moved to supplementary section. For example, Figure 4 and the corresponding paragraph comparing agains DataMaps is quite intuitive and may not need to be part of the main paper. The y-axis is closely related to the metric for $v_{al}$ and the shape is quite expected. Similarly, Fig 8 can be argued to be intuitive and a direct consequence of the selected measure.
- Table 1 and the corresponding paragraph may need further context. The discussion ignores the context around the fidelity of the synthetic data to actual ground truth. E.g. guarding against trivially Easy data
- Finally, while the paper is quite well written, the actual method to calculate the Easy, Ambiguous, and Hard groups is not presented properly - it should be present at least as an algorithm in the supplementary section. 
questions: Some questions for the authors:
- while the authors rely on the training dataset performance, its not evident that their results and/or formulation is dependent on it. For example, the results in the supplementary section shows how the $v_{al}$ stabilizes after a few epochs. One can also draw corollary from the steady state dynamics for state space model and so on. Equation 1 also approximates the aleatoric uncertainty as an expectation. One can argue that for converged models, the steady state value may be enough. In contrast would a burn-in period have an impact on this estimate?
- It seems that the entire formulation is true while comparing models of similar performance. How does this formulation play into Eqns (1) and (2).
- Section 4.3: How many ambiguous datapoints were in each dataset? The absolute number is important while analyzing the importance of the effect of removing such datapoints on model performance
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a framework to identify examples in the datasets that a classifier tends to distinguish correctly, wrongly, and randomly.  The paper decomposes the accuracy of a given example into a summation of epistemic uncertainty and aleatoric uncertainty. The paper then uses aleatoric uncertainty to classify examples into the three aforementioned categories, as opposed to existing work that uses epistemic uncertainty as the criterion.
strengths_and_weaknesses: Strengths:
* the paper deals with a significant problem towards a better understanding of how difficult a data point is to classifiers in terms of classify it accurately.
* The proposed approach that decomposes the accuracy of a given example into a summation of epistemic uncertainty and aleatoric uncertainty is clean and informative of the difficulty of classifying a given example.
* The presentation of the paper is clear. Experiments are illustrative.

Weakness:
* Given the generality of the proposed method, the paper may consider evaluating the proposed method beyond tabular data.

questions: * In equation (2), is the estimate based on the empirical process a good estimate for the two uncertainty quantities given that the samples in the empirical process might be correlated with each other rather than iid?
limitations: The paper discusses the limitation of the proposed method. Finding the attributes that are responsible for the hardness of the data points seems to be particularly interesting and relevant for the interpretability of machine learning algorithms.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors proposed a framework of data characterisation and data sculpting during model training based on the outcome and feature characteristics of the subgroups within a dataset. The authors separate the subgroups into the Easy, Ambiguous and Hard groups. Via a set a of experiments on different datasets, they showed that the the framework was able to identify the uncertainties due to the data characteristics, insights regarding the subgroup feature characteristics, and can potentially improve model generalisability by excluding Ambiguous group from training.
strengths_and_weaknesses: The authors provides a different perspective, which puts the characteristics of data in the centre, to analyse the factors that influences the performance of a model. They provide a tool, Data-IQ, to help the researcher understand data better before diving into modelling process. To me, it is in innovative and should be encouraged. However, what the authors propose might not be sufficiently mature and can be used as framework for assessing datasets and guide model training process by the community. Regarding the writing, the paper was trying to put too much ideas into the limited length, including variation vs robustness, subgroup characterisation, model generalisability, etc.. It would be much clearer if there could focus on one single point, e.g. data sculpting based on the identified subgroups.
questions: 1. To my understanding, the "Hard" group are sample with similar feature as the "Easy" group, but opposite outcomes. How is this different from the "Ambiguous" group, which includes samples that have similar features but different outcomes? 
2. Regarding the data insights, are the findings clinical meaning full? Are their any evidences (related biomedical researches), can be referenced?
3. The improved in Table 2 is marginal. Also the comparison involves two factors: Data-IQ and modelling method Group-DOR. Could the authors show a baseline model + Data-IQ?
limitations: The authors discussed the limitations of the method in the section 5. However, the discussion was limited, maybe due to the length of the paper. I personally would be interested to understand how the framework may influence the inclusivity of the dataset used for model training.
review_title: null

Meta-review: metareview: This easy-to-follow paper has been thoroughly evaluated by five competent reviewers. Four of them rated the work as acceptable (two full and two weak accepts), while one recommended a rejection. In my opinion, the reviewer with the negative assessment has not raised fundamental issues that would disqualify this paper from being considered for NeurIPS. The authors provided extensive clarifications to all the reviewers, including the one with a negative opinion. That reviewer did not engage in discussion with the authors. I recommend accepting this paper without reservations
confidence: Certain
Decision: Accept