Title: TRIAGE: Characterizing and auditing training data for improved regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: data-centric AI

data characterization

data quality
Abstract: Data quality is crucial for robust machine learning algorithms, with the recent interest in data-centric AI emphasizing the importance of training data characterization. However, current data characterization methods are largely focused on classification settings, with regression settings largely understudied. To address this, we introduce TRIAGE, a novel data characterization framework tailored to regression tasks and compatible with a broad class of regressors. TRIAGE utilizes conformal predictive distributions to provide a model-agnostic scoring method, the TRIAGE score. We operationalize the score to analyze individual samples' training dynamics and characterize samples as under-, over-, or well-estimated by the model. We show that TRIAGE's characterization is consistent and highlight its utility to improve performance via data sculpting/filtering, in multiple regression settings. Additionally, beyond sample level, we show TRIAGE enables new approaches to dataset selection and feature acquisition. Overall, TRIAGE highlights the value unlocked by data characterization in real-world regression applications.

TRIAGE: Characterizing and auditing training data for improved regression

Abstract

Data quality is crucial for robust machine learning algorithms, with the recent interest in data-centric AI emphasizing the importance of training data characterization. However, current data characterization methods are largely focused on classification settings, with regression settings largely understudied. To address this, we introduce TRIAGE, a novel data characterization framework tailored to regression tasks and compatible with a broad class of regressors. TRIAGE utilizes conformal predictive distributions to provide a model-agnostic scoring method, the TRIAGE score. We operationalize the score to analyze individual samples' training dynamics and characterize samples as under-, over-, or well-estimated by the model. We show that TRIAGE's characterization is consistent and highlight its utility to improve performance via data sculpting/filtering, in multiple regression settings. Additionally, beyond sample level, we show TRIAGE enables new approaches to dataset selection and feature acquisition. Overall, TRIAGE highlights the value unlocked by data characterization in real-world regression applications.

1 Introduction

Data characterization, an important problem. Over a decade ago, it was recognized that 'more data beats clever algorithms, but better data beats more data'[1], emphasizing the key role of training data in the performance and robustness of ML algorithms [2-4]. In particular, training data from the 'wild' may have various errors or lack coverage, rendering it unsuitable for the ML task [5].

These observations motivate the need for data characterization , which aims to score each training sample based on its impact on the performance of the ML task. The training data can then be partitioned or rank ordered according to the score to give a high-level description of the sample and its utility. Data characterization provides a natural way for data sculpting [6, 7], which filters training samples that have a low or even detrimental effect on the ML task. The goal is that models built on the sculpted/filtered (and smaller) training data have improved performance.

In this work, we address the understudied data characterization problem in regression settings . In contrast, existing works on data characterization have primarily focused on the classification setting [8-13]. These methods leverage the probability distribution over discrete labels (softmax or logits) for scoring. However, in the equally important regression setting, the probability distribution over the continuous labels are not typically given by standard regression models. In fact, the majority of the regression models only output a point estimate, e.g. the predicted mean. Without access to probabilistic estimates, one cannot simply apply the existing classification-tailored methods for

Over

Well

Under

Trained ML model

with checkpoints

Epochs (NN)/Estimators (GBT)

Analyze TRIAGE training dynamics

Compute TRIAGE scores via CPS

i

iii

ii

Characterize samples

Under

CPS Probablity

Retrain & use w/

Over

Under

downstream tasks

Data sculpting/Filtering

discrete labels to the regression setting. We provide further details why classification-tailored methods are inapplicable to the regression setting in Sec. 2 and Table 1. This non-applicability motivates the need for regression-tailored methods for data characterization, which are 'not only principled, but also practical for a larger collection (...) of ML models' [4].

To anchor the need for data characterization in regression, let's consider a few concrete examples. In predicting house prices, an outlier might be a house with an extremely high price due to a unique or under-represented feature in the data. This could affect the model's performance, causing it to overestimate prices for similar houses without the unique feature. Alternatively, imagine predicting patient length of stay based on clinical features. If the dataset contains some erroneous features, the regression predictions may be suboptimal, potentially leading to incorrect clinical decisions.

In designing a data characterization framework for regression, we outline the following desired properties, drawing motivation from the capabilities of works in the classification setting [8-13]:

(P1) Consistency: The characterization of data examples should be stable and consistent across runs and for similar performing models with different parametrizations.

(P2) Improve performance: The characterization should guide the improvements to model performance. For instance, via data sculpting/filtering.

(P3) Informative for data collection: The characterization score should be informative to enable selection between datasets, as well as, feature collection/acquisition.

(P4) Regressor versatility: The characterization should be principled and practical for a large collection of ML models (e.g. not just neural networks).

To fulfill P1-P4 , we propose a framework to characterize data in regression settings, called TRIAGE (s.f. TR aining I dentification & A uditing of G ood E xamples) - see Fig.1. TRIAGE introduces a novel regression scoring method -TRIAGE score , building on the recent development of Conformal Predictive Systems (CPS) [14]. CPS is a model-agnostic method that can turn the output of any regressor into a probability distribution for continuous labels, termed conformal predictive distribution (CPD). Unlike point estimates such as errors/residuals or conventional conformal prediction intervals, the CPD provides richer information through a full predictive distribution. This enables us to assess the probability of a prediction being over- or under- estimated.

We then operationalize the CPD (and TRIAGE score) in a novel way for data characterization, computing the trajectory of TRIAGE scores over training checkpoints, as a measure of how samples are learned differently during training. Such training dynamics , shown in Fig 1 and have been shown to contain a salient signal about the value of the sample to a learner [15-17]. By analyzing the training dynamics, we then categorize training samples as: under-, overor well-estimated by the model. This is useful going back to our motivating example - where erroneous values could cause the model to overestimate house price. Using the TRIAGE score, we can identify such a sample, as it would likely have a low probability of being correct based on the model's predictive distribution. Such identification unlocks a variety of use cases. For instance, sculpting/filtering such samples can improve model performance (even with less data), as we empirically show in Sec. 5.

Contributions. 1 ⃝ Conceptually , we present TRIAGE, the first data characterization framework tailored to regression settings, addressing the need for an ML-aware data quality that is principled and practical across a large collection of ML regressors. 2 ⃝ Technically , we introduce a principled regression scoring method called TRIAGE score, that leverages conformal predictive distributions in a novel way for data characterization. We operationalize the score to analyze how different samples are

learned during training, thereby permitting characterization of individual samples as: under-, overor well-estimated by the model. 3 ⃝ Empirically , we demonstrate the utility of TRIAGE across multiple use cases satisfying P1-P4 , including consistent characterization, sculpting to improve performance in a variety of settings, as well as, guiding dataset selection and feature acquisition.

2 Related work

This work engages with recent efforts in data-centric AI [6, 7, 19, 20] on data characterization. However, prior methods have focused on classification settings. In contrast, TRIAGE addresses the regression setting where tailored data characterization has not been explored. We now discuss related works and outline the challenges of applying prior classification focussed methods to regression, for an overview see Table 1. Prior classification scoring methods can be divided into two groups:

■ Tailored methods , assign scores based on aspects specific to classification, including logits [12], probabilities over labels [8, 9] or changes to predictions on discrete labels [10, 21]. However, these methods are inapplicable to our regression setting because these aspects are not present in regression. ■ General purpose methods , while focussed on classification, they assign scores using generic measures and can be repurposed for regression. However, these general methods based on backpropagated gradients [11, 13] or losses [18] have two limitations. Firstly, scores like gradients can only be used with differentiable models (e.g. neural networks) and are not compatible with many widely used regressors, such as XGBoost - i.e. does not satisfy (P4) Versatility . Secondly, even if they can be used, general-purpose scores may not distinguish between samples that have similar magnitude scores (e.g. high loss scores), even if the underlying reasons for these scores may be different with respect to the task. We show the value of such a differentiation between samples in Sec.5.2.

Data valuation. A related area to data characterization is that of data valuation, for example: Shapley based valuation methods [22, 23]. We first outline some conceptual differences: (a) TRIAGE unlocks other data-centric tasks beyond just data selection, e.g. feature-level acquisition/collection and selection between datasets. These are out of scope for data 'sample' selection; (b) Unlike TRIAGE, these methods are not tailored to regression; (c) method differences: computationally Shapley-based methods need to assess multiple sample permutations and need to retrain the model many times. Hence, they struggle to scale to very high-samples sizes (e.g. 100k), unlike TRIAGE where the cost is cheap comparatively. Recently, a valuation method LAVA [24] has been introduced to address this which uses an additional embedding model to reduce dimensionality before the optimal transport step. Beyond conceptual differences, we compare TRIAGE experimentally and in terms of computation time in Appendix C.

3 Data Characterization Preliminaries

In this section, we formulate the data characterization problem for regression settings. Consider the standard regression setting, where the aim is to assign an input x ∈ X ⊆ R d X to a continuous valued label y ∈ R (often a point estimate for y ). This is done using a regressor f θ : X → Y , parameterized by θ ∈ Θ . We assume f θ is trained on a dataset D with M ∈ M ∗ observations, i.e. D = { ( x m , y m ) | m ∈ [ M ] } . For the purposes of later notation, we consider a sample space Z := X × R , where each observation z m = ( x m , y m ) ∈ Z . Typically, parameters θ are learned in regression, via empirical risk minimization (ERM) with a mean squared error (MSE) loss, i.e. ERM ( θ ) = arg min θ ∈ Θ 1 M ∑ M m =1 ℓ ( x m , y m ; θ ) , with loss function ℓ : X × Y × Θ → R + .

ERMensures regressor f θ performs well on average , however it goes without saying that the model is unlikely to perform equally well for all samples in the dataset. Consequently, we desire to characterize such types of samples. Formally, for each sample x m , we aim to assign a group label g m ∈G , where

G = { Over-estimated, Well-Estimated, Under-estimated } . We then use these groups for downstream tasks, or to sculpt/filter samples in the dataset D based on their group assignment. Before giving a precise description of how the group labels are assigned, we provide some context.

Prior works have shown the behavior of samples over the course of model training, called training dynamics contains signal about the nature of a sample itself [15-17]. For instance, some samples are easily learned and stable through training, whereas other samples might be challenging and have high variance. Many classification-based methods leverage this concept and assign g m by analyzing training dynamics e.g. Data-IQ, Data Maps, AUM etc. However, as discussed in Secs. 1 and 2, the aforementioned classification methods do not apply to regression. Consequently, this requires us to formalize a new principled scoring method for regression. After that, we can operationalize the scoring method to analyze the training dynamics and assign group labels g m to each x m .

For regression, such a principled scoring method could evaluate the probability that the true target y is less than or equal to observation f ( x ) , where the categories of samples are: 1 ⃝ Under-estimated : P ( y ≤ f ( x )) ≫ 0 . 5 , 2 ⃝ Well-estimated : P ( y ≤ f ( x )) ∼ 0 . 5 , 3 ⃝ Over-estimated : P ( y ≤ f ( x )) ≪ 0 . 5 . In the next section, we discuss how we design TRIAGE to achieve this, with the final goal in mind - assigning group labels g m to each sample x m .

4 TRIAGE: characterizing data for regression settings

We now introduce TRIAGE, a regression-focused data characterization framework, compatible with any iteratively trained ML regressor (Sec. 4.4). TRIAGE follows a three-step procedure:

Compute TRIAGE scores (Sec 4.1): Compute the TRIAGE score for every sample x m using conformal predictive distributions. Do this for every regression checkpoint to obtain a trajectory.

Analyze TRIAGE score training dynamics (Sec 4.2): Operationalize the TRIAGE score for data characterization by analyzing the TRIAGE score's trajectory for each training sample.

Characterize & assign groups (Sec 4.3): Characterize samples based on their training dynamics, assigning each sample x m to a group g m . The groups inform filtering/sculpting or other use cases.

4.1 Computing TRIAGE scores via Conformal Predictive Systems (Step 1).

Recall, we want to compute P ( y ≤ f ( x )) as a principled scoring method to characterize samples. We want to do so without changing the chosen regressor type, i.e. P4: Regressor versatility . Of course, the full information of an uncertain label exists in its probability distribution, allowing us to compute probabilities of various outcomes. Unfortunately, regular regressors ( point predictions ) or conventional conformal regressors ( prediction intervals ) do not provide this distribution. While Bayesian methods offer distributions, they require fundamentally changing the regressor.

These challenges motivate defining TRIAGE around Conformal Predictive Systems (CPS)[14]. CPS combines predictive distributions [25, 26] with conformal prediction [27], crucially enabling us to compute the required probability distribution of a continuous variable-called Conformal Predictive Distribution (CPD). CPS also fulfills our versatility goal (i.e. P4 ) as it applies to any regressor. To clarify Conformal Predictive Systems output valid cumulative distribution functions, which are termed Conformal Predictive Distributions (CPD). This is the cumulative probability with respect to a label y , given some x and regressor f . With CPDs denoted as Q , the conformal p-values get arranged into a probability distribution which has the properties of a CDF - thus essentially becoming probabilities, see [28]. Since the CPD has the properties of a CDF, we use the CPD to estimate probabilities that the true target is less than or equal to a specified threshold/value.

Computing CPDs. TRIAGE formalizes the CPD computation in the framework of the computationally efficient Split Conformal Predictive Systems [28]. A proper training set D tr = { ( x i tr , y i tr ) = z i tr | i ∈ [ M ] } is used to train regressor f . A separate, often smaller, calibration set D cal = { ( x i cal , y i cal ) = z i cal | i ∈ [ q ] } , is used for conformal calibration. Finally, we define a conformity measure µ , where µ : Z → R ∪ {-∞ , ∞} , quantifies the dataset's agreement with the observation. For each label y i cal | i ∈ [ q ] , we compute q conformity scores α i = µ ( z i cal ) . We define the conformity measure ( µ ) as per Eq. 1, which is a balanced isotonic conformity measure. We discuss this choice in Appendix A.3, a necessary condition s.t the CPD can be interpreted as a probability distribution [28] 1 .

where y is the label, f ( x ) a predicted label, σ an estimate of prediction quality, computed with a separate normalization model, enabling CPDs to reflect task difficulty. σ is kNN fitted on the residuals of D cal, similar to [29].

We compute the CPD denoted Q m (Eq. 2) for each sample x m via Algorithm 1, where the CPD is defined for y ∗ ∈ R [28]. We extract the TRIAGE score using the CPD. We repeat the CPD computation for all samples. Next, we describe our novel approach to operationalize the TRIAGE scores, for the purpose of data characterization.

Algorithm 1 Computing a CPD

Require: Training set ( x i tr , y i tr ) ∈ D train , i ∈ [ M ] and Calibration set ( x i cal , y i cal ) ∈ D cal , i ∈ [ q ] . 1: Regressor f is trained on D train . ▷ Training 2: Compute residuals on D cal : R i = y i cal - f ( x i cal ) ,i ∈ [ q ] 3: Define σ ( x ) ; which finds the average residuals of the k-nearest neighbors x , where KNN is fit on D cal . 4: Calculate conformity scores α 1 ,α 2 ...α q over D cal using µ ; i.e. α i = µ ( x i cal ,y i cal ) ,i ∈ [ q ] . ▷ Calibration 5: Sort α 1 ,...α q in ascending order ; s.t α (1) <...<α ( q ) for eval sample x ∈ X train do ▷ Evaluation Let S ( i ) := f ( x ) + α i σ ( x ) for i ∈ [ q ] . Let S (0) := -∞ and S ( q +1) := ∞ . end for Return the CPD as per Equation (2).

where τ ∈ [0 , 1] , i ′ := min { j | S ( j ) = S ( i ) } and i ′′ := max { j | S ( j ) = S ( i ) } to account for ties.

4.2 Analyzing TRIAGE score trajectories - training dynamics (Step 2)

Novel use of CPD's for data characterization. The TRIAGE score , denoted as T ( x, y, θ ) = P ( y ≤ f θ ( x )) , represents a novel use of CPDs for data characterization, diverging from their original predictive uncertainty use. Additionally, while CPDs are normally calculated post-model fitting, we observe from Fig.1(ii) that samples' learning varies, leading to different convergence rates of scores. As discussed in Sec. 3, such dynamics reveal sample characteristics [15-17]. Hence, we propose to examine TRIAGE score trajectories to capture the learning dynamics of different samples.

To enable this analysis, we utilize the model checkpoints of any ML regressor trained over iterations (e.g. neural networks, XGBoost etc). Specifically, during iterative training, the model parameters θ vary over E ∈ N ∗ iterations, taking E different parameter values at each checkpoint, i.e. θ 1 , θ 2 , ..., θ E . We wish to account for the impact of these changing parameters. At each checkpoint θ e , we compute the CPD to obtain the TRIAGE score for each sample x m . We repeat the computation for every checkpoint to obtain a TRIAGE score trajectory over checkpoints per-sample of dimensionality E . i.e. we evaluate the function Q for a specific f θ e ( x ) to get the estimated probability P ( y ≤ f θ e ( x )) . We repeat for all f θ e ( x ) checkpoints where e ∈ E to get the trajectory of TRIAGE scores for sample.

To quantify and contrast the behavior of the different sample trajectories, we compute two metrics to summarize the per-sample trajectory (see Fig. 2), namely (1) Confidence : reflects the degree of confidence wrt to P ( y ≤ f θ ( x )) . It is computed as the mean of the TRIAGE score over training, i.e. C ( x i , y i ) = 1 E ∑ E e =1 T ( x i , y i , θ e ) . (2) Variability : reflects the consistency or fluctuation of the score for a specific sample. It is computed as the standard deviation of the TRIAGE score over training, i.e. V ( x i , y i ) = √ ∑ E e =1 ( T ( x i ,y i ,θ e ) -C ( x i ,y i )) 2 E . We describe next how these two metrics form the basis of sample characterization and group assignment.

4.3 Characterization & group assignment (Step 3)

We now discuss how the two characterizing metrics, C and V , are used to assign a group label g ∈ G to each training sample x m . Let's intuitively define each group. 1 ⃝ Under-estimated (UE) : low variability V (i.e. decisive about) and HIGH average TRIAGE score C , i.e. P ( y ≤ f ( x )) ≫ 0 . 5 , 2 ⃝ Over-estimated (OE) : low variability V (i.e. decisive about) and LOW average TRIAGE score C , i.e. P ( y ≤ f ( x )) ≪ 0 . 5 , and 3 ⃝ Well-estimated (WE) : average TRIAGE score C is uncertain P ( y ≤ f ( x )) ∼ 0 . 5 . In addition, these samples are often linked to high variability, implying uncertainty about P ( y ≤ f ( x )) . We empirically observe samples with these different traits per group, depicted in the characteristic curve, see Fig. 2.

We take a threshold-based approach to assign g m , similar to previous characterization works [8, 9]:

where C up and C low are upper and lower confidence threshold resp. and P n the n-th percentile. We discuss the selection of C up and C low in Appendix A.

In Sec. 5, we empirically show how the groups assigned to each sample can be used to sculpt/filter the training dataset or for other downstream tasks such as dataset selection or feature acquisition.

4.4 Using TRIAGE with a variety of regressors (P4)

The general-purpose characterization methods discussed are largely compatible with differentiable models (neural networks). However, in real-world settings (e.g., healthcare, finance), practitioners often employ other iterative algorithms, like XGBoost/GBDTs [30]. TRIAGE addresses this limitation, offering a model-agnostic approach using CPS - as the conformity scores can be computed for any regressor. Of course, our use of training dynamics assumes regressor f can be checkpointed. This still confers compatibility with a broader class of models: neural nets, XGBoost, GBDTs, linear regression satisfying P4 . We underscore TRIAGE's versatility beyond neural networks, allowing practitioners to use TRIAGE with their preferred model for a specific application. Appendix A.2 details the ease of usage and space and time considerations.

5 Empirical investigation

This section presents an empirical evaluation demonstrating TRIAGE 2 satisfies (P1) Consistency, (P2) Improved regression performance and (P3) Informative for data collection. Recall (P4) Regressor versatility is satisfied by TRIAGE's design. Additional use-cases and results are in Appendix C.

Datasets. We conduct experiments on 10 real-world regression datasets with varying characteristics. i.e. different sample sizes (500-100k), dimensionality (8-85), and task difficulty. The datasets are drawn from diverse domains, including safety-critical medical regression: (i) Prostate cancer from the US [31] and UK [32], (ii) Hospital Length of Stay [33] and (iii) MIMIC Antibiotics [34]. Additionally, we analyze general UCI regression datasets [35], including Bike, Boston Housing, Bio, Concrete, Protein and Star. The datasets are detailed in Appendix B, along with further experimental details. We observe similar performance across different datasets. However, due to space limitations, we sometimes highlight results for a subset and include results for the remainder in Appendix C.

5.1 (P1) Consistent characterization

A key aspect of data characterization is: the ordering of samples matter [9, 10]. This is important when the scores are utilized for data sculpting/filtering. For example, if we rank sort samples by their scores and filter a subset of samples. Consequently, it is crucial to ensure the scoring method used for data characterization is stable and consistent. This consistency is desired across different seeds, but also different practitioners may adopt diverse model architectures/parameterizations for data characterization.

Consistency of TRIAGE. We compare the consistency and robustness to variation of TRIAGE vs baseline general purpose methods: VoG [11], GradN [13], Loss [18]. We also consider simply computing the end of training Residual Errors (used in the

1.0

0.92

0.8

0.45

Correlation

0.6

[0.25

0.4

一

亚王亚亚王

VoG

LoSS

0.2

GradN

Errors/Residual

0.0

Triage (Ours)

Metrics

Well-estimated

Over-estimated

Under-estimated

1.0

0.8

9

0.6

0.4

0.2

0.0E

0.00

0.05

0.10

0.15

0.20

0.25

V(x,y)

TRIAGE conformity score). We assess the consistency of the scores across different model architectures/parameterizations (described in Appendix B.3). All models are trained to convergence with early stopping. Similar to prior classification-based characterization methods [9, 10], we assess consistency based on the Spearman rank correlation of the scores evaluated for all model combinations.

Fig. 3, shows TRIAGE is the most consistent compared to baselines, with the highest Spearman correlation (0.91), satisfying P1 . This consistency means insights derived using TRIAGE scores will also remain consistent. Analyzing each of the 10 datasets (see Appendix C.1), further shows that TRIAGE is stable in magnitude and rank order, whereas the baselines do not have such consistency both in magnitude or ordering across datasets, making the baselines challenging to use in practice.

Takeaway 1. TRIAGE has the most consistent scoring method compared to baselines over different datasets and model combinations, satisfying P1.

TRIAGE v Errors/Residuals. We note that TRIAGE and Errors/Residuals (considered an ablation of TRIAGE's dynamics component) have similar correlations. This raises two questions: (i) are the samples identified the same? We stratify the samples in D train into three equal sized groups ordered by TRIAGE or Error scores. The overlap is ∼ 0.77, highlighting a difference. This leads us to ask (ii) what added value does TRIAGE offer? Consider the case of different samples that have the same error magnitude. Errors/Residuals treat these samples the same, based solely on magnitude. Can TRIAGE offer greater precision to further differentiate samples with the same magnitude?

Examples w/ same

error magnitude can

be differentiated via

Triage scores

As shown in Fig.4, TRIAGE scores evidently offer a way to handle samples differently, showing that samples with similar residuals/errors are associated with different TRIAGE scores. We show the phenomena holds across other general-purpose scoring rules in Appendix C.10. This result demonstrates the added value of the training dynamics aspect, to differentiate samples with the same error magnitude at a more granular level. This contrasts, errors/residuals which are an ablation of TRIAGE, computing the conformity score only at the final training checkpoint, instead of for all checkpoints. We explore the value of this differentiation to improve regressor performance next.

5.2 (P2) Improve regression performance

We now evaluate the ability of TRIAGE to improve regression performance by sculpting/filtering the data by keeping only well-estimated samples . We study two setups using TRIAGE: (1) fine-grained filter (as described above) and (2) sculpting the data based on a deployment purpose.

5.2.1 Fine-grained filtering

In the previous section, we showed TRIAGE's ability to differentiate samples with more precision. Does this differentiation with TRIAGE scores benefit model performance? To find out, we sort the training samples from least to most challenging based on the baseline computed scores: Residuals/Errors, Loss, VoG, GradN. We then evaluate test MSE performance, as we increase the proportion p of samples retained for training. i.e. increasingly retain more challenging samples with higher loss/VoG etc. As expected, the baselines (shown in Fig 5, blue) have lower MSE with more sam-

0.0

0.2

0.4

0.6

0.8

1.0

Baseline proportion of data retained.

*Triage is applied on top (i.e. less data)

0.10

0.15

0.20

Test MSE

Baseline (errors filter)

Triage Filter

(a)

Errors

0.0

0.2

0.4

0.6

0.8

1.0

Baseline proportion of data retained.

*Triage is applied on top (i.e. less data)

0.10

0.15

0.20

Test MSE

Baseline (loss filter)

Triage Filter

(b)

Loss

0.0

0.2

0.4

0.6

0.8

1.0

Baseline proportion of data retained.

*Triage is applied on top (i.e. less data)

0.075

0.100

0.125

0.150

0.175

0.200

Test MSE

Baseline (grad filter)

Triage Filter

(c)

Grad

0.0

0.2

0.4

0.6

0.8

1.0

Baseline proportion of data retained.

*Triage is applied on top (i.e. less data)

0.075

0.100

0.125

0.150

0.175

0.200

Test MSE

Baseline (vog filter)

Triage Filter

ples retained for training. However, the elbow around p = 0 . 5 , after which MSE does not decrease, shows that the added high-magnitude samples are not beneficial.

We now assess the benefit of differentiating these scores using TRIAGE. To do so, we apply TRIAGE as an additional filter on top of the baseline retained samples only keeping the well-estimated subset of samples . i.e. at each baseline proportion p , TRIAGE reduces the number of samples retained compared to the baseline s.t. D Triage ⊂ D Baseline . Fig 5 shows, for all proportions p , we improve MSE performance over baselines simply by differentiating between samples using TRIAGE and filtering to keep only well-estimated samples. This performance benefit holds for all 4 general-purpose scoring methods, illustrating the value of TRIAGE as a fine-grained filter, where even with less data than baselines, the careful selection of samples by TRIAGE leads to improved regression performance. Results on more datasets mirroring the above are in Appendix C.11.

Takeaway 2. TRIAGE's differentiation of samples can be used to augment other approaches as a fine-grained filter; improving regressor test performance, simply by keeping well-estimated samples.

5.2.2 Data sculpting to fit a deployment purpose.

In many situations, we might want to take a regressor trained on a large D train and deploy it in an area with limited labeled data. For instance, repurposing a regressor in a low-resource healthcare setting. This raises the question, instead of acquiring new data - which in some cases might be impossible - could we sculpt the existing large D train to be more 'fit for purpose'. The goal is to enable better regressor performance. Our setup assumes limited access to a small amount of data from the deployment environment, which is insufficient in sample size to directly train a highly performant regression model, yet we could use this data for calibration D cal purposes. Can TRIAGE help?

As a case-study, we consider a regression task similar to [36-38], using patient covariates to predict PSA blood levels, a known predictor of prostate cancer mortality. This task has utility in low-resource areas with limited medical funds [38]. We use multi-country prostate cancer data in which their is a distribution shift and train a regressor on data from the US (SEER)[31]. We evaluate performance on data from the UK (CUTRACT)[32], where the smaller D cal is also drawn from. In addition to the normal train-large, test-small baseline, we evaluate the utility of sculpting D train from the US using TRIAGE only keeping well-estimated samples ( D WE = D Triage ), s.t D Triage ⊂ D train. Conformal calibration uses D cal from the UK.

We compare TRIAGE to several methods, as detailed in Appendix B.3, including: (i) data-driven training: training a model directly on the D cal or combining the two datasets ( D train ∪ D cal ). (ii) prediction-based sculpting: train on D cal (UK data) and then subsequently filter D train (US data) based on predictive uncertainty scores 3 . We assess NGBoost, Bayesian Neural Network, Gaussian Process, Bayesian ridge regression, conformal intervals and residuals/errors. We only consider general purpose methods whose criteria allow us to leverage D cal and are compatible beyond neural nets for flexibility.

We then evaluate MSE test performance of regressors trained on the different 'sculpted' D train. We test sensitivity to varying sizes of D cal, as in many settings, we might not have access to many calibration samples. Table 2 shows that TRIAGE consistently outperforms model-driven methods across the board. This suggests that predictive uncertainty scores are insufficient to select the samples with the highest utility in D train . This contrast implies TRIAGE's scoring method better selects samples in D train with greater utility, leading to improved performance on D test .

Compared to data-driven approaches, we find that TRIAGE has the best MSE , specifically, in the small sample regime , where D cal only contains a limited number of samples n =10-200. Of course, when sufficient data is available ( ≥ 300 samples), sculpting has a reduced benefit. Rather, training directly on the sufficient D cal is preferable. The result highlights the viability of sculpting a larger D train with respect to a small D cal to improve regression in settings with limited data access - with TRIAGE offering the best performance benefits.

In addition, sculpting especially in healthcare settings should account for both (i) imbalanced feature subgroups and (ii) long-tailed outcomes. We assess this in Appendix C.7 and show that TRIAGE does retains strong performance on minority subgroups and long-tailed outcomes by virtue of the calibration dataset.

.

D

cal

sample size

10

20

30

40

50

100

200

300

400

500

Ours (Sculpting)

TRIAGE

(

WE

)

0.051+-0.003

0.050+-0.003

0.047+-0.002

0.046+-0.002

0.046+-0.001

0.046+-0.002

0.046+-0.001

0.045+-0.001

0.045+-0.002

0.046+-0.002

Not TRIAGE (

OE

∪

UE

)

0.088+-0.014

0.068+-0.005

0.066+-0.003

0.066+-0.005

0.068+-0.006

0.080+-0.011

0.087+-0.015

0.093+-0.015

0.082+-0.012

0.078+-0.009

Data-driven

Baseline (

D

train

)

0.064+-0.002

0.064+-0.002

0.064+-0.002

0.064+-0.002

0.064+-0.002

0.064+-0.002

0.064+-0.002

0.064+-0.002

0.064+-0.002

0.064+-0.002

Baseline (

D

cal

)

0.092+-0.023

0.070+-0.012

0.070+-0.010

0.064+-0.003

0.066+-0.009

0.052+-0.004

0.047+-0.001

0.043+-0.001

0.043+-0.001

0.041+-0.001

Baseline (

D

train

∪ D

cal

)

0.060+-0.002

0.059+-0.002

0.058+-0.002

0.056+-0.002

0.055+-0.001

0.049+-0.001

0.047+-0.001

0.044+-0.001

0.044+-0.001

0.042+-0.001

Prediction based

sculpting of

D

train

Error Sculpt

0.066+-0.010

0.058+-0.010

0.060+-0.010

0.060+-0.010

0.057+-0.010

0.058+-0.010

0.055+-0.011

0.055+-0.011

0.057+-0.010

0.057+-0.010

CP Intervals Sculpt

0.064+-0.006

0.081+-0.023

0.082+-0.008

0.106+-0.043

0.073+-0.013

0.059+-0.005

0.055+-0.002

0.063+-0.011

0.054+-0.002

0.051+-0.003

NGBoost

0.066+-0.006

0.121+-0.041

0.085+-0.015

0.080+-0.008

0.094+-0.008

0.196+-0.030

0.132+-0.010

0.096+-0.011

0.095+-0.006

0.103+-0.008

Bayesian ridge

0.080+-0.008

0.118+-0.023

0.111+-0.012

0.129+-0.011

0.129+-0.016

0.111+-0.017

0.106+-0.006

0.110+-0.010

0.114+-0.009

0.103+-0.009

BNN

0.068+-0.005

0.063+-0.006

0.066+-0.004

0.066+-0.004

0.064+-0.005

0.057+-0.006

0.072+-0.010

0.056+-0.006

0.064+-0.003

0.068+-0.008

GP

0.051+-0.006

0.066+-0.008

0.069+-0.005

0.077+-0.008

0.072+-0.006

0.071+-0.009

0.085+-0.011

0.085+-0.003

0.095+-0.004

0.089+-0.006

Data insights from sculpting. We seek to understand what types of samples are 'sculpted'. Such insights are especially useful in clinical settings. We see that US patients ( D train) typically have higher cancer stages and higher PSA scores than their UK counterparts ( D cal / D test ). It is precisely these dichotomous samples that are filtered by TRIAGE to improve predictive performance. These insights can be illustrated to stakeholders via a radial diagram, as in Appendix C.5.

Additionally, given the potential data shift (around exchangeability), Appendix C.6 analyzes CPS calibration, along with continuous ranked probability score (CRPS) quantifying the quality of the predictive distribution.

Beyond predictive performance to fairness. We also show that sculpting the data has potential beyond simply improving predictive performance. In Appendix C.4, we illustrate how TRIAGE can be used to sculpt data to improve fairness of regressors based on access to a limited set of samples.

Takeaway 3. TRIAGE can help to improve regression performance by sculpting a larger dataset to be fit-for-purpose, simply by leveraging a small number of calibration samples.

5.3 (P3) Informative for data collection

We now demonstrate the value of TRIAGE for dataset selection and feature collection/acquisition.

5.3.1 Dataset selection: From sample-level to dataset level.

In real-world applications, we might want entire dataset characterization, rather than just sample-level characterization. An understudied scenario is how to compare and select between different datasets. This need arises in organizations where data is stored in isolated silos and it is difficult to access; with synthetic data a common solution [39, 40] or alternatively when purchasing data via data marketplaces [41]. We study the former scenario, where we have multiple 'synthetic' data versions, produced by different ML models or data vendors. However, our assessment could easily apply to the latter of data marketplaces.

We explore the setting where the real D train is not accessible to calculate statistical similarity measures, yet we still wish to compare synthetic datasets and select a version to train on. The underlying assumption is that the synthetic datasets reflect the true underlying distribution. However, depending on the generation process, some synthetic datasets might prove superior to others.

The question we address is: Can TRIAGE's data characterization allow us to compare and select the synthetic training dataset that yields the best test performance on real data?

We simulate this scenario by generating synthetic training data using (V1) CTGAN and (V2) TVAE [42], representing 2 synthetic data vendors. We employ TRIAGE to analyze and compare them, based on the percentage of well-estimated examples recommended for retention. As is standard, we train a regressor on synthetic data and test it on real data (TSTR) [43, 44]. The best synthetic data should yield the lowest MAE on real test data. Table 3 shows that where TRIAGE's recommendation has a higher retention percentage (i.e. more well-estimated samples), that training with that dataset, leads to lower MAE on real test data. This illustrates that even if the real data is

Dataset

(V1) CTGAN

(V2) TVAE

Bike

0.13 (45% Retained)

0.09 (58% Retained):

Bio

0.24 (45% Retained)

0.21(52% Retained)

Boston

0.16 (38% Retained)

0.13(60% Retained)

Concrete

0.19 (46% Retained)

0.11 (48% Retained)

LOS

0.16 (47% Retained)

0.09(56% Retained)

MIMIC

0.05

(41% Retained)

0.08 (

48% Retained

)

Prostate

0.21 (46% Retained)

0.18 (52% Retained)

Protein

0.22 (52% Retained)

0.20 (54% Retained)

Star

0.18 (38% Retained)

0.14 (51% Retained)

unavailable, TRIAGE's characterization can be used by practitioners to select the synthetic training dataset that will give the best real test performance.

Takeaway 4. TRIAGE scores allow us to assess utility between different synthetic datasets in a principled manner. We show that synthetic datasets that TRIAGE suggests retaining more samples (i.e. more well-estimated), allow for better generalization when testing on real-data (i.e. MAE).

5.3.2 Value of feature collection/acquisition.

Previous settings considered assume a fixed dataset. What if we can actively improve the data's utility by acquiring features? e.g., a doctor runs extra tests to improve a regressor. Understanding the value of features one might acquire is useful, especially if acquisition is costly. We note feature acquisition differs from feature selection (e.g. [45]), where all features are present and we select useful ones. It also differs from active learning, which quantifies sample acquisition value, not features.

We now show that TRIAGE enables a principled approach to assess the benefit of acquiring a specific feature. We posit that acquiring a valuable feature wrt. the task should increase the proportion of well-estimated samples (which we've shown is linked to better performance).

To illustrate the potential of TRIAGE scores, we set up an experiment where we rank the features based on their correlation to the target (i.e. increasing utility). We then simulate the sequential "acquisition" of features with increasing utility (based on correlation). Concurrently, we observe how the groups change as features are acquired.

Fig. 6 shows that as we acquire 'valuable' features, the proportion of the well-estimated samples increases, with the over- and under-estimated proportions dropping. This result, repeated on other datasets in Appendix C.12, shows

5

10

0.0

0.2

0.4

0.6

Feature Corr

0.0

0.2

0.4

0.6

0.8

1.0

Proportion of data

UNDER

WELL

OVER

Features from least to most important added

TRIAGE scores are sensitive to the value of a feature, quantified by the increase in the proportion of well-estimated samples.

Takeaway 5. TRIAGE offers a principled way to assess active dataset improvement when we acquire new features. The feature's value links to the ability to increase the well-estimated proportion.

6 Discussion

We present TRIAGE, a regression-focused data characterization framework, an understudied area in data-centric AI. TRIAGE offers a new scoring method to overcome the limitations of classificationcentric scores. We show TRIAGE's characterization can unlock a variety of use cases to enhance regression, with minimal effort, beyond standard training on a dataset and exchangeability assumptions. We highlight that a key aspect of ensuring good performance of TRIAGE is careful construction of the calibration set - which should be as representative as possible. Rather than automating or replacing a data scientist's role, TRIAGE as a 'data-centric AI' tool aims to empower data scientists to perform the 'data' work in a more principled manner.

Limitations and future opportunities : (1) understanding the attributes contributing to characterization, (2) provide theoretical guarantees taking into account the challenging interaction between the CPD and Triage trajectory (for more see Appendix C.9), (3) in high-stakes settings, to prevent harm we could leverage domain expertise, where a human-in-the-loop could audit the samples surfaced by TRIAGE, before sculpting.

References

Peter Norvig. On big data - the unreasonable effectiveness of data, 2011.

Abhinav Jain, Hima Patel, Lokesh Nagalapatti, Nitin Gupta, Sameep Mehta, Shanmukha Guttula, Shashank Mujumdar, Shazia Afzal, Ruhi Sharma Mittal, and Vitobha Munigala. Overview and importance of data quality for machine learning tasks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 3561-3562, 2020.

Nitin Gupta, Hima Patel, Shazia Afzal, Naveen Panwar, Ruhi Sharma Mittal, Shanmukha Guttula, Abhinav Jain, Lokesh Nagalapatti, Sameep Mehta, Sandeep Hans, et al. Data quality toolkit: Automatic assessment of data quality and remediation for machine learning datasets. arXiv preprint arXiv:2108.05935 , 2021.

Cedric Renggli, Luka Rimanic, Nezihe Merve Gürel, Bojan Karlas, Wentao Wu, and Ce Zhang. A data quality-driven view of mlops. IEEE Data Engineering Bulletin , 2021.

Haihua Chen, Jiangping Chen, and Junhua Ding. Data evaluation and enhancement for quality improvement of machine learning. IEEE Transactions on Reliability , 70(2):831-847, 2021.

Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence , 4(8):669-677, 2022.

Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar. Dc-check: A data-centric ai checklist to guide the development of reliable machine learning systems. arXiv preprint arXiv:2211.05764 , 2022.

Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Conference on Empirical Methods in Natural Language Processing , 2020.

Nabeel Seedat, Jonathan Crabbé, Ioana Bica, and Mihaela van der Schaar. Data-iq: Characterizing subgroups with heterogeneous outcomes in tabular data. In Advances in Neural Information Processing Systems , 2022.

Pratyush Maini, Saurabh Garg, Zachary Chase Lipton, and J Zico Kolter. Characterizing datapoints via second-split forgetting. In Advances in Neural Information Processing Systems , 2022.

Chirag Agarwal, Daniel D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10368-10378, 2022.

Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems , 33:17044-17056, 2020.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems , 34:20596-20607, 2021.

Vladimir Vovk, Jieli Shen, Valery Manokhin, and Min-ge Xie. Nonparametric predictive distributions based on conformal prediction. Machine Learning , 108(3):445-474, 2019.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017.

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pages 322-332. PMLR, 2019.

Yuanzhi Li, Tengyu Ma, and Hongyang R Zhang. Learning over-parametrized two-layer neural networks beyond ntk. In Conference on learning theory , pages 2613-2682. PMLR, 2020.

Shoaib Ahmed Siddiqui, Nitarshan Rajkumar, Tegan Maharaj, David Krueger, and Sara Hooker. Metadata archaeology: Unearthing data subsets by leveraging training dynamics. arXiv preprint arXiv:2209.10015 , 2022.

Andrew Ng, Lora Aroyo, Cody Coleman, Greg Diamos, Vijay Janapa Reddi, Joaquin Vanschoren, Carole-Jean Wu, and Sharon Zhou. Neurips data-centric ai workshop, 2021.

Nabeel Seedat, Jonathan Crabbe, and Mihaela van der Schaar. Data-suite: Data-centric identification of in-distribution incongruous examples. In International Conference on Machine Learning . PMLR, 2022.

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations , 2018.

Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning , pages 2242-2251. PMLR, 2019.

Ruoxi Jia, David Dao, Boxin Wang, Frances A Hubis, Nezihe M Gürel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. Proceedings of the VLDB Endowment , 12(11):1610-1623, 2019.

Hoang Anh Just, Feiyang Kang, Tianhao Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. In The Eleventh International Conference on Learning Representations , 2022.

Jieli Shen, Regina Y Liu, and Min-ge Xie. Prediction with confidence-a general framework for predictive inference. Journal of Statistical Planning and Inference , 195:126-140, 2018.

Tore Schweder and Nils Lid Hjort. Confidence, likelihood, probability , volume 41. Cambridge University Press, 2016.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Conformal prediction. Algorithmic learning in a random world , pages 17-51, 2005.

Vladimir Vovk, Ivan Petej, Ilia Nouretdinov, Valery Manokhin, and Alexander Gammerman. Computationally efficient versions of conformal predictive distributions. Neurocomputing , 397:292-308, 2020.

Ulf Johansson, Henrik Boström, Tuve Löfström, and Henrik Linusson. Regression conformal prediction with random forests. Machine learning , 97(1):155-176, 2014.

Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. arXiv preprint arXiv:2110.01889 , 2021.

Máire A Duggan, William F Anderson, Sean Altekruse, Lynne Penberthy, and Mark E Sherman. The surveillance, epidemiology and end results (seer) program and pathology: towards strengthening the critical relationship. The American journal of surgical pathology , 40(12):e94, 2016.

Prostate Cancer UK. Cutract. https://prostatecanceruk.org , 2019.

Microsoft. Predicting hospital length of stay, 2017.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Arthur Asuncion and David Newman. Uci machine learning repository, 2007.

Thomas A Stamey, John N Kabalin, John E McNeal, Iain M Johnstone, Fuad Freiha, Elise A Redwine, and Norman Yang. Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate. ii. radical prostatectomy treated patients. The Journal of urology , 141(5):1076-1083, 1989.

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

Gongli Li and Han Li. Linear model selection and regularization for serum prostate-specific antigen prediction of patients with prostate cancer using r. IEEE Access , 9:97591-97602, 2021.

Khaled El Emam, Lucy Mosquera, and Richard Hoptroff. Practical synthetic data generation: balancing privacy and the broad availability of data . O'Reilly Media, 2020.

Andre Goncalves, Priyadip Ray, Braden Soper, Jennifer Stevens, Linda Coyle, and Ana Paula Sales. Generation and evaluation of synthetic patient data. BMC medical research methodology , 20(1):1-40, 2020.

Markus Spiekermann. Data marketplaces: Trends and monetisation of data goods. Intereconomics , 54(4):208-216, 2019.

Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems , 32, 2019.

Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv:1706.02633 , 2017.

Michael Platzer and Thomas Reutterer. Holdout-based empirical assessment of mixed-type synthetic data. Frontiers in big Data , page 43, 2021.

James Jordon, Jinsung Yoon, and Mihaela van der Schaar. Knockoffgan: Generating knockoffs for feature selection using generative adversarial networks. In International conference on learning representations , 2018.

Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 'everyone wants to do the model work, not the data work': Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1-15, 2021.

Sezal Chug, Priya Kaushal, Ponnurangam Kumaraguru, and Tavpritesh Sethi. Statistical learning to operationalize a domain agnostic data quality scoring. arXiv preprint arXiv:2108.08905 , 2021.

Neoklis Polyzotis and Matei Zaharia. What can data-centric ai learn from data and ml engineering? arXiv preprint arXiv:2112.06439 , 2021.

Nabeel Seedat, Alan Jeffares, Fergus Imrie, and Mihaela van der Schaar. Improving adaptive conformal prediction using self-supervised learning. In International Conference on Artificial Intelligence and Statistics , pages 10160-10177. PMLR, 2023.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems , 34, 2021.

Michael Redmond and Alok Baveja. A data-driven software tool for enabling cooperative information sharing among police departments. European Journal of Operational Research , 141(3):660-678, 2002.

Daniel Steinberg, Alistair Reid, and Simon O'Callaghan. Fairness measures for regression via probabilistic classification. arXiv preprint arXiv:2001.06089 , 2020.

Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-based algorithms. In International Conference on Machine Learning , pages 120-129. PMLR, 2019.

Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 259-268, 2015.

Equal Employment Opportunity Commission et al. Uniform guidelines on employee selection procedures. Fed Register , 1:216-243, 1990.

Hans Hersbach. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting , 15(5):559-570, 2000.

Jacob Steinhardt. Robust learning: Information theory and algorithms . Stanford University, 2018.

Ricardo A Maronna, R Douglas Martin, Victor J Yohai, and Matías Salibián-Barrera. Robust statistics: theory and methods (with R) . John Wiley & Sons, 2019.

Mengjie Chen, Chao Gao, and Zhao Ren. A general decision theory for huber's contamination model. 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper present a data characterization method for regression task. The method leverages conformal predictive systems literature and proposed to estimate training data scores by thresholding percentile of  calibration data points given their conformity measure. The method is interesting in terms of leveraging calibration dataset CPD to measure the training data score. The threshold of grouping (4.3) is less explained, but empirical results show it works. The paper is self-claimed to be the first data characterization method that is suitable for regression task.\n\nSTRENGTHS:\nThe proposed method is novel to me. Leveraging calebration dataset to re-score training data is interesting. I am quite curious how the choice of calibration dataset would impact the method's performance.\n\nThe paper is very well motivated. And, maybe over motivated, considering justification and explanation missing in the section 4.2 and 4.3.\n\nThe experimental results show the proposed method works well in benchmark datasets and could be a good tool to do data selection or feature selection.\n\nWEAKNESSES:\nThe paper takes pretty long paragraph to highlights its novelty and difference between other data characterization method, which looks like an ads. The motivations part should be reduced and focus more on algorithm explanation. There is very less justification on why thresholding CPD works for the data characterization purpose, and where the rules in Eq 3 come from. They need further justification and explanation.\n\nAlgorithm 1 need to further adjustment and state what the \"eval sample\" is. I think it is one training data in D_train. The output of Algorithm should be a matrix |X| x |M| right? please state it somewhere to help understanding. \n\nDesign of 4.2 is not explained. Why mean over training steps? What if we put more weights to higher iteration? What if there are multiple candidate regression methods? would this alter the outcome?\n\nQUESTIONS:\nWhy mean over training steps when computing C(x,y)?  What if we put more weights to higher iteration?\nWhat if there are multiple candidate regression methods? would this alter the outcome? \nWhy does the group assignment threshold designed as Eq 3? Any explanation or justification?\n\nLIMITATIONS:\nNo concern. But the authors should describe further in terms of mis using this method, where decision maker can remove data from underrepresented population based on the solution.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The task of data characterization aims to address variations in individual-level performance despite achieving good average performance. Existing methodologies have predominantly focused on classification, leaving a gap in data characterization approaches for regression. In this paper, the authors propose the TRIAGE framework to bridge this gap. Extensive experiments demonstrate the framework's superior performance in regression tasks.

STRENGTHS:
- The paper is well-written.

- This is the first paper to introduce a principled data characterization framework in regression settings, supported by extensive experiments.

WEAKNESSES:
Many contents are in the appendix. It would be helpful more discussions about the appendix in the main text.

QUESTIONS:
- In simplified setting (e.g., linear (kernel) regression?), is it possible to derive further theoretical guarantees on the behavior of C and V, defined in Section 4.2?

- Is the group assignment consistent? (in the sense that under what conditions are the samples accurately assigned their true group label?)

LIMITATIONS:
refer to weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a new data characterization framework, TRIAGE, for regression models. The method utilizes conformal predictive distributions to compute the training examples' scores. To compute TRIAGE scores, the authors use predictive distributions and conformal prediction. A proper training set is used to train a regressor, and a separate calibration set is used for conformal calibration. Conformity measures the dataset's agreement with the observation. Consequently, the conformal predictive scores are computed at each epoch for each training point. Afterward, TRIAGE measures the mean and standard deviation for each training point. The method analyzes each example's training dynamics at each epoch to group each point into one of the groups: under/over/well estimated by the model based on the thresholds. TRIAGE can reduce the number of samples to train the regressor compared with the baseline methods. Specifically, it is observed that the MSE performance is improved with the selected data. It holds an advantage steadily over four different scoring methods.\n\nSTRENGTHS:\n+ The authors propose a novel data characterization method for any regression models that analyze the training dynamic of each training point. Using conformal predictive systems for this problem is intuitive and effective.\n+ TRIAGE effectively reduces the number of samples to train regressors compared with the baseline methods. It shows that the MSE performance is improved with data keeping and holds an advantage for four different scoring methods.\n+ The paper is well written.\n\nWEAKNESSES:\n+ Time complexity can be enormous for large neural networks to compute scores over multiple epochs.\n+ Only simple datasets and simple regressors are performed. The data sculpting experiment is also limited to 500 available samples.\n+ Some more challenging points might be important in the medical field, but if TRIAGE discards those samples, the model would not learn the critical medical case.\n\nQUESTIONS:\n+ How much does the calibration dataset affect the data sculpting performance?\n+ How exactly are distances computed in KNN for residuals of the calibration dataset? Does it use some embedding space?\n+ What exactly do authors mean by model-agnostic? It seems that conformity scores are computed on residuals which are based on trained regression model predictions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe problem studied in this work is the following: Given a dataset $\\lbrace (x_i, y_i ) \\rbrace_{i=1}^M$ and a regressor $f_\\theta$ trained on this dataset, assign a group label $g_i$ to each sample that specifies whether the regressor under or overestimates on the sample. Such group labels can be used to identify and remove outliers in the dataset. The key formula of the proposed method is Eqn. (1), which is similar to Eqn. (5) in [23] that takes into account the \"prediction accuracy\" over each sample. The authors proposed to first put each sample into one of the several bins defined on a calibration set, and then assign a CPD score accordingly. The group label is then assigned using the CPD score as well as its variance. In the experiments, the authors use these group labels to remove outliers in the training set, and observe that it can improve the regression performance for a variety of regressors.\n\nSTRENGTHS:\n- The problem studied in this work is very important for real applications since most datasets contain outliers.\n- The desiderata for data characterization, as listed in (P1)-(P4) after line 44, are very clear and define the overall goal of this work.\n- The experimental results presented in Section 5 support that the proposed method satisfies the desiderata.\n\nWEAKNESSES:\n- The overall presentation is not good enough and there are many confusing points in this work, which I will discuss in the Questions section.\n- The authors claim TRIAGE to be \"the first data characterization framework tailored to regression settings\" (lines 63-64), yet in [23] cited in this work, a similar approach for regression conformal prediction was proposed. In fact, the proposed method in this work largely resembles the method in [23], including the CPD score definition Eqn. (1) (versus Eqn. (5) in [23]), the use of calibration scores $\\lbrace \\alpha_1,\\cdots,\\alpha_q \\rbrace$ (versus Eqn. (1) in [23]), and the use of KNN for estimating $\\sigma$ (versus Eqn. (11) in [23]). Thus, I am not sure about the contributions of this work. However, I am sure that \"TRIAGE is the first data characterization framework tailored to regression\" is an overclaim.\n- I feel that the writing is too wordy, especially in Section 5, and the authors use some LaTeX tricks which make the manuscript looks more compact than necessary. I think the authors are able to make Section 5 more concise, so that the main text could easily fit in 9 pages and the paper would have a much better shape.\n\nQUESTIONS:\n1. There are several confusing points in the manuscript that require clarification:  \n(a) In Algorithm 1, line 3, what do you mean by \"nearest neighbor residuals of KNN\"? Is it the average distance to the k nearest neighbors?    \n(b) In Eqn. (11) of [23], the definition of $\\sigma$ considers two factors: (i) Whether a sample is close to its k nearest neighbors; (ii) Whether the k nearest neighbors have consistent labels. Does this work use the same definition for $\\sigma$? If not, what are the differences?  \n(c) In the definition of the TRIAGE score $T(x,y,\\theta)$ in lines 161-162, what do you mean by \"$P(y \\le f_\\theta(x))$\"? What is this probability taken over? If $x, y, f_\\theta$ are all deterministic, why should there be a probability?   \n\n\n2. Can the authors make a thorough comparison between this work and [23], including the methodology, the algorithm, the experimental setup and results? This could help clarify the contributions of this work.\n\n3. (Bonus) It would be great to make a theoretical connection between the proposed method and robust statistics. Consider a simple case, where $x$ is 1-dimensional, and the goal is to fit $y=f(x)$ on a dataset $(X,Y)$ with outliers. The dataset could follow Huber's contamination model. In such a scenario, I am curious about whether TRIAGE is provably better than the error baseline (as empirically compared in lines 244-260), and whether there is any guarantee on the performance of the predictor with the outliers removed using TRIAGE. A good starter for robust statistics could be the thesis of Jacob Steinhardt: https://cs.stanford.edu/~jsteinhardt/publications/thesis/paper.pdf. This is a bonus point, but such theoretical analysis could greatly enhance the contributions of this work.\n\n**Post rebuttal note:** I have read the rebuttal and had a discussion with the authors. The authors have addressed most of my questions. I have raised my rating from 4 (original) to 6 (current).\n\nLIMITATIONS:\nLimitations are discussed in Section 6.\n\n**Review summary:** I think this work is interesting overall, and there are some good takeaways. However, the writing is confusing at times, and it also seems to me that a large part of this work is very similar to [23]. I believe many people will find this work useful, but I also believe that this work still needs improvement. So I give a borderline rating to this submission though I rarely give borderline ratings in my reviews. If the authors could address my questions during the rebuttal, I am willing to raise my score to 6 or 7, depending on the quality of the rebuttal.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of training data characterization for regression problems. The authors noted that existing research on training data characterization mostly focuses on classification problems and there remains an absence of research for regression problems. This work proposes TRIAGE, a novel framework designed exclusively for regression settings and applies to a variety of tasks.\n\nTRIAGE leverages conformal predictive distributions to provide a model-agnostic scoring method for evaluating how the model performs on each training sample. This framework also enables analysis into the training dynamics from checkpoints, visualizing the samples being under-/well-/over- estimated by the model during the training process. \n\nThe proposed framework is useful for a variety of data curation tasks, such as improving the model performance by throwing poorly-predicted training samples. Beyond sample-wise data analysis, the framework also applies to broader tasks such as dataset selection/feature acquisition. The work validates the proposed method with a range of empirical studies on 10 tableau datasets with 500-100k samples, showcasing its value in data characterization in practical cases.\n\nSTRENGTHS:\nThe problem being studied and data-centric perspectives are attractive and important. The work is well-motivated and nicely written with easy-to-follow illustrations.\n\nThe narrative of the paper is highly structured, skillful, intriguing and informative. I did enjoy reading this paper. The importance of the work is highlighted. References are substantial and high quality.\n\nObjectives P1-P4 and the comparison table with baselines are clear and precise which can be informative and beneficial for a broader audience. And the improvements in these objectives are later validated in empirical studies, facilitating the verification of this work. \n\nThe technical body is clear. The analysis of training dynamics from checkpoints is interesting. Novel insights are provided in the takeaways. The work is practical with many interesting use cases.\n\nWEAKNESSES:\nI am combining this part with the questions. \n\n1. Since this work is situated in the context of data-centric AI and also studies data characterization problems, it is a bit unnatural to miss out on the comparisons with data valuation methods. \n\nFor example, Shapley-based methods are a common benchmark for the general interpretability of ML applications and also apply to characterizing the effects of training data on regression tasks. Also, I was wondering how model-agnostic data valuation ([1]) pipelines work in this setting.\n\nI would like to see discussions both conceptually and empirically on how those methods perform in the context this paper studies and how they differentiate from the proposed approach.\n\n[1] Lava: Data valuation without pre-specified learning algorithms, ICLR 2023\n\n2. What is the rationality for throwing out data that is over/under-predicted? It could be outliers, but would simply throwing out such data hurt the generalizability of the model or its robustness against distributional shifts, which is prevalent in real-world applications? Do the authors have any empirical study results on such cases?\n\n3. Data selection is an established field of research. This work showcases the proposed approach apply to dataset selection tasks. It would be beneficial if the authors could provide a direct comparison with data selection baselines and visualize their performance.\n\nIt won't harm the contribution of this paper if its data selection performance is suboptimal compared to methods designed exclusively for data selection tasks. But I think it is important to know how they compare exactly and benchmark the gap, which would be helpful for future works to compare with or improve over.\n\n4. What is the computational overhead of the proposed framework? Can the authors provide the computation time? How scalable is the proposed framework? What is the largest scale the authors are able to apply it to?\n\nThe text in Figure 1 can be made larger.\n\nQUESTIONS:\nSee Weaknessnes.\n\nLIMITATIONS:\nNo major limitations. Questions for discussion are listed above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-06"}