TRIAGE: Characterizing and auditing training data for improved regression
Authors: Nabeel Seedat
Jonathan Crabbé
Zhaozhi Qian
Mihaela van der Schaar
Venue: NeurIPS 2023 poster (2023)
Keywords: data-centric AI
data characterization
data quality
TL;DR: TRIAGE provides systematic data characterization for regression settings; compatible with a variety of regressors

Abstract: Data quality is crucial for robust machine learning algorithms, with the recent interest in data-centric AI emphasizing the importance of training data characterization. However, current data characterization methods are largely focused on classification settings, with regression settings largely understudied. To address this, we introduce TRIAGE, a novel data characterization framework tailored to regression tasks and compatible with a broad class of regressors. TRIAGE utilizes conformal predictive distributions to provide a model-agnostic scoring method, the TRIAGE score. We operationalize the score to analyze individual samples' training dynamics and characterize samples as under-, over-, or well-estimated by the model. We show that TRIAGE's characterization is consistent and highlight its utility to improve performance via data sculpting/filtering, in multiple regression settings. Additionally, beyond sample level, we show TRIAGE enables new approaches to dataset selection and feature acquisition. Overall, TRIAGE highlights the value unlocked by data characterization in real-world regression applications.

TRIAGE: Characterizing and auditing training data for improved regression
Nabeel Seedat University of Cambridge ns7410cam ac uk
Jonathan Crabbé
University of Cambridge jc21330cam. ac.
Zhaozhi Qian University of Cambridge zq2248cam. ac. uk
Mihaela van der Schaar University of Cambridge
mv4726cam ac. uk
Abstract
Data quality crucial for robust machine learning algorithms, with the recent interest in data-centric Al emphasizing the importance of training data character- ization. However, current data characterization methods are largely focused on classification settings, with regression settings largely understudied. To address this, we introduce TRIAGE, novel data characterization framework tailored to regression tasks and compatible with broad class of regressors. TRIAGE utilizes conformal predictive distributions to provide a model-agnostic scoring method, the TRIAGE score. We operationalize the score to analyze individual samples' training dynamics and characterize samples as under-, over-, or well-estimated by the model. We show that TRIAGE's characterization is consistent and highlight its utility improve performance via data sculpting/fiteerigg in multiple regression settings. Additionally, beyond sample level, we show TRIAGE enables new approaches dataset selection and feature acquisition. Overall, TRIAGE highlights the value unlocked by data characterization in real-world regression applications.
Introduction
Data characterization, an important problem. Over decade ago, it was recognized that "more data beats clever algorithms, but better data beats more data"[1], emphasizing the key role of training data in the performance and robustness of ML algorithms [2-4]. In particular, training data from the "wild" may have various errors or lack coverage, rendering it unsuitable for the ML task [5].
These observations motivate the need for data characterization, which aims score each training sample based on its impact on the performance of the ML task. The training data can then be partitioned rank ordered according the score give high-level description of the sample and its utility. Data characterization provides a natural way for data sculpting [6, 7], which filters training samples that have low the sculpted/fittered (and smaller) training data have improved performance.
detrimental effect the MIL task. The goal is that models built on
In this work, we address the understudied data characterization problem in regression settings. In contrast, existing works on data characterization have primarily focused on the classification setting [8-13]. These methods leverage the probability distribution over discrete labels (softmax or logits) for scoring. However, in the equally important regression setting, the probability distribution over the continuous labels are not typically given by standard regression models. In fact, the majority of the regression models only output a point estimate, e.g. the predicted mean. Without access to probabilistic estimates, cannot simply apply the existing cisssficatiio-taillree methods for
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Compute TRIAGE scores via CPS
ii
Analyze TRIAGE training dynamics
Characterize samples
CPS Probabiity
Preere
Daniin
Any Nenatierly Trained mode' $..42..42
cownstream sasks
Dest
Epochs (NNoEstimttor (GBT)
Data SOU Fihorin
Figure 1: TRIAGE systematically characterizes data in regression setlings via a 3-step pipeline. (i) Compute TRIAGE scores: of each sample in the datasct, for each regression checkpoint, via a Conformal Predictive System (CPS), Over the training epochs/iterations, we (ii) Analyze the per-sample training dynamics, studying the trajectory of CPS probability scores for cach training sample, with the scoring method operationalized characterize the samples. (iii) Characterize samples, assigning them different groups (over-, under- and well-estimated) based on the TRIAGE scores. These groups inform sculpting & other use-cases.
discrete labels to the regression setting. We provide further details why cetiifiiaationtaallrree methods inapplicable to the regression setting in Sec. 2 and Table 1. This non-applicability motivates the need for pnarressioottaillrre methods for data characterization, which are "not only principled, but also practical for larger collection (...) of ML models" [4].
To anchor the need for data characterization in regression, let's consider few concrete examples. In predicting house prices, an outlier might be house with an extremely high price due to a unique or undee--epresented feature in the data. This could affect the model's performance, causing it to overestimate prices for similar houses without the unique feature. A Alternatively, imagine predicting patient length of stay based on clinical features. If the dataset contains some erroneous features, the regression predictions may be suboptimal, potentially leading to incorrect clinical decisions.
In designing a data characterization framework for regression, we outline the following desired prop- erties, drawing motivation from the capabilities of works in the classification setting [8-13]:
(P1) Consistency: The characterization data examples should be stable and consistent across runs and for similar performing models with different parametrizations.
(P2) Improve performance: The characterization should guide the improvements to model performance. For instance, via data sculpting/filtering.
(P3) Informative for data collection: The characterization score should be informative to enable selection between datasets, as well as, feature collection/acquisition.
(P4) Regressor versatility: The characterization should be principled and practical for a large collection of ML models (e.g. not just neural networks).
To fulfill P1-P4, we propose a framework to characterize data in regression settings, called TRIAGE (s.f. TRaining Identification & Auditing of Good Examples) - see Fig. TRIAGE introduces a novel regression scoring
score, building on the recent development of Conformal Predictive Systems (CPS) [14]. CPS is a model-agnostic method that can turn the output of any regressor into a probability distribution for continuous labels, termed conformal predictive distribution (CPD). Unlike point estimates such as errors/residuals or conventional conformal prediction intervals, the CPD provides richer information through full predictive distribution. This enables us to assess the probability of a prediction being over- or under- estimated.
We then operationalize the CPD (and TRIAGE score) in novel way for data characterization, computing the trajectory of TRIAGE scores over training checkpoints, as measure of how samples are learned differently during training. Such training dynamics, shown in Fig 1 and have been shown to contain salient signal about the value of the sample to a learner [15-17]. By analyzing the training dynamics, we then categorize training samples as: under-, over- or well-estimated by the model. This is useful going back to our motivating example- - where erroncous values could cause the model to overestimate house price. Using the TRIAGE score, we can identify such sample, as it would likely have a low probability of being correct based on the model's predictive distribution. Such identification unlocks a variety of use cases. For instance, houlptingffiltrring such samples can improve model performance (even with less data), as we empirically show in Sec. 5.
Contributions. ! Conceptually, we present TRIAGE, the first data characterization framework tailored to regression settings, addressing the need for an ML-aware data quality that is principled and practical across large collection of ML regressors. ② Technically, we introduce a principled regression scoring method called TRIAGE score, that leverages conformal predictive distributions in a novel way for data characterization. We operationalize the score to analyze how different samples are

---

Table 1: Comparing data characterization for classification to TRIAGE (a tailored regression method)
Type method
Approaches
Scoring
repurposed regression
Applicable trined inerativelyy
for non non-applicabilit
mecitod
Tailend regression
TRIAGE (Ouns)
CTS pcububility
Dasa Mags [3]
Midel unceriuity
Needs label procabulty distrination
Tailored classification
Data IQ [9]
SSFT [10]
Forgetting scores
Data unceriiinty
Needs label probubility distribution
Needs discrete class changes
AUM (12)
Logit
Needs logis output
Mazadax Arzloclogy [1]]
For-ssmple
General purprus purpuee
VoG
Vasiance
Needs differentiable model (e.g. Nezds differentiable moodel (e.g. NN)
GraNd [13]
Gradient Nocm
learned during training, thereby permitting characterization of individual samples as: under-, over- or well-estimated by the model. (3) Empirically, we demonstrate the utility of TRIAGE across multiple use cases satisfying PI-P4, including consistent characterization, sculpting to improve performance variety of settings, as well as, guiding dataset selection and feature acquisition.
2 Related work
This work engages with recent efforts in data-centric AI [6, 7, 19, 20] on data characterization. However, prior methods have focused on classification settings. In contrast, TRIAGE addresses the regression setting where tailored data characterization has not been explored. We now discuss related works and outline the challenges of applying prior classification focussed methods to regression, for an overview see Table 1. Prior classification scoring methods can be divided into two groups:
Tailored methods, assign scores based on aspects specific to classification, including logits [12]. probabilities over labels [8, 9] or changes to predictions on discrete labels [10, 21J. However, these methods are inapplicable to our regression setting because these aspects are not present in regression.
General purpose methods. while focussed on classification, they assign scores using generic measures and can be repurposed for regression. However, these general methods based on backpropa- gated gradients [11, 13] or losses [18] have two limitations. Firstly, scores like gradients can only be used with differentiable models (e.g. neural networks) and are not compatible with many widely used regressors, such XGBoost - i.e. does not satisfy (P4) Versatility. Secondly, even if they can be used, general-purpose scores may not distinguish between samples that have similar magnitude scores (e.g. high loss scores), even if the underlying reasons for these scores may be different with respect to the task. We show the value of such a differentiation between samples in Sec.5.2.
Data valuation. A related area to data characterization is that of data valuation, for example: Shapley based valuation methods [22, 23]. We first outline some conceptual differences: (a) TRIAGE unlocks other data-centric tasks beyond just data selection, e.g. feature-level acquisition/colleciio and selection between datasets. These are out of scope for data "sample" selection; (b) Unlike TRIAGE, these methods are not tailored to regression; (c) method differences: computationally Shapley-based methods need to assess multiple sample permutations and need to retrain the model many times. Hence, they struggle to scale to very high-samples sizes (e.g. 100k), unlike TRIAGE where the cost cheap comparatively. Recently, a valuation method LAVA [24] has been introduced to address this which uses additional embedding model to reduce dimensionality before the optimal transport step. Beyond conceptual differences, we compare TRIAGE experimenaally and in terms of computation time in Appendix C.
3 Data Characterization Preliminaries
In this section, we formulate the data characterization problem for regression settings. Consider the standard regression setting, where the aim is to assign an input E X C Rdx to continuous valued label y E R (often a point estimate for y). This is done using regressor fo : V. parameterized by 0 E e. We assume fo is trained on a dataset D with M E M observations, i.e. D == ((.m ") m E [M]] For the purposes of later notation, we consider a sample space Z = X' x IR, where each observation " = (x²" ")) E Z. Typically, parameters 0 are learned in regression, via empirical risk minimization (ERM) with a mean squared error (MSE) loss, i.e.
ERM = arg mingge 1 ΣM f(x"" ; with loss function I X x y x θ R+
ERM ensures regressor fo performs well on average, however it goes without saying that the model is unlikely to perform equally well for samples in the dataset. Consequently, we desire to characterize such types of samples. Formally, for each sample zh we aim to assign a group label g" G. where
3

---

g - {Over-estimatee Well- EellmEstiaate,, Under-estimaatdd- We then use these groups for downstream tasks, or sculpt/filter samples in the dataset DD based on their group assignment. Before giving a precise description of how the group labels are assigned, we provide some context.
Prior works have shown the behavior of samples over the course of model training, called training dynamics contains signal about the nature of a sample itself [15-17]. For instance, some samples are easily learned and stable through training, whereas other samples might be challenging and have methods leverage this concept and assign g by analyzing training dynamics e.g. Data-IQ. Data Maps. AUM etc. However, as discussed in Secs. 1 and 2, the aforementioned classification methods do not apply to regression. Consequently, this requires us to formalize a new principled scoring method for regression. After that, we can operationalize the scoring method to analyze the training dynamics and assign group labels g" to each z"
high variance. Many classification--art
For regression, such a principled scoring method could evaluate the probability that the true target y is less than or equal to observation f(x), where the categories of samples are: I entimaest Ply < f()) >> 0.5, @ Well-estimated: Ply I f()))
0.5, 3 Over-estimated: P(y S f())) < 0.5. In the next section, we discuss how we design TRIAGE to achieve this, with the final goal in mind assigning group labels gr to each sample x.
4 TRIAGE: characterizing data for regression settings
We now introduce TRIAGE, perressionffcuusee data characterization framework, compatible with any iteratively trained ML regressor (Sec. 4.4). TRIAGE follows a three-step procedure:
1. Compute TRIAGE scores (Sec 4.1): Compute the TRIAGE score for every sample . using conformal predictive distributions. Do this for every regression checkpoint to obtain a trajectory.
2. Analyze TRIAGE score training dynamies (Sec 4.2): Operationalize the TRIAGE score for data
characterization by analyzing the TRIAGE score's trajectory for each training sample.
Characterize & assign groups (Sec 4.3): Characterize samples based on their training dynamics. assigning each sample x. to a group gr The groups inform filtering/sculpting or other use cases.
4.1 Computing TRIAGE scores via Conformal Predictive Systems (Step 1).
Recall, we want to compute P(y S fx)) as a principled scoring method to characterize samples. We want to do so without changing the chosen regressor type, i.e. P4: Regressor versatility. Of course, the full information of an uncertain label exists in its probability distribution, allowing us to compute probabilities of various outcomes. Unfortunately, regular regressors (point predictions) or conventional conformal regressors (prediction intervals) do not provide this distribution. While Bayesian methods offer distributions, they require funndamentally changing the regressor.
These challenges motivate defining TRIAGE around Conformal Predictive Systems (CPS)[14], CPS combines predictive distributions [25, 26] with conformal prediction [27]. crucially enabling us to compute the required probability distribution of a continuous variable called Conformal Predictive Distribution (CPD). CPS also fulfills our versatility goal (i.e. P4) as it applies to any regressor. To clarify Conformal Predictive Systems output valid cumulative distribution functions, which are termed Conformal Predictive Distributions (CPD). This is the cumulative probability with respect to a label y given some x and regressor f With CPDs denoted as Q the conformal p- p-values get arranged into probability distribution which has the properties of a CDF thus essentially becoming probabilities, see [28]. Since the CPD has the properties of a CDF, we use the CPD to estimate probabilities that the true target is less than equal to specified threshold/value.
Computing CPDs. TRIAGE formalizes the CPD computation in the framework of the com- putationally efficient Split Conformal Predictive Systems [28]. A proper training set Drr ((o Vir) 2 i [M] is used to train regressor| f. separate, often smaller, calibra- tion sei D..al = [(real) Veal) = =cal E [q]), is used for conformal calibration. Finally, we define a conformity measure (4, where μ : 2 -> R U (-oc, 20), quantifies the dataset's agreement with the observation. For each label y i E [g]. we compute q conformity scores
(=(((a)) We define the conformity measure (J1) as per Eq. 1, which is balanced isotonic conformity measure. We discuss this choice in Appendix A.3, a necessary condition s.t the CPD can be interpreted as probability distribution [28]
p(x, v) (g- f(x))//(())
(1)
Under the standard conformal exchangeability assumption (see Appendix A.3), the CPDs are valid.
4

---

where y is the label, f(r) predicted label, an estimate of prediction quality, com- puted with separate normalization model, enabling CPDs to reflect task difficulty. a is KNN fitted on the residuals of Deal sim- ilar to [29].
Algorithm 1 Computing CPD
Require:
Training set (Rf, Vi.) E Denin i E [M] and
Calibration set (d Yeal) E Deal. i E [g].
1: Regressor f trained on Prrin-
> Training
2: Compute residuals on Deut -
3: Define (x)); which finds the average residuals of
We compute the CPD denoted Q" (Eq. 2) for each sample xd via Algorithm I, where the CPD is defined for y* R [28]. We ex- tract the TRIAGE score using the CPD. We repeat the CPD computation for all sam- ples. Next, we describe our novel approach to operationalize the TRIAGE scores, for the purpose of data characterization.
the k-nearest neighbors x, where KNN is fit Deal
4: Calculate conformity scores a,,,, over Deal
using fe; i.e. oe -iel)
>. Calibration
5: Sort a.y..iy in ascending order s.t
for eval sample 2: Xerain do
Evaluation
Let Sig) f(x + a2)((z) for [g)- Let S(o) == -c and S(g+1) == oC.
end for
Return the CPD as per Equation (2).
if yf (S S(4))
Qly =
(2)
2/r
yf = Soi (1:g]
q+1
where T E [0, == min Stg) = ) and 8" == max (j (1 { account for ties.
4.2 Analyzing TRIAGE score trajectories - training dynamics (Step 2)
Novel use of CPD's for data characterization. The TRIAGE score, denoted as T(z, y, 0) Ply S fo(x)), represents a novel use of CPDs for data characterization, diverging from their original predictive uncertainty use. Additionally, while CPDs are normally calculated post-model fitting, we observe from Fig. I(i) that samples' learning varies, leading to different convergence rates of scores. As discussed in Sec. 3, such dynamics reveal sample characteristics [15-17]. Hence, we propose to examine TRIAGE score trajectories to capture the learning dynamics of different samples.
To enable this analysis, we utilize the model checkpoints of any ML regressor trained over iterations (e.g. neural networks, XGBoost etc). Specifically, during iterative training, the model parameters vary over E E N* iterations, taking E different parameter values at each checkpoint, i.e. 01, .,...... 0g. We wish to account for the impact of these changing parameters. At each checkpoint 0c. we compute the CPD to obtain the TRIAGE score for each sample xt We repeat the computation for every checkpoint to obtain a TRIAGE score trajectory over checkpoints per-sample of dimensionality E. i.e. we evaluate the function Q for specific Se_ (1)) to get the estimated probability P(y S Se. (z))) We repeat for all fe, (z) checkpoints where e E E to get the trajectory of TRIAGE scores for sample.
To quantify and contrast the behavior of the different sample trajectories, we compute two metrics to summarize the per-sample trajectory (see Fig. 2), namely (1) Confidence: reflects the degree of confidence wrt to Ply < fo(z)). It is computed as the mean of the TRIAGE score over training. i.e. Cl y²) = 1 -e= T(x', ,y", 0z). (2) Variability: reflects the consistency or fluctuation of the score for a specific sample. It is computed as the standard deviation of the TRIAGE score over
training, i.e. V( v²) / EE (T(a" -((((")))) We describe next how these metrics
form the basis of sample characterization and group assignment.
4.3 Characterization & group assignment (Step 3)
We now discuss how the two characterizing metrics, and V. are used to assign a group label E g to each training sample =² Let's intuitively define each group. ! Under-estimated (UE): low variability V (i.e. decisive about) and HIGH average TRIAGE score C, i.e. P(y S fxx) > 0.5,
2 Over-estimated (OE): low variability V (i.e. decisive about) and LOW average TRIAGE score i.c. P(y < f(x)) < 0.5, and 3 Well-estimated (WE): average TRIAGE score C' is uncertain P(y S f(x) 0.5. In addition, these samples are often linked to high variability, implying uncertainty about P(y < f())). We empirically observe samples with these different traits per group, depicted in the characteristic curve, see Fig. 2.
5

---

We take a previous characterization works [8. 9]:
approach to assign g"", similar to
Under-aalmmtted
Wollendimmted
Overeeatintedd
UE
if C(""", ,y") 2 Cup A V(=", v²") < (Puu))
9"h
QE
if CF"""" ,y") I Clow A V(x", "(""","" ,y") < Pn (Pam))))
WE
otherwise
0.6
0.4
where Cup and Clow are upper and lower confidence thresh- old resp, and Pn the n-th percentile. We discuss the selection of Cup and Clow in Appendix A.
022
0.0
0.05
0.30
0.15
0.20
V(2.y)
In Sec. 5, we empirically show how the groups assigned to each sample can be used to sculpt/filter the training dataset or for other downstream tasks such as dataset selection or feature acquisition.
Figure 2: Training dynamics of samples
(above) are summarized a characteristic curve (below-gre)
4.4 Using TRIAGE with a variety of regressors (P4)
The general-punn purproe- characterization methods discussed are largely compatible with differentiable models (neural networks). However, in real-world settings (e.g.. healthcare, finance), practitioners often employ other iterative algorithms, like XGBoost/GBDTT [30]. TRIAGE addresses this limita- tion, offering model-agnosti approach using CPS as the conformity scores can be computed for any regressor. Of course, our use of training dynamies assumes regressor can be checkpointed. This still confers compatibility with broader class of models: neural nets, XGBoost, GBDTs, linear regression - satisfying P4. We underscore TRIAGE's s versatility beyond neural networks, allowing practitioners to use TRIAGE with their preferred model for specific application. Appendix A.2 details the ease of usage and space and time considerations.
Empirical investigation
This section presents an empirical evaluation demonstrating TRIAGE satisfies (P1) Consistency, (P2) Improved regression performance and (P3) Informative for data collection. Recall (P4) Regressor versatility is satisfied by TRIAGE's s design. Additional use-cases and results are in Appendix C.
Datasets. We conduct experiments on 10 real-world regression datasets with varying characteristics. i.c. different sample sizes (500-100k), dimensionality (8-85), and task difficulty. The datasets are drawn from diverse domains, including safery-critical medical regression: (i) Prostate cancer from the US [31] and UK [32], (ii) Hospital Length of Stay [33] and (iii) MIMIC Antibiotics [34]. Additionally, we analyze general UCI regression datasets [35], including Bike, Boston Housing, Bio, Concrete, Protein and Star. The datasets are detailed in Appendix B, along with further experimental details. We observe similar performance across different datasets. However, due to space limitations, we sometimes highlight results for a subset and include results for the remainder in Appendix C.
5.1 (P1) Consistent characterization
1.0
A key aspect of data characterization is: the ordering of samples matter [9, 10]. This is important when the scores are utilized for data sculpting/filtering For example, if we rank sort samples by their scores and filter subset of samples. Consequently, it is cru- cial to ensure the scoring method used for data characterization is stable and consistent. This consistency is desired across different seeds, but also different practitioners may adopt diverse model for data characterization.
0.8
0,63
Correlation
0.6
0.4
VeG
Loss
0.2
Gredtt
Erors/Resicull
Tlage (Ours)
Metrics
Figure 3: Stability of scoring meth- ods. TRIAGE is most consistent (highest Spearman (highestSSeermmnn correlation of 0.91 & lowest variation) averaged across datasets. Results on all 10 datasets are in Appendix C.I.
Consistency of TRIAGE. We compare the consistency and ro- bustness to variation of TRIAGE vs baseline general purpose methods: VoG [11], GradN [13], Loss [18]. We also consider simply computing the end of training Residual Errors (used in the
"Code:

---

TRIAGE conformity score). We assess the consistency of the scores across different model architec- uures/paramelerizations (described Appendix B.3). All models are trained convergence with early stopping. Similar to prior classification-basee based characterization methods [9, 10], we assess consistency based on the Spearman rank correlation of the scores evaluated for all model combinations.
Fig. 3, shows TRIAGE is the most consistent compared to baselines, with the highest Spearman correlation (0.91), satisfying P1. This consistency means insights derived using TRIAGE scores will also remain consistent. Analyzing each of the 10 datasets (see Appendix C.I), further shows that TRIAGE is stable in magnitude and rank order, whereas the baselines do not have such consistency both in magnitude or ordering across datasets, making the baselines challenging to use in practice.
Takeaway 1. TRIAGE has the most consistent scoring method compared to baselines over different datasets and model combinations, satisfying P1.
TRIAGE Errors/Residuals. We note that TRIAGE and Errors/Residuals (considered an ablation of TRIAGE's dy-
Triage Score
0.75
Examples w/ same error magnitude can be differentiated via Triage scores
namics component) have similar correlations. This raises two questions: (i) are the samples identified the same? We stratify the samples in Dusin into three equal sized groups ordered by TRIAGE or Error scores. The overlap is ~~0.77 highlighting difference. This leads to ask (ii) what added value does TRIAGE offer? Consider the case of different samples that have the same error magnitude. Er- rors/Residuals treat these samples the same, based solely on magnitude. Can TRIAGE offer greater precision to further differentiate samples with the same magnitude? As shown in Fig.4, TRIAGE scores evidently offer way handle samples differently, showing that samples with similar residuals/errors are associated with different TRIAGE scores. We show the phenomena holds across other general-purpose scoring rules in Appendix C.10. This result demonstrates the added value of the training dynamics aspect, to differentiate samples with the same error magnitude more granular level. This contrasts, errors/residuals which are an ablation of TRIAGE, computing the conformity score only at the final training checkpoint, instead of for all checkpoints. We explore the value of this differentiation to improve regressor performance next.
0.50
0.25
0.2
0.4
0.6
Error
Figure 4: Samples with the same error, have different Triage scores, highlighting the po- tential to differentiate samples
5.2 (P2) Improve regression performance
We now evaluate the ability of TRIAGE to improve regression performance by sculpting/fltering the data - by keeping only well-estimated samples. We study two setups using TRIAGE: (1) fine-grained filter (as described above) and (2) sculpting the data based on a deployment purpose.
5.2.1 Fine-grained filtering
0.20
0.20
In the previous section, we showed TRIAGE's ability to dif- ferentiate samples with more pre- cision. Does this differentia- tion with TRIAGE scores bene- fit model performance? To find out, we sort the training sam- ples from least most challeng- ing based on the baseline com- puted scores; Residuals/Errors, Loss, VoG, GradN. We then eval- uate test MSE performance, as we increase the proportion p of samples retained for training. iee increasingly retain more challenging samples with higher loss/VoG etc. As expected, the baselines (shown in Fig 5, blue) have lower MSE with more sam-
Test MSE
Toxt MSE
0.15
0.15
0.10
0.10
Basaline propartion retained.
retained. Baceline
"Trage applied top less data)
"Triage app lied top (i.e. less data)
(a) Errors
(b) Loss
0.209
Baseline fiter
Boseline
fiter
0.375
MSE
9:375
0310
0.125
0:100
5100
02
2.4
00
02
04
0.
0.3
Baseline propartion data jotained. "Thage is applied on top (1.o. less dota)
Baseline proportion of data retained.
*Triage is applied on top ll.c less datal
(c) Grad
(d) VoG
Figure 5: TRIAGE improves test MSE, fine-grained filter. For each haseline proportion retained, TRIAGE, is applied on top such that TRIAGE has less data (Drriage C DBseeline), only keeping well- estimated samples for cach proportion
7

---

ples retained for training. However, the elbow around 0.5, after which MSE does not decrease, shows that the added high-magnitude samples are not beneficial.
We now assess the henefit of differentiating these scores using TRIAGE. To do so, we apply TRIAGE as additional filter on top of the baseline retained samples only keeping the well-estimated subset of samples. i.c. at each baseline proportion TRIAGE reduces the number of samples retained compared the baseline s.t. Drriage C Doseline- Fig 5 shows, for all proportions we improve MSE performance over baselines simply by differentiating between samples using TRIAGE and filtering to keep only well-estimated samples. This performance benefit holds for all general-purpee purpose scoring methods, illustrating the value of TRIAGE as ame-inaieed filter, where even with less data than baselines, the careful selection of samples by TRIAGE leads to improved regression performance. Results on more datasets mirroring the above are in Appendix C.11.
Takeaway 2. TRIAGE's differentiation of samples can be used to augment other approaches fine-grained filter; improving regressor test performance, simply by keeping well-estimated samples.
5.2.2 Data sculpting to fit a deployment purpose.
In many situations, we might want take regressor trained on a large Dunin and deploy it in an area with limited labeled data. For instance, repurposing a regressor in a low-resource healthcare setting. This raises the question, instead of acquiring new data which in some cases might be impossible- could we sculpt the existing large Daain to be more "fit for purpose". The goal is to enable better regressor performance. Our setup assumes limited access to a small amount of data from the deployment environment, which is insufficient in sample size to directly train a highly performant regression model, yet we could use this data for calibration Deat purposes. Can TRIAGE help?
As a case-study, we consider a regression task similar to [36-38], using patient covariates to predict PSA blood levels, known predictor of prostate cancer mortality. This task has utility in low-resource areas with limited medical funds [38]. We use multi-country prostate cancer data in which their is a distribution shift and train a regressor on data from the US (SEER)[31] I]. We evaluate performance on data from the UK (CUTRACT)(32), where the smaller Deal is also drawn from. In addition to the normal train-large, test-small baseline, we evaluate the utility of sculpting Duain from the US using TRIAGE - only keeping well-estimated samples (Dw @0w==Drrige)) S.t Drriage C train Conformal calibration uses Deal from the UK.
We compare TRIAGE to several methods, as detailed in Appendix B.3, including: (i) data-driven training: training a model directly on the Deal or combining the two datasets (Dunin U Deai). (ii) prediction-based sculpting: train on Dant (UK data) and then subsequently filter Drain (US data) based on predictive uncertainty scores We assess NGBoost, Bayesian Neural Network, Gaussian Process, Bayesian ridge regression, conformal intervals and residuals/errors. We only consider general purpose methods whose criteria allow us to leverage Deal and are compatible beyond neural nets for flexibility.
We then evaluate MSE test performance of regressors trained on the different "sculpted" Dtrain- We test sensitivity to varying sizes of D in many settings, we might not have access to many calibration samples. Table 2 shows that TRIAGE consistently outperforms model-driven methods across the board. This suggests that predictive uncertainty scores are insufficient to select the samples with the highest utility in Dmmnn- This contrast implies TRIAGE's scoring method better selects samples in Dirii with greater utility, leading improved performance on D
Compared to data-driven approaches, we find that TRIAGE has the best MSE, specifically, in the small sample regime, where Deal only contains a limited number of samples n=100200 Of course, when sufficient data is available (2300 samples), sculpting has a reduced benefit. Rather, training directly on the sufficient Deat is preferable. The result highlights the viability of sculpting larger Durain with respect to a small Deal to improve regression in settings with limited data access - with TRIAGE offering the best performance benefits.
In addition, sculpting especially in healthcare settings should account for both (i) imbalanced feature subgroups and (ii) long-tailed outcomes. We assess this in Appendix C.7 and show that TRIAGE does retains strong performance on minority subgroups and long-tailed outcomes by virtue of the calibration dataset.
MModel-driven methods are fit on Dest such that we assess how uncertain samples in Dinin are with respect to Dea
8

---

Table 2: Comparison of test MSE for different approaches, changing the size of Dea TRIAGE's approach to data sculpting of the large Dinin with respect small Deal in shown to outperform alternatives with lower test MSE, especially when Pei from the target domain has small sample sizes. (4 better.. Note [Paiin| > [Peal
Fi: sample size
10
20
30
to
50
130
200
X0
400
200
Our (Sealpeing)
Naz TELAGE (OE U.S)
01884-6001
0.053e
d.d0s
0.065+
0.00
0.065-0.000
0.05--0000
0.000-0001
0AM6--U 0.955-80001
00.7-0.0:5
0.00-00011
0.45--000
0082-40022
0078-00009
Baacite
(10044 000
0,00
00544-0.00 0a54, 0.00 03154
0,054-0.00 0100 0,044-0000 0.02
00044-00002
0064--00022
Bawilin (Paa)
I (1905-4855
0100 0.012
012 0.000 0.044-0:00 61056-8100
0.055-00044
0947-09001
6843-41111
lowilte: (12a U 12g)
8 0:09--40000 0.864-00000 nas e 60
000--000
0047-0999
0044-0001
0065-0001
5017-4001
Erox
00000 01551 0:000
0:000 (0010
0.000
0:55
0011
0011
0021-00010
CP lntervals Scotgs
8 01064+ 0:506
0.00++ 0.533
0.106 0.043
0/073
n.013
6255-00002
6003-00011
00054-0003
DC51 --055
Prediction haad
NGBooe:
8 8.085-400006
0 0.121+-40041
0.086-100115
0.080--0.005
0.0944-0:005
0.196x00000
6.132-00:00
0.00-00111
0006--00055
0. 0.103--00006
scipping of Dus.
Basesian Dalez
0.003--0:000 0.115--4:223
0.101e-0.012
0.129-0..011
6.1294-0.016
0.111-000:3
0.006.00.005
6:110-00:0
0.114-00000
0.103--00000
BNN
01054-00200
01 0.006 01000--0.005 0107-000065 0.07--01000 0,071-0005 0.05--0011 0,05-00003
0/955-0101
010806-01006
Data insights from sculpting. We seek understand what types of samples are "sculpted". Such insights are especially useful in clinical settings. We see that US patients (Dmin) typically have higher cancer stages and higher PSA scores than their UK counterparts (Deal PPest) is precisely these dichotomous samples that are filtered by TRIAGE to improve predictive performance. These insights can be illustrated stakeholders via a radial diagram, as in Appendix C.5.
Additionally, given the potential data shift (around exchangeability). Appendix C.6 analyzes CPS calibration, along with continuous ranked probability score (CRPS) quantifying the quality of the predictive distribution.
Beyond predictive performance to fairness. We also show that sculpting the data has potential beyond simply improving predictive performance. In Appendix C.4, we illustrate how TRIAGE can be used to sculpt data to improve fairness of regressors based on access to a limited set of samples.
Takeaway 3. TRIAGE can help to improve regression performance by sculpting a larger dataset to be fit-for-purpose, simply by leveraging a small number of calibration samples.
5.3 (P3) Informative for data collection
We now demonstrate the value of TRIAGE for dataset selection and feature collectioon/aquuisttion.
5.3.1 Dataset selection: From sample-level to dataset level.
In real-world applications, might want entire dataset characterization, rather than just sample-level characterization. An understudied scenario is how to compare and select between different datasets. This need arises in organizations where data is stored in isolated silos and it is difficult to access; with synthetic data a common solution [39, 40] or alternatively when purchasing data via data marketplaces [41]. We study the former scenario, where we have multiple 'synthetic' data versions, produced by different ML models or data vendors. However, our assessment could easily apply to the latter of data marketplaces.
We explore the setting where the real Dirain is not accessible to calculate statistical similarity measures, yet still wish to compare synthetic datasets and select a version to train on. The underlying assumption is that the synthetic datasets reflect the true underlying distribution. However, depending on the generation process, some synthetic datasets might prove superior to others.
The question we address is: Can TRIAGE's ss data characterization allow us to compare and select the syuthetic training dataset that yields the best test performance on real data?
We simulate this scenario by generating synthetic training data using (V1) CTGAN and (V2) TVAE [42], representing 2 synthetic data vendors. We employ TRIAGE to analyze and compare them, based on the percentage of well-estimated exam- ples recommended for retention. As is standard, we train regressor on synthetic data and test it
Table 3: Comparing performance & (Triage retention).
Training on synthetic data w/ better utility (1 retained) has the lowest real test data MAE across datasets.
Dataset
(V1) CTGAN
(V2) TVAE
Bike
0.13 (45% Retained)
0.09 (58% Retained):
Bio
0.24 (45%
Retained)
0.21(52% Retained)
on real data (TSTR) [43, 44]. The best synthetic data should yield the lowest MAE on real test
Boston
0.16 (38%
Retained)
0.13(60%
Retained)
Concrete
0.19 (46%
Relained)
0.11 (48% Retained)
LOS
0.16 (47% Retained)
0.0%556% Retained)
data. Table 3 shows that where TRIAGE's rec-
MIMIC
0,05 (41% Retained)
0.08 (48%
Retained)
ommendation has a higher retention percentage (i.e. more well-estimated samples), that training with that datasst,, leads to lower MAE on real test data. This illustrates that even if the real data is
Prostale
0.21 (46% Relained)
0.18 (52% Retained)
Protein 0.22 (52%
Retained)
0.20 (54% Retained)
Stur 0.18 (38% Retained)
0.14 (51% % Retained)
9

---

unavailable, TRIAGE's characterization can be used by practitioners to select the synthetic training dataset that will give the best real test performance.
Takeaway 4. TRIAGE scores allow us to assess utility between different synthetic datasets a principled manner. We show that synthetic datasets that TRIAGE suggests retaining more samples (i.e. more well-estimated), allow for better generalization when testing on real-data (i.e. MAE).
5.3.2 Value of feature collection/acquisition.
Previous settings considered assume fixed dataset. What if we actively improve the data's utility by acquiring features? e.g., a doctor runs extra tests to improve regressor. Understanding the value of features one might acquire is useful, especially if acquisition is costly. We note feature acquisition differs from feature selection (e.g. [45]), where all features are present and we select useful ones. also differs from active learning, which quantifies sample acquisition value, not features.
We now show that TRIAGE enables principled approach to assess the benefit of acquiring specific feature. We posit that acquiring valuable feature wrt. the task should increase the proportion of well-estimated samples (which we've ve shown is linked to better performance).
UNDER
WELL
OVER
1.0
0.6
0.8
Proportion 0f data
Feature Corr
0.6
To illustrate the potential of TRIAGE scores, we set
0.4
an experiment where we rank the features based on their correlation to the target (i.e. increasing utility). We then simulate the sequential "acquisition" of features with in- creasing utility (based on correlation). Concurrently, we observe how the groups change as features are acquired.
0.2
0.2
0.0
0.0
10
Features least to most important added
Figure 6: The value of acquiring a feature can he quantified by its ability to increase the proportion of well-estimated samples.
Fig. shows that as we acquire "valuable" features, the proportion of the well-estimated samples increases, with the over- and under-estimated proportions dropping. This result, repeated on other datasets Appendix 12 shows TRIAGE scores are sensitive to the value of a feature, quantified by the increase in the proportion of well-estimated samples.
Takeaway 5. TRIAGE offers a principled way to assess active dataset improvement when we acquire new features. The feature's value links to the ability increase the well-estimated proportion.
6 Discussion
We present TRIAGE, a regression- focusee data characterization framework, an understudied area in data-centric AI. TRIAGE offers new scoring method to overcome the limitations of classification- centric scores. We show TRIAGE's characterization can unlock a variety of use cases to enhance regression, with minimal effort, beyond standard training a dataset and exchangeability assump- tions. We highlight that a key aspect of ensuring good performance of TRIAGE careful construction of the calibration replacing a data scientist's 's role, TRIAGE as a "data-centric AI" tool aims to empower data scientists to perform the "data" work in a more principled manner.
which should be representative as possible. Rather than automating or
Limitations and future opportunities: (1) understanding the attributes contributing to characteriza- tion, (2) provide theoretical guarantees taking into account the challenging interaction between the CPD and Triage trajectory (for more see Appendix C.9). (3) in high-stakes settings, to prevent harm could leverage domain expertise, where a human-in -oman-in-the-logg could audit the samples surfaced by TRIAGE, before sculpting.
Acknowledgements
The authors are grateful to Fergus Imrie, Boris van Breugel, Paulius Rauba and the anonymous NeurIPS reviewers for their useful comments & feedback. The authors would also like to thank Richard Samworth, Marco Scutari, Qingyuan Zhao and Yao Zhang for their insightful discussion on robust statisties linked to TRIAGE. Nabeel Seedat supported by the Cystic Fibrosis Trust, Jonathan Crabbe by Aviva, Zhaozhi Qian by Cancer Research UK.
10

---

References
[1] Peter Norvig, On big data the unrcasonable effectiveness of data, 2011.
[2] Abhinav Jain, Hima Patel, Lokesh Nagalapatti, Nitin Gupta, Sameep Mehta, Shanmukha Guttula, Shashank Mujumdar, Shazia Afzal, Ruhi Sharma Mittal, and Vitobha Munigala. Overview and importance of data quality for machine learning tasks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3561-3562, 2020.
[3] Nitin Gupta, Hima Patel, Shazia Afzal, Naveen Panwar, Ruhi Sharma Mittal, Shanmukha Guttula, Abhinav Jain, Lokesh Nagalapatti, Sameep Mchta, Sandeep Hans, al. Data quality toolkit: Automatic assessment of data quality and remediation for machine learning datasets. arXiv preprint arXiv:2 108. 108.05935 2021.
[4] Cedric Renggli, Luka Rimanic, Nezihe Merve Gürel, Bojan Karlas, Wentao Wu, and Ce Zhang. A data quality-driven view of mlops. IEEE Data Engineering Bulletin, 2021.
[5] Haihua Chen, Jiangping Chen, and Junhua Ding. Data evaluation and enhancement for quality improvement of machine learning. IEEE Transactions on Reliability, 70(2):831-847, 2021.
[6] Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, 4(8):669-677, 2022.
[7] Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar. De-check: A data-centric ai checklist guide the development of reliable machine learning systems. arXiv preprint 4,5554221,.05764 2022.
[8] Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Conference on Empirical Methods in Natural Language Processing, 2020.
[9] Nabeel Seedat, Jonathan Crabbé, Ioana Bica, and Mihaela van der Schaar. Data-iq: Characteriz- ing subgroups with heterogeneous outcomes tabular data. In Advances in Neural Information Processing Systems, 2022.
[10] Pratyush Maini, Saurabh Garg, Zachary Chase Lipton, and J Zico Kolter. Characterizing datapoints via second-split forgetting. In Advances in Neural Information Processing Systems, 2022.
[11] Chirag Agarwal, Daniel D' ssuuza, and Sara Hooker, Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10368-10311 2022.
[12] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:1 13:17044-77056, 2020.
[13] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34:20596-20600 20607, 2021.
[14] Vladimir Vovk, Jieli Shen, Valery Manokhin, and Min-ge Xie. Nonparametric predictive distributions based on conformal prediction. Machine Learning, 108(3):445-474 2019.
[15] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization deep networks. In International conference on machine learning. pages 233-242. PMLR, 2017.
[16] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference Machine Learning, pages 322-332. PMLR, 2019.

---

[17] Yuanzhi Li, Tengyu Ma, and Hongyang R Zhang. Learning over-parametrized two-layer neural networks beyond ntk. In Conference on learning theory, pages 2613-2682. PMLR, 2020.
[18] Shoaib Ahmed Siddiqui, Nitarshan Rajkumar, Tegan Maharaj, David Krueger, and Sara Hooker. Metadata archaeology: Unearthing data subsets by leveraging training dynamics. arXiv preprini arXiv:2209. 10015, 2022.
[19] Andrew Ng, Lora Aroyo, Cody Coleman, Greg Diamos, Vijay Janapa Reddi, Joaquin Van- schoren, Carole-Jean Wu, and Sharon Zhou. Neurips data-centric ai workshop, 2021.
[20] Nabeel Seedat, Jonathan Crabbe, and Mihaela van der Schaar. Data-suite: Data-centric iden- tification of in-distribution incongruous examples. In International Conference on Machine Learning. PMLR, 2022.
[21] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2018.
[22] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pages 2242-2251 PMLR, 2019.
[23] Ruoxi Jia, David Dao, Boxin Wang, Frances A Hubis, Nezihe M Gürel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. Proceedings of the VLDB Endowment, 12(11) 11111:161---2,,, 2019.
[24] Hoang Anh Just, Feiyang Kang, Tianhao Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. In The Eleventh International Conference on Learning Representations, 2022.
[25] Jieli Shen, Regina Y Liu, and Min-ge Xie. Prediction with confidence a general framework for predictive inference. Journal of Statistical Planning and Inference, 195:126-144 140, 2018.
[26] Tore Schweder and Nils Lid Hjort. Confidence, likelihood, probability, volume 41. Cambridge University Press, 2016.
[27] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Conformal prediction. Algorithmic learning in random world, pages 17-51, 2005.
[28] Vladimir Vovk, Ivan Petej, Ilia Nouretdinov, Valery Manokhin, and Alexander Gammerman. Computationally efficient versions of conformal predictive distributions. Neurocomputing, 397:292-308, 2020.
[29] Ulf Johansson, Henrik Boström, Tuve Löfström, and Henrik Linusson. Regression conformal prediction with random forests. Machine learning, 97(1):155-17) 2014.
[30] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug. Martin Pawelczyk, and Gjergii Kasneci. Deep neural networks and tabular data: A survey, arXiv preprint arXiv:2110.01 889, 2021.
[31] Máire A Duggan, William F Anderson, Sean Altekruse, Lynne Penberthy, and Mark E Sher- man. The surveillance, epidemiology and end results (seer) program and pathology: towards strengthening the critical relationship. The American journal of surgical pathology, 40(12):e94, 2016.
[32] Prostate Cancer UK. Cutract. https:/ppostaaeeanceenk..or 2019. 2019.
[33] Microsoft. Predicting hospital length of stay, 2017.
[34] Alistair EW Johnson, Tom Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[35] Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
12

---

[36] Thomas A Stamey, John N Kabalin, John E McNeal, lain M Johnstone, Fuad Freiha, Elise A Redwine, and Norman Yang. Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate. ii. radical prostatectomy treated patients. The Journal of urology, 1476551006-1083, 1989.
[37] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining. inference, and prediction, volume 2. Springer, 2009.
[38] Gongli Li and Han Li. Linear model selection and regularization for serum prostate-speciffc antigen prediction of patients with prostate cancer using r. IEEE Access,
2021.
[39] Khaled El Emam, Luey Mosquera, and Richard Hoptroff. Practical synthetic data generation: balancing privacy and the broad availability of data. O'Reilly Media, 2020.
[40] Andre Goncalves, Priyadip Ray, Braden Soper, Jennifer Stevens, Linda Coyle and Ana Paula Sales. Generation and evaluation of synthetie patient data. BMC medical research methodology. 20(1):1-40, 2020.
[41] Markus Spickermann. Data marketplaces: Trends and monetisation of data goods. Intereco- nomics, 54(4):208 216, 2019.
[42] Lei Xu, Maria Skoularidou, Alfredo Cuesta- and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems, 32, 2019.
[43] Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint urXiv:1706.02633 2017.
[44] Michael Platzer and Thomas Reutterer. Holdout- empirical assessment of mixed-type synthetic data. Frontiers in big Data, page 43, 2021.
[45] James Jordon, Jinsung Yoon, and Mihaela van der Schaar. Knockoffgan: Generating knockoffs for feature selection using generative adversarial networks. In International conference on learning representations, 2018.
[46] Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "everyonee wants to do the model work, not the data work": Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-15, 2021.
[47] Sezal Chug, Priya Kaushal, Ponnurangam Kumaraguru, and Tavpritesh Sethi. Statistical learning to operationalize a domain agnostic data quality scoring. arXiv preprint arXiv:2108 0et::2208..00000 08905, 2021.
[48] Neoklis Polyzotis and Matei Zaharia. What can data-centric learn from data and ml engineer- ing? arXiv preprint arXiv:2112.06439, 2021.
[49] Nabeel Seedat, Alan Jeffares, Fergus Imrie, and Mihaela van der Schaar. Improving adaptive conformal prediction using self-supervised learning. In International Conference Artificial Intelligence and Statistics, pages 10160-10177. PMLR, 2023.
[50] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34, 2021.
[51] Michael Redmond and Alok Baveja. data-driven software tool for enabling cooperative information sharing among police departments. European Journal of Operational Research, 141(3))660-677 2002.
[52] Daniel Steinberg, Alistair Reid, and Simon O'Callaghan. Fairness measures for regression via probabilistic classification. arXiv preprint arXiv:2001 27X00:2001.00089, 2020.
[53] Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. Fair regression: Quantitative defini- tions and reduction- based algorithms. International Conference on Machine Learning, pages 120-129. PMLR, 2019.
13

---

[54] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. Certifying and removing disparate impact. proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259-268. 2015.
[55] Equal Employment Opportunity Commission et al. Uniform guidelines on employee selection procedures. Fed Register, 1:216-243, 1990.
[56] Hans Hersbach. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting, 15(5)5599-570, 2000.
[57] Jacob Steinhardt. Robust learning: Information theory and algorithms. Stanford University, 2018.
[58] Ricardo Maronna, R Douglas Martin, Victor J Yohai, and Matías Selibián-Barrens Robust statistics: theory and methods (with R). John Wiley & Sons, 2019.
[59] Mengjie Chen, Chao Gao, and Zhao Ren. A general decision theory for huber's contamination model. 2016.
14


Reviews:
[{"review_id": "5dxAMXSmkO", "reviewer": "Reviewer_ckq8", "summary": "This paper investigates the problem of training data characterization for regression problems. The authors noted that existing research on training data characterization mostly focuses on classification problems and there remains an absence of research for regression problems. This work proposes TRIAGE, a novel framework designed exclusively for regression settings and applies to a variety of tasks.\n\nTRIAGE leverages conformal predictive distributions to provide a model-agnostic scoring method for evaluating how the model performs on each training sample. This framework also enables analysis into the training dynamics from checkpoints, visualizing the samples being under-/well-/over- estimated by the model during the training process. \n\nThe proposed framework is useful for a variety of data curation tasks, such as improving the model performance by throwing poorly-predicted training samples. Beyond sample-wise data analysis, the framework also applies to broader tasks such as dataset selection/feature acquisition. The work validates the proposed method with a range of empirical studies on 10 tableau datasets with 500-100k samples, showcasing its value in data characterization in practical cases.", "questions": "See Weaknessnes.", "limitations": "No major limitations. Questions for discussion are listed above.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The problem being studied and data-centric perspectives are attractive and important. The work is well-motivated and nicely written with easy-to-follow illustrations.\n\nThe narrative of the paper is highly structured, skillful, intriguing and informative. I did enjoy reading this paper. The importance of the work is highlighted. References are substantial and high quality.\n\nObjectives P1-P4 and the comparison table with baselines are clear and precise which can be informative and beneficial for a broader audience. And the improvements in these objectives are later validated in empirical studies, facilitating the verification of this work. \n\nThe technical body is clear. The analysis of training dynamics from checkpoints is interesting. Novel insights are provided in the takeaways. The work is practical with many interesting use cases.\n\n", "weaknesses": "I am combining this part with the questions. \n\n1. Since this work is situated in the context of data-centric AI and also studies data characterization problems, it is a bit unnatural to miss out on the comparisons with data valuation methods. \n\nFor example, Shapley-based methods are a common benchmark for the general interpretability of ML applications and also apply to characterizing the effects of training data on regression tasks. Also, I was wondering how model-agnostic data valuation ([1]) pipelines work in this setting.\n\nI would like to see discussions both conceptually and empirically on how those methods perform in the context this paper studies and how they differentiate from the proposed approach.\n\n[1] Lava: Data valuation without pre-specified learning algorithms, ICLR 2023\n\n2. What is the rationality for throwing out data that is over/under-predicted? It could be outliers, but would simply throwing out such data hurt the generalizability of the model or its robustness against distributional shifts, which is prevalent in real-world applications? Do the authors have any empirical study results on such cases?\n\n3. Data selection is an established field of research. This work showcases the proposed approach apply to dataset selection tasks. It would be beneficial if the authors could provide a direct comparison with data selection baselines and visualize their performance.\n\nIt won't harm the contribution of this paper if its data selection performance is suboptimal compared to methods designed exclusively for data selection tasks. But I think it is important to know how they compare exactly and benchmark the gap, which would be helpful for future works to compare with or improve over.\n\n4. What is the computational overhead of the proposed framework? Can the authors provide the computation time? How scalable is the proposed framework? What is the largest scale the authors are able to apply it to?\n\nThe text in Figure 1 can be made larger.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear ``R-ckq8``. \n\nThank you for your thoughtful comments to improve the paper. We provide answers (A)-(D) & highlight updates to the paper\n\n# (A) Additional comparisons\nThanks for suggesting an empirical & conceptual comparison to valuation methods (e.g. Shapley-based & LAVA) to strengthen the results. \n\nFirst, we clarify that the Gradient Norm (GraNd) baseline [12] from the main paper falls into the data selection literature. Besides showing TRIAGE improves upon GraNd. One key disadvantage of selection via GraNd links to (P1) Consistent characterization, where we show in Fig 4 that the GraNd selector is not consistent across model parameterizations with a Spearman correlation of 0.6 compared to TRIAGE’s 0.91. This means [12] would select different subsets for different models compared to TRIAGE. Additionally, GraNd only applies to neural nets which is less flexible than TRIAGE.\n\n*We now compare to data selection w/ data valuation.*\n\n**(i) Experimentally:** We compare two Shapley valuation methods: (i) TMC-Shapley [R1] & (ii) KNN-Shapley [R2] and LAVA [R3] as suggested, which uses optimal transport. We compare in the context of Table 2- data selection for sculpting. The TRIAGE $D_{cal}$ is used as the validation set for these methods.\n\nNote - TMC-Shapley: was computationally unfeasible for all 20k samples. Hence, we sample 5k and compare it to TRIAGE separately. (ii) KNN-Shapley & LAVA: run on the original 20k samples.\n\n**Performance:** The results are shown in the **response pdf (Table 1 & 2)**. These methods are competitive with TRIAGE. However, TRIAGE tailored to regression outperforms them on downstream MSE performance.\n\n**Computational time:**  We assess compute time vs TRIAGE and show for different sizes of $D_{cal}$ that TRIAGE is more time efficient & unlike Shapley methods, our cost doesn’t increase with the size of the $D_{cal}$. KNN-Shapley is 1-3X & TMC-Shapley 600X more expensive than TRIAGE. We show the results in **Fig. 2 in the response PDF**.\n\n**UPDATE:** Include new results in Table 2 and add a new Appendix to discuss the new computational time experiments. \n\n**(ii) Conceptually:** Differences between valuation methods & TRIAGE are: (a) TRIAGE unlocks other data-centric tasks beyond just data selection, e.g. feature-level acquisition/collection and selection between datasets. These are out of scope for data “sample” selection; (b) Unlike TRIAGE, these methods are not tailored to regression; (c) method differences: computationally Shapley-based methods need to assess multiple sample permutations & need to retrain the model many times. Hence, they struggle to scale to very high-samples sizes (e.g. 100k), unlike TRIAGE where the cost is cheap comparatively (we experimentally compare this later). LAVA uses an additional embedding model to reduce the dimensionality before the optimal transport step.\n\n**UPDATE:** Include a discussion with references on data valuation (Shapley & LAVA) in the related work (Sec. 2).\n\n# (B) Data sculpting under distribution shift  \nWe agree that the distribution shift setting is interesting to assess.  Our experiment in **Section 5.2.2: \"Data sculpting to fit a deployment purpose\"** evaluates this setting, where we sculpt $D_{train}$ of US patients to perform well on a different distribution of UK patients at deployment time. We apologize that this was unclear and will update the text to clarify. \n\nTable 2 shows that with a small UK calibration set, TRIAGE sculpts the US data so that the regressor generalizes well on the UK test set, achieving the lowest MSE compared to baselines.\nThis highlights the importance of calibration set construction: even in a scenario with a distribution shift, good performance is attainable with TRIAGE by calibrating with a handful of relevant samples.  We will include a discussion on point in Sec. 5.2.2.\n\nAdditionally, in **Appendix C.6.1**, we look at generalization for distinct patient groups, specifically the minority sized group. In this imbalanced setup, TRIAGE due to calibration preserves the performance for the minority group, contrasting favorably with baselines. We apologize that this was obscured, given the numerous appendices and will refer to this more prominently than our initial mention on L330-331.\n\nFinally, on insights, we refer the reviewer to **Appendix C.4**, where we show a radial plot (Fig. 14) providing insights into why we sculpt certain samples (over/under-estimated) — thereby allowing us to match our deployment setting better and generalize better. We use the extra camera ready page to add Fig 14 to the main paper to anchor Table 2 with the insights.\n\n**UPDATE:** Improve description of Sec. 5.2.2 to convey this is a distribution shift setting, better flag the results of Appendix C.6.1 in the main paper, which assesses minority and majority sized group performance, move the radial plot (Fig. 14)  to the main paper to provide further insights on sculpting.\n\n# (C) Computational time\nThank you for the suggestion. We agree that analyzing the time cost to compute TRIAGE scores is important. We have run a new experiment where we vary the dataset size from 2000-100k samples and assess the TRIAGE computational time. Naturally, as the size increases, so does computational time. However, even at 100k samples computing the TRIAGE scores for all 100k takes <2min highlighting TRIAGE’s time efficiency to scale to larger data sizes. We show the results in **Fig. 2(a) of the response pdf uploaded**. We have also compared computational time to valuation methods as mentioned in (A). \n\n**UPDATE:** Add a new Appendix to include the computational time results.\n\n# (D) Text size Fig. 1\nWe will increase the font size in Fig. 1 for readability.\n\n[R1] Ghorbani, A., & Zou, J. Data shapley: Equitable valuation of data for machine learning. \n\n[R2] Jia, R et al. Efficient task-specific data valuation for nearest neighbor algorithms. \n\n[R3] Just, H. A et al. LAVA: Data valuation without pre-specified learning algorithms.\n"}, {"review_id": "cQmzVCuiwO", "reviewer": "Reviewer_oVcJ", "summary": "The problem studied in this work is the following: Given a dataset $\\lbrace (x_i, y_i ) \\rbrace_{i=1}^M$ and a regressor $f_\\theta$ trained on this dataset, assign a group label $g_i$ to each sample that specifies whether the regressor under or overestimates on the sample. Such group labels can be used to identify and remove outliers in the dataset. The key formula of the proposed method is Eqn. (1), which is similar to Eqn. (5) in [23] that takes into account the \"prediction accuracy\" over each sample. The authors proposed to first put each sample into one of the several bins defined on a calibration set, and then assign a CPD score accordingly. The group label is then assigned using the CPD score as well as its variance. In the experiments, the authors use these group labels to remove outliers in the training set, and observe that it can improve the regression performance for a variety of regressors.", "questions": "1. There are several confusing points in the manuscript that require clarification:  \n(a) In Algorithm 1, line 3, what do you mean by \"nearest neighbor residuals of KNN\"? Is it the average distance to the k nearest neighbors?    \n(b) In Eqn. (11) of [23], the definition of $\\sigma$ considers two factors: (i) Whether a sample is close to its k nearest neighbors; (ii) Whether the k nearest neighbors have consistent labels. Does this work use the same definition for $\\sigma$? If not, what are the differences?  \n(c) In the definition of the TRIAGE score $T(x,y,\\theta)$ in lines 161-162, what do you mean by \"$P(y \\le f_\\theta(x))$\"? What is this probability taken over? If $x, y, f_\\theta$ are all deterministic, why should there be a probability?   \n\n\n2. Can the authors make a thorough comparison between this work and [23], including the methodology, the algorithm, the experimental setup and results? This could help clarify the contributions of this work.\n\n3. (Bonus) It would be great to make a theoretical connection between the proposed method and robust statistics. Consider a simple case, where $x$ is 1-dimensional, and the goal is to fit $y=f(x)$ on a dataset $(X,Y)$ with outliers. The dataset could follow Huber's contamination model. In such a scenario, I am curious about whether TRIAGE is provably better than the error baseline (as empirically compared in lines 244-260), and whether there is any guarantee on the performance of the predictor with the outliers removed using TRIAGE. A good starter for robust statistics could be the thesis of Jacob Steinhardt: https://cs.stanford.edu/~jsteinhardt/publications/thesis/paper.pdf. This is a bonus point, but such theoretical analysis could greatly enhance the contributions of this work.\n\n**Post rebuttal note:** I have read the rebuttal and had a discussion with the authors. The authors have addressed most of my questions. I have raised my rating from 4 (original) to 6 (current).", "limitations": "Limitations are discussed in Section 6.\n\n**Review summary:** I think this work is interesting overall, and there are some good takeaways. However, the writing is confusing at times, and it also seems to me that a large part of this work is very similar to [23]. I believe many people will find this work useful, but I also believe that this work still needs improvement. So I give a borderline rating to this submission though I rarely give borderline ratings in my reviews. If the authors could address my questions during the rebuttal, I am willing to raise my score to 6 or 7, depending on the quality of the rebuttal.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The problem studied in this work is very important for real applications since most datasets contain outliers.\n- The desiderata for data characterization, as listed in (P1)-(P4) after line 44, are very clear and define the overall goal of this work.\n- The experimental results presented in Section 5 support that the proposed method satisfies the desiderata.", "weaknesses": "- The overall presentation is not good enough and there are many confusing points in this work, which I will discuss in the Questions section.\n- The authors claim TRIAGE to be \"the first data characterization framework tailored to regression settings\" (lines 63-64), yet in [23] cited in this work, a similar approach for regression conformal prediction was proposed. In fact, the proposed method in this work largely resembles the method in [23], including the CPD score definition Eqn. (1) (versus Eqn. (5) in [23]), the use of calibration scores $\\lbrace \\alpha_1,\\cdots,\\alpha_q \\rbrace$ (versus Eqn. (1) in [23]), and the use of KNN for estimating $\\sigma$ (versus Eqn. (11) in [23]). Thus, I am not sure about the contributions of this work. However, I am sure that \"TRIAGE is the first data characterization framework tailored to regression\" is an overclaim.\n- I feel that the writing is too wordy, especially in Section 5, and the authors use some LaTeX tricks which make the manuscript looks more compact than necessary. I think the authors are able to make Section 5 more concise, so that the main text could easily fit in 9 pages and the paper would have a much better shape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear ``R-oVcJ``. \n\nThank you for your thoughtful comments to improve the paper. We provide answers (A)-(D) & highlight updates to the paper.\n\n# (A) Comparing TRIAGE to Ref [23]\nWe discuss differences & similarities to illustrate TRIAGE’s contribution.\n\n**Differences:**\n1. **Objective:** TRIAGE performs data characterization, scoring samples on their impact on a regressor, enabling data-centric tasks like data sculpting, dataset selection & feature acquisition. In contrast, [23] is conventional conformal prediction for predictive uncertainty estimation.\n\n2. **Algorithm:** (1) TRIAGE uses conformal predictive distributions (CPDs) providing a full predictive distribution. This contrasts [23]’s prediction intervals, which are less informative than a predictive distribution (see L50-52). (2) TRIAGE’s novelty is studying the training dynamics of scores. In contrast, [23] computes prediction intervals *once* after training, not reflecting dynamic changes, which we show are vital to characterize differences between samples. \n\n3. **Experiments:** TRIAGE tackles data-centric tasks like data sculpting (Sec 5.2),  dataset selection (Sec 5.3.1) & feature collection/acquisition (Sec 5.3.2). In contrast, [23] evaluates the prediction intervals for predictive uncertainty (coverage & efficiency).\n\n4. **Results:** While [23] tackles different tasks from TRIAGE, we adapt the CP intervals for sculpting (Sec 5.2.2). Table 2's baseline \"*CP Intervals Sculpt*\" corresponds to a conformal regressor as in [23]. We will clarify this in the revision. Table 2 shows that porting CP intervals for sculpting are less effective than TRIAGE tailored for this task.\n\n**Similarities:** (i) We clarify that Eq 1 (similar to Eq.5 in [23]) is *NOT* the CPD score but simply a conformity score. The CPD is per Eq 2. We will amend L145-150 to clarify. (ii) we clarify that the similarities noted on conformity score & calibration sets usage are general elements fundamental to  conformal prediction itself & not unique to any method. This is akin to loss functions (conformity scores) & validation sets (calibration sets). \n\n**UPDATE:** Add Appendix discussing differences between TRIAGE & [23].\n\n# (B) Clarifications\n(i) Clarify KNN residuals: We compute the errors for all samples in $D_{cal}$. For sample $x_i$ we estimate its difficulty score $\\sigma$ as the mean absolute errors of $x_i$’s k-nearest neighbors in $D_{cal}$, where K=5\n\n(ii) Do we use the same KNN definition as [23]: [23] evaluates multiple variants of the KNN score, including Eq 11 (see Papadopoulos et al. (2011)). To clarify, we use the KNN definition in Sec 3.1 of [23], which we describe in (i)\n\n(iii) Clarifying the probability in the TRIAGE score: The CPD in Eq. 2, denoted $Q(y)$, returns a predictive (probability) distribution. We discuss the necessary condition for this interpretation on L148-150.  Then for a specific value $y$, the function returns the estimated probability $P(Y \\leq  y)$, where $Y$ - true target and $y$ - prediction $f_{\\theta}(x)$. This score is then computed for the $E$ different checkpoint parameter values.\n\n# (C) Streamline Sec. 5\nBased on your suggestion, we will streamline the writing in Sec. 5, allowing us to expand the Figs side-by-side across the page width.\n\n# (D) Bonus - TRIAGE & Robust Statistics\nWe are grateful for the reviewer's suggestion & for sharing the thesis resource. \n\n(i) **TRIAGE vs. Robust Statistics**:\nMotivated by your question, we contrast TRIAGE & Robust Statistics. (1) Post-hoc vs Built-in:  TRIAGE wraps a regressor to detect and sculpt outlier samples, whereas Robust Statistics embeds outlier resilience within the model [R1,R2], e.g. via Huber loss [R2].  (2) Additional data-centric applications: TRIAGE tackles diverse \"data-centric AI\" tasks, like comparing synthetic data (Sec. 5.3.1) and feature acquisition/collection (Sec. 5.3.2), which is beyond the scope of robust statistics.\n\n(ii) **Theoretical Analysis**:\nConnecting TRIAGE theoretically to robust statistics is an intriguing question. However, we highlight two important challenges:\n\n-\tInterdependence of CPD and training dynamics in TRIAGE. Disentangling their impacts theoretically is non-trivial. \n\n-\tScores across training epochs are correlated due to iterative model training. This highlights the complexity of any theoretical guarantee, given the dynamic nature of the scores & their correlation.\n\nGiven the complexities, we will discuss the theoretical proof as future work in Sec. 6, citing the provided thesis and the suggested Huber contamination setting. We will also add an Appendix outlining the aforementioned challenges.\n\nAs an initial step, we provide simulations. \n\n(iii) **Simulation using Huber's Contamination Model**:\nOur simulation setup mirrors [R3], generating data from a linear model ($y = X\\beta + \\eta$), with $X\\sim U[0,1]$ and $\\eta\\sim \\mathcal{N}(0,1)$. Mimicking Huber's model we contaminate the response $y$ corrupting $\\epsilon$ samples, replacing ($\\eta_i$) with ($\\eta_i + c_i$), where ($c_i$) comes from a different distribution.\nWe compare TRIAGE with:\n•\tError baseline\n•\tTraining with Huber Loss\n•\tTRIAGE applied to a Huber Loss trained model.\n\nThe results in the **uploaded pdf (Fig 1)** show TRIAGE has lower MSE vs the error baseline as $\\epsilon$ rises. TRIAGE is also stable in response to contamination by virtue of the clean calibration set.  Interestingly, combining TRIAGE with a model trained using Huber’s loss proves superior to using either alone, highlighting the compatibility of TRIAGE with robust techniques.\n\n**UPDATE**:\n - Add discussion about proof to Sec.6, citing the Huber setting & the resource (thesis)[R1]\n\n - New Appendix to discuss the challenges of a theoretical proof\n\n- New Appendix with the simulation results\n\n[R1] J. Steinhardt. Robust learning: Information theory and algorithms\n\n[R2] R. Maronna, et al. Robust statistics: theory and methods \n\n[R3] M. Chen,, et al. A general decision theory for Huber’s ∈-contamination model\n\n\n"}, {"review_id": "RoJLVd0sf9", "reviewer": "Reviewer_pEXt", "summary": "The authors introduce a new data characterization framework, TRIAGE, for regression models. The method utilizes conformal predictive distributions to compute the training examples' scores. To compute TRIAGE scores, the authors use predictive distributions and conformal prediction. A proper training set is used to train a regressor, and a separate calibration set is used for conformal calibration. Conformity measures the dataset's agreement with the observation. Consequently, the conformal predictive scores are computed at each epoch for each training point. Afterward, TRIAGE measures the mean and standard deviation for each training point. The method analyzes each example's training dynamics at each epoch to group each point into one of the groups: under/over/well estimated by the model based on the thresholds. TRIAGE can reduce the number of samples to train the regressor compared with the baseline methods. Specifically, it is observed that the MSE performance is improved with the selected data. It holds an advantage steadily over four different scoring methods.\n", "questions": "+ How much does the calibration dataset affect the data sculpting performance?\n+ How exactly are distances computed in KNN for residuals of the calibration dataset? Does it use some embedding space?\n+ What exactly do authors mean by model-agnostic? It seems that conformity scores are computed on residuals which are based on trained regression model predictions.\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The authors propose a novel data characterization method for any regression models that analyze the training dynamic of each training point. Using conformal predictive systems for this problem is intuitive and effective.\n+ TRIAGE effectively reduces the number of samples to train regressors compared with the baseline methods. It shows that the MSE performance is improved with data keeping and holds an advantage for four different scoring methods.\n+ The paper is well written.\n", "weaknesses": "+ Time complexity can be enormous for large neural networks to compute scores over multiple epochs.\n+ Only simple datasets and simple regressors are performed. The data sculpting experiment is also limited to 500 available samples.\n+ Some more challenging points might be important in the medical field, but if TRIAGE discards those samples, the model would not learn the critical medical case.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear ``R-pEXt``. \n\nThank you for your thoughtful comments which have helped improve the paper. We provide answers (A)-(E) & highlight updates to the paper\n\n# (A) Computational time\nThank you for bringing up this point. We agree that analyzing the time cost to compute TRIAGE scores is important. We have run a **new experiment** where we vary the dataset size from *2000-100k* samples and assess the TRIAGE score computational time. Naturally, as the dataset size increases, so does computational time. That being said, even at 100k samples computing the TRIAGE scores for all samples takes <2min, highlighting the time efficiency and capability of TRIAGE to scale to large data sample sizes. We show the results in **Fig. 2 of the response pdf**. \n\n**UPDATE:** Add the new result as a new Appendix, as an important analysis of the time cost of TRIAGE.\n\n# (B) Clarifying data sculpting experiment sample size (“limited to 500 samples”)\nWe wish to clarify the sample size (Sec 5.2.2). The limited sample size up to 500 samples is just for the $D_{cal}$ (calibration dataset) in order to demonstrate that we only require a few calibration samples. The training set ($D_{train}$) which TRIAGE audits and sculpts, is the SEER medical dataset from the US which has a large sample size of **20k samples**. We apologize if this was unclear. \n\n**UPDATE:**  We will update Sec 5.2.2 to include the sample size of the SEER dataset to make it clear that we audit and sculpt a large dataset and that small samples are confined to $D_{cal}$.\n\n# (C) Clarifying simple datasets & regressors\nWe wish to clarify that the datasets used are not simple but rather real-world datasets. As mentioned on L212-219 & Table 4 (Appendix B.2), our *10 datasets span sample sizes (500-100k) & dimensionality (8-85)*. In addition, the medical datasets used are reflective of actual regression settings. For instance, (i) SEER [25] and CUTRACT are Prostate cancer datasets from US and UK hospitals, (ii) Hospital Length of Stay [27] and (iii) MIMIC Antibiotics [28] are also real-world medical datasets.\n\nWe also study a variety of powerful regressors, including neural networks, XGBoost and CatBoost. These regressors are likely to be used in practice on these large and high-dimensional datasets. \n\n**UPDATE:**  We will clarify these two aspects at the start of Sec. 5.\n\n# (D) Discarding critical samples (e.g. medical domain)\nWe appreciate the important consideration around data sculpting, especially in medical domains where minority sized groups are often critical samples.  Our evaluation on precisely this issue can be found in **Appendix C.6.1**. We realize that our mention of L330-331 may not have drawn enough attention to it.  To summarize, we show the prostate cancer setting where younger patients are a minority-sized group.  In this imbalance scenario, TRIAGE still retains strong performance on the minority-sized group (with the lowest MSE) compared to baselines. The reason is that the calibration set contains a few minority samples, ensuring they aren't overlooked and discarded by TRIAGE. This underscores the importance of a representative calibration set for capturing critical cases. We will include a discussion on this as guidance to users of TRIAGE to promote safe and responsible usage.\n\n**UPDATE:**  Better flag our experiment in Appendix C.6.1 (on data imbalance) & include a discussion in Sec 6 around constructing a calibration set for safe usage of TRIAGE.\n\n# (E) Clarifications\n\n*(i) Clarifying the KNN in TRIAGE:*\n\nWe compute the errors (residuals) for all samples in $D_{cal}$. For sample $x_i$ we estimate its difficulty score ($\\sigma$) as the mean absolute errors of $x_i$’s k-nearest neighbors in $D_{cal}$, where K=5\n\n\n\n*(ii) Clarifying the term model-agnostic:*\n\nBy stating that Conformal Predictive Distributions offer a \"model-agnostic\" score, we mean that the score can be computed using any regressor. While the reviewer rightly points out that the core is to compute conformity scores (i.e., residuals), these scores can be derived post-hoc from any regressor, requiring only its output predictions. This \"model-agnostic\" approach contrasts methods like Bayesian ones, which necessitate specific modeling assumptions, thereby limiting their applicability across different regressors (see L130-132).\n\n**UPDATE:**  Explain in the main manuscript that by model-agnostic, we mean the conformity scores can be computed for any regressor just needing the outputs.\n\n\n*(iii) Effect of the calibration dataset:*\n\nThe calibration dataset is important to data sculpting as discussed in point (D). i.e. performance on minority sized groups. We have investigated another aspect **Appendix C.5**: how $D_{cal}$ affects the  “validity/calibration”, especially if we violate the exchangeability assumption.  \n\n- Appendix C.5.1 shows we have valid CPDs if exchangeability is satisfied; hence will provide good sculpting. \n\n- Appendix  C.5.2 then shows the effect as $D_{cal}$ gets more non-exchangeable. We find that in the non-exchangeable setting when $D_{cal}$ is small e.g. <300 samples we still empirically achieve coverage and have high quality CPDs based on CRPS score & calibration curves. Moreover, we have good MSE sculpting performance, as shown in Table 2. That said, above 0.3 (> 300 samples) we have sufficient samples that violate exchangeability. This reduces the coverage below 0.9. Interestingly, this matches the change-over point in Table 2, where training directly on $D_{cal}$ will lead to better performance than sculpting. The difference though is not significant, with the MSE not drastically harmed. Additionally, our CPDs are still of high quality, based on the low CRPS score.\n\n**UPDATE:** For clarity, we'll revise our reference on L337-339 to explicitly mention that Appendix C.5 evaluates the effects of $D_{cal}$.\n\n"}, {"review_id": "jHyrVh9rT3", "reviewer": "Reviewer_pNo8", "summary": "The task of data characterization aims to address variations in individual-level performance despite achieving good average performance. Existing methodologies have predominantly focused on classification, leaving a gap in data characterization approaches for regression. In this paper, the authors propose the TRIAGE framework to bridge this gap. Extensive experiments demonstrate the framework's superior performance in regression tasks.\n", "questions": "- In simplified setting (e.g., linear (kernel) regression?), is it possible to derive further theoretical guarantees on the behavior of C and V, defined in Section 4.2?\n\n- Is the group assignment consistent? (in the sense that under what conditions are the samples accurately assigned their true group label?)", "limitations": "refer to weakness", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written.\n\n- This is the first paper to introduce a principled data characterization framework in regression settings, supported by extensive experiments.", "weaknesses": "Many contents are in the appendix. It would be helpful more discussions about the appendix in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear ``R-pNo8``. \n\nThank you for your thoughtful comments which have helped improve the paper. We provide answers (A)-(C) & highlight updates to the paper\n\n# (A) Many appendices - include discussion in the main text\nThank you for suggesting better flagging our numerous appendices' contents.\n\n**UPDATE:**  In addition to the references interspersed in the text, in the additional camera ready page we will include a section outlining new discussions and results covered in the Appendix. \n\n# (B) Consistency of TRIAGE\nWe assess the consistency of the scores computed from the training dynamics for different parameterized models in Fig.4 (Sec 5.1). By virtue of consistency of the scores, this would lead to consistency of group assignment since we use a threshold based mechanism to assign the groups. We find that TRIAGE is much more consistent than the baselines with a Spearman rank correlation or 0.91 averaged across datasets for different parameterized models. \n\nAdditionally, in **Appendix C.2**, we further assess consistency using different model classes — namely XGBoost and CatBoost. We apologize that this was unclear linked to your first point about many contents in the Appendix. We will flag this additional set of results in Appendix C.2 in the main manuscript as an additional consistency assessment. The Spearman rank correlation of scores between these different models is similarly high, as shown in Table 5 with a mean of 0.88 +− 0.04. Given the consistency and stability of the scores, this would mean the subsequent assigned groups would be stable. We visualize this in Fig. 13 (Appendix C.2), which illustrates the stability and consistency of the TRIAGE characteristic curves. \n\n**UPDATE:** Flag the additional results in Appendix C.2 around consistency in the main manuscript in Sec 5.1 and in the dedicated discussion on contents in the Appendix.\n\n# (C) Theoretical guarantees\nWe agree that providing theoretical guarantees would be valuable beyond just the strong empirical performance of TRIAGE. We wish to highlight two important aspects which make this process challenging: \n\n- Since there is an interdependence of CPD and training dynamics in TRIAGE. Hence, disentangling their impacts theoretically is non-trivial.\n\n- Scores across training epochs are correlated due to iterative model training. This highlights the complexity of any theoretical guarantee, given the dynamic nature of the scores and the correlation of scores. Especially, since independence is often assumed when providing theoretical results — whereas we cannot assume this with a training dynamics based perspective.\n\n**UPDATE:** Given the complexities, we propose to discuss a theoretical guarantee as future work in Sec.6. We will also add an Appendix outlining the aforementioned challenges.\n\n\n"}, {"review_id": "DvHKBvPHso", "reviewer": "Reviewer_N4d8", "summary": "This paper present a data characterization method for regression task. The method leverages conformal predictive systems literature and proposed to estimate training data scores by thresholding percentile of  calibration data points given their conformity measure. The method is interesting in terms of leveraging calibration dataset CPD to measure the training data score. The threshold of grouping (4.3) is less explained, but empirical results show it works. The paper is self-claimed to be the first data characterization method that is suitable for regression task.", "questions": "Why mean over training steps when computing C(x,y)?  What if we put more weights to higher iteration?\nWhat if there are multiple candidate regression methods? would this alter the outcome? \nWhy does the group assignment threshold designed as Eq 3? Any explanation or justification?", "limitations": "No concern. But the authors should describe further in terms of mis using this method, where decision maker can remove data from underrepresented population based on the solution. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is novel to me. Leveraging calebration dataset to re-score training data is interesting. I am quite curious how the choice of calibration dataset would impact the method's performance.\n\nThe paper is very well motivated. And, maybe over motivated, considering justification and explanation missing in the section 4.2 and 4.3.\n\nThe experimental results show the proposed method works well in benchmark datasets and could be a good tool to do data selection or feature selection.", "weaknesses": "The paper takes pretty long paragraph to highlights its novelty and difference between other data characterization method, which looks like an ads. The motivations part should be reduced and focus more on algorithm explanation. There is very less justification on why thresholding CPD works for the data characterization purpose, and where the rules in Eq 3 come from. They need further justification and explanation.\n\nAlgorithm 1 need to further adjustment and state what the \"eval sample\" is. I think it is one training data in D_train. The output of Algorithm should be a matrix |X| x |M| right? please state it somewhere to help understanding. \n\nDesign of 4.2 is not explained. Why mean over training steps? What if we put more weights to higher iteration? What if there are multiple candidate regression methods? would this alter the outcome? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear ``R-N4d8``. \n\nThank you for your thoughtful comments which have helped improve the paper. We provide answers (A)-(E) & highlight updates to the paper\n\n# (A) Evaluation over all training steps vs looking at higher iterations [Design motivation]\n\nTRIAGE aims to analyze the behavior of different samples through the training process. Specifically, as shown in **Fig. 9 (Appendix A)**, while at the end of training, samples can converge to the same score — the trajectories of TRIAGE scores (training dynamics) are different for different samples. Some samples converge quickly, while others might take longer and oscillate more. We aim to use these differences to delineate samples. As shown in Fig. 9, most of the variability arises in the earlier training steps, which is why we chose to compute the score over all steps to capture this early behavior. While we discuss this in Appendix A.4, we will further update it to be clearer and reference this point in the main manuscript as suggested.\n\n**Experiment:** Beyond explanations, we also thank the reviewer for bringing up this interesting question. It has spurred us to conduct a new experiment to provide additional validation for why we want to evaluate over the entire training trajectory rather than focusing on later/higher iterations. \n\nWe specifically have repeated the data sculpting experiment (original results in Table 2). We evaluate computing the TRIAGE score starting at later and later points in the training trajectory after: (i) 33% of training, (ii) 50% of training, (iii) 70% of training and (iv) 80% of training. In contrast to our version in the manuscript, which we will term TRIAGE (Base) is computed from the beginning of training starting at 0% of training). We then compute the TRIAGE scores for each variant (i)-(iv). We then select samples for sculpting and evaluate test MSE for the regressor after sculpting. \n\nWe find an increase in MSE (worse performance) the further in the trajectory that we start the score computation. The results averaged over all calibration sizes are shown in **Fig.3 in the uploaded response PDF** and are reflected as MSE increase vs TRIAGE (Base). This highlights the importance of capturing the variability between sample training dynamics early in training before they reach steady state, to differentiate them. We thank the reviewer for the suggestion, as it helps further motivate our use of training dynamics & computing the mean over the whole trajectory.\n\n**UPDATE:** Refine the discussion in Appendix A (referring to it in the main paper) and include the new result in Appendix C.\n\n\n# (B) Multiple candidate regressors\nWe evaluate the performance of TRIAGE and specifically the stability of the TRIAGE scores given differently parameterized regressors. Our result in **Fig. 4** highlights that the TRIAGE scores are more consistent compared to baselines — with a Spearman correlation of 0.91. The consistent scores would mean the outcomes of tasks using the scores would be similar — i.e. we have similar outcomes. We have also conducted a similar analysis across different types of regressors in **Appendix C.2 (Table 5)**. We will add a discussion on this in the main manuscript and link it better to bring this result to the fore.\n\n# (C) Adjustments and Clarifications \nThank you for catching the ambiguity in Algorithm 1. We will update it to be explicit that an evaluation sample is indeed a sample from $D_{train}$. On the output dimensions, yes after all samples are computed, we have an $M$ x $q$ matrix representative of the different CPDs. We then note when we compute the TRIAGE score itself, the dimensionality is $M$ x $E$, where $E$ are epochs. \n\n**UPDATE**: We will update the text with these clarifications and notation updates.\n\n# (D) Streamline Introduction \nThank you for the suggestion. We will streamline the introduction to make space for further discussion on the algorithm. Specific to your suggestion to include further motivation on the thresholding and curve — this additional space (along with the additional camera ready page) will allow us to bring the algorithmic motivations from Appendix A.5 and Appendix A.6 into the main paper. \n\n# (E) Add discussion on potentially removing under-represented populations\n\nWe agree that this is an important consideration, especially around data sculpting when there is a data imbalance between majority and minority groups. We have evaluated this setting in **Appendix C.6.1**. We apologize that this assessment and discussion got lost in the numerous appendices. We will endeavor to better flag it beyond L330-331. Based on your suggestion, we will include it in the discussion Sec.6. To summarize the experimental result in Appendix C.6.1. \n\nWe show under this imbalance scenario that TRIAGE still retains strong performance (with the lowest MSE) on the minority group — compared to baselines. This is because of the calibration set containing these a few minority samples, permitting TRIAGE not to filter them and hence retain good performance. This finding highlights an important aspect of calibration set construction to be as representative as possible. We will include a discussion on this as guidance to users of TRIAGE to promote safe and responsible usage."}]

Author rebuttal:
We thank the reviewers for their insightful and positive feedback!

We are encouraged that they found the "problem being studied and data-centric perspectives attractive and important” (**R-ckq8**) for "real applications"  (**R-oVcJ**) and that TRIAGE is a principled (**R-pNo8**) and “novel”  (**R-pEXt,R-N4d8**) data characterization framework in regression settings. Further, they found TRIAGE’s use of conformal predictive systems as "intuitive and effective” (**R-pEXt**) with " analysis of training dynamics from checkpoints is interesting” (**R-ckq8**) — with the "extensive experiments" (**R-pNo8**) supporting the “many interesting use cases”  (**R-ckq8**), such as being a "good tool to do data selection or feature selection” (**R-N4d8**).

We address specific questions and concerns below and highlight updates based on reviewer suggestions that will be incorporated into the revised manuscript.

We have uploaded a ``response pdf`` with additional experiments. These include:

* Simulation study with Huber’s contamination model

* Computational time for 2000-100k samples and compared to Data Valuation methods.

* Motivation for computing the TRIAGE score over all training steps

* Comparison to data valuation methods

On the basis of our clarifications and updates, we hope we have addressed the reviewers' concerns. 

Thank you for your kind consideration!

Paper 11604 Authors


Meta-review: This paper studies an important problem to provide a score for each data point which reflects how useful each observation is for the learning algorithm. It does so by using conformal inference ideas (more specifically CPD). This paper was quite well received by the reviewers. While some concerns were raised by the reviewers most of them were answered during the discussion period. The presentation of the paper is pointed out to be limited by reviewer oVcJ. While the paper motivates well their problem the writing is at times sloppy and lacks rigor. Notably, while the algorithm for CPD is well-defined, the definition of the triage score is never clearly defined and stated. However I still recommend acceptance because the proposed method is interesting. However I ask the authors to take the concerns of oVcJ into account to improve the writing.
Decision: Accept (poster)