Enhancing Prediction Performance through Influence Measure
Authors: Shuguang Yu
Wenqian Xu
Xinyi Zhou
Xuechun Wang
Hongtu Zhu
Fan Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: local influence measure
training robustness
active learning
Primary area: interpretability and explainable AI

Abstract: In the field of machine learning, the pursuit of accurate models is ongoing. A key aspect of improving prediction performance lies in identifying which data points in the training set should be excluded and which high-quality, potentially unlabeled data points outside the training set should be incorporated to improve the model's performance on unseen data. To accomplish this, an effective metric is needed to evaluate the contribution of each data point toward enhancing overall model performance. 
This paper proposes the use of an influence measure as a metric to assess the impact of training data on test set performance. Additionally, we introduce a data selection method to optimize the training set as well as a dynamic active learning algorithm driven by the influence measure. The effectiveness of these methods is demonstrated through extensive simulations and real-world datasets.

Published as a conference paper at ICLR 2025
ENHANCING PREDICTION PERFORMANCE THROUGH INFLUENCE MEASURE
Shuguang Yu"; Wenqian Xu Xinyi Zhou", Xuechun Wang", Hongtu Zhu', and Fan Zhou "Shanghai University of Finance and Economics, Shanghai, China UUniversity of North Carolina at Chapel Hill, North Carolina, USA
ABSTRACT
In the field of machine learning, the pursuit of accurate models is ongoing. key aspect of improving prediction performance lies in identifying which data points in the training set should be excluded and which high-quality, potentially unlabeled data points outside the training set should be incorporated improve the model's performance on unseen data. To accomplish this, an effective metric is needed to evaluate the contribution of each data point toward enhancing overall model performance. This paper proposes the use of an influence measure as a metric assess the impact of training data test set performance. Additionally, we introduce data selection method to optimize the training set as well as a dynamic active learning algorithm driven by the influence measure. The effectiveness of these methods is demonstrated through extensive simulations and real-world datasets.
INTRODUCTION
To build effective machine learning models, the significance of individual training data points cannot be overstated. Each data point the training set contributes uniquely to the model's learning process, shaping its performance, generalization, and resilience to various challenges (Blum & Langley 1997). Simply evaluating the model's performance on the provided data is insufficient; understanding the influence of individual training examples and making informed decisions about their inclusion or exclusion critical for developing effective and reliable models.
Recent ad advacements in machine learning have highlighted the importance of strategic data selection and management during training. Techniques such as active learning (Settles 2009) promote the iterative selection of the most uncertain unlabeled data points for labeling and inclusion in the training set. However, estimating uncertainty in deep neural networks (DNNs) is challenging due to their tendency to exhibit overconfidence (Ren et al. 2021). To address this, methods like deep Bayesian approaches (Gal et al. 2017), ), muery-by-commitee (Gorriz et al.) 2017). Variational Auto-Encoders (Sinha al. 2019), adversarial learning (Ducoffe & Precioso 2018 Mayer & Timofte (2020), graph convolutional networks (Caramalau et al.,,2021] and noise stability (Li et al. (0024) have been proposed to improve uncertainty estimates. However, these approaches assume well-trained model and fail to consider how model parameters might evolve when the training data is modified.
To bridge this gap, recent studies have focused on quantifying the impact of individual training examples on model behavior, with the main challenge being the identification of an appropriate evaluation metric. The Shapley value has emerged as a promising solution, inspiring number of
Shapley- valuue based approaches (Ghorbani & Zou 2019 Jia et al. 2019a by Ghorbani et al. 2020
Kwon & Zou 2022, Wann & Jial (023). However, these methods often require multiple model retrainings and evaluations, making them computationally expensive.
Techniques such as influence functions (Koh Liang 2017 Pruthi al., 2020 Yeh et al. 2018
Chen et al. (222T) offer insights into the effect of individual data points on model predictions, helping to identify and mitigate harmful or overly influential examples. For instance, CChabra et al. (2024) apply influence functions to measure the impact of training data a specific model, improving
'The first four authors have contributed equally to this paper.
1 Corresponding author. Email: zhoufan @ www...suff..duu.comm

---

Published as a conference paper at ICLR 2025
performance by pruning detrimental data points. Their method is highly efficient it avoids the need for model retraining. However, these algorithms may still fail in certain cases, as demonstrated the following simple example.
Binary Classification via Logistic Regression. We consider a logistic regression model p(r) = (1 + e --- where 3 R represents the coefficients. For training set 2 { 9:) and validation set V, the influence function in Chhabra et al. (0024) for example is defined as
I(-x;)= = Σ
OgL x; B) L au (0 24 B),
(x.y)EV
where dg- 0/03, x R {0,1} a binary classification label, L represents the cross-entropy loss, and 3 = arg minger n- Ei=1 L(y;, x;; According to Chhabra et al. (2024), samples zạ with negative influence function values negatively impact the model's per- formance on the validation set and should be removed. For all training samples Iú, the term (x 0gL(v, .r; 3) 07 z4; B) remains constant, with only Ly 245 () varying.
In the simple logistic regression scenario, have 03 Lyy, B) [(1 === u Assum- ing E(ayiyy (y x; /) Ly, n4 B}] -1 and m; > 0, the training sample influence clearly incorrect to solely remove data points from
negative if and only if y; = 1. However, one class, as outliers in the other class may also negatively influence the model's performance. For higher-dimensional Iú, as shown Section Jusing both simulated and real- world data, our method consistently outperforms the approach proposed by Chhabra et al. (0224) and others.
Given the limitations of existing methods, we draw inspiration from local influence measures developed in the statistical community (Zhu al., 2007 2011 [Shu & Zhu| 2019: Sui et al. 2023) to propose a novel metric in this article. Our approach differs from previous ones that typically evaluate data samples model parameters with a fixed model assumption.
the impact of perturbations Instead, our measure directly assesses the effect of minor perturbations to training samples on the model's performance implement this, we have developed a new perturbation manifold and expanded the local influence framework. Using this innovative approach, we introduce two key metrics: one for data trimming, aimed at identifying and removing training set anomalies that compromise model performance on the test set, and another for active learning, which focuses on selecting the most impactful unlabeled data to enhance the prediction performance. Moreover, acknowledging the challenges of slow computation and high memory usage inherent in calculating exact local influence measures, we propose two approximation methods. Our experiments real-world datasets demonstrate that these approximations achieve performance comparable to exact calculations while significantly reducing computational overhead.
validation set, allowing the model adapt these changes. To
We summarize our contributions as follows,
Unlike existing local influence measures, we propose new metric that evaluates the impact of perturbations to training samples on the model's performance on validation data.
(ii) The proposed metric is applicable to both data trimming and active learning. For data trimming, evaluates the effects of minor perturbations each sample, offering deeper insights into how individual samples impact model performance. In the context of active learning, the method ca captures the relationship between training samples, unlabeled data, and parameter updates.
(ii) We propose two approximation methods alleviate the high computational cost of calculating local influence measures. These algorithms significantly reduce computational overhead while maintaining better performance than other methods, as demonstrated in our experiments.
2 A NEW INFLUENCE MEASURE
In this section, we begin by presenting essential background information, including the Perturbation Manifold, before formally defining the proposed metric.
Perturbation Manifold. Our definition of the perturbation manifold closely follows that of Shu & Zhu 2019). Given an input sample (xx, in the training set Z = and machine
2

---

Published as a conference paper at ICLR 2025
learning model with an estimated parameter vector 0, which is trained on 2, the prediction probability for class e {1...., K} is denoted as P(c|ær, 0). Let w (wi...... wp)) be perturbation vector that varies within an open subset Ω C RP. The perturbation w is applied to æ, thereby affecting the learning of the parameter vector Ô. We denote the parameter vector obtained by the model after perturbing the training sample æ with w as ((x with (xx) = 0. We define P(ck ( +()+w)) as the prediction probability under the perturbation such that -c=1 P(c|æ + w. 0( + w)) = 1. It
assumed that there exists a w/ Q such that P(c|æ + wo, 0(x: a: + we)) = P(c|æ, 0). Additionally, we assume that {P(c|z + w. ((x + +w)) is positive and sufficiently smooth for all w E Q.
Following the development in (Zhu et al. 2007: 2011). we define M ( + w. 0(x + w))
{t} as perturbation manifold. The tangent space of M at is denoted by Tw. which is spanned by [ot(wle cc, æ. ((a))//w>//i where [[w]c, xx (zz) log P(e/æ + w, 0(x + w)). Let Gz(w)
EK 0 (wlc, æ, If (w) is positive definite, then M is a Riemannian manifold (Shu & Zhu 2019) with (2(W)
æ, ((x)) P())) + w, (x + w)) with a.,. - (0/ 0/Owp).
serving as the Riemannian metric tensor (Amari 2012: Amari & Nagaoka 2000p.
Although Gz(w) is often not positive definite in classification problems, we can still reduce the dimensionality of the perturbations and reconstruct a Riemannian manifold (Shu & Zhu 2019).
The Influence Measure. Let Lyy', x'; 0) denote the loss function of the model with parameter on z' (x',y) v/) & Z, we can get the expression for the (first- influence measure:
FI(=', =) = Ly' x': ((x x + ww)) (w)) where (w)) is the pseudoinverse of (_(w))
[ L(y,, x': Ō(x + wo)),
(2.1)
We consider linear perturbation approach where applying perturbation w to the training sample transforms (ær, y) into (æe + w, y). Consequently, the parameter vector updates to ((x + w) := arg minøge (Ei= LWv, ax;; 0) + L(y, x + w; 0) L(y, 0)), with wa = 0. Using the chain rule, we can derive the expression for L x'; ((x + wo)):
Ow x'; x'; 0(x + wo)) 0g Lyy x 0 (( 0 w) S
(2.2)
Here, 0a + w)w 2 n H 0g Le xz (x)). whose derivation is detailed in Appendix A where Hg = n Dm 0g x;: (z)) The term 0g Lyy a; (()) represents the gradient of the loss function first taken with respect to the model parameters and then with respect to æ, evaluated at the perturbed training sample (z, y).
For the computation of Gz (w)). P(c|æ + wo, Ō(x + wo)) can be directly obtained using the learned model parameter vector θ and the unperturbed sample point æ. The calculation of @wl((wole, æ, ((x))
requires the application of the chain rule:
@wl(wole, x, (x))) =āw log P(c|æ wo. 0(x x + wo))
(2.3)
=ūg log P(c|æ + wo Ō(x + wo)) 0(x + ww/l=ww
+ 02 log P(c|æ + Wo. 0(x Wo)).
All differentiation operations can be easily computed using backpropagation (Goodfellow et al. 2016) in deep learning libraries such as TensorFlow (Abadi et al.) (016) and PyTorch (Paszke et al. 2017). This entire process efficient and does not require retraining the model parameters.
Theorem 1. If so represents a diffeomorphism of w, then FI(z', 2) is invariant under any reparame- terization associated with yp.
Compared to widely used measures in Fuclidean spaces, such as the Jacobian norm (Novak et al. 2018) and Cook's local influence measure (Cook) 2018), Theorem demonstrates that FT(2', =) remains invariant under any diffcomorphic transtormation (c.g.. scaling) of the perturbation vector w. The proof of Theorem can be found in Shu & Zhu (019)).
The significance of Theorem 1 is especially pronounced when there are scale differences among the dimensions of æ. For instance, certain dimensions have significantly larger values than others, the contribution of perturbations to those dimensions may appear exaggerated. However, our FI(z', 2) mitigates this scaling issue by employing the metric tensor of the perturbation manifold instead of that of the standard Fuclidean space.
3

---

Published as a conference paper at ICLR 2025
3 FI FOR DATA TRIMMING.
The primary goal of data trimming is to eliminate training samples that may compromise the model's performance on datasets beyond the training set. Since our proposed influence measure (FI) quantifies the impact of each training sample on the model's performance on validation sets, it serves as natural tool for data trimming.
From model robustness perspective. if a small perturbation in a training sample leads to significant effect on the model's performance on the validation set, that sample should be excluded, which aligns with the principle of our proposed metric. Given the inherent challenge of ensuring that all samples in the training set are entirely accurate, sample with excessive influence could severely degrade the model's s overall performance if it contains any contamination. Therefore, to enhance prediction performance on an unseen data set, such samples should be removed from the training set.
Following the setup of CChabra et al. (2024), we introduce training set E, validation set V, and a base model F. In this context, we assess the impact of each training sample on the model's performance by computing the FI for each training tuple 2; E Z with respect to V. To achieve this, we extend Equation equation [.1 to encompass the entire validation set. This involves replacing the loss function for individual validation sample with the mean loss function across the entire validation set, as follows:
FI (z) = LVV (W L(V; 0( + wo)),
(3.1)
where aw LVV (e + wo)) = To Derwee 0 s'; Ō(x x + wo)).
The algorithm for computing FI is outlined in Algorithm After computing these values, we can sort all data points in the training set in descending order based on their FI values and remove the top b points enhance the model's performance on the test set. For the detailed data trimming algorithm, please refer to Algorithm
Algorithm 1 Calculation of Fl
Input: Training set Z. Validation set V. Base model F Output: Influence measure vector Fl E |
1: procedure FT -AALCULATIIOOON.. V.
2: Train FF with 2, and obtain the parameter vector 0
3: Generate empty vector FI of size |Z| 1
+ 4: Calculate [V] E(rraree 0 0yL(y', x';
and H&
5: for every 2; in do
6: Calculate Gz, (wo) and Og (yi, x;;
7: L, 0 [ Eiewver 0 Oo x'; 0) H 1,0000 nz;
8: FI + O. L(V: θ) (w))
9: end for
10: return Fl
11:
end procedure
4 FI FOR ACTIVE LEARNING.
In active learning, the primary objective is to identify the most uncertain or informative samples from an unlabeled pool for annotation, typically in sequential batches. After each round of annotation, the newly labeled data are combined with the existing labeled set to retrain the model and improve its performance. Li et al. (2024) argue that a small perturbation to the model parameters leads to significant changes in the prediction for given sample, this indicates high uncertainty for that sample under the current model, suggesting that should be labeled and added to the training set. Our approach is more fundamental: since the model parameters derived from the training data, we directly assess how perturbations to the training samples influence the model's predictions for the unlabeled samples. If slight perturbations most training samples substantially alter the model's
4

---

Published as a conference paper at ICLR 2025
prediction for given sample, it likely contains missing information from the training set and should therefore be included.
We now introduce method for active learning using the proposed FI. For unlabeled sample Tunlabel. we first assign it a predicted label and treat it as a validation sample. Next, we calculate the influence measure of Damlabel with respect each point in the training set. The overall influence measure Flucéie (#unlabe!) is derived by aggregating these individual measures either by averaging or using specific quantiles of these FI values. The Fl (æunlabel) is defined follows:
Fl (Iunlabel) g (([FF(( Zuntabel zi) 3+)
(4.1)
where represents the aggregation function, Zunlabel = (Iuntabell Upred), and Ypred is the predicted label assigned by the current model for Funiahel-
During each round of active learning, we begin by applying Algorithm 2 to compute F1 We then sort all the samples in the unlabeled pool in descending order based on their FI values. The top-ranked samples are labeled and added to the training set. In the subsequent round, the model retrained, Fl is recalculated, and the process is repeated. For the detailed active learning algorithm, please refer to Algorithm
Algorithm 2 Calculation of Fiacieve
Input: Labeled pool of training data L, Unlabeled pool of training U, Base model F, Aggregation function g
Output: Influence measure vector Fl R
procedure F1addt--CLLCLLEC F
g)
2: Train F with L, and obtain the parameter vector 0
3: Generate empty vector Freetive of size (4]]
4: Calculate H
5:
for every < in 14 do Obtain an estimated label d with Calculate (gL( x/'
6:
7:
8:
J 0
9:
for every of in L do
10:
Calculate Gz4 (wa) and d2 af:
11:
O. T at 0) de (i,, ax!: 8) H)H, 0- I. (yf .f:
12:
J JU (ūr, al! 0) (w)) T. ;
13:
end for
14: Flactive [1] 9(J)
15:
end for
16: return Fi
17:
end procedure
5
APPROXIMATION METHODS
The data selection processes described in the previous two sections have two main drawbacks. First, computing FI requires substantial storage for second-ordes derivatíves, particularly in models with numerous parameters or high-dimensional data. To address this issue, we propose the KFSVD approximation method, which combines the Kronecker-factored (K-FAC) approximation (Martens & Grosse 2013 Nickl et al.) 2023) and Truncated Singular Value Decomposition (Truncated-SVD) approximation (Golub & Reinsch 1971) to effectively reduce storage requirements. Sencond, the necessity to compute the FT for all data points in the data trimming algorithm, as well the need to recalculate Flactive for all unlabeled data in each round of the active learning algorithm, significantly decreases computational efficiency. To mitigate this, we implement a subsampling approximation that enhances overall performance. The details of these two approximation methods are follows.
KFSVD approximation, which combines K-FAC approximation with Truncated-Se -WD approxima- tion mitigate memory consumption associated with the Hessian matrix H and the second-order
5

---

Published as a conference paper at ICLR 2025
partial derivatives dg x; (())). The K-FAC algorithm is widely recognized technique for approximating the Hessian matrix, which not only accelerates computations but also significantly reduces storage requirements. Meanwhile, the Truncated-SVD approximation employs power itera- tion and related techniques to compute the top-k eigenvalues and their corresponding eigenvectors of 0z0g Lyy, æ; (())), thereby providing an effective approximation of these second-order partial derivatives. Assuming the dimensionality of the sample covariates is d and the number of model parameters is the Truncated-SVl SVD approximation enables the decomposition of the second-ore order partial derivatives as 0_0oT((u a; (ax)) = Uxx Vo where A is diagonal matrix containing the top-k eigenvalues of 0, (y, ax; (a:)) and Upxx and Vax& are comprised of orthogonal vectors. Algorithm 5 and provide the detailed procedures for calculating FT and Flaeine using the KFSVD approximation.
Subsampling approximation, which utilizes subsampling and random forest techniques to enhance computational efficiency. Specifically, we extract a small subset of samples (c.g., 20%) and compute their FI using Algorithm or 7 These computed values are then used to sort the samples. The features of each sample, along with their corresponding ranks, serve as covariates and target variables to create new dataset. Subsequently, dataset and leverage the trained model to predict the ranks of other samples. Selections are based on these predicted ranks. Since the computation of Fl is the most time-consuming part of the workflow, the overall acceleration is directly correlated with the proportion of samples for which we choose to compute FI accurately. For instance, selecting 20% of the samples can reduce the total processing time to of the original duration when the dataset is sufficiently large.
train regression model using random forests on this
In practice, integrate both approximation methods to develop the comprehensive FI based data trimming algorithm and Fi Fladdd--based active learning algorithm. These algorithms effectively address the storage and computational efficiency challenges inherent in calculating FI. The detailed procedures outlined in Algorithm [ and
To better illustrate the computational advantages of the proposed two approximation methods, we analyze their complexity in terms of both memory usage and computational speed. This analysis is also compared with the complexity of Influence Value (IV) as introduced by CChabra et al. (2024).
Time Complexity. The following are the time complexities of Fl and IV:
FT without approximation methods: O(p² + nd2 + ndp). Here, n is the maximum value of the sample sizes of the training set and the validation set, is the dimension of the covariates, and p is the dimension of the model parameters. The term (pp)) comes from computing the inverse of the Hessian matrix. The ((nd)) term is due to inverting G, repeated n times. The O(ndp) term corresponds to computing Og for times.
Fl with approximation methods: ( ((gnn)) + an d)ks and + +E p.). Here, a (0, 1] is the proportion of data for which we compute Fl accurately during subsampling. T is the number of decision trees the random forest, & is the Truncated-SVD parameter, 8 the number of iterations for computing each eigenvalue during power iteration, and (P1, PP..P.....,PP
represent the number of parameters in each layer of an L-layer neural network.
Fl without approximation methods: Opp + nd f + dd)p). Here, n is the maximum value of the sample sizes of the labeled dataset and the unlabeled dataset.
Flactive with approximation methods: O log(an) + an²(p + d)ks + an El El_pp pt).
IV: O(p" + mp).
By employing two approximation techniques, we can significantly reduce the computational gap between our method and compared methods like IV. When the dataset is large enough, setting an as a constant allows us to achieve better computational complexity than IV. For complex models (e.g.. neural networks) where p is sufficiently large. the n term dominates the computational complexity of IV. In such cases, our method's 's computational costs could be lower than IV.
Memory Usage. Below is a discussion of the space complexities associated with FI and IV:
In the computation of FI, the highest storage requirements are for the Hessian matrix and 0, d. which together occupy a space of Opp² dp).

---

Published as a conference paper at ICLR 2025
After using the KFSVD approximation method, the space requirement of FI is reduced to O P (p + d)k)-
In the computation of IV, is necessary to store sample information and the Hessian matrix, resulting in a space complexity of O(p² + d).
After applying approximation techniques, our algorithm achieves space complexity similar to that of the IV. In fact, when is large, our method offers an advantage.
6 EXPERIMENTAL RESULTS
In this section, we present experimental results that illustrate how our newly proposed metrics, Fi and Fi contribute to enhancing model prediction performance. We compare our algorithms with ofatc-of-the-art strategies in both data trimming and active learning scenarios, thereby validating the effectiveness of our approach on both simulated and real-world datasets.
6.1 DATA TRIMMING
In this subsection, we conduct simulations using both linear and nonlinear models to demonstrate how our algorithm enhances data trimming efficiency and evaluate the effectiveness of the proposed FI on real-world datasets. The latest data trimming method, IV, serves the primary baseline for comparison in these experiments.
Validation on 2D Linear Model. Logistic regression is utilized for this binary classification task. We begin by generating several datasets by sampling from two isotropic 2D Gaussian distribu- tions. Each dataset comprises 150 training samples, 100 validation samples, and 600 test samples. The experimental settings for this scenario consistent with those of the study by Chhabra et al. (2024). To account for the ran- domness inherent in the sampling process, we analyze our method on datasets generated under the same distribution but with different random seeds. As shown in Table [ our method consistently enhances model performance compared to theirs in most cases, particularly when trimming 5, 10, and 20 samples, with thirty different random seeds employed each time. Moreover, it is evident from Figure [ that IV tends to trim samples from a specific class under certain conditions. In this context, Figure TD clearly demonstrates that in some scenarios IV fails, while our method continues to perform effectively.
Table 1: Comparison of two methods linear model. Number of cases where Fl outperforms IV across 30 random seeds, along with performance improvements. Acc Ff: the mean accuracy by Fl, and Acc IV: by IV.
deleted points
# of better case
Acc_FI(%)
Acc_IV(%)
23
96.22-10.65
95.77-10.76
10
28
96.20-10.65
94.84
20
30
96.23:10.4
93.36#2.21
Frstur
Feature
Feature:
Feature:
Feature:
Figure 1: Performance under Linear Model. Different colored points represent different classes. A shows the training set. B and C respectively denote the samples to be trimmed by Fl and IV. D denotes test set. Green line: boundary without trimming; Red line: boundary after FI trimming; and Blue line: boundary after IV trimming.
7

---

Published as a conference paper at ICLR 2025
Validation on 2D Nonlinear Model. After demonstrating the effectiveness of our method in linear scenarios, we now extend our examination to nonlinear cases. To achieve this, we construct a binary classification dataset that is non-linearly separable, with each class represented by crescent-shaper region. A more intuitive understanding can he obtained from Figure We employ a neural network that includes input layer, two hidden layers with ReL.U activation functions, and output layer with sigmoid activation function. Similar to the linear case, we conduct repeated experiments in the nonlinear scenario. Each dataset comprises 500 training samples, 250 validation samples, and 250 test samples. As shown in Table 2 our method achieves a higher average accuracy and outperforms the other methods in most cases across the 20 repetitions. Figure P Iillustrates instances where utilizing IV to remove training points can lead to a deterioration in model performance under certain conditions.
Validation on the World Datasets. In this study, we assess the efficacy of our method by utilizing four real-world datasets: two tabular datasets, Adult (Kohavi 1996) and Bank (Moro et al. 2014); a visual dataset, CelebA (Liu et al. (015);; and textual dataset, Jigsaw Toxicity (Noever
2018). Additional details regarding the datasets and experiments can be found in Appendices D.1
Both Fi and IV are evaluated on the validation set, with Lo- gistic Regression serving as the base model. The results on the test sets of these datasets are pre- sented in Figure 2 Ablation experiments are also conducted in the Appendix F to assess the impact of perturbations on model performance.
Table 2: Comparison of two methods on nonlinear model. Number of cases where FI outperform IV across 20 random seeds, along with performance improvements. Acc_FI: the mean accuracy by Fl, and Acc_/V: by IV.
# of deleted points
# of better case
Acc.FI((5)
Acc_IV(S5)
5
17
89,90=2.16
87.50=2.46
10
17 15
20.24
88.02.2.622
20
90.72.11.67
27.8
As illustrated in Figure 2 our findings indicate that under a limited budget b, our F! data trimming method consistently outperforms two baseline models, IV and Random Trimming. Among the four datasets examined, Random Trimming demonstrates improvement in model performance. Although IV exhibits a notable enhancement on Adult and Bank, it tends to remove important data Jigsaw Toxicity
points CelebA, resulting in decreased performance, and shows no improvement compared to Random Trimming. This suggests that IV may fail in certain scenarios. In contrast, our method consistently achieves the maximum improvements across all datasets, particularly on the Bank dataset, where accuracy increases by more than 10%. Even on CelebA, where IV fails completely, our model still enhances the test set accuracy, further demonstrating the robustness of our approach. To ensure a more rigorous comparison between FI and IV, we also perform K-fold cross-validation in Appendix E.S] The results demonstrate that our method consistently outperforms IV, highlighting the robustness and stability of our approach.
The experimental results above show that our method performs better on real-worle datasets than on simulated ones. This is largely due to the simplicity of the simulated datasets, which are two- dimensional with clear boundaries effectively separating the classes and contain no erroneous data points. In such cases, sufficiently large dataset allows the model to easily find the optimal boundary, minimizing the advantages of data trimming. dimensional and more complex, often containing errors. Here, the benefits of removing potentially erroncous high-influence points become more evident. To support this, we introduce noise into the real-world datasets and conduct further experiments; the details of the noise addition method given in Appendix [ and the results are shown in Figure & The findings clearly demonstrate that both FI and IV outperform Random Trimming, with our method retaining significant advantages over IV, further confirming its effectiveness.
contrast, real-world datasets are typically high-
Finally, although the Shapley-value-basee method is time-consumingg, we conduct a comparison with our approach using specific datasets. And we also examine the potential impact of the masking effect. Details of both analyses are presented in Appendix E.4
E.4
6.2 ACTIVE LEARNING
In this section, we primarily conduct numerical experiments three simple tabular datasets and three complex image datasets to demonstrate the effectiveness of the proposed FI metric in active learning. We compare method with two state-of-the-art active learning baselines including IV
8

---

Published as a conference paper at ICLR 2025
Adult
Dank
CelebA
agaon Tosicity
0413
0.70
0.90
A390
4390
300
1.0
Figure 2: Accuracy curves of three data trimming methods on test sets of Adult, Bank, CelebA and Jigsaw Toxicity.
(Chhabra et al. 2024 and BALD (Gal et al., 2017 Kirsch al., 2019 2023). Random Selection is also included as a baseline. All results are averaged over three runs to ensure a reliable evaluation.
Note that, IV, as proposed by Chhabra al. (202)) originally calculates influence values only once during the initial selection, which can hinder overall performance since these values are not updated with subsequent labelings. To ensure fairness in comparisons, modified IV to recalculate influence values in each round of selection, and this updated version is used in our experiments.
For our method, the aggregation function o (Zuntabb:: 2;) () in Equation 4.1 denotes the oper- ation of calculating the mean of the data points from the set {FI Zumlubel, }=) the 10th and 90th percentiles, sorted in descending order. In other words, compute the mean of the middle 80% of the data after sorting. This approach effectively excludes extreme values, thereby enhancing the robustness of our final result.
i= that fall between
Tabular Datasets. For the tabular datasets, we employed Logistic Regression model. The total number of annotation rounds for each experiment, along with the settings for the unlabeled pool size and acquisition size, are detailed in Table 4 As shown Figure our method outperforms the other approaches on these simple tabular datasets. Random Selection performs reasonably well initially, but as data volume increases, improvements in model performance diminish. BALD shows comparable performance to Random Selection on tabular datasets. IV initially underperforms compared to Random Selection, but as more data is added, its performance improves and eventually becomes comparable to BALD. Our method starts similarly to BALD, but its focus more challenging samples allows to quickly address missing information once the model achieves a certain accuracy. As result, in the latter stages of each graph, our method distinctly diverges from the others.
Adult
CelebA
Jigsaw Toxicity
0.900
5.620
0.525
0.500
0.800
6780
0.900
6.600
0.753
0.760
0.725
Bandom
Sancom (koun)
0.700
Random
0.702
Flous)
0.300
IV
0.680
Bald
0.575
Bui
Buld
1000
2030
3000
4202
50d0
6050
1050
1500
2000
1000
2050
1030
4000
Dutaset Size
Figure 3: Classification performance due to the different active learning methods on Adult, CelebA, and Jigsaw Toxicity.
Image Classification. The data in the tabular datasets have been processed, resulting in high test accuracy with logistic regression. In contrast, for the image datasets -MNia (Lecun et al. 1998). EMNIST (Cohen et al., (017), and CIFAR-IO 10 (Krizhevsky) 2012) the input consists of raw images, and we employ a Convolutional Neural Network (CNN) as the classifier. To simplify the computation of Fl we utilize the outputs from the last layer of the neural network and focus on the parameters of that layer. However, during each training iteration, all parameters of the neural network updated,

---

Published as a conference paper at ICLR 2025
not just those of the final layer. The settings for the unlabeled pool size and acquisition size for each image dataset are provided in Table 5 Experimental results indicate that our method maintains comparable time consumption to other approaches, even with complex image data (see Table for details). Our primary focus active learning is enhancing model accuracy rather than speed. We developed CNNs tailored for MNIST' and EMNIST, with specifications in Table For CIFAR-IO, adopted a model architecture from Trockman & Kolter (2022). As shown in Figure 4 both BALD and IV methods suited to different scenarios, but our method consistently outperforms others across all datasets, particularly in more complex situations. In fact, the MNIST dataset, being relatively straightforward, does not fully showcase the advantages of our proposed method. To address this limitation, we constructed more challenging variants of the MNIST dataset, specifically unbalanced and redundant versions. Furthermore, we evaluated our method on two more complex datasets, Office-31 and AG News, and experimented with fine-tuning pre-trained models. Comparative analyses conducted on these datasets demonstrate the superior efficacy of our method, as detailed in Appendix E.3. Additionally, Figure visually represents the selection preferences of Fq highlighting its tendency to identify points that are more challenging for the current model to distinguish.
To deepen our analysis, we present ablation experiments Appendix E. to better understand the trade-offs between potential performance reduction and quantified computational cost reduction, and we also conduct a comparison with the Chapley-value-bes method in Appendix
MNIST
EMNIST
CIAR10
0.550
0.825
5,000
0.803
0.840
0.775
4,850
0.800
0.000
0.255
Random Fitur
0.709
Rancon Fkoun)
Randem Floum)
0.675
0.260
6.700
Dals
0.553
Said
Bald
202
500
32.00
20202
Dataset
Figure 4: Classification performance due the different active learning methods on MNIST, EMNIST; and CIFAR-1O.
Ground Truth: 5
Ground Truth: 5
Ground Truth:
S
S
4
1.0
1.0
1.0
Probabilky
0.5
0.5
0.5
0
1
33
5
1
4
2
3 4
Predicted
Prodicted
Predicted
Figure 5: Images with the highest Fiactive selected in the first round of active learning for the MNIST problem, along with their corresponding prediction probability distributions.
7 CONCLUSION
In this paper, we introduce novel local influence metric that evaluates the impact of perturbations to training samples on model performance concerning validation samples. This metric is applicable in both data trimming and active learning, offering valuable insights into the contributions of individual samples and their relationships with unlabeled data. Furthermore, we propose two approximation methods to mitigate the computational costs associated with calculating local influence measures. Our experimental results demonstrate that these algorithms effectively reduce costs while outperforming other otate-of-the-art methods.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg. Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: system for large-scale machine learning, arXiv preprint, arXiv:1605.0865 00600, 2016.
Shun-ichi Amari. anfferetiil-geomeenn methods in statistics, volume 28. Springer Science Business Media, 2012.
Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. American Mathematical Soc., 2000.
Avrim L Blum and Pat Langley. Selection of relevant features and examples in machine learning. Artificial intelligence, 97(1-2):245-271, 1997.
Razvan Caramalau, Binod Bhattarai, and Tae-Kyun Kim. Sequential graph convolutional network for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9583-958885 9592, 2021.
Yuanyuan Chen, Boyang Li, Han Yu, Pengcheng Wu, and Chunyan Miao. Hydra: Hypergradient data relevance analysis for interpreting deep neural networks. 35:7081-7089, May 2021.
Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. "what data benefits my classi- fier?" enhancing model performance and interpretability through influence-based data selection. In The Twelfth International Conference on Learning Representations, 2024.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST to handwritten letters. arXiv preprint,
2017.
R. Dennis Cook. Assessment of local influence. Journal of the Royal Statistical Society: Series B (Methodological), 1832):133-155, 12 2018. ISSN 0035-9246.
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09888 09841, 2018.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In Proceedings of the 34th International Conference Machine Learning, volume 70, pp. 1183-1192, 2017.
Amirata Ghorbani and James Y. Zou. Data shapley: Equitable valuation of data for machine learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 2242-2251 2019.
Amirata Ghorbani. Michael P. Kim, and James Zou. distributional framework for data valuation. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 3535-3544 2020.
Gene H Golub and Christian Reinsch. Singular value decomposition and least squares solutions. In Handbook for Automatic Computation: Volume II: Linear Algebra, pp. 134-151. Springer, 1971.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Mare Gorriz, Axel Carlier, Emmanuel Faure, and Xavier Giró Nieto. Cost-effective active learning for melanoma segmentation. arXiv preprint, 1rXiv:1711.09168. 2017.
Yufen Huang, Mei-Ling Kuo, and Tai-Ho Wang. Pair-perturbation influence functions and local influence in pca. Computational statistics & data analysis, 51 (112))5886-58899 2007.
Ruoxi Jia, David Dao, Boxin Wang. Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J. Spanos, and Dawn Song. Efficient specific data valuation for nearest neighbor algorithms. arXiv preprint, 100.:1908.00611 2019a.

---

Published as a conference paper at ICLR 2025
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J. Spanos. Towards efficient data valuation based on the shapley value. In International Conference on Artificial Intelligence and Statistics, volume 89, pp. 1167-1176, 2019b.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
Andreas Kirsch, Sebastian Farquhar, Parmida Atighehchian, Andrew Jesson, Frédéric Branchaud- Charron, and Yarin Gal. Stochastic batch acquisition: A simple baseline for deep active learning. Trans. Mach. Learn. Res., 2023, 2023.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 1885-1894, 2017.
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pp. 202-207, 1996.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 2012.
Yongchan Kwon and James Zou. Beta shapley: unified and noise-reduced data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, volume 151, pp. 8780-8802, 2022.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998. doi: 10. 10.1109/5.72679..
Xingjian Li, Pengkun Yang, Yangcheng Gu, Xueying Zhan, Tianyang Wang, Min Xu, and Chengzhong Xu. Deep active learning with noise stability. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 13655 13663, 2024.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In IEEE International Conference Computer Vision, pp. 3730-3738, 2015.
James Martens and Roger B. Grosse. Optimizing neural networks with moonecker-factoree approx- imate curvature. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 2408 2417,241,222 2015.
Christoph Mayer and Radu Timofte. Adversarial sampling for active learning. In IEEE Winter Conference on Applications of Computer Vision, pp. 3060-3068, 2020.
Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22-31, 2014.
Peter Nickl, Lu Xu, Dharmesh Tailor, Thomas Möllenhoff, and Mohammad Emtiyaz Khan. The memory- pertuurtt equation: Understanding model's sensitivity data. In Advances in Neural Information Processing Systems, volume 36, pp.
2023.
David Noever. Machine learning suites for online toxicity detection. arXiv preprint, arXiv:1810001869, 2018.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.08866) 2018
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zach DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, volume 33, pp. 19920-19922 19930, 2020.
12

---

Published as a conference paper at ICLR 2025
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM computing surveys, 54(9):1 100, 2021.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models new domains. In European Conference on Computer, volume 6314, pp. 213-226, 2010.
Burr Settles, Active learning literature survey. 2009.
Hai Shu and Hongtu Zhu. Sensitivity analysis of deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4943-4950, 2019.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In IEEE International Conference on Computer Vision, pp. 5972-5981, 2019.
Yang Sui. Yukun Huang, Hongtu Zhu, and Fan Zhou. Adversarial learning of distributional reinforee- ment learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 32783-32796 2023.
Asher Trockman and J. Zico Kolter. Patches are all you need? arXiv preprint, arXiv:2201.09792 2022.
Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, volume 206, pp. 6388 6421. PMLR, 2023.
Jiachen T. Wang, Tianji Yang, James Zou, Yongchan Kwon, and Ruoxi Jia. Rethinking data shapley for data selection tasks: Misleads and merits. In Proceedings the 41st International Conference on Machine Learning, volume 235, pp. 52033-52068 2024.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, volume 33, pp. 5776-5788, 2020.
Chih-Kusa Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pp. 649-657, 2015.
Hongtu Zhu, Joseph G. Ibrahim, Sikyum Lee, and Heping Zhang. Perturbation selection and influence measures in local influence analysis. The Annals Statistics, 35(6):2565-2588) 2007.
Hongtu Zhu, Joseph G Ibrahim, and Niansheng Tang. Bayesian influence analysis: geometric approach. Biometrika, 98(2):377332, 2011.
A DERIVATION OF EXPRESSION FOR 0, + w)
Recall that 0(x) minimizes the empirical risk: R(0) = = Σ" L(Vi, T.; 0). We assume that R is twice-differentiable and strongly convex in 0, i.e Hg:= (( = Σ 0g Lyy, æ; ô) exists and is positive definite. This ensures the availability of H which will be utilized in the subsequent derivation. The perturbed parameter vector ((x + ww can be written as
θ(x - w) = arg min R(0) + L(y. x w L(y, x; 0)
(A.1)
Define the parameter change 8(w) - 0(x w) - ((x), and note that, since ((x) does not depend w, the quantity aim to calculate be expressed in terms of it: 0, w) aso(w))
13

---

Published as a conference paper at ICLR 2025
According to the definition of Ō(x + w), we know
1 - 1 w)) + 0o [-y,, w; 0(x + w)) -g 1 o; 0( + w)).
(A.2)
n
Next, since 0(x + w) - ((x) as w 0, we perform Taylor expansion of the right- side: - L. x; Ō(x + w)),
1
= 0.l(
+ + + 00 (y, wi Ō(x + w))
(A.3)
n
n
(00R(0(x)) 2y wx 1 0y x; 0(x))]
"
0g T w: 0())) _ 00 + +(83R(((=))
(A.4)
Solving for ((w), we get
8(w) = - (0)R(0(x)) + ( - Lo x: ((x))]-
(A.5)
(0) + T w ((x))
_ do Do 0(x)))
Since ((z) minimizes R, we have R( = 0. We further assume that 0g Lly x; d(x)) is continu-
æ. then have
8(w) - -
w; ((x)) 0oL(y, (()))]
(A.6)
After differentiation, the final expression can obtained,
0 w) aw dww) w=w-
n Hi (g00 0(z)))
(A.7)
B ALGORITHMS
Algorithm Data Trimming
Input: Training set Z. Validation set V, Base model F, Budget b
Output: Trimmed Dataset Z'
1: procedure DATATRIMMING Z, V. F, b)
2: Call the algorithm FT -CALCULATIION2,, V, F) to obtain Ff 2
3: Select b samples 2; Z as (Z,), whose Fi rank in the top b
4: Z' e' {2s)
5: return Z'
6: end procedure
Algorithm 4 Active Learning
Input:
Labeled pool of training data L. Unlabeled pool of training data 4, Base model F, The number of samples for annotation (per round) N, Aggregation function
Output: Updated labeled pool L
procedure ACTIVELEARNINGLL. U, F.N.N..) g.
2: for 4 1 to NUM M. rounds do -CA.CCLLTIONN......C. U, F. to obtain Flactive E RM 4: Select N samples arf as {x whose Fractive ranks in the top N
3:
Call the algorithm Flactive_Ca
5: Take {Xx} out of L, and query their labels (Yx)
6: Update L CU (XN, ((X,NNN) YN}
7:
end for
8: return
9:
end procedure
14

---

Published as a conference paper at ICLR 2025
Algorithm 5 Calculation of FTuiil
approx
Input: Training set Z. Validation set V. Base model F, Truncated-Si parameter
Output: Influence measure vector Fluit
apprax E 1
1: procedure F Flagp apprex CALCULATIONE V, F)
2: Train FF with 2, and obtain the parameter vector 0
3: Generate an empty vector FI Flpppeex size |2]
4: Calculate H with K-FAC approximation
5: Calculate + "Eri d 2
6:
for every zi in do Calculate (w)(w))
7:
8:
Calculate Akxi, Upxk and Vax with power itcration
9:
T;; 0) t Upx Vaxk
L 0) - M (a0 wev 0. .'' x'; 0) Hj 0000gL
10:
FTuai
aa L(V; Gt (w0)
apprax
12: end for
13: return Fluit appma
14:
end procedure
Algorithm 6 Data Trimming with Approximation
Input: Training set Z, Validation V/ Base model F, Budget b, Pruncated-SVD parameter Subsampling rate E (0, 1] Output: Trimmed Dataset 2'
1: procedure 2MMIINAPPONG,,,, V. F. k, a))
2: Subsample Z, C S.t. Zn w.p. a for all
3: Call the algorithm Fiuii -CALCULATION V. F, k) to obtain FJuti!
RR| 22/2xX
approxx
apprax
4: for 2n (worr yi-) 2a do
T3 the rank of Fjuai Fipppeee [''] sorted in ascending order
5:
6:
end for
7:
Train a random forest with ((rerr))
8:
for 2; -a;, E Z do
9:
f: < h(x:))
10:
end for
11:
Select samples 2; E Z, whose î rank in the top Z' <- 2 }
12:
13:
return Z'
14: end procedure
15

---

Published as a conference paper at ICLR 2025
Algorithm 7 Calculation of F Flactive
upprox
Input: Labeled pool of training data L. Unlabeled pool training datal, Base model F, Aggregation
function g. Truncated-ST VVD parameter &t
Output: Influence measure vector Fi
appnax E R x
1: procedure Flappre CALCULATION(C.,,FF
2: Train FF with L, and obtain the parameter vector θ
3: Generate an empty vector Fl apprex size [4]]
4: Calculate H with K-FAC approximation
5:
for every ad! in I do Obtain an estimated label y with Calculate Lii, 27; ; 0)
6:
7:
8:
J 4
9:
for every z in L do
Calculate Gzg(wo) Calculate Abxks Upxk and Vaxk with power iteration
12:
00 (yVk ar: 0) c- UpxxAAxt Vdxt
0a Lyv wh: J I+ JU L () (w)) L 2!! Q)}
O. L att 0) + daL(in att 0) 10
14:
15:
end for
Flaetiie end for
[]] 9(J)
17:
return F FInetive
19:
end procedure
Algorithm Active Learning with Approximation
Input:
Labeled pool of training data L, Unlabeled pool of training data U. Base model F, Number of samples for annotation (per round) N, Aggregation function g. Truncated-SVD parameter Subsampling rate E (0, 1]
Output: Updated labeled pool C
1: procedure
WAW.LLANIINNPAPPO........ k.
2: for 1 to NU -rounds do Subsample U. C s.t. E Ua w.p. for all U
3:
4: Call the algorithm Fractive -CALCULATION Ua F, g, k:) to obtain Flactive approxx
approx E IR U4 4a1x1
5: for Øz E Ula do
6:
Tyr + the rank of Freeire [:'] sorted in ascending order
7:
end for
Train random forest h with ((ay, rx) [hal
8:
9:
for x; 24 do P; 4- hxxs) end for
10:
12:
{XN}, whose f. ranks in the top N Take }Xx} out of, and query their labels {YN}
Select N samples
13:
14:
Update L + (X(N.. Yn}
15:
end for
16:
return L
17:
end procedure
C DATA SOURCES
Adult. The Adult dataset consists of 48,842 instances and 14 features, including categorical and integer types. Extracted from the 1994 Census database by Barry Becker, focuses on predicting whether individual's annual income exceeds $50,000. Records were filtered based criteria such
16

---

Published as a conference paper at ICLR 2025
age, gross income, and work hours, making this dataset valuable resource for classification tasks in social science and Access Link: Adult Database
Bank. The Bank dataset pertains to direct marketing campaigns conducted by Portuguese banking institution, focusing on phone call outreach. The primary objective of this dataset is to classify whether client will subscribe dataset is derived from multiple marketing campaigns, which often required several contacts with the same client to ascertain their interest in the product. Access Link: Bank Database
term deposit, indicated by the binary variable (yes/no). This
The CelebA dataset is a large-scale facial attribute dataset containing over 200,000 celebrity images, each annotated with 40 attribute labels. Due to its rich annotations and diversity in facial appearances, CelebA has been widely used in various computer vision tasks, such as facial recognition, attribute prediction, and generative modeling. Access Link: CelebA Database
CelebA.
Jigsaw Toxicity. The Jigsaw dataset of Wikipedia consists of comments from online platforms that have been labeled for toxicity. It contains a large number of comments, with 28 features of syntax, sentiment, emotion and outlier word dictionaries. The dataset is commonly used for training and evaluating machine learning models aimed at detecting harmful or inappropriate content in user-generated text. Access Link: Jigsaw Toxicity Database
The MNIST database a large collection of handwritten digits that is widely used for training and testing in the field of machine learning. This dataset contains 70,000 images of handwritten digits (0-9), each of which is a 28x22 28 pixel grayscale image. Access Link: MNIST Database
MNIST.
EMNIST. The Extended MNIST database consists of handwritten character digits sourced from the NIST Special Database 19, formatted as 28x22 pixel images to align with the structure of the MNIST dataset. There six different splits provided in this dataset, and use the EMNIST Letters with 145,600 characters. Access Link: EMNIST Database
CIFAR10. The CIFARIO database, developed by the Canadian Institute for Advanced Research, is widely utilized collection of images for training machine learning and computer vision algorithms. consists of 60,000 color images, each measuring 32 32 pixels, categorized into 10 classes: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Each class contains 6,000 images. Access Link: CIFARIO Database!
Office-31. The Office-31 dataset is a widely used benchmark dataset for domain adaptation tasks. contains 1,110 images categorized into 31 classes, collected from three different domains: Amazon (A), Webcam (W), and DSLR (D). These domains represent images from an online retailer, web camera, and a digital SLR camera, respectively. In our experiments, we combine all domains together, treating the dataset as a single unified classification task. Access Link: 3 Dataser
AG News. The AG News dataset is a widely used text classification dataset that contains news articles categorized into four distinct classes: World, Sports, Business, and Science/Technolog. The dataset consists of 120,000 training samples and 7,600 test samples, with each class containing an equal number of articles. It is commonly used for benchmarking text classification models in natural language processing (NLP), Access Link: AG News Dataset
D EXPERIMENT DETAILS
D.I EXPERIMENT DETAILS IN
Data Construction for
Datasets. In this experiment, the preprocessing methods for all real-word datasets are consistent with those used in Chhabra et al. (0024). For CelebA, we use the extracted features provided by the authors Liu et al. (2015). and for Jigsaw Toxicity, we obtain text embeddings using the MiniLM transformer model (Wang et al. 2020). Further details are provided below.
Adult. This dataset contains 37,692 samples, with 30, 162 for training and 7,530 for testing.
There are 102 features, and the target to predict if income exceeds $50k (yes) or not (no).
Bank. This dataset consists of 30,490 samples, divided into 18,292 training samples and 12,198 test samples. There are 50 features, and the target is to predict if the client will subseribe term deposit (yes/no).
17

---

Published as a conference paper at ICLR 2025
CelebA. This dataset includes 104,163 samples, split into 62,497 training samples and 41,666 testing samples. There are 39 features, and the aim to predict whether person is smiling (yes) or not (no).
Jigsaw Toxicity. This dataset consists of 30,000 samples, split into 18,000 training samples and 12,000 test samples. There are 385 features, and the target to determine a tweet is toxic (yes) not (no).
Given that the initial test accuracy of the model on the original datasets generally exceeds 90%, the impact of data trimming is minimal. Accordingly, we randomly sample 5,000 instances from the original training set to serve as the new training set, and similarly, 4,200 instances from the original test set to serve as the new test set.
Data Construction for Noisy Real-World Datasets. Consider the additional experiments on data trimming presented Figure [ For Adult, Bank, and Jigsaw Toxicity, our construction method follows the same approach as in the Real-World Datasets experiment, with the additional step of randomly sampling 500 instances from the training sets and adding white noise with variance of 0.1. For CelebA, since the variables are binary (-1, 1), we introduced noise by randomly selecting six feature columns and flipping their values (transforming -1 and 1 to -1). This approach generates noisy dataset for supplementary experiments.
Random Trimming. We randomly remove (budget) data points from the training sets under five random seeds, and evaluate the average performance the test sets in each iteration.
Experimental Procedure and Parameter Settings. First, model is trained on the initial dataset. Based on this trained model, we then implement three data trimming strategies, removing data points. Finally, retrain the model to evaluate the effectiveness of the different trimming strategies. The experimental procedure is illustrated in Figure 66 highlighting the key steps involved our study. Additionally, the parameter settings are detailed in Table
Trimming Strategy
Table 3: Parameter Settings of Data Trim- ming.
Fl
Remove points
Model Evaluation
Adult
Bunk
CelebA
Jigsaw Toxicity
optimizer
SGD
SGD
Adam le-6
Adams
points randonty
learning, weight_deccay
16-2
Adult
Bunk inoise
CelebA +noise
Jigsaw Tosicily
Modified
optimiser
SGD
heorning.rate rute
le:2
1e-2
le:2
le-l
deight_decay
le-3
le-4
le-6
le-4
Figure 6: Flowchart of data trimming.
18

---

Published as a conference paper at ICLR 2025
EXPERIMENT DETAILS IN 6.2 D.2
Active Learning Strategy
[ours]
IV
Model Tuning
BALD
Random
model parameters
pool
Acquire label query
labeid pool
Figure Flowchart of active learning.
Data Construction for Unbalanced and Redundant MNIST. For Unbalanced MNIST, set the total sample size to 27,500. Categories 4 are each allocated an equal sample size, representing 1/55 of the total sample. Similarly, categories 5 to 9 assigned equal sample sizes, with each constituting 10/55 of the total sample. This allocation strategy ensures a deliberate imbalance among the classes. Samples systematically drawn based these ratios from the original dataset to create this new unbalanced dataset. For Redundant MNIST, the task is delineated to classify solely the digits and presented in equal proportions. If the acquisition function selects input from any class other than or 7, the labeling function designates "neither" category. This setup leads to three-way classification scheme during training, categorized us to assess the model's effectiveness in dealing with class imbalance and partial class information, critical aspects real-world applications where similar conditions are often encountered.
vs. 7 vs. neither. This design allows
Active Learning Experiment. Tables 4 and dettail the parameter settings for tabular and image data, respectively. The tabular dataset experiments were based on the approach used in Chhabra et al. (2024). with parameter settings derived from their open-source code. The experimental plots in their Appendix G guided our choices, and we adopted similar parameter scales. For image classification, we followed the design framework proposed in Kirsch et al. (2023), aligning with the parameter scales outlined in their Appendix E. Additionally, o the neural network architectures employed for MNIST and EMNIST.
Table 4: Active Learning Experiment Configuration for Tabular Datasets
Attribute
Adult
CelebA
Jigsaw Toxicity
Number of Classes
2
2
2
Rounds
12
11
Initial Pool
300
300
180
Unlabeled Pool Size
5000
3000
5000
Acquisition Size
500
150
500
E ADDITIONAL EXPERIMENTS
E. ABLATION EXPERIMENTS
Here, we supplement ablation experiments to evaluate the impact of perturbations on model per- formance, taking the Bank dataset case study. Specifically, introduce Gaussian noise to b
19

---

Published as a conference paper at ICLR 2025
Table 5: Active Learning Experiment Configuration for Image Datasets
Attribute
MNIST
EMNIST
CIFARIO
Number of Classes
10
37
10
Rounds
14
10
9
Initial Pool
60
600
6000
Unlabeled Pool Size
420
2000
10000
Acquisition Size
30
100
2000
Table 6: Architecture of MNIST CNN
Layer Type
Activation
Output Dimensions (inel. Padding)
Conv2d
ReLU
(32, 14, 14), P=1 (64, 7, 7), P=1
Conv2d
ReLU
Dropout
Linear
ReLU
256 num classes
Linear
samples in the training with the highest and the lowest Fl values, respectively, where b takes values from [50, 100, 150, 200, 250, 300]. After introducing the perturbations, we evaluate the model's performance on the test set. The results are presented Table beble. can be observed that samples with higher FI values are more sensitive to perturbations, as their accuracy decreases more significantly compared to samples with lower values.
Table 7: Impact of Perturbations Model Accuracy for Data Trimming.
Acc. after Perturbations (Top b-FI Samples)
Acc. after Perturbations (Bottom b-FI Samples)
b
Change in Acc.
Change in Acc.
0
78.36%
/
78.36%
50
77.59%
-0.77%
78. 78.49%
0.14%
100
77.19%
-1.17%
78.33%
-0.03%
150
77.07%
-1.29%
77.45%
-0,91%
200
75.47%
-2.89%
77.42%
-0.93%
250
75.42%
-2.93%
77.19%
-1.17%
300
75.45%
-2.91%
77.04%
-1.31%
To address the trade-off between computational cost and performance, we add ablation experiments on two datasets. As shown in Table & the approximate method is over three times faster than the exact computation, with minimal impact on performance. It even achieves better results on the CelebA dataset. Additionally, the dataset size increases, the proportion of time spent training and prediction with the random forest decreases, making the speedup even more significant.
E.2 ADDITIONAL EXPERIMENTS FOR DATA TRIMMING
Here, we present experimental results on four real-world datasets with noise in Figure g and nonlinear experimental results in Figure
20

---

Published as a conference paper at ICLR 2025
Table Comparison of Approximate and Exact Methods on Different Datasets
Dataset
Approx. Accuracy
Exact Accuracy (%)
Approx. Time (s)
Exact Time (s)
MNIST CelebA
96.0 83.2
96.2
396
1225
81.5
260
1054
Adult
CalebA
Josaw Todecty
0.0
0,773
0.99
12.
0:3
20
10
10
100
200
210
135
Figure 8: Accuracy curves of three data trimming methods on test sets of Adult, Bank, CelebA and Jigsaw Toxicity.
Training Set
Trimmed Samples(Oum)
Trimmed SamplesiIV)
Test Set Performance
0.004
Acc: 8.505
Acc: 9.380
Feature
resture
Feature
Feature
Feature 1
Feature
Figure 9: Performance under Nonlinear Model. Different colored points represent different classes. shows the training set. and C respectively denote the samples to be trimmed by Fl method and IV method. D denotes test set. Green line: boundary without trimming. Red line: boundary after FI trimming. Blue line: boundary after IV trimming.
E.3 ADDITIONAL EXPERIMENTS FOR ACTIVE LEARNING
Unbalanced MNIIT
Redundant MNIST
0.950
0.300
Tost Accurary
0.850
0.940
0.000
Random Fi(ours)
Random Fliours)
0.920
0.750
0.900
100
200
300
400
500
100
200
300
400
500
Dataset Size
Dataset Size
Figure 10: Classification performance due to the different active learning methods on Unbalanced MNIST, and Redundant MNIST.
21

---

Published as a conference paper at ICLR 2025
The AG News dataset (Zhang al. 2015) is widely used benchmark in natural language processing (NLP), while the Office-31 dataset Saenko et al. (010)) presents more complex challenge due to its diverse domains. As shown in Figure our method achieves consistently strong performance both the AG News and Office-31 datasets, demonstrating its effectiveness across different types of classification tasks.
Office-31
News
0.850
0.825
0.840
Test Accursyy
0.775
0.820
0.750
0.725
Random Fl(ours)
Random
0.780
Filours)
0.700
IV
Bald
Bold
0.675
700
Size
Size
Figure 11: Classification performance due to the different active leamning methods on Office-31, and AGNews.
Table Time (seconds) for active learning algorithms over toy case on MNIST, EMNIST, and CIFAR-10.
Methods
MNIST
EMNIST
CIFAR-10
FI(ours)
19
45
372
IV
13
15
204
BALD
9
15
214
E.4 ADDITIONAL EXPERIMENTS FOR COMPARISON WITH MASKING EFFECT AND DATA SHAPLEY
Masking Effect is key concept in model diagnostics, referring to the phenomenon where the impact of one outlier may be obseured by nearby outlier, making detection difficult. Algorithms like FI, which are based on single perturbations, inevitably face interference from the masking effect when addressing model diagnostic issues. However, the primary focus of this research is enhancing the predictive performance of the model, suggesting that the masking effect is unlikely to have significant impact the process.
To address this, an algorithm that accounts for the masking effect has been developed, referred to Fl-paired. This approach is inspired by Huang et al. (2007), where perturbations are applied simultancously to two samples, and the corresponding FI values for the sample pairs are calculated. Subsequently, all sample pairs are sorted based on their values, and points are removed in descending order of FI. Given that the masking effect arises from outlier detection issues, which differ significantly from the context of active learning, comparisons are made solely regarding the data trimming problem, as illustrated in Figure 12
The results indicate that, across both the Adult and Bank datasets, the performance of significantly surpasses that of Fl-paired. Furthermore, the requirement to compute FI for each sample pair substantially increases computational costs. Therefore, in this context, utilizing FI directly is the more effective choice.
Data Shapley indeed a well-known metric for data valuation (Ghorbani & Zou 2019) and is applicable in this context. However, based do not demonstrate any advantage over our approach or IV. [Wang et al. (2024) share similar perspective applying Data Shapley to data selection tasks, noting that its performance can be
experimental results, Shapley value-basee methods
22

---

Published as a conference paper at ICLR 2025
comparable to random selection. Additionally, their high computational cost presents significant drawback. Specifically:
Adult
Bank
0.790
0.00
0.785
0.760
Acciracy
0.84
0.770
0.765
0.78
0.760
50
100
150
203
50
100
159
200
Fitours)
TV
Flgaired
Shaprey Value
Random
Figure 12: Performance comparison of different methods on Adult and Bank, including the newly added Masking Effect and Shapley method.
In our experiments on data trimming with Adult and Bank, we observe that there is almost no significant difference between the performance Data Shapley method and Random selection, with slight improvement on Bank and even worse on Aduli (see Figure 12).
In our experiments on active learning with MNIST, we found that Data Shapley performed the worst, even slightly underperforming compared Random selection (see Figure
In terms of time consumption, Data Shapley method takes approximately 860 times longer than FI.
MNIST
0.950
0.900
0.850
0.800
Random
Fl(ours)
0.750
IV
0.700
Shapley
100
200
300
400
500
600
Figure 13: Performance comparison of different methods on MNIST, including the newly added
Shapley method.
E.5 K-FOLD CROSS- VALIDATION FOR DATA TRIMMING
Although all the datasets used in our experiments have clearly defined training and test sets, we further conduct 10-fold cross-validation on Data Trimming to verify the robustness of our method across different splits of the data.
In this experimental setup, the initial training set was randomly divided into 10 equally-sized subsets. In each iteration, one subset was held out as the validation set, while the remaining 9 subsets were used the training set. The test set was fixed and remained the same in the previous experiments, ensuring consistency across all evaluations. The results of 10-fold experiments are detailed in Figure 14 where compare the overall performance and stability of our approach Fl against the IV method.
23

---

Published as a conference paper at ICLR 2025
0.90
0.33
Acaraay
0.96
0.503
0.80
2.77
3.75
100
123
275
50
325
152
175
200
100
150
203
230
Plouat
Figure 14: K-Fold for Data Trimming: Performance Comparison of Fl and IV on Adult, Bank and CelebA
24


Reviews:
[{"review_id": "c4R9hwh6Ho", "reviewer": "Reviewer_mDX6", "summary": "This paper proposes a local influence-like case influence measure as a metric to assess the impact of training data on test set performance. \n\nThe measure is applied in two areas: data trimming and active learning. Based on the numerical studies, both applications help enhance the model's robustness and outperform the state-of-art strategies. \n\nTwo approximation techniques are introduced to address the high computational cost. They reduce memory and processing requirements while maintaining effectiveness.", "questions": "Please check the Weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "It's a useful study that includes \n\n1. the intuition and details of how this new influence measure is designed\n\n2. scenarios where it can be used \n\n3. approximation methods that can make it really practical\n\nExtensive examples with detailed simulation settings are included bringing persuasiveness.", "weaknesses": "1. I wonder if it is possible to provide the complexity analysis in terms of both memory and speed for the proposed method and the state-of-art strategies for examples in the paper. \n\n2. I am a bit confused about what robustness means in the paper. Is it about how the model would behave after removing or perturbing any data? Or its about after doing these procedures, the prediction performance is consistently better than not doing it? If it is the latter case, improving prediction performance should be a better way of describing it. \n\nMoreover, since we are talking about robustness, should the standard deviation of prediction accuracies in Table 2 also be included? also error bars in figure 2 and 8.\n\n3. Unlike the influence function in Chhabra et al, it seems that there is no clear threshold or boundary that differentiates samples that positively or negatively affect the model performance. In the examples, the number/portion of samples removed is also a tunable parameter. I speculate that the accuracy won't always increase monotonically with the portion of samples removed in the training set. And sometimes, it the data are perfectly separable, there is no need to remove any of the samples. With this being said, is there a guideline on how much data should be removed when working on a real-world dataset? \n\n4. I don't know if the masking effect is considered or not. Is there a way to incorporate this into your procedure? \n\n5. I am also curious about the choice of \\omega_0. In some other case influence literature, people would simply this from vector to scalar by considering it as the weight of the observation of interest. I wonder if it is treated similarly here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VHP15Xylh9", "reviewer": "Reviewer_pxvt", "summary": "The paper introduces a novel influence function based method to guide the data selection for data trimming and active learning. For data trimming, the proposed method applies perturbations to samples and measures the impact on the model's performance on test data. Here if the impact on test performance from perturbations in a sample input is large, it is encouraged to exclude that instance from the training set. \nIn the case of active learning, they perturb the model parameters and measure the changes in prediction for a new instance. If the changes in prediction is significant, the new instance will be added to the training data.\nAuthors study the effectiveness of their proposed method on synthesized and real world data sets for linear and non linear models and multiple domains. Experimental results show superior performance for the proposed method compared with other alternative approaches.", "questions": "- Based on equation 1 it looks like the perturbations applied to the training instances is by adding noise to them versus changing the training data's weight/importance. Can authors explain the rationale behind picking this type of perturbation over changing training instances' weights?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The problems addressed by the paper, i.e. filtering training data and choosing new training instances for active learning to boost model performance are real world problems with significant impact on a wide range of machine learning problems with real world applications.\n\n- Authors have provided sufficient literature review and the main claims of the paper are clear. \n\n- Authors propose two approximation methods for their local influence measure to reduce the computational cost of their proposed method.\n\n- Experimental results shed light on performance of the proposed method compared with two (depending on the test data set) state of the art approaches and a baseline of the random selection.", "weaknesses": "- The main weakness of this work is lack of ablation studies to better understand the impact of the contributions of this work on its performance.\n  - For example, the first contribution of the paper is to evaluate the impact of perturbations against the model performance for the test data set. It would have been useful to see how much this slight modification is helping with improving the proposed method's performance for data trimming.\n  - As another example, authors utilize two approximation techniques to reduce the computational cost of their proposed method, however, there is no experiment providing more insights into the trade offs between the potential reduction in the proposed method's performance vs. the quantified computational cost reduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NBXXPsloan", "reviewer": "Reviewer_LQN4", "summary": "This paper proposes a better influence measure to evaluate the impact of the training dataset on the test set. This metric mainly improves the flaws of the influence measure in Chhabra et al. (2024). The experimental results demonstrate that the FI proposed in this paper outperforms the IV metric in Chhabra et al. (2024).", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The contribution of this paper is simple and clear.\n2. the experimental results demonstrate that the FI proposed in this paper is superior to the IV metric in Chhabra et al. (2024).", "weaknesses": "1. The paper claims \"Our metric directly assesses the effect of small perturbations in the training samples on the model's performance on the test set\", whether this leaks information about the test samples during the training process, which needs to be clarified by the authors.\n2. The parameter settings in the experiments need further clarification.\n3. What are the limitations of FI that need to be further explained in this paper?\n4. I think the FI proposed in this paper is not enough to compare with IV only. The argument of this paper is to propose a better metric, however there are many similar metrics, such as the Shapley value. Therefore, this paper needs to further demonstrate the value of the proposed scheme.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJ2HVl8Vtv", "reviewer": "Reviewer_2oe6", "summary": "This study presents a First-order Influence (FI) measure designed to enhance the generalization capabilities of supervised machine learning models. To achieve this, the proposed measure assesses the contribution of each individual training data point to the model's average performance on a given validation set. The FI measure is first applied to Data Trimming, a process that identifies and removes non-representative training data points to improve model performance. Additionally, the FI measure is utilized in Active Learning (AL) to select the most informative, unlabeled data points that will be added to the training dataset, thereby optimizing the model's learning process with limited samples, iteratively.", "questions": "Q. What scenarios or conditions are likely to result in FI to underperform?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed applications of the First-order Influence (FI) measure demonstrate superior performance compared to its recent counterparts, particularly the Influence Value (IV) method [Chhabra et al. (2024) / chhabra2024data], showcasing the effectiveness of FI against a very recent work.\n\n@inproceedings{chhabra2024data,\n  title={\" What Data Benefits My Classifier?\" Enhancing Model Performance and Interpretability through Influence-Based Data Selection},\n  author={Chhabra, Anshuman and Li, Peizhao and Mohapatra, Prasant and Liu, Hongfu},\n  booktitle={The Twelfth International Conference on Learning Representations},\n  year={2024}\n}", "weaknesses": "- The evaluation is conducted for binary classification with logistic regression, supplemented by additional experiments using CNN on existing image datasets. However, the analysis is somewhat limited in scope. Although this setup is based on a related ICLR work [Chhabra et al. (2024) / chhabra2024data] and compares against the method named Influence Value (IV) from that published work, I find the dataset evaluation to be relatively insufficient. To thoroughly assess the performance of using FI or related methods, I believe it is essential to test it on a diverse range of datasets with varying characteristics, which would help to confirm its strengths and potentially reveal its limitations.\n\n- Following that, while FI appears to outperform the compared methods (IV and Random for Data Trimming; IV, Random, and Bald for Active Learning), its superiority is not necessarily absolute (at least this is not clearly shown), as performance varies across datasets. Notably, on the CelebA dataset for Data Trimming, Random actually outperforms IV and achieves results comparable to FI. This dataset-specific variation raises concerns about the generalization of the FI's performance across different datasets, leaving questions about how they would perform on other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kFEtrBM7SI", "reviewer": "Reviewer_Sa7V", "summary": "Authors propose the use of an influence measure based metric to determine the importance of samples in a dataset towards generalisation performance of models. An approach for selecting such samples for optimal training (data trimming) and an active learning based approach is also presented. Relevant metrics to this end have been introduced and evaluated on datasets.", "questions": "Details of the complexity and scalability of the proposed method, and evaluation on larger datasets would help establish practical utility of the method.\n\nCode for proposed method could be made available.\n\nAdditionally, addressing comments mentioned as weaknesses would be helpful.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The initial discussion introducing the proposed method is detailed and well motivated. Authors present the approach with sufficient rigor. The algorithms are presented well and the experimental results reported on the chosen datasets as well as the experiments conducted have been described in adequate detail with supplementary results.", "weaknesses": "My primary concern with the work presented is the limited scalability analysis of the proposed methods for larger datasets, and possibly the lack of discussion of the computational complexity of the algorithms. Practical datasets for most real-world applications could have much higher number of samples, as well as features, so it would be important to benchmark with such datasets.\n\nIt may be useful to provide accuracy results in a tabular form reported as mean +/- std. dev. as an appendix for the sake of completeness.\n\nThe authors should possibly consider making the code for the methods proposed available publicly to ensure reproducibility.\n\nAs far as the datasets where train and test sets are not explicitly provided, and authors have chosen a split to conduct the experiments and report the results, are these k-fold validated?\n\nWhile a time comparison is provided in Table 7, could the authors comment on possible methods for scalability of the proposed approach for practical applications, especially when inference is required from streaming datasets in test.\n\nThe equations in the appendix could be numbered for ease of reference. Proof for Theorem 1 does not seem to be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)