Distributionally Robust Coreset Selection under Covariate Shift
Authors: Tomonari Tanaka
Hiroyuki Hanada
Hanting Yang
Aoyama Tatsuya
Yu Inatsu
Akahane Satoshi
Yoshito Okura
Noriaki Hashimoto
Taro Murayama
Hanju Lee
Shinya Kojima
Ichiro Takeuchi
Venue: Accepted by TMLR ()

Abstract: Coreset selection, which involves selecting a small subset from an existing training dataset, is an approach to reducing training data, and various approaches have been proposed for this method. In practical situations where these methods are employed, it is often the case that the data distributions differ between the development phase and the deployment phase, with the latter being unknown. Thus, it is challenging to select an effective subset of training data that performs well across all deployment scenarios. We therefore propose Distributionally Robust Coreset Selection (DRCS). DRCS theoretically derives an estimate of the upper bound for the worst-case test error, assuming that the future covariate distribution may deviate within a defined range from the training distribution. Furthermore, by selecting instances in a way that suppresses the estimate of the upper bound for the worst-case test error, DRCS achieves distributionally robust training instance selection. This study is primarily applicable to convex training computation, but we demonstrate that it can also be applied to deep learning under appropriate approximations. In this paper, we focus on covariate shift, a type of data distribution shift, and demonstrate the effectiveness of DRCS through experiments.

Published in Transactions on Machine Learning Research (06/2025)
Distributionally Robust Coreset Selection under Covariate Shift
Tomonari Tanaka
Nagoya University
Hiroyuki Hanada
Nagoya Universiby
RIKEN
Hanting Yang
hanting. yaaggngggnnngaannnnnngnn man...gnggommmm@mml..comm
Nagoya University
Tatsuya Aoyama
ooyama. wogamma.aaauuy..aaaggoeeo www.ggp...agyyyymmmmmmalloolm
Nagoya University
Yu Inatsu
inatsu..nn@@steeh.ac...
Nagoya Instilute of Technology
Satoshi Akahanc
Nagoya Universily
Yoshito Okura
okuma. hwwp.yyossttonnggoyaaallgggggg
Nagoya University
Noriaki Hashimoto
RIKEN
Taro Murayama DENSO CORPORATION
toro. hoo.murryamm..7z.2hp..deee..
Hanju Lee
lcc.hanjju.8dd0ppp.de dew.://......p.deedooomo com
DENSO CORPORATION
Shinya Kojima
DENSO CORPORATION
Ichiro Takcuchit
takeachi, ichiro. nb n6 mail maw.aaaaa.aaaaua..... nagoya-u.oeueeooo ac.
Nagoya University
RIKEN
Reviewed on OpenReview: https: // wttp://ewepeeeeeeeeemeeee/ net/ forum? id= Bu7XMLJssC
Abstract
Coreset selection, which involves selecting a small subset from an existing training dataset, an approach to reducing training data size, and varions approaches have been proposed for this method. In practical situations where these methods are employed, it often the case that the data distributions differ between the development phase and the deployment phase, with the latter being unknown. Thus, it is challenging to select an effective subset
"Corresponding author. i "Correspondigg author.

---

Published in Transactions on Machine Learning Research (06/2025)
of training data that performs well across all deployment scenarios. We therefore propose Distributionally Robust Coreset Selection (DRCS), which theoretically derives an estimate of the upper bound for the worst-case test error, assuming that the future covariate distribution may deviate within a defined range from the training distribution. Furthermore, by selecting instances in a way that, suppresses the estimate of the upper bound for the worst-case test error, DRCS achieves distributionally robust training instance selection. This study primarily applicable
convex training computation, but we demonstrate that it can also be applied to deep learning under appropriate approximations. In this paper, we focus on covariate shift, type of data distribution shift, and demonstrate the effectiveness of DRCS through experiments.
1 Introduction
Coreset selection is a technique designed to reduce the size of training data while maintaining the performance of predictive models (Guo et al., 2022). By carefully identifying a representative subset of the original datasct, this approach addressos critical challenges such as computational efficiency and memory limitations. Coresets are constructed capture the most informative and diverse samples, ensuring that the essential characteristics of the underlying data distribution are preserved. This makes the method especially valuable in scenarios involving large-scale datasets or resouree-constranncl environments, where processing the entire dataset may be infeasible. Coreset selection finds broad applications in areas such as data summarization, accelerating model training, reducing annotation costs, and improving model interpretability. Tt. is also practically applied to technologies that require retaining useful training instances, such as continual learning and active learning (Sener & Savareso 2017 Tonova et al., 2019 Paul et al., 2021 Ducoffe & Precioso, 2018 Margatina et al., 2021). As machine learning advances into more complex domains, coreset. selection will become increasingly critical for balancing efficiency and scalability.
This paper addresses the problem of coresct selection in scenarios where the future deployment environment of the model is uncertain. This challenge frequently arises in applications that require models to be tailored to specific, and often unpredictable, conditions or environments. For example, it includes tasks such as assessing investment risks under fluctuating economic conditions, predicting medical outcomes across diverse demographic groups, and estimating agricultural yields under varying climate scenarios. In such settings, the ability to create representative and compact subset of the data that ensures reliable model performance is crucial. Traditional corcset selection does not make the assumption that future test distributions are uncertain, and it is unclear whether the model will work elfectively under such conditions. The objective of this study is to develop a methodology for selecting a coreset from the original dataset while explicitly accounting for the need for worst-case robustncss. This enables the model to perform effectively even under uncertain and variable future deployment conditions.
In this study, we focus on a distributionally robust setting under covariate shift conditions, which is a critical challenge in many real-world machine learning applications. Covariate shift occurs when the distri- bution of imput features changes between the training and deployment (test) phases. Distributionally robust learning (Goh & Sim 2010 Delage & Ye 2010 Chen & Paschalidis 2021 aims to tackle this problem by optimizing model performance under worst-case distributional shifts, enabling models to perform effectively across a wide range of potential data distributions and ensuring robustness to variability and uncertaintios. Within this context, we address the specific problem of selecting a robust coreset, assuming that the future covariate distribution may deviate within defined range from the distribution of the original datasct. To this end, we propose a novel method, termed the Distributionally Robust Coreset Selection (DRCS) method, which focuses constructing representative and robust subset of data tailored for these challenging con- ditions.
The basic idea of the DRCS method is to select a coreset that minimizes the worst-case test error under uncertain future test distributions, addressing the challenge of ensuring robustness in the face of potential distributional shifts. To achieve this, we derive a novel and theoretically grounded upper bound for the worst-case test error in the context of distributionally robust covariate shift. This upper bound serves as a critical foundation for guiding the coreset. selection process. Building on this theoretical insight, we propose
2

---

Published in Transactions on Machine Learning Research (06/2025)
efficient algorithm designed to select coresct that approximately minimizes this upper bound, ensuring that the resulting subset of data is both compact and robust to changes in future test. distributions. Although
the method is primarily developed for problems formulated within a specific class of convex optimization frameworks, it is versatile and can extended to coreset selection for deep learning models by leveraging the neural langent kernel (NTK) or fine-tuning scheme.
[A] Naive method (retraining)
Distribution1
Distribution 2 w
Distribution 3
w
W1
U2
W3 w
Original
the worst-case test error situation
Selection
Selection 2
Figure 1: The concept of coreset selection in this study. In the left panel, each plot shows the distribution of the training data, where cach column represents patterns of distribution changes, while cach row represents pallerns of instance selection. Let gray lines represent the learned results with specified weights and all instances, while green lines the retrained results with specified weights and selected instances. The goal is to obtain selection pattern that can suppress the degradation of test crror, even the worst-case distribution for each selection pattern. In this figure, through the three distributions, Selection can be considered a better choice than Selection 2. It is practically impossible since need to explore such worst-case test error for all distributions and selection patterns.
3

---

Published in Transactions on Machine Learning Research (06/2025)
[B] Our method (calculate bound of model parameters)
Distribution
Distribution 2
Distribution 3
7
W1
w 3
Original
upper bound worst-case tast error situation
Selection
Selection 2
The concept of coreset selection in this study. These figures also show the distribution of the training data. The green area indicates the bound of model parameters obtained when retraining is per- formed, and we can calculate algorithmically the bound before retraining. We perform coreset selection to minimize the worst-case test error in the distribution, where the bound becomes the largest among all possible distributions.
Figure 2:
Distributio(r=0.200 unknawn=3)
Distributie(t=0..80 unknawn=28)
0.5
0.5
5 0.0
51 0.0
0.5
0.55
-1.0
-1.0
connect
cornec incoreett
Incerfect
-2.02
-2.0
(a) Training instance selection with a re- duced bound of model parameters
(b) Training instance selection with ran- dom sampling
Figure 3: This figure illustrates an upper bound of the validation error in this study. Both figures show the distribution of the validation data. We calculate bound of model parameters and use this to derive an upper bound of the validation error. In this figure, the blue and red areas represent that the validation data in these areas have a determined classification. On the other hand, the validation data in the green area does not have a determined classification. As a result, the upper bound of the validation error can be reached cases where all instances in the green area are incorrectly classified. Since the bound of the model parameters depends how to select instances such as (a) and (b), we perform coreset selection in distribution where the upper bound of the validation error is minimized.
4

---

Published in Transactions on Machine Learning Research (06/2025)
1.1 Contribution
In this study, we address the challenges discussed in Section by introducing model parameter bounding techniques. Furthermore, we propose a distributionally robust coreset selection method that provides theo- retical guarantees for model performance in binary classification problems. The contributions of this study are as:
We consider the problem of coreset selection under covariate shift environments while taking distri- butional robustness into account, and propose a method to address this challenge.
In the proposed method, we derive an upper bound of the validation error under the worst-case covariate shift and perform coreset selection to minimize this upper bound.
2
Problem Settings
In this section, we formulate distributionally robust coreset selection (DRCS) problem in the context of uncertain covariate-shift settings.
2.1 Preliminaries and Notations
We consider a binary classification problem with the training dataset = {(z;, y/s bela) where n denotes the number of training instances, z; E X C Rd represents the i-th feature vector defined on the input domain X, and ys E {=1} denotes the corresponding label for [n], with the notation [n] indicating the set of natural numbers up to n. Similarly, let D' {(a), sejn be the validation dataset of size n', where each validation instance is assumed to follow the same distribution as the training instances. For binary classification problems, we consider classifier parameterized by set of parameters 8, defined
f(;B): Rd e x f(x; B) R,
(1)
where the binary label for an input vector æ predicted as - f(;; (3 < O, and +1 otherwise. Furthermore, denote the loss function for binary classification (e.g., binary cross-entropy hinge loss)
8: {+1} (y. fxx; + Ely, f(:; B)) E RR,
(2)
where R4 indicates the set of nonnegative numbers !
In this study, we investigate distributionally robust learning under an uncertain covariate shift setting, where the input distributions for the caaiing/validdae datasets and the test datasets differ, with the discrepancy between these distributions known to lie within specified range. In such a covariate-shift setting, it is well known that the difference in input distributions can be addressed through weighted learning. We denote the weight for the i-th training instance as > 0, E [ni], and represent the r-dimensional vector of these weights as w E [0, Typically, these weights are determined based on the ratio of the lest input density to the training/valien validation input density (Shimodaira 2000 Sugiyama et al. 2007). Therefore, a binary classification problem in a covariato-shift. setting generally formulated as (regularizod) weighted empirical risk minimization problem in the form of
min B
Σ =E(m)
willys, + p(,3),
(3)
Emi Wy
For example, in the case of the binary cross-entropy loss for logistic regression model, it is expressed
fly, fax 8)) log(1 e ((m.))
As another example, the hinge loss for support vector machine (SVM) is expressed Eyy f(x; B)) max vf(x;
5

---

Published in Transactions on Machine Learning Research (06/2025)
where p(B) denotes a regularization function.
In a conventional covariate-shift setting where the test input distribution is known, the weight vector w can be predetermined based on the density ratio, enabling the optimal model parameters to be obtained by directly solving the minimization problem in equation 3 On the other hand, in the distributionally robust setting, the density ratio and hence the weight vector un are unknown. In this study, we consider a distributionally robust learning scenario where the weight vector lies within hypersphere of a certain radius S, centered around the uniform training data weights (i.., In = [I...., 1] This distributionally robust learning problem is formulated
min man NEW Eieln] Wi
Σ ic(n)
willyn, f(a; B)) + p(3),
(4)
where
W: (w E I2
|| - 1,112 /2 v 5}
(5)
represents the hypersphere within which the weight vector can exist.
2.2 Applicable Class of Problems by the Proposed Method
Before presenting the proposed method, we define the class of classification problems which the DRCS method applies. The proposed DRCS method is applicable when the classifier in equation the loss function in equation - and the regularization function in equation 3 satisfy certain conditions. As class of applicable classifiers, we focus the basis function model in the form of
f(x;
Σ ((0 JE(K]
(6)
where oj : Rd 3 x + (z( E RR [k] is the j-th basis function, l is the number basis functions, and ((x)) E R& the k-dimensional vector that gathers the basis functions. Furthermore, the loss function f(;; B)) assumed to be convex with respect to its second argument, and the regularization function p(B) must also be convex with respect B.
While these conditions may seem restrictive, kernelization is achievable by considering the dual form of the basis function model in equation 0 enabling extensions to popular nonlinear classifiers such nonlinear logistic regression and kernel support vector machines. When applying the proposed method to deep learning models, tools such as Neural Tangent Kernel (NTK) (Novak et al.) 2020) and Neural Network Gaussian Processes (NNGP) (Lee et al.) 2017 can be utilized. These tools bridge traditional kernel methods with deep learning, offering both thcorctical insights and practical tools for tackling high-dimensional and complex learning problems. Furthermore, in practical DRCS problems, since the training and test data are lypically assumed to share certain degree of similarity, fine- fine-tuning approach that updates part of the model parameters during testing is beneficial. The proposed method is applicable in such scenarios and serves valuable tool for coresct selection in deep learning models as well.
2.3 Distributionally Robust Coreset Selection (DRCS) Problems
To formulate the coreset selection problem, let us introduce an n-dimensional binary vector E {0, 13". where = 1 indicates that the i-th training instance is included in the coreset, while v, = 0 indicates that the i-th training instance is excluded. Hereafter, we refer to v as the coreset vector. Let us write the training error of our DRCS problem as
Po. ( Eie(n] U200 vE[N]
Dawe (yi, f(x;;))) + P(B).
We assume S 1, focusing on scenarios where the differences between the training and input distrihutions not substantial.

---

Published in Transactions on Machine Learning Research (06/2025)
Given coreset vector v {0, 1}" and weight vector w W, the classifier's optimal model parameter vector is written as
(v, w) = arg min Pe (3).
(8)
Given a weight vector w' E R" for validation dataset, the weighted validation error for the optimal model parameter vector in equation can be defined as
VaEr(v, w. w/) Exerng w/ VE[N!
Σ
w (vl sgn(f(a); w))))
(9)
where
W (w R. || - 1,/112 S Q}
(10)
represents the hypersphere of a certain radius Q within which the weight vector can exist. Here, I(-) is the indicator function that returns if the argument is true and otherwise, and sgn IR + [+1} the sign function that returns the sign of the given scalar input. the weight vector w' for the validation dataset is appropriatcly determined based on the density ratio of the input distributions of the validation and the test datasets, the weighted validation error VaEr in equation g can be used as test error estimator.
In distributionally robust setting where the weight vector is uncertain, minimizing the worst-case test error is necessary. As an estimator of the worst-case test error, it is reasonable to use the worst-case weighted validation error (WrVaEr) in equation g
expressed as
WrVaEr(w)
max max VaEr(u, w. w²).
(11)
w W'EW' wEW
Based on the above discussion, the goal of the DRCS problem is formulated as the problem of finding the coreset vector v E {0, 13" that minimizes the worst-case weighted validation error in equation 11 Let m < n the size of the coreset, i.., the number of remaining training instances after coreset selection. Then, the DRCS problem we consider formulated
2 = arg min WrV subject to ||l||1 = m.
(12)
By summarizing equations can be expressed as:
9
and
12
minimizing the worst-case weighted validation error (WrVaEr)
min max max min VaEr(w, w, w)
(13)
Unfortunately, the problem in equation 12 is highly challenging. First, it involves complex optimization over four types of vectors: v, w, w', and B. This optimization is technically complex as it requires solving nested optimization problems across four hierarchical levels, involving a large number variables and significantly increasing computational complexity, along with the difficulty of ensuring convergence and global optimality, Moreover, the fourth-level optimization for {0, 1) is combinatorial optimization problem, which becomes infeasible to solve optimally when is large. To address these technical challenges, our approach involves the following two steps:
(i) Deriving an upper bound for the worst-case weighted validation error in equation 11
(ii) Greedily identifying a coresct that minimizes this upper bound of worst-case weighted validation error.
In the next section, we describe methods for (i) and (ii) in Section 3.1 and Section 3.2 respectively.

---

Published in Transactions on Machine Learning Research (06/2025)
2.4 A Working Example: Lz-regularized Logistic Regression
As working example of such classification problem, let consider Læ-regularized logistic regression of basis function models, i.e., consider basis function model in equation 6 binary cross-entropy as the loss function E(y, fx; ()) and L2 regularization function as the regularization function p(B). Considering the coreset vector v, the optimization problem for Le-regularized logistic regression is defined as
f0w arg min P., (g,)
(14)
8E3*
where
w,w, log(1 =e >
11118118
Pe,w (B)
(15)
EiE(na 0:W; iE|m]
which is referred to as the primal problem and (B) is called primal objective function.
3 Proposed DRCS Method
In this section, we present the proposed method to solve DRCS problem in equation 12 As discussed in
the previous section, since it difficult to directly minimize the validation error (equation 11 , we show how to derive and minimize its upper bound. Also, since it still includes difficult problems (a combina- torial optimization and an alternative optimization), we show specific algorithm to solve it using greedy optimization.
3.1
Upper Bound for the Worst-case Test Error
In this subsection, we present our main result.
3.1.1
Main Theorem
An upper bound of the worst-case weighted validation error, which can serve as the worst-case test error esti- mator, WrVaBr(v) in equation II is presented the following theorem. Our main result can be formulated as equation T8
Definition 3.1. For a convex function f IR" + R, its convex conjugate f R" R U [+oo} is defined as (B) su (BT 3' f(A"))
Definition 3.2. A function f _ + R is called p-strongly convex if f(3) 1118113 / is conver.
Definition 3.3. For arg ming P. (8) in equation 7 define f(x; B) g (xxx). Then, its dual problem is defined as
(e) Eie(n) W:W,s
Σ (,w,t((-aa) pp" Ere(ny w,w; (diag(v & w ( T
(16)
ig|m
where क- ()(x)) ((x2) b. ((tn))) E Rank and & is the element-wiss product. Here, especially, if p(3) = 1)1 1 it is calculated
1
De Exe(n) 1:W%
mwwl (-as)
[[diag( 8 6 T) Tal (17)
VE(M)
2A (Eig(n) 07i0;
This derivation presented in Appendix B.1 Then state the main theorem follows.

---

Published in Transactions on Machine Learning Research (06/2025)
Theorem 3.4. Assume that p in the primal objective function Pin, is u--trongly conver with respect to B. Let us denole the optimal primal and dual solulions for the entire training set (i.e., vi Vi E [m]) with uniform weights (i.e., W; = Vi (m)) as
Bi arg min 11 (3) and aia, = max D1..
BERK
QER
respectively. Then, an upper bound of the worst-case weighted validation error is wrillen as
WrVaEr(o) S WrVaEr (()) =1 1 Q
[((())]B -V
(1,5(
1
(18)
n
where,
((0) (0.13%; ((v) 1 11 ((a{) - ( / DG w) > 208
(19)
wEW
DG(v, w) Po, ( (3) Du Du.ul (a
(20)
Here, the quantity DG is called the duality gap and it plays main role of the upper bound of the volidation error.
Especially, if we use I.2reeglaaizaaionn p(3) = 2 11 DG(v, w),, the mazimization max DG(v, w) can algorithmically computed.
We sketch the proof of Theorem 3.4 in the next section. The complete proof is given in Appendix
3.1.2 Proof Sketch
The proof sketch of Theorem 3.4 is primarily divided into the following steps.
1. First, as a premise, consider a model where Pw is p-strongly convex.
2. Second, under the point above, derive the bound of model parameters. The model parameter vector 3 trained using the coreset vector w and weight vector w, is guaranteed to converge within the range By which represented by an L2-norm hypersphere (Hanada et al. 2023). A
hypersphere of the radius is given by R = V DG w), which is calculated in equation 20
Third, calculate a bound of the weighted validation error. Using the hypersphere range B computed for a specific coreset vector v and weight. vector w, an upper bound of the weighted validation error can be analytically calculated. This upper bound is determined by DG. The larger DG, the greater the upper bound becomes.
4. Fourth, maximize DG with respect to the weight vector w for training dataset in equation 21 We can obtain an upper bound of the worst-case weighted validation error. We can casily confirm that DG(u, w) is a convex function with respect to w, and therefore its maximization is difficult in general. However, if we use I-2rregularization, it becomes convex quadratic function and its maximization can be algorithmically computed by solving an eigenvalue problem. In fact,
max DG w) max (u w) A w) T (v 2 w) + wEW
(21)
wEW
where the matrix A of the quadratic term, the vector of the linear term, and the constant term respectively given
A 1
2X diag(ai,.. v) T K diag(ai.... @ v),
b = [[n, f(x .... aj a In [e(n]'
1
2X (In @ ai... @ v) K @ aj..,, & v).
Here, K E Ruxn is kernel matrix, where is defined K,, ((x;) Tp(;))
9

---

Published in Transactions on Machine Learning Research (06/2025)
5. Finally, maximize upper bound of weighted validation error with respect to the weight vector wn for the validation dataset. We can derive an upper bound of the worst-case weighted validation error, leading to equation 19 and therefore equation [8
3.2 Greedy Coreset Selection Based on the Upper Bound
The basic idea of the proposed DRCS method is to select the coreset vector e E {0, that minimizes the upper bound of the worst-case weighted validation error represented by equation [8] Since this is combinatorial optimization problem, finding the global optimal solution within a realistic timeframe is challenging; thus, we adopt greedy approaches to obtain approximate solutions.
A naive greedy approach, referred to as greedy approach 1, repeats the followings: we remove one training instance that minimizes equation 18 and update w to maximize DG(v, w) in equation 20 (Algorithm
Although greedy approach 1 is sufficient for small datasets, the compulational cost. becomes prohibitively high for larger datasets. The most significant computational cost in calculating the bound in equation [8 lies in the eigendccomposition of A in oquation 21 required to determine max w), which needs (
time with naive computation, or Onn²) il is approximated. To circumvent this cost, another approach, referred to as greedy approach 2, does not update whenever instance is removed, but instead fixes optimized with initial v (i.e., 1n) (Algorithm 2). Then, it recalculates the values of min WrVVaErBB (v)
after the removal of cach instance to dynamically update the selection process. Morcover, as much simpler approach, referred to as greedy approach 3, determines the instances to be removed based solely on the initial values of min, Wr UBB (v) without recalculating them after each removal (Algorithm 3/. These greedy approaches are heuristics and do not guarantee optimality. However, in Section 5 we demonstrate that these approaches facilitate the selection of coreset, effectively mitigating the increase in worst-case test error.
The time complexities of these algorithins are as follows (if eigendeccopposition is not approximated), where del is the number of deletions: Onenndel) for greedy approach 1, (nn for greedy approach 2, and
Onn3 + ndel) for greedy approach 3. The details of these algorithm is given in Appendix C.4 In this Appendix, we provide the pseudocode of them.
4 Related Works and Limitations
4.1 Related Works
Coreset Selection. Coreset selection is a technique for selecting important data samples in training to enhance data efficiency, reduce the computational cost, and maintain or improve model accuracy. Cur- rently, several approaches exist, each differing in how they evaluate the importance of data. Representative methods are outlined below. Geometry-Based Methods: Geometry-Based Methods utilize the data distribution in the feature space improve learning efficiency by reducing redundant samples. Examples include k- center-Greede (Sener & Savarese 2017), which minimizes the maximum distance between samples, and Herding Welling 2009), which iterat ively selects samples based on the distance between the coreset cen- Methods: Uncertainty-Based Methods prioritize
ter and the original dataset center. Uncertainty-Bant selecting samples for which the model has the least confidence. Methods such as Least Confidence, Entropy, and Margin select high-uncertainty samples based on these metrics Coleman et al. 2019). Error / Losss- Based Methods: Frror/Loss--Bar Basod Methods select important samples based on loss function values or gradient information. Examples include GraNd (Paul al., 2021) and EL2N (Paul al. 2021). which are based the magnitude of the loss. censitivity-Based Methods: Sensitivity-
Methods are similar to Stror/Loss- Based Methods, however, for each sample, its importance value is defined by the maximum of the relative loss (ratio of the loss of the sample to the sum of the losses of all samples) among all possible model parameters. It can assure the upper bound of the model parameter shifts by the coresel selection in more strict manner (called the e-coreset) (Munteanu et al. 2018 Tukan et al. 2020 2021 Tolochinksy ct al. 2022 Alishahi & Phillips 2024). Decision Boundary-Basoo Methods: Decision Boundary- -Basdd Methods focus on selecting samples near decision boundaries that are difficult to classify, Examples include Adversarial DeepFool (Duoofff (Ducoffe & Precioso 2018) and Contrastive Active Learning Margatiin et al.
10

---

Published in Transactions on Machine Learning Research (06/2025)
2021p. Gradient Matching-Based Methods: Gradient
Based Methods aim to approximate the gradient of the entire dataset with a small number of samples. MMrzasiaizal et al. 2020
and GradMatch (iilaamsttt et al. 2021a) utilize this approach by leveraging gradient information. Bilevel
Optimization- Based Methods: Bilevel Optimization-Be Basddd Methods formulate corcsct selection as a
bilevel optimization problem. SntrriveeKillansserr
et al. 2021c) and Glister Sester(Killamsett,, et al. 2021b) have
Methods: Optimization methods based on Submodular functions(Iyer & Bilmes 2013) enable combinatorial optimiza- tion by introducing submodular functions to avoid the combinatorial explosion, allowing for the optimization of diverse sets of samples. Examples of submodular functions include Graph Cut (GC), Facility Location (FL), and Log-Determinant (LD) [yer et al.) 2021). Other approaches can be found, for example, in Guo
been applied to continual learning and active learning. Submodularity Optimization-Basse
et al. 2022).
Distributionally Robust. DR has been studied various machine learning problems to enhance model robustness against variations in data distribution. The DR learning problem generally formulated as a worst-case optinization problem to account for potential distributional shifts. Consequently, techniques that integrate DR learning and optimization have been explored in both fields. The proposed method builds upon such DR techniques, emphasizing effective training sample selection even when the future test distribution is unknown. It incorporates DR. considerations during sample selection rather than during training computation. While the primary goal of the proposed method is to reduce computational resources through sample removal, this process also has practical implications in other scenarios. For example, in continual learning (e.g., see Wang et al. (2022)), ) managing data by selectively retaining or discarding samples is erucial, especially when anticipating shifts in future data distributions. Improper deletion of important data may result in catastrophic forgetting (Kirkpatrick et al.) 2017), where the model loscs previously acquired knowledge after learning new data. Our proposed coreset selection method explicitly addresses this DR setting, and, to the best of our knowledge, no existing studies explore coreset selection under such a framework. Morcover, many existing coreset selection methods are heuristic in nature and lack robust theoretical guarantees.
4.2 Limitations
The proposed DRCS method is developed for binary classification problems and is applicable to models (such as SVM and logistic regression) where the primal objective function (loss function + regularization term) possesses strong convexity. Therefore, it cannot be directly applied to deep learning models. However, by appropriately selecting a regularization function, the method can also be extended to kernel methods. Consoquently, we can well approximate the deep learning by utilizing the recently popular Neural Tangent Kernel(NTK) NNovak et al. )020).
The proposed DRCS method utilizes a bound of the model parameter, and a similar approach has been studied in instance selection by Hanada et al. (2023). The method proposed by Hanada et al. (2023) (DRSSS) is effective for sample-sparse models due to its characteristics, but it is limited to models that are
both strongly convex and instance-sparse, and can remove only samples that do not change the training results at all.
In contrast, the proposed method is applicable to any model with strong convexity, making it a more versatile approach.
5 Numerical Experiment
5.1 Experimental Settings
In this section, we numerically evaluate the proposed DRCS through experiments. For this experiment, Ii...., i, is learned by solving equation 7 and 16 We perform cross-validation with a training-to-ter to-tes data ratio of 4:1 in the experiments. Next, we define the training weight range as W == (w ||w - 1, - $} and set as follows. The datasets are primarily designed for binary classification, and for multi-class datasets, two classes are extracted and used. We set. S with a parameter as S Vntla [atla-> 1/, where n is the number
11

---

Published in Transactions on Machine Learning Research (06/2025)
of positive instances in the training dataset. We defined as above since it is equivalent to the case when the class balance is changed; if the weights of all positive instances (i y/- +1) change from to while the weights for negative instances (i yi = -1) remain at 1, S is calculated as above. a = 1 means that no distribution change occurs. Similarly, we define the validation weight range as W == || - 1nll2 - Q} and set Q in the same way as above. In the learning selup of this paper; we adopt the empirical risk determined basod
minimization approach as shown in equation 7 and thus the regularization parameter on the number of instances. Specifically, we first present results obtained by retraining with À that is determined by valsdaaiiia Results for other values of À will be discussed later in Section 5.4 Details of implementations are presented in Appendix D.1. and details of data setups and hyperparameters are presented in Appendix D.2
5.2 Coreset Selection for Tabular Data
In this section, we present the experimental results for tabular datasets. The experiments in this section were conducted using logistic regression model (logistic loss + L2 regularization), with Radial Basis Function (RBF) kernels (Schölkopf et. al. 2001) applied to the datasets listed in Table 1 In this experiment for tabular data, we adopt Algorithm for calculation of the upper bound in equation 18
Baselines. The baseline methods for comparing coreset selection are as follows: geometry based meth- ods: (a)Herding (Welling 2009). (b)k-Center-Greedl) (Sener & Savarese 2017), uncertainty-based methods: (c) Margin(:a method that selects instances closest to the decision boundary (margin).) Other method: (d)Random sampling. For table datasets, methods that do not rcly deep learning are chosen.
The results are shown in Figure 4 and 5 The horizontal axis represents the size of the selected dataset, where moving to the right indicates a smaller selected dataset. The vertical axis represents the weighted validation accuracy (1 - VaEr) minimiized with respect lo w' by using equation 43 The retraining settings of all methods are controlled to be the same. In Figure ] the proposed method demonstrates the higher weighted validation accuracy compared to other methods. We also found that, depending on sctups, existing methods (that do not consider DR) may end in worse performance than random sample removal. This implies that, to achieve good coreset selection under DR, the consideration of DR is essential. All results for other datasets, other À and support vector machine model are presented in Appendix D.3 and D.4
5.3 Coreset Selection for Image Data
This section discusses coreset selection for image datasets. As discussed in Section 2.2 although the proposed method is not specifically designed for deep learning, it can be applied image data by leveraging Neural Tangent Kernels (NTK) Novak et al. 2020) using the layers preceding the final layer of deep learning model as a feature extractor. In this experiment, considering future extensibility, we apply the proposed method to features extracted from images using a fcature extractor. The experimental results using NTK are included in the Appendix D.6. In this paper, we evaluate the DRCS method using the CIFARIO dataset [Krzzheskyy] 2009). CIFAR10 consists of 50,000 training instances and 10,000 test instances, divided into 10 categories. Since the proposed method designed for binary classífication, we extract subset of the CIFARIO dataset. Specifically, we create a binary classification dataset consisting of 40,000 training instances calegorized as vehicles (airplane, automobile, ship, truck) and animals (cat, deer, dog, horse). The test dataset is similarly divided into two classes, resulting in a total of 8,000 test instances. The generalization performance of the datasets selected by the coresct selection methods is evaluated using the widcly used deep neural network ResNet50 (He et al. 2016). In this experiment, all hyperparannters and experimental set settings are kept consistent before and after instance selection and model retraining. Specifically, for all experiments, we use a batch size of 128, a learning rate of 0.01, weight decay of 0.001, and train the model using the Adam optimizer for 100 epochs. In the proposed method, the selected instances are transformed using a fixed feature extractor, and classification is performed using the DRCS algorithm. The feature extractor is trained under the same settings described in the earlier experimental setup. In this experiment for image data, we adopt Algorithm 2 for calculation of the bound in equation 18
3 Hyperparameters RBF kernel are determined with heuristics; see Appendix D.2
12

---

Published in Transactions on Machine Learning Research (06/2025)
- 1.05, australian
2- 1.05, breast-cancer
a 1.05, heart
1.0
1.0
0.8
0.8
Proposcd
Propeszd
0.8
Herding
0.8
Herding
Koenter
Koenter
Margin
Margin
Random
0,4
Proposed
044
Sandim
Herding
Guarantze
Kconker
0.2
0.2
Kargin
Random
Guarartee
0.0
0.
0.4
0.6
0.0
0.2
0.4
0.6
1.0
0/3
0.2
0.4
0.0
1.5
Sample celeciion
Semole deleed ratio
Sample deletion radoo [-)
Figure 4: We compare our proposed method with several instance selection baselines with respect to the weighted validation accuracy (1 VaEr). Our method exhibits superior performance.
australian, 1.5
breast-cancer, m
1.0
heart, 227
1.0
2=1.
a=1.0
1.01
.1000
a= 1.01
0.8
1.00
0.8
08
a= 1.02
1.09
2-1.88
1.03
.=1.
1.04
0.5
1.00
2.4
.=1.
0.6
1.05
8..0
0.00
0,3
detetion
deletion rabl
Sumple usion ratio
Figure 5: We show the lower bound of the worst-case weighted validation accuracy (1 Wr This graph indicates that the lower bound of test accuracy is theoretically guaranteed under covariate shift with an unknown test distribution.
Baselines. The baseline methods for comparing coreset selection are as follows: uncertainty- based method:(a))oaas Confidence (Coleman et al. 2019), error prror/loss-based methods: (b)GraNd (Paul et al. (021)) (e)DeelFool (Ducoffe & Precioso] 2018); Gradient matching-based method: Gradrahtttch (Killamsetty
et al. 2021a). Bilevel optimization- based method: (e)Glister (Killamsetty et al. 2021b). Submodularity- based method: Dettrmnntttnnn (Tyer et al. 2021 For image datasets, comparative experiments were conducted, including methods designed for deep learning. In these experiments, the baseline methods were implemented using Guo et al. (2022)
The results are shown in Figure D The method for visualization and calculation is the same as that used in Section 5.2 In this figure, the proposed method demonstrates the higher weighted validation accuracy compared to other methods.
5.4 Discussion on Regularization Parameters
In the previous section, we presented results using a specific preset value for À based on a predefined criterion. Ideally, À should be selected to optimize model performance. Here, we discuss the relationship between the choice of X, the level of theoretical guarantces, and the resulting model performance. Here, we present the results for table data.
Figure [ iillustrates the number of removed instances that can be theoretically guaranteed for specific accuracy levels across different values of X. First, let us compare the columns in the top row of Figure 7 In the first and fourth columns, it can be observed that the performance of the proposed method deteriorates in the later stages. When strong regularization is applied, the model's performance remains poor even with the entire training set, which diminishes the effectiveness of the proposed method. On the other hand, when the regularization is small, the bound of the model parameters becomes broader, making instance selection less effective.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Tabular datasets for numerical experi- ments. All are binary classification datasets from LIBSVM dataset (Chang & Lin 2011
1.0
0.8
Name
n
n²
Accuracy
0.7-
splice
1000
517
61
Proposed
0.6
DrepFool
australian
690
307
15
Forgetting
breast-cancer
683
239
11
0.5-
heart
270
120
14
GraNd
ionosphere
351
225
0.4-
Submodular
Uncertaintty
0.3
Sample deletion ratio [-]
Figure 6: We compare proposed method with several instance selection baselines with respect to the weighted validation accuracy (1 VaEr). Our method exhibits superior performance generally.
A-1 surrra Iv1
1,05 audiii byn
1.
1-105 aueisi
10-
1,55 aumi kn
Popoad
Ramas
leding
kader
smn
10
80
sutrolan 6455
sutreian 2015
Saltiler 1
Sustralan 552
68.
1.00
0.
52
04 C8 10
06
Lepi (ep-000innoe1
30
60
Figure 7: The results represent the model performance across varying values of lambda. The top row corresponds to the weighted validation accuracy (1 VaFr), and the bottom row to the lower bound of the worst-case weighted validation accuracy (1 WrVaEr The first column shows results for = n 10-3 the second column for A Abest by vaoss-validation, the third column for 1.5 and the fourth column for À
Second, let us compare the columns the bottom row of Figure 7 In the first column, the model provides strong performance guarantees without any sample deletion or weight change (a 1.0). However, even slight deletions small weight changes cause the theoretical guarantees to break down. In contrast, in the third and fourth columns, where X sample deletions or weight changes. As the sample delction ratio increases, the method can provide wider guarantees, although this comes at the cost of reduced guaranteed performance. These observations reveal a trade-off between weight changes, sample deletion ratios, and guaranteed model performance, This trade-off is likely inffuenced by the choice of the regularization parameter. As shown in equation 19 smaller X results in larger parameter bound, while a larger À leads to tighter parameter bound. This effect directly impacts the range of accuracy lower bounds that can be guaranteed.
1.5 n, the method still provides broader guarantees even with
14

---

Published in Transactions on Machine Learning Research (06/2025)
6 Conclusions
In this paper, we proposed DRCS as robust coresct selection method when the data distribution in de- ployment enviromment is uncertain. The proposed DRCS method effectively reduces data storage and model update costs in a DR learning environment. Our technical contribution is deriving an upper bound of the worst-case weighted validation error under covariate shift, and then, we perform coreset selection aimed at minimizing this upper bound. As a result, the upper bound of the test error under future uncertain covari- ale shift. was estimated, and its theoretical guarantee was provided. Furthermore, the effectiveness of the proposed DRCS method was also demonstrated in the experiments.
We are thinking of the following possible future directions. One is the application to the regression problems or multi-class classification problems. Although the computation of the upper bound of the validation error become different, we believe that the same strategy is available. Another is the extension to dataset distillation, which allows not only removing samples but also newly creating samples that gives similar model parameters to that by the original dataset. Also, a part of existing methods (Munteanu et al. 2018 Tukan et al.
2020 2021 Tolochinksy et al., 2022 Alishahi & Phillips) 2024) imply two possible future directions: One is that, since part of coreset selection methods allows not only to select coresets but also weight the selected samples, we consider exlending our method to employ the strategy. The other is that, since these methods provide probabilistic guarantee in the performance, we consider deriving the performance guarantee of our mothod. Since current upper bound of the validation error in the proposed method is deterministic and therefore tends to be loose, probabilistic analysis may help this.
References
Meysam Alishahi and Jeff M. Phillips. No dimensional sampling coresets for classification. In Proceedings of the 4 1st International Conference on Machine Learning, 2024.
Chil-Chung Chang and Chih-Jen Lin. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27, 2011. Datasets are provided in authors' website: attpps//wwe csie. ntu. edu.
Ruidi Chen and Ioannis Ch. Paschalidis. Distributionally robust learning. arXiv Preprint, 2021. URL
https://arxiv. http://xxivvo.r//bbaaaaaa http/bbb/21000 2091//91008808993333
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv
preprint Xrxi::1906. 11829, 2019.
Frick Delage and Vinyu Ye. Distributionally robust. optimization under moment uncertainty with application data-driven problems. Operations Research, 58(3):595 612, 2010. doi: 10.1287/opre..000.01
Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimiza- tion. Journal of Machine Learning Research, 2016. URL
porperpaper... pdf pdf To appear.
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: margin based
approach. arXiv preprint orXiu: 182200002 2018.
Joel Goh and Melvyn Sim. Distributionally robust optimization and its tractable approximations. Operations Research, 38(4-1):902-99) 2010. doi: 10.1287 00.1287/opre..
10900000000 1090.0795.
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference Database and Expert Systems Applications, pp. 181-195.
Springer, 2022.
Hiroyuki Hanada, Noriaki Hashimoto, Kouichi Taji, and Ichiro Takeuchi. Generalized low-rank update: Model parameter bounds for low-rank training data modifications. Neural Computation, 35(12):1970- 2005, 2023.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Hiroyuki Hanada, Aoyama Tatsuya, Akahane Satoshi, Tomonari Tanaka, Yoshito Okura, Yu Inatsu, Noriaki Hashimoto, Shion Takeno, Taro Murayama, Hanju Tee, Shinya Kojima, and Ichiro Takeuchi. Distribu- tionally robust safe sample screening, 2024. URL
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathanicl J. Smith, Robert Kern, Matti Pi- cus, Stephan Hoyer, Marten H. Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre
Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585( 3:57 September 2020. doi: 10.1038 100038//4158--000 120-266492. URL https:
7/doi. org. 10. 00188//411800000000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Jean-Baptiste Hiriart-Urrue and Claude Lemaréchal. Convez Analysis and Minimization Algorithms II: Advanced Theory and Bundle Methods. Springer, 1993.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial infor- mation measures with applications machine learning. Tn Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Rishabh K lyer and Jeff A Bilmes. Submodular optimization with submodular cover and submodular knapsack constraints. Advances in neural information processing systems, 26, 2013.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad- match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Gen- eralization based data subset selection for eflicient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8118 2021b.
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh lyer. Retrieve: and robust sumi-supervised learning. Advances in neural information processing systems, 34:1448--144 14501, 2021c.
Coreset selection for efficient
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in noural networks. Proceedings of the National Academy of Sciences, 114(13):3521 3526, 2017. doi:
Alex Krizhevsky. The cifar-10 dataset, 2009. URL https://wwwt www. http::///wwwccc..tooooo..ee.e www.otorooo.ddee Serronto.etu//kriiicffee html
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jelfrey Pennington, and Jascha Sohl- Dickstein. Deep neural networks as gaussian processes, arXiv preprint arXiv:1711.0001 2017.
Katerina Margatina, Giorgas Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. arXiv preprint arXiv:2100. 100.044000.0764 2021.
Baharan Mirzasolciman, Jeff Bilmes, and Jure Leskovee. Corescts for data-cfficient training of machine learning models. In Proceedings of the 37th International Conference on Muchine Learning, pp. 6950 6960, 2020.
Alexander Munteamu, Chris Schwiegelshohn, Christian Sohler, and David Woodruff. On core- sets for logistic regression. ume 31, 2018. URL 035766
In Advances Neural Information Processing Systems, vol- www.wng...eurrss..mm htwp://ggg..oorpps..e/epeeeeeeee
16

---

Published in Transactions on Machine Learning Research (06/2025)
Eugene Ndiaye, Olivier Fercoq, Alexandre Gramfort, and Joseph Salmon. Gap safe screening rules for sparse multi-task and multi-class models. In Advances in Neural Information Processing Systems, pp. 2015.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha. Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2020. URL https://githu: hitgggtth...... com/ wooa//googl////ooo wwwggooooouralleleeeee
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NIPS 2017 Workshop Autodiff, 2017.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in newral information processing systems, 14220596-20001 2021.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine lcarning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Ralph Tyrell Rockafellar. Convez analysis. Princeton university press, 1970.
Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A generalized representer theorem. In International conference on computational learning theory, pp. 1166-26..62 Springer, 2001.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint Xrxi 1.0x:1008.00 00889 2017.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the likeliood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 5(35):985-1000) 2007. URL http:
//jmlr httpppppprr/////iiiaaaaaaaa hhmmm
Elad Tolochinksy, Tbrahim Jubran, and Dan Feldman. Generic coreset for scalable learning of monotonic kernels: Logistic regression, sigmoid and more. In Proceedings of the 39th International Conference on Ma- chine Learning, pp. 21520 21547, 2022. httttt:t////pcteeeeet http:/eeeediinglmm.e.eeeeeeeeeee mlr. mlt.:pprees///t///ooooolhilile
html
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoff Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference Learning Representations, 2019.
Murad Tukan, Alaa Maalouf, and Dan Feldman. Coresets for near-convex functions. In Advances in Neural Information Processing Systems, volume 33, pp. 997 1009, 2020. URL http/ttpp:///rrreeeingi.ee
Murad Tukan, Cenk Baykal, Dan Feldman, and Daniela Rus. On coresets for support vector machines.
Theoretical Computer Science, 890:171-191, 2021. URL http:///wwwwccceeeerreettttt hwt.::/wc.ceeeee....oooocsceeee.
article/pii/SOaie /ttpp/////0004975000022277
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Ev- geni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbrogt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10. 10.1038/s415924 019-0096-----22.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing HONG, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. In International Conference on Learning Representations, 2022.
Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th ansoual international conference machine learning, pp. 1121-1128, 2009.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net, Journal of the Royal Statistical Society, Series B, 67:301-322 2005.
18

---

Published in Transactions on Machine Learning Research (06/2025)
A Proofs of general lemmas
In this appendix, we present collection of lemmas that are utilized in other sections of the appendix.
Lemma A.1. (Fenchel-Moreau theorem) Let f: R [U{+ be conver function. The biconjugate f
(= (f()) coincides with f if f is convez, proper (i.e., 3p E R f(p) < +00), and
Proof. Refer to Section 12 of Rockkaellarr 1970 for details.
As specific instance of Lemma A.1 the result holds when f is finite-valued (vp E Re: f(p) < +00) and convex.
Lemma A.2. For conver function f IR + RU {+oo}, the following statements hold:
If f is proper and u-smooth, then f is -stroglyy conver.
f is proper, lower-semicontinnous. and K-strongly conver, then f is (1/16)
Proof. See Section X.4.2 of (Hiriart-Urruty & Lemaréchal 1993 for further explanation.
Corollary A.3. Assume that the regulorization function p in equation 0 is k-strongly convex, a condition necessary for applying DRCS. Then, by Lemma A.2 p" must be (1/k) -smotth. This ensures that p* connot take infinite values in this context.
Lemma A.4. Let f :. [+00+00) be a k-strongly convex function, and let p" argmin f(p) be minimizer. For any E IRd, the following inequality holds:
- 1o-p'122 V (1(pp) f(p')].
Proof. Refer to (Ndiaye al., 2015) for a detailed proof.
Lemma A.5. For any vectors a, e R and a posilive scalar O, the following holds:
min PER": c
a p a c Slaa|12
max T p= a T c 5|aa|/2
PERm -
Proof.
Using the Cauchy-Schwarz inequality, we derive:
||a|22lpp c||2 S a T (p-c) c) ||l|2lle cl2--
The first inequality becomes an equality if 3w 0: (w c)2 and the second inequality becomes an
equality if Ew' > 0: w/ ('
Since pp c|| < S, we also have:
- SSlaalla S <a T(pp c) s SS1aa
with equality when |p - c||2 = S.
The optimal p satisfying these conditions is:
c- (S/[|aa))))) for the minimum, and
(S/[|a)) 2 for the maximum.
Thus, the minimum and maximum values of TTpp (p-c) are -S[[a|lz and Si|a||2, respectively, completing the proof.
19

---

Published in Transactions on Machine Learning Research (06/2025)
B Proofs of Definition 33
B.1 Derivation of Dual Problem by Fenchel's Duality Theorem
We follow the formulation of Fenchel's duality theorem provided in Section 31 of Rockafellar 1970
Lemma B.1 (A specific form of Fenchel's duality theorem: f:9 < +00). Let f R and g : RR. + R
be conver functions, and let A Raxd be matrir. Define
pd = min (Ap) + g(p)),
(22)
PE
u
max
m g (AA (ATu)).
(23)
WER
According to Fenchel's duality theorem, the sollowing equalities and conditions hold:
f(Ap') + -f(("') 9-' o'(AA
u (Ap(Ap'), p' E 0g" (A uu).
Proof. Introducing dummy variable sb E R and Lagrange multiplier E R", the problem can be rewritten as:
min [f(Ap) 9()) PERd
max min [f(b] + ((p) - (Ap ab)l
(24)
uERe PER", $ER"
min max
[-f((b) g(p) u (Ap +)]
MER" PERd, DERn
min
max
[((-u) T - f(bb) + {(A T u) T g(p)}
wER" peR DER
min "ER"
(-u) + (( u)] -max mmx - g"(AA uA)
(25)
mER"
The optimal solutions p", ,,, and u² must satisfy the following conditions based on the optimality criteria:
Ap = vb. Auu E 0g(p'), -u" -u" =
Similarly, intuoing a dummy variable e Rdd and a Lagrange multiplier p Rdd the dual problem can be reformulated as:
max MERn
| (- - (A' ")] -A] = min PERd uERn .ER!
max
- -g"())) p (AT Φ)1
(26)
min PER uER" DER
max
(1(Ap) (-u) (- (-
+ (p (Φ)))
min] [ ( (p)) min (fAp) + g())] (:. Lemma A.1
PER
pER
The optimal solutions u" Φ², and pt for the dual problem satisfy:
AT = Φ², p² E e (x)) (A (ATu*) u²),
Ap E af(-u)).
Lemma B.2 (Dual problem of weighted regularized empirical risk minimization). Let us consider linear predictions including the kernel method. For the minimization problem
8 arg min Pu (9), where (B, Eie(n) UeW, BER
1
Σ
w.w,((i B7 TTΦ(xx)) P(B).
se(m
20

---

Published in Transactions on Machine Learning Research (06/2025)
The corresponding dual problem, obtained by applying Fenchel's duality theorem (Lemma B.1 is given by
a u,8
arg max (w))),
acR"
where (0) Eie(n] 0:00
vrw; (-a;((-aa)) p¹
Naw; (diag(e &o 60 y)
iE(n)
16 restated)
Here, let us denote :=
:= (((x1))
Φ(x2) 6( E ERixk The primal and dual solutions, f and a
satisfy the following conditions:
Pw, (Bm = Du,u(a
B Op" ((diag() & w @ 4) a
E [n] -owss de Be ((x.)).
Proof. To apply Fenchel's duality theorem, we set f, 9, and in Lemma B.1 as:
f(u): = Eie(n) waw sE[n]
wwilluu) 9(B) = p(B), A = diag(y)
The conjugate function of f computed as:
f'(u) sup w'ER"
u
T u
Σ
1
Σ
v,w
Uii v,
E
E sEin]
E
KEIM
where let us denote E = DiEin] 0:Wls. Thus, the dual objective from equation
23 becomes:
(- - g" uA
Σ
(- w;W;;
U1
E
((diag(y) Tu).
E
Rewriting 4, + h,waa or equivalently u + [ & a), we obtain the dual problem of equation
The relationships between the primal and dual problems can be expressed as:
-u' =0''0'AAp)) = - ( w w. E (ddiiag(y))
1
- n,w,Welyr, T = a Öl(y,, S (xxx))
p² E (A u²) + Be. 0g" ((diag(y) w 2 )w,ww) = Op" ((diag(v w www& 2 4) a.
C Proofs and additional discussions of Section 3
C.1 Proof of Theorem 3.4
In this appendix, we provide the complete proof of theorem 3.4
Theorem C.1. Assume that in the primal objective function PI,. is p-strongly conver with respect to
B. Let us denote the optimal primal and dual solutions for the entire training set (i.e., Us = Vi E [n]) with wniform weights (i.e., w; = Vi [m]) as
Bin. arg min PI... (3) and ain, arg max DI,,
BERK
aER"
21

---

Published in Transactions on Machine Learning Research (06/2025)
respectively. Then, upper bound of the worst-case weighted validation error is written
WrVaEr(b) S WrVaEr UB =1 -QV [((())1 (1
18 restated)
n
n'
where,
2
max DG((, >0)
((0) E (0,1)"; ((()= =1 1/ 1n 2(x{) (0(()))22
19 restated)
x weW
DG(U, () = Po, (3 (3im)1.) (aw
20 restated)
Here, the quantity DG is called the duality gap and it plays a main role of the upper bound of the validation error.
Especially, if we use L2-regularization p(3) = 1111112 DG w), the mazimization wax DG(u, w) can
be algorithmically computed.
This be proved follows.
First, derive the bound of model parameters.
Lemma C.2. Without loss of generality, we assume that the last (nold neww) instances are removed (nuew < nlld, and ao xiee vi E [n=eW]). Let Pw be u-strongly conver, and suppose Bi E ER and
ai 1n are given. Then, we can assure that the following Bu,w C IR* must satisfy 3:. E Bu,w:
Be == {
2 D w)
BE
IR
|| - Bi, 1, 12 SSRR:
(27)
Proof. See Section 4.2.1 of (Hanada et al. 2023
Here, DG takes various values depending on the possible weights w and coreset vector V.
Next, we caleulate a bound of the weighted validation error using B
Theorem C.3. The range of the weighted validation error is derived using the bound of model parameters after relruining, B C CR where Bi, E Bu,w, as follows:
(w)) surelyincorrect
(w]) surelyincorrect
+ (w")
ni' (w))
unknoww
ssurelycorrect
VaFr <
(28)
Σ wo
Σ vo
Σ w/
AE|nn)
AE(n*)
sE(n')
where
(w)) Ti surreyycorrect
Σ
will
min
T (bxx)) >
(29)
BEBn,
iE/n'l
surelyyncoorect
(wi)
Σ *E|n')
wtl
max BEB,,00
v ((x)) <0
(30)
(w') Punknown
Σ
will
min BEB
3/ (xx{) < 0,
max
319 ((x()
(31)
BEB.,..
*E(n!]
n' Psurelycorrect
+ n (w)) Msurelyincorred
(w') unkknown
1 1 w'
(32)
Here, equation 30 indicates that if the bound of model parameters after retraining is known, some of the test instances can be predicted. Moreover, equation 31 indicates that knowing only the bound of model
parameters is insufficient to determine the correctness of classification. The interpretation of equation 28 is as
follows. If (w')
unknown is assumed to be entirely misclassified, the validation reaches its maximum possible
22

---

Published in Transactions on Machine Learning Research (06/2025)
value, resulting in the rightmost side of the inequality. Conversely, if all are assumed to be correctly classified, the validation error reaches its minimum possible value, resulting in the leftmost side of the inequality. That is, the range of the worst-case test accuracy after retraining can be expressed as equation 28
Next, using Bu we describe the method for computing an upper and lower bounds of the linear score v(B3" )(xr),, which necessary for the classification equation 30 and equation 31 In general, when B is represented a hypersphere, these bounds can be explicitly obtained.
Lemma C.4. Tf Be. is given a hypersphere of radius R E R20 centered at the original model porameter Ii..... (Br.u := {B E R2 |13 - 11,,1,,12 S R}), an upper and lower bounds of the linear score y/8 xt can analytically calculated as
min Tf ((x)) = v 2(x)) - (1v(
(33)
BEB..w
y/8 ((x)) = -/ )(x)) + 11v( (1(( 12R. BEB
max
(34)
The proof is shown in Lemma A.5 In equation 33 and equation 34 these show that upper and lower bounds of the linear score v Φ ((x))) depend on the value of R. As R increases, a lower bound is more likely to take negative values, while an upper bound is more likely to take positive values.
Corollary C.5. If the bound of model parameters after retraining, B C R specifically represented as hypersphere with an L2 norm radius R maazimizing an upper bound of the worsl-case weighted volidation error, shown in equation 28. is equivalent to mazimizing R.
Finally, considering the worst-case weighted validation error, the optimization of the weights w' assigned to the validation instances is performed.
Lemma C.6. Assume that the range of validation weights wt is an Lй-norm hyperspphre, defined as W == {w' ||00 - 1wll2 S {Q} (Q > 0), and that the sum of the weights is constant. In this case, the marimization problem for the validation weights that results in an upper bound of the weighted validation error can be formulated as follows:
n' 12 (10')
surelycorrect
where (w') aaurelycrrret = ((v, =(( Tw', G min T > 0).
max w WW
(35)
w/
BEB
subject | 1 SQ, Dm==n'
(36)
RE|x2
and the marimization of this problem can be analytically computed as follows:
(w]) surelooorrect
[T - @ [K((0, (I un))
max w EW
(37)
w
ni'
KE
The proof is shown in Appendix maximizing It with respect to w allows us to maximize an upper bound of the worst-case weighted validation error. This expressed as
C.2 Here, Ç is a function of v and w. From equation 33 and equation 35
Gi(D) = (1 s(x)) - |10()))22 0}.
(38)
2 max DG(v,w)
where R = V î
(39)
weW
Therefore, an upper bound of the worst-case weighted validation error is written
W-VaFr(b) 11 [,((()) -qy (((())]18
(())) n'
1
(40)
n
23

---

Published in Transactions on Machine Learning Research (06/2025)
This provides theoretical upper-bound guarantee on the worst-case test error after retraining.
C.2
Proof of Lemma C.6
In this appendix, provide the proof of following constrained maximization problem:
(w)) n1 surelycorrect
n'
max
subject to 1|'' - 1 11112 - Q. 1 n',
(41)
w
iE|n
where
suereyoorrecct
(w²)
CTw' w
(42)
Then, this problem can be transformed and rewritten as follows:
72' n (w))
surelycorrect eWw STw" ww'
max wr EVV
(43)
n
8EIN
Thus, we prove that this minimization problem can be solved analytically.
Proof. First, we write the Lagrangian function with Lagrange multiplier μ4, v E IR as:
-Tw L(w', I(w',,,V) = =5 w/ + ju (1leo' 1. Q²) (1,,,,"'' n²).
(44)
Then, due to the property of Lagrange multiplier, the stationary points of equation 43 are obtained
ÖL(w', v,v)) S 24 (w 1,,) + vlw 0, Oao
(45)
2p (w' - 1nn) = -C -vlw'
(46)
Squaring both sides of equation 46 we get:
4p² 1wo' 1 = ||<122 2 1n v2
(47)
Multiplying both sides of equation
46 by 1 T we obtain:
2μ w1 - n²) = -1 - v1 In
0 = 1 1,,6 - vn',
1 1/5
(48)
n¹
Substituting equation
48 into equation 47 we obtain:
2
T S
4p² |/w0 1.1 = ||<112
n (1T)5)²
+
n'.
44² I/w2 1w ||<11 (1T)C)²
n'
2μ = 2 |1c11 (IT
(49)
n'
Substituting equation 49 into equation 45
we obtain:
IT. 1, (
' 1
(50)
(1 n'
n'
V ||<13
24

---

Published in Transactions on Machine Learning Research (06/2025)
Multiplying both sides of equation 50 by obtain:
Q
1
S'w' = 5
- -11112
5 1n
V |cc1 (I)))
ni
n
= 5 @V |15112 51
(51)
Finally, we obtain the minimum value of Tw'':
w':
S'w' w = 1 - QV ||51|2 (ITT)) w'EW"
min
n
C.3 Use of strongly-convex regularization functions other than L2-regularization
Let consider the use of regularization functions other than 12-regularization, assuming that the fune- tion is k-strongly convex to satisfy the conditions for applying DRCS. For L2-regularization, the term
pa Exenn sew, (. (diag(v w ② y ( ai....) in the duality gap equation 20 simplifies to a quadratic func- tion with respect w. However, for other regularization functions, even if they are k-strongly convex, the behavior differ significantly, potentially complicating the application of DRCS.
As famous example, consider the elastic net regularization Zou & Hastie (2055). With hyperparameters
À > 0 and X' > 0, the regularization function and its convex conjugate are defined as follows:
X
0(3) 2
1111112 X||3||I
(p(p) 2A jE(d
Σ
[max 1p,1 1
In this case, the term
0"
(diag(o 2 2 y) T ai..
2
max (0.
(v 2 w @ ai 2 1 - -X Exe(n) U:Wl
2X
jE[d]
requires maximization with respect to w, which is nontrivial and introduces additional complexity.
Next, discuss sufficient condition for regularization functions that allows weighted regularized empirical risk minimization to support both kernelization and DRCS.
Lemma C.7. weighted regularized empirical risk minimization as defined in 7 the regularization function p can support both DRCS and kernelization if described as:
p(
K 111111 H(11)122)
where H. R>0 + R is an increasing function and x is a positive constant.
Proof. According to the generalized representer theorem Schölkopf et al. 2001 weighted regularized em- pirical risk minimization can be kernelized if the regularization function p can be expressed in terms of a strictly increasing function 1220 + IR as p(B) = ((1)
By combining this requirement with the condition for applying DRCS, which demands that p be n-strongly convex, obtain the stated form of p.
25

---

Published in Transactions on Machine Learning Research (06/2025)
C.4 Methods for Greedy Coreset Selection
A naive greedy approach involves removing the training instance that minimizes equation 18 one al time (this approach is referred to as greedy approach 1). First, the pseudocode of greedy approach 1 for small datasots given in Algorithm
Algorithm Distributionally Robust Coreset Selection for Small Datasets
Input: Dataset D == {(a,, Vi) matrix A, vector b, constant c
1: Initialize v < {1}"
Set desired number of deletions, dell
while number of deletions is less than del do
for each where e2 1 do
Set o < and 2 +- 0
Remove i-th element from
6:
Compute the duality gap when instance is removed:
DG, max {(v' & w) A(u' & ()) + b & w) +c}
Store DG;
end for
Find the index i" corresponding to the smallest DG, value
10: Set 0
& Remove the element with the smallest maximum value
11:
Update and repeat the process
12: end while
Construct the subset D = (x,, yi (oi, E D. 10 1};
13:
Output:
Selected datasct: D
Next, the pseudocode of greedy approach 2 for large datasets is given in Algorithm 2 It performs the maximization calculation once at the beginning, and then sequentially performs instance selection.
26

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm 2 Distributionally Robust Coreset Selection for Large Datasets
Input:
Dataset D ((z,, Vi)he(()) matrix A, vector b, constant
I: Initialize < {1)
2:
Compute worst-case weight that maximize the duality gap:
w worst
arg max {(o w) 8 w)) + ( +c)
wEW
Set desired number of deletions, de while number of deletions is less than del do for each where & 1 do
6: Set o' <- and ny 4-6 0
b Remove i-th element from v
7:
Calculate DG; + (v' @ wwrsst) A(v' & w worst bb (v'e
@ w worst +
8: Store DG;
9: end for
10: Find the index i" corresponding the smallest DG, value
11: Set Vp- +
Remove the element with the smallest maximum value
12: Update and repeat the process
13:
end while
Construct the subset D {I,, Vi (a,, E D, 1};
14:
Output:
Selected dataset: D
Then, the pseudocode of greedy approach for large datasets is given in Algorithm 3
Algorithm 3 Distributionally Robust Coreset Selection for Large Datasets
Input:
Dataset D = {(x,, Wi]ie(m), matrix A, vector b, constant c 1: Initialize v + {1)m
2:
Compute worst-case weight that maximize the duality gap
w worst
arg max {(v @ AA A(v @ w) + b ( ( +c}
wEW
Set desired number of deletions, del
4: for each i E [n]] do
5: Set v + and of +- 0
b Remove i-th element from v
6: Calculate DG, < (v' @ w worst T A(v' @ w worst) + b (v' & w worst) +
7: Store DG,
end for
Identify del smallest DG,'s, and v; 0 for these i's, or is otherwise
10:
Construct the subset D {x,, Vi (x,, y1 D, u.u. 1};
Output: Selected dataset: D
Here, the duality gap DG; computed within the algorithm quadratic convex function with respect to w. As method to solve the constrained maximization of DG, in W, apply method of Lagrange multiplier. In this case, since all stationary points be computed algorithmically, maximization is achievable. For the proof, please refer to the Appendix of Hanada et al. (2024). This maximization calculation requires O (n²) time. For this reason, applying Algorithm 1 which repeatedly requires maximization calculations, Lo large datasets computationally expensive. Therefore, Algorithm 2 which reduces the number of maximization computations, is also considered. Furthermore, in large datasots, the computational cost incrcases even the instance selection process by order avoid an increase in computational cost, we adopt Algorithm for large datasets in Section 5.3
27

---

Published in Transactions on Machine Learning Research (06/2025)
D Details of Experiments
D.1 Experimental Environments and Implementation Information
We used the following computers for experiments: For experiments except for the image dataset, we run experiments on a computer with Intel Xeon Silver 4214R (2.40GHz) CPU and 64GB RAM. For experiments using the image dataset, we run experiments on a computer with Intel(R) Xeon(R) Gold 6338 (2.00GHz) CPU, NVIDIA RTX A6000 GPU and 1TB RAM.
Procedures implemented in Python, mainly with the following libraries:
NumPy Harris et al.
2020
Matrix and vector operations
CVXPY Diamond & Boyd 2016
Couvex optimizations (Iraining computations with weights)
SciPy (Virtanen al. equation 21 (by module
2020 :
Solving equations to maximize the quadratic convex function cottmiiceroot_sclllll
Defining the source neural net work (which will be converted to a kernel by NTK) for image prediction
PyTorch Paszke et al.
2017 ::
newral-tangents NNvak al.
2020 NTK
D.2 Data and Learning Setup
The criteria of selecting datasets (Table and detailed setups are as follows:
All of the datasets are downloaded from LIBSVM dataset (Chang & Lin 2011). We used training datasets only if test datasets are provided separately ("splice").
In the table, the column "d" denotes the number of features including the intercept feature.
The choice of the regularization hyperparameter X, based on the characteristics of the data, is as follows: We set n, n 10-1.5 10 and best which is decided by cross-validation.
The choice of the hyperparameter in RBF kernel is fixed as follows: we set ( d V(Z) suggested in sklearn.c svm SVC of scikit-learn (Pedrogosa et al. 2011), where V denotes the clementwise sample variance.
28

---

Published in Transactions on Machine Learning Research (06/2025)
D.3 All Experimental Results of Section 5.2 using logistic regression model
In this appendix D.3 we show all experimental results using logistic regression model(logistic loss + 1.2 regularization). First, show model performance.
Dataset: australian, À Abest
Dataset:
australian, À = 10 10-16 2-1 (6, austration
Dataset: australian, À a= 1.25 austn lan
2 1,35 austration
Heeding
furdam Gas ortes
Icerar
Kostar
32
62
Vargh
02
Perga
Gameetes
Carariss
3.8
Senole deedon rebo
SerpleMekk Mee
Samph didio rioo
Datasel: breast-cancer,
Dataset: breast-cancer, A Abrert
A = 71 10-15
Dataset: breast-cancer, 7
3-1.05 cruse
02
Pipesst
Hending
Hirsing
Harling
0.8 20
10
0.3
Dataset: heart, A Abrect
Dataset: heart, X 105
Dataset: heart,
1.5
--1.0 1.05, heart
1005 wear
3.0
1-15 heat
Korter
Popesst
Costriee
Reeder
42
Morge
Morga
18
Sanch deeke (2)
Makk Meg
Seyple (1
Dataset: ionosphere, X = Abest
Dataset: ionosphere, A = n 10
Dataset: ionosphere, X =
2.06, tomosheree scae
#.255 iorosphare scale
#=105 Innashere scale
Ses
3.4
04
lledin
Heasing
Herdin
4.2
fardsm
fander
Sample aato
Dataset: splice, X Abest
Dataset: splice, X n. 1.5
Dataset: splice, n
saace
sale
3.1.64 spice scale
Presseed
Propased modio
Pepcord Hixting
Herling
Perge Koadenn
Kondam
33
33
m3
12
R.S
Figure 8: Model performance for RBF--ernel logistic regression models, under the settings described in Section and Appendix D.2
Next, show a guarantee of model performance.
29

---

Published in Transactions on Machine Learning Research (06/2025)
Dataset: australian, À n2- 10-a 3
Dataset: australian, À = n- 10-10-11 10-1.
Dataset: australian, 7
1.
austra 2555
auatsulann 17.45
surraton, 552
Eos
6-4
%8
Dataset: craner, = 72 10-3
Dataset: breast-cancer,
A A= 72 10 1.5
Datasel: breast-cancer, X
3.0
breach-cwntee, 2- 25065
1.2
trwaciecaeer 17.56
3.-
tewser-ancee J. 506
3.33
64
20.5
0.4
12
58
To
085
F-6
Dataset: heart, X n- 10 10-3
Dataset: heart, À n 1001011
Dataset: heart,
1.0
10
heart 212
2-
2-16
Eex
02
2.
25
12
86
24
10
%8
64
26
Dataset: ionosphere, À = n 10-3
Dataset: ionosphere, À = 10-1.5
Dataset: ionosphere, X
unesphere sale, A-2.8
ineepphre Am
knoogefere 200 260
10-
3-1
Fee
206
54
60
35 11
933
02
14
86
33
30
685
EA
13
Dataset: splice, X >= n 10
Dataset: splice, X == n0 1.5
Dataset: splice, X
10
1=03
spice 29.20
2-
Tas
02
64
2.5
605
02
34
23
30
085
62
EA
26
2.
Sanil
Figure Model performance for RBF-kernel logistic regression models, under the scttings described in Sent 5 and Appendix D.2
30

---

Published in Transactions on Machine Learning Research (06/2025)
D.4 All Experimental Results of Section 5.2 using support vector machine model
In this appendix D.4 we show all experimental results using support vector machine model(hinge loss + L2 regularization). First, we show model performance.
Dataset: australian, À Abest
Dataset: australian, == n- 10 1.5
Dataset: australian, À = a- 25, ausn
3.8555 austosiion
a=1 austraiion
2.4
Convrite
Gamentos
Garartes
He:ding
Herdigg
Hocke
32
0.2
Tordem
Reedyy
Tandon
Dataset: breast-cancer, À
Dataset: breast-cancer, À Abest
10-
Dataset: breast-cancer,
05 barce
1.3
breast
2.8
0.3
60-
Garadtes
Frassted
Popetad
Kargin
Marge
Porga
0.8
3.0
64
26
64
13
0.3
33
64
86
66
10
60
04
24
66
Dataset: heart, X Abest
Dataset: heart, X 1051.
Dataset: heart,
2:25, heart
#=100 neart
1-10 heat
Carerto
2.44
Hectro
Hurdeg
32
0,2
fander
Bandor
Tandor
Sampa
Dataset: ionosphere, À Abeet 1005,
Dataset: ionosphere, n 10 1..
Dataset: ionosphere, X n
6.3
60-
Sueratee
Guarnnee
Popesed
Resmed
0.2
Couter
Sandor
Nandom
Parcem
0.8
0.
60
Sanch ml
Dataset: splice, X Abest
Dataset: splice, X = n.
Dataset: splice, X
1.03, spice scale
05 spice
a.100 sp acale
Propeaed
3.4
Eas
Parge
Comurtes
0.2
Sample
Sample delitinn
Figure 10: Model performance for RBF-kernel SVMs, under the settings described in Section 5 and Appendix
D.2
Next, show guarantee of model performance.
31

---

Published in Transactions on Machine Learning Research (06/2025)
Dataset: australian, À n2- 10-a
Dataset: australian, À = n- 10010-111 10-16
Dataset: australian, 7
1.
australar, 0505
auatrulann 17.45
susratonn 552
68.
Eos
6-4
0.2
093
%
Dataset: breast-cancer,
Dataset: breast-cancer,
72 - 10-3
A == 72 10 1.5
Datasel: breast-cancer, X
3.0
breach-caneer, du 25065
1.2
mrwarvaater 2u 17.26
3.0
tewsercancee J. 506
2.
2-33
077
18
25
21
0,5
62
F-6.
Dataset: heart, X n2 10-3
Dataset: heart, X n 10 1051.
Dataset: heart,
10
22215
1-1
1.5
2-
10
heart 212
2-16
16
12
48
62
8%
24
08
64
Dataset: ionosphere, A= == n 10-3
Dataset: ionosphere, À = 10-1.
Dataset: ionosphere, X
unisphere solle A-228
ineepphre Am
knoogefere 20f 260
10-
3-1
54
1.6
35
12
933
02
12
10
685
02
Dataset: splice, X = n1 10-3
Dataset: splice, X = n0 1.5
Dataset: splice, X
1.0
stice 1-032
splie, 29
spice.
2-
Tax
6A.
02
C44
2.5
3.
605
2.
23
30
08.5
62
64
26
25
Sumpa
Figure 11: Model performance for RBF-kernel SVMs, under the scttings described in Scction and Appendix D.2
32

---

Published in Transactions on Machine Learning Research (06/2025)
D.5 The effectiveness of the proposed method in different models
Here, we compare the results between logistic regression in Appendix Regarding model performance, there is no significant difference between the two models, and the trends when regularization is large or small are generally consistent for both (Figure and 10p. On the other hand, in terms of theoretical evaluation, there are significant differences between the two models (Figure and 11. The major difference between SVM and logistic regression lies in whether the model is instance-sparse. In SVM, training instances where di..1..f Therefore, more effective instance selection is possible, contrast, logistic regression is not sparse model, and as result, the theoretical lower bound of the worst- case weighted validation accuracy is expected to be lower than that of SVM.
D.3 and SVM in Appendix D.4
0 do not affect the duality gap during instance selection.
the duality gap can be suppressed further. In
D.6 Experimental Results Using NTK in Section 5.3
We also experimented the DRCS method with NTK for an image dataset. We composed 5,000-sample binary classification dataset from "CIFAR-10" dataset by choosing from classes "airplane" and 'antomobile". A total of 1000 instances were sampled, and experiments were conducted using Algorithm [ It was demonstrated that the proposed DRCS method could be approximately applicd to doep learning by using NTK.
Dalaset: CIFAR1O, A n 10-3 -- 2.05, diario
Dalaset: CIFAR1O, X x= n 10 1.5
Datasel: CIFARIO, À = 3- 1.05 clarto
2-1.05 diaris
Proposed
Pmprove
Reedem
Pandon
Sas
2.4
003
42
02
0.2
Gurories
Sample Sampleddetke ddaie 90b B
Samph diliton rooo c
Dataset: CIFARIO, A = TL. 10 3
Dalaset: CIFARIO, À x= TL. 10 10-1.5 -15
Datasel: CIFARIO, X = T1
1050
64
3.55
62
0.2
62-
0//
00/
68
Sande (110010011111
Figure 12: Model performance for NTK, under the settings described in Section and Appendix D.2
33


Reviews:
[{"review_id": "KjSzI6xFrR", "reviewer": "Reviewer_EyTh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper proposes to address a key problem in distribution shift robustness. The theoretical result looks rigorous. Fig. 1-2 are helpful for understanding. \n\nWeaknesses: The strong convexity assumption is an obvious weakness. The applicable task is only binary classification. The above two weaknesses limit the scope of this paper. Calculating the upper bound of the worst-case weighted validation error can be computationally expensive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Distributionally Robust Coreset Selection (DRCS) to address the problem of selecting a small subset of training data (a coreset) when there is a distribution shift between the deployment environment and the training one (i.e., covariate shift). As the data distribution during deployment is often unknown, selecting an effective subset of training data that performs reasonably well across all possible test distributions is a challenge. The paper derives an upper bound for the worst-case test error and designs DRCS that suppresses the estimate of this upper bound for the worst-case test error. The experiment is run on both tabular and image data and shows the effectiveness of DRCS.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could the authors discuss the connection between coreset selection and dataset distillation? \n\nCould the authors explain how to extend the work to non-convex functions?\n\nPlease also give the computational cost of the proposed method along with baselines'.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r2jYPTiVl9", "reviewer": "Reviewer_Mf39", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper presents a novel and rigorous approach to coreset selection under covariate shift, addressing a challenge in distributionally robust learning. The paper derives an upper bound for the worst-case test error, providing a rigorous guarantee on the robustness of coreset selection under uncertain test distributions. It introduces a trilevel optimization framework to formally model the worst-case risk minimization problem, accounting for both coreset selection and distributional shifts. The use of duality theory and strong convexity assumptions enables an efficient computation of the worst-case bound.  The paper identifies the computational intractability of the exact trilevel optimization problem and proposes efficient greedy heuristics to approximate optimal coreset selection. To efficiently compute the worst-case test error bound, the upper bound of the weighted validation error is maximized with respect to the weight vector $\\omega’$ for the validation dataset. This maximization is performed through eigenvalue solving under $L_2$ regularization, allowing to derive an upper bound for the worst-case weighted validation error.  The authors conduct extensive experiments on tabular and image datasets, demonstrating that DRCS outperforms existing coreset selection methods in terms of robustness to distributional shifts. \n\nWeaknesses:\n1. One of the primary limitations of the paper is the absence of a formal theoretical analysis quantifying how tight or loose the proposed upper bound on the worst-case test error is. While the bound ensures robustness, it remains unclear how closely it approximates the true worst-case error in practical settings. An empirical study comparing the upper bound with actual worst-case errors would further validate the robustness and practical applicability of the proposed methods. This could involve evaluating the computed worst-case validation error bound alongside the empirically observed worst-case validation error, providing insight into how well the bound reflects real performance.\n\n2. A recent paper examined coresets for classification in a regularized setting:\n\nMeysam Alishahi and Jeff M. Phillips. 2024. \"No Dimensional Sampling Coresets for Classification.\" In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 43, pp. 1008–1049.\n\nGiven the similarity in setting, the authors may find it useful to refer to this paper and explore potential connections between its results and their own findings. Furthermore, a promising direction for future study would be to investigate whether it is possible to establish a dimension-free bound for the worst-case test error.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Coreset selection is an approach used in machine learning to select a small representative subset of a training dataset while preserving model performance. In coreset selection, it is generally assumed that the training and test data distributions are the same i.e., they follow the same underlying probability distribution. The paper ``Distributionally Robust Coreset Selection under Covariate Shift'' introduces a novel method (called DRCS) to address the challenge of selecting a coreset of training data under a scenario where the input distribution at deployment differs from training. The key contribution of DRCS is a theoretical upper bound for the worst-case test error, which guides the selection of a coreset that maintains model performance under uncertain distribution shifts. In a conventional covariate shift setting, the test input distribution is assumed to be known, allowing weights $\\omega$ to be determined based on the density ratio between test and training distributions. In the distributionally robust setting, the density ratio (and thus $ \\omega $) is unknown but is assumed to lie within a defined uncertainty set:\n$$W :=  \\{ w \\in \\mathbb{R}^n \\mid \\| w - \\mathbf{1}_n \\|_2 \\leq S \\}$$\nwhere $S$ controls the degree of allowed deviation from uniform weights. The final DRCS optimization problem is designed as a trilevel optimization problem with the third level as a combinatorial optimization which becomes infeasible for large cases. \nTo make the selection process computationally efficient, the authors propose (1) an upper bound for the worst-case weighted validation error and (2)  three greedy heuristics that approximate the optimal coreset selection. Through experiments on tabular and image datasets, DRCS demonstrates superior performance compared to existing coreset selection methods, proving its robustness and practicality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be helpful for the authors to either address the weaknesses outlined above under \"Weaknesses\" or provide an explanation if addressing them is particularly challenging.", "broader_impact_concerns": "No Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WlLBwwaB4V", "reviewer": "Reviewer_g4Vy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In what follows, the strengths of the paper are given:\n* The first coreset that handles distribution shift, via suggesting DRCS (Distributionally Robust Coreset Selection).\n* The idea leverages the test error into the coreset generation process which is a naturally greedy algorithm that would remove the least impactful point iteratively until the number of remaining points is the desired coreset size.\n\nIn what follows, the weaknesses of the paper are given:\n* The paper is somewhat hard to follow as some terms are ill-defined (see the following section).\n* The coreset itself lacks any theoretical guarantees concerning the training data (unlike $\\varepsilon$-coresets which are problem-dependant and generated to approximate (multiplicatively) the training cost function of the entire data).\n* Considering the papers mentioned in the paper, it is indeed true that they are mostly heuristic, however, for the problems chosen to experiment with, the authors did not choose to test against $\\varepsilon$-coresets papers where the coreset is usually a weighted subset. E.g., see [1-4].\n* The theoretical running time of either of Algorithms 1,2,3 is not given explicitly.\n* The experiments are somewhat weak. Specifically:\n  1) In most cases, especially when the coreset size is small (< 20% of the data), uniform sampling performs better than the presented coreset. While it is evident from papers in the fields of $\\varepsilon$-coresets, that uniform sampling and the coreset would behave almost the same for large coreset sizes, however, in the small regime after some very small coresets sizes, the proposed coreset should be much better than uniform sampling, showcasing the efficacy and need for alter from random sampling. \n2) The experiments were carried out with very small sizes. This defeats the purpose of using a coreset. All data sizes were too small. As for CIFAR10, the data was then post-processed to contain at max 5000 points, which again is too small. One can split the data into mainly three groups where one would be vehicles, the second would be animals, and the remaining can be discarded for example, and run your experiment then on this. The whole idea of a coreset is to handle large-scale datasets, to provide faster training and/or higher quality trained model (due to the data being of higher quality, i.e., lacking noise and incorrectly labeled points for example). \n\n----------------------------------------\n[1] Munteanu, A., Schwiegelshohn, C., Sohler, C., & Woodruff, D. (2018). On coresets for logistic regression. Advances in Neural Information Processing Systems, 31.\n   \n[2] Tukan, M., Maalouf, A., & Feldman, D. (2020). Coresets for near-convex functions. Advances in Neural Information Processing Systems, 33, 997-1009.\n   \n[3] Tukan, M., Baykal, C., Feldman, D. and Rus, D., 2021. On coresets for support vector machines. Theoretical Computer Science, 890, pp.171-191. \n   \n[4] Tolochinksy, Elad, Ibrahim Jubran, and Dan Feldman. \"Generic coreset for scalable learning of monotonic kernels: Logistic regression, sigmoid and more.\" International Conference on Machine Learning. PMLR, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper suggests the first coreset that handles distribution shift, specifically covariate shift tailored for machine learning applications that use strongly convex functions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Equation 11, should $\\mathcal{W}$ be $\\mathcal{W}^\\prime$?\n2) What is $\\ell^\\ast$ is at Equation 16?\n3) What is $\\rho^\\ast$ is at Equation 16? Is it related to the conjugate convex of $\\rho$?\n4) At the second item list of the sketch of proof of Theorem 3.3 (section 3.1.2), what is assumption 1?\n5) what is the motivation behind setting $S:= \\sqrt{n^\\+}\\left| a-1 \\right|$?\n6) In Figure 5, setting a to 1 means that $S=0$?\n7) Regarding section 5.3, I didn't quite follow the experiment. Did you train the model on the entire data, then take the features from ResNet50, choose a subset as the coreset, and then retrain the entire model on the subset? Why not simply use the pretrained model of ResNet50, and then train on the subset similar to either [5] or [6] for example?\n8) Page 12, did you mean to write \"calculation\" instead of \"calcuration\"?\n9) Page 17, In Lemma A.1., did you mean to write $f^\\ast$ instead of $f^{\\ast\\ast}$?\n10) The paper uses $\\otimes$, which I assume is the element-wise multiplication. If that's the case, please make sure to explicitly state this.\n11) Page 19, what should be instead of the \"equation ??\"?\n12) In the proof of Lemma C.6, is $\\sum_{i \\in [n^\\prime]} w_i^\\prime = n^\\prime$? Please connect between equation 41 and equation 32.\n13) Page 26, what does $d^\\prime$ represent/hold?\n14) In most of the experiments, the random sample does almost as good as the given coreset, and in some cases, even better. Do you have an explanation of why this would happen? \n15) I suggest the authors use problem-dependent coresets to see how the given coreset is better than a coreset that would aim to approximate the training cost function.\n16) I understand that this is the first coreset, and the running time is around $\\Omega(n^3)$ which is quite high, however, I suggest the authors provide a graph of time of construction + model fitting for every coreset size for each experiment in the paper.\n17) Is there a way to incorporate weights in your approach, i.e., having a weighted coreset? \n\n-----------------------------------\n[5] Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G. and Iyer, R., 2021, May. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 9, pp. 8110-8118).\n\n[6] Tukan, M., Zhou, S., Maalouf, A., Rus, D., Braverman, V. and Feldman, D., 2023, July. Provable data subset selection for efficient neural networks training. In International Conference on Machine Learning (pp. 34533-34555). PMLR.", "broader_impact_concerns": "I have no concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is on topic for the audience, formulates an interesting problem space, and provides useful results in addressing the problem.  I reviewers agree it fits the requirements for TMLR after the responses.  There are no outstanding issues, so can be accepted as is.
Decision: Accept (poster)