Data Pruning by Information Maximization
Authors: Haoru Tan
Sitong Wu
Wei Huang
Shizhen Zhao
XIAOJUAN QI
Venue: ICLR 2025 Poster (2025)
Keywords: Data Pruning
Deep Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: In this paper, we present InfoMax, a novel data pruning method, also known as coreset selection, designed to maximize the information content of selected samples while minimizing redundancy. By doing so, InfoMax enhances the overall informativeness of the coreset. The information of individual samples is measured by importance scores, which capture their influence or difficulty in model learning. To quantify redundancy, we use pairwise sample similarities, based on the premise that similar samples contribute similarly to the learning process.
We formalize the coreset selection problem as a discrete quadratic programming (DQP) task, with the objective of maximizing the total information content, represented as the sum of individual sample contributions minus the redundancies introduced by similar samples within the coreset.
To ensure practical scalability, we introduce an efficient gradient-based solver, complemented by sparsification techniques applied to the similarity matrix and dataset partitioning strategies. 
This enables InfoMax to seamlessly scale to datasets with millions of samples. 
Extensive experiments demonstrate the superior performance of InfoMax in various data pruning tasks, including image classification, vision-language pre-training, and instruction tuning for large language models.

Published as a conference paper at ICLR 2025
DATA PRUNING BY INFORMATION MAXIMIZATION
Haoru Tan *1 Sitong Wu 2, Wei Huang', Shizhen Zhao', Xiaojuan Qi"
I The University of Hong Kong
The Chinese University of Hong Kong
ABSTRACT
In this paper, we present InfoMax, a novel data pruning method, also known coreset selection, designed to maximize the information content of selected samples while minimizing redundancy. By doing so, InfoMax enhances the overall informativeness of the coreset. The information of individual samples is measured by importance scores, which capture their influence or difficulty in model learning. To quantify redundancy, we use pairwise sample similarities, based on the premise that similar samples contribute similarly to the learning process. We formalize the coreset selection problem as discrete quadratic programming (DQP) task, with the objective of maximizing the total information content, represented as the sum of individual sample contributions minus the redundancies introduced by similar samples within the coreset. To ensure practical scalability, introduce efficient gradient-base basedd solver, complemented by sparsification techniques applied to the similarity matrix and dataset partitioning strategies. This enables InfoMax to seamlessly scale to datasets with millions of samples. Extensive experiments demonstrate the superior performance of InfoMax in various data pruning tasks, including image classification, vision-language pre-training, and instruction tuning for large language models.
INTRODUCTION
InfoMax (Ours)
D2 Prening
CCS
ImegeNin1KK
Large-scale datasets have been pivotal in the re- cent breakthroughs in deep learning (Brown, 2020; Kirillov et al., 2023; Radford et al., 2021; Rom- bach et al., 2022). However, the growing size of training data substantially increases both training costs and storage demands. Moreover, significant redundancies within these datasets highlight the importance of data- aata-pruning methods that remove redundant samples and identify a compact yet in- coreset, to enable
(Zan-shall)
FlidetoK
CLR ST18/32
ImngeNa-IK
InassNa-1K
CIFAR-100
(Iiza--rob)
formative subset, known more efficient model training and data storage.
MMLLE
Research in this field can be broadly divided into two categories: score-based methods (Sorscher al., 2022; Radford et al., 2021) and geometry- based methods (Sener & Savarese, 2017; Ash et al., 2020). Score-based methods focus on developing metrics to evaluate a sample's informativeness, such as prediction uncertainty (Har-Peled et al., 2007), loss value (Cody Coleman et al., 2019), influence score (Xia et al., 2024; Tan et al., 2023). However, as shown in Figure 2(a), these methods often select samples densely concentrated in re- gions with the highest scores, leading to redundancies and failing to consider simpler samples with lower scores, which results in biased selections. Recent work (Sorscher et al., 2022) highlights that even simple samples are important for improving model generalization. On the other hand,
TYDIAQA
Figure 1: Summarization of InfoMax's perfor- mance in vision-language pre-training and image classification (both at 10% selection ratio), and instruction fine-tuning for large language mod- els (5% selection ratio). The results show that InfoMax has demonstrated substantial progress in various scenarios, see Section 4 for details.
$ Equal Contribution. Corresponding Author

---

and Published as a conference paper at ICLR 2025
FFosuur re valuee
Festure vaku
Festure evraus
Fealure
vallurevauuu
Fantraaaali
Fosturevalus
Feature valuó
Firquencyy
Fregg sergaerroy
Frequeccy
Frequency
Score
Score
Score
Score
(4) Soore-based
(b) Diversity-bnsed
(c) D--Prrnng
(d) InfoMax
The coreset distributions from InfoMax, and, the score-based method (output margin (Har-Peled et al., 2007), and diversity- method (k-Median clustering (Feldman & Langberg, 2011; Har-Peled Mazumdar, 2004)), hybrid method (D*-Pruning (Maharana et al., 2023)). In the upper figures, the background illustrates the distribution density of the data in the space after PCA dimensionality reduction. Brighter colors indicate higher density of samples. Simultaneously, we present the locations of the samples selected by different methods using scatters. The figures below show the distribution of the scores of the corresponding coreset. This set of experiments conducted CIFAR-100 (Krizhevsky, 2009). The coreset selected by our InfoMax is capable of covering both high-density and low-density regions in terms of spatial uniformity, and also low-information data in terms of score distribution.
Figure 2:
geometry- -based methods aim to select diverse subset of the data, reducing redundancies between samples (Sener & Savarese, 2017). As illustrated in Figure 2(b), while these methods prioritize diversity, they often overlook informative samples with high-importance scores, leaving large number of low-scoring samples in the selection.
Recently, hybrid approaches have emerged that combine importance scores with diversity to design more effective algorithms (Ash et al., 2020; Maharana et al., 2023; Zheng et al., 2022; Yang et al., 2024). One of the most notable examples is D²-Pruning (Maharana al., 2023), which models dataset as a graph. In this framework, node scores represent a sample's 's informativeness, such as difficulty or importance, while edges capture the similarities between samples. The data pruning process is formulated as an iterative node selection problem, where at each step, nodes with the highest scores are selected, and the scores of neighboring nodes are reduced to account for redundancy. However, due its greedy selection process, the algorithm is prone to getting stuck in suboptimal solutions, making it challenging to maintain a proper balance between importance and diversity. For example, as shown in Figure 2(c), many samples remain concentrated in high-density areas with low scores, leaving significant portions of the space uncovered.
In this paper, we introduce InfoMax, approach for data pruning and coreset selection. Our core insight is to find subset of samples that maximizes overall information by simultaneously considering each sample's information contribution and the information overlap among them. First, a sample's informativeness can be explained by its importance or difficulty. For instance, a sample with intricate structures, complex backgrounds, and occlusions provides more information than one with simple patterns see Figure 3. Therefore, we explore acore-based metrics that evaluate a sample's difficulty or importance, providing an information assessment. Second, the information overlap among samples is evaluated in a pairwise manner and quan- tified by their pairwise similarities. Samples with greater similarity will have higher degree of information overlap. Using these considerations, we formulate coreset selection discrete quadratic programming (DQP) problem with equality constraints that
and effective
Low-informative
High-informative
Figure Cases samples with low informativeness (left) and high infor- mativeness (right).
2

---

Published as a conference paper at ICLR 2025
specify the desired number of selected samples, see Eq. (2). The objective function represents the overall information as the sum of each sample's individual information, reduced by redundancies introduced by the presence of similar samples within the coreset. Furthermore, propose an efficient and robust gradient-based solver to address scalability. This solver is enhanced by sparsification techniques applied the similarity matrix and dataset partitioning strategies, enabling practical and scalable computation. For instance, our method processes 12 million data points in just 37 minutes (see Section B.2 for details). By solving this problem, InfoMax identifies a subset of samples that maximizes overall information, thereby reducing the likelihood of suboptimal solutions often seen with greedy- based methods, see the discussion in Section 3.3. As shown in Figure 2(b), the coreset generated by our approach strikes a good balance between diversity (i.e., well-distributed across the entire space) and informativeness (i.e., containing a reasonable number of important samples). The superior performance across multiple tasks, including image classification, vision-language pretraining, and instruction fine-tuning for large language models, see Section 4, further supports the effectiieness of our new formulation. We have summarized the overall pipeline in Figure 4 and Algorithm 1. In summary, our contributions are:
We propose InfoMax, new coreset algorithm designed to maximize overall information by accounting for each sample's 's individual contribution while reducing information overlap, with simultancous focus on maintaining diversity and importance.
We propose efficient gradient- based solver enhanced by sparsification techniques and dataset partitioning strategies to make InfoMax scale large-scale datasets.
Extensive experiments show that InfoMax exhibits the best performance and consistently out- performs the state-of-the-ard schemes in series of tasks, including image classífication, and vision-language pre-training (Radford et al., 2021), large language model supervised fine-tuning (Xia et al., 2024) experiments. Notably, brings about a significant improvement of approximately 5.5% compared to the previous best methods 5% selection rate in instruction fine-tuning experiments. Additionally, it shows around 2% performance enhancements on classification tasks a 10% selection rate, see Figure 1.
2 PRELIMINARIES
In this section, we first present the problem definition for data pruning, followed by a discussion of existing methods, including based, svvessty-bbsedd and hybrid approaches.
2.1 PROBLEM DEFINITION
Before diving into the literature review of existing methods, we first define the problem of data pruning, also known as coreset selection (Sener & Savarese, 2017; Mirzasoleiman et al., 2020; Killamsetty et al., 2021). Let
(z:) represent the training set, drawn independently and identically distributed (i.i.d.) from an underlying data distribution. The goal of dataset pruning is to identify the most informative subset of the training data that minimizes information loss while maximizing model performance. Formally, the problem can be stated as:
S arg max I(S): ScD.
(1)
where p the budget of the target coreset and - represents the cardinality of set. I(S) measures the set-level information of the candidate subset S. There are multiple choices (Sorscher et al., 2022; Wei et al., 2015; Kaushal et al., 2021) for the instantiation for the set information I(S). For example, given the loss function ((-) and the network parameters w, previous works Yang al. (2023); Zheng et al. (2022) often use the overall test loss reduction caused by training the model
on S as the information measure I(S) Eg-P [(2, w²) 1(=, wx) Eq. (1) would be identical as
S* arg min Ez-r ((z, wg) where w is the initial parameter and w indicates the optimal S=D,
network parameter learned on S. The key distinction between various data pruming methods lies in how they define T(S), which is detailed below.
3

---

Published as a conference paper at ICLR 2025
Step 1
Step 2
Step
Step 4
Caloulate
cumple (S)) eear sample festare similaitty
Information Madimization Dijective Construction
InfoMae
Sa. (4)
1/00- K
Pelee die lerslive Ietolla velva L
Dataset [t....R]]
Neurl
Featrre
inae-corplla nakkninncy
Correct
Newook
See
Figure 4: The overall pipeline of our InfoMax. In the first two steps, we use a network to extract features calculate the similarity matrix and the intra-sample information terms. Then, in step 3, construct the quadratic optimization problem for infoMax, see Eq. (2) for details. Finally, we perform the iterative InfoMax solver defined by Eq. (3) to obtain the final InfoMax coreset.
2.2 ACORE-BASED METHOD
The score- score-based method (Paul et al.. 2018; Tan et al., 2023; Toneva et al., 2018) often selects samples solely based on the score values. They generally model I(S) as: I(S) = [(2,] ) + [(22) +... ((2p). where S (=1.-..,Zp) =p} C D, with each representing a data sample, and T(=) denoting the information associated with that sample. The primary task then becomes identifying suitable score metric to evaluate a sample's individual contribution, (z). of each sample. Various scores have been proposed to quantify (z)), such as model uncertainty (Har-Peled et al., 2007), loss value (Cody
Coleman et al., 2019), and influence score (Tan et al., 2023; Pruthi et al., 2020).
The assumption underlying the score-based formulation is that there is no information overlap between different samples, allowing the individual contributions of samples to be summed to represent the overall contribution. However, this approach fails to account for overlap in the information provided by different samples. For instance, if two identical samples, 2; and 2j. both receive high scores, the information gained from adding 2j after selecting 2; would be negligible. As result, this method cannot ensure that the selected samples offer broad coverage the data space, leading to loss of diversity and suboptimal solutions. This limitation is illustrated in Figure 2(a), they select samples densely concentrated in regions with the highest scores, leading to redundancies and failing to consider simpler samples with lower scores, which results in biased selections.
2.3 GEOMETRY-RASED METHOD AND HYBRID METHOD
Geometry-base -base Method In contrast to score-based methods, gcometry-based methods de- sign Z(S) to consider maximizing the diversity and minimizing the information overlap among samples. For many works (Lloyd, 1982; Tan et al., 2006; Coates & Ng, 2012; Har-Peled & Mazumdar, 2004; Feldman & Langberg, 201 1), the set-level information I(S) is regarded as:
I(S) < (=,,,)) - (2;;²z), where [(2;, =j) measures the similarity of two samples, indicating information overlap (also the mutual information). To solve the above problem, Sener & Savarese (2017) applied greedy k-center to choose the coreset with good data coverage.
This formulation assumes that all individual samples carry equal amounts of information, disregarding the varying significance of each sample. Consequently, tends to overlook critical samples while retaining large number of non non-informative ones. As illustrated in Figure 2(b), while these methods prioritize diversity, they often overlook informative samples with high-importance scores, leaving a large number of low-scoring samples in the selection.
Hybrid Method More recently, hybrid methods (Zheng et al., 2022; Ash et al., 2020; Maharana et al., 2023) have been developed that account for both the individual importance and diversity of samples simultaneously. For instance, CCS (Zheng et al., 2022) sets different sampling ratios for samples with different scores to enhance data coverage and balance both easy and hard samples. D2-PPuning (Maharana et al., 2023) proposes to select data on the graph. One of its core steps is the Inverse Message Passing operation. Specifically, it selects the sample with the highest score among all unselected candidates and subsequently reduces the scores of the neighborhood. However, the above approaches are primarily based on heuristic designs and are often solved using greedy algorithms. These methods are prone to local optima, as the algorithm lacks a holistic view of the problem at each step, resulting in suboptimal outcomes. In Figure 2, the selected samples demonstrate
4

---

Published as a conference paper at ICLR 2025
inadequate coverage of the entire space, with many concentrated in low-importance regions, hence results in the suboptimal performance in practice, see Figure for details.
In this paper, we propose unified formulation with scalable solver for coreset selection, aimed at maximizing the information of the selected subset by accounting for both the individual contributions of samples and their redundancies.
METHOD
3.1 INFOMAX: FORMULATIONS AND SOLUTIONS
The objective of InfoMax is to identify a subset of data samples that maximize intra-sample infor- mation while minimizing inter-sample redundancies caused by similar samples, thereby achieving an optimal balance between diversity and importance. For sample 2, we represent its information content as [==). For two samples and 8, their information overlap or redundancy is denoted as 8),,8). The total information is measured by summing the individual information of the samples, with deductions for inter-sample redundancy, as outlined below. We will discuss in Section 3.3 that under some mild settings, this optimization problem in Eq. (2) is equivalent to solving the original information maximization problem for data pruning as defined in Eq. (1).
Problem Formulation for InfoMax Here, we consider inter-sample redundancy in pairwise manner using (=,,s) and the matrix containing all pairwise sample similarities is denoted as K....
Then, given the dataset D {21, 22, ... 2N3, we introduce a variable X, E {0, to represent
whether a sample is selected (X; = 1) or pruned (X; 0), the overall information is formulated as
a quadratic function of variable X as:
max Ex.((=- -a Xc [ ZED
Σ K,X,X I,SED
s.t.
Σx= 2ED
(2)
where p (1- 8) the size budget of the selected set. In the formulation, the first-order term (2) measures the intra-sample information of the sample 2, which can be instantiated using any existing sample-wing wise scores, e.g., EL2N (Paul et al., 2018) or SSP (Sorscher al., 2022). The quadratic term K2, measures the inter-sample redundancy between the sample and 8.
Objective Construction For the intra-sample information I RVV and the pairwise feature similar- ity K e N we introduce two calculation modes by following Maharana et al. (2023), namely the supervised mode and the unsupervised mode.
The supervised mode. We initially train a surrogate model on the dataset. Subsequently, we employ to calculate sample-wise scores, such the loss value (Cody Coleman et al., 2019) and gradient norm (Paul et al., 2018). Additionally, we use features before the classification layer as features F xd where d is the feature dimension. We utilize the inner product the pairwise feature similarity, that is, K FTF. However, the supervised mode is somewhat unfriendly as training an additional model incurs a non-negligible expense, especially on large-scale datasets.
The unsupervised mode. We extract features using existing open-source models such as DINO (Oquab al., 2023). Additionally, we use the inner product the pairwise feature similarity. For the intra-sample information, we employ the SSP score (Sorscher et al., 2022). Specifically, it entails first conducting clustering in the feature space. The distance between a sample and the corresponding cluster center constitutes the SSP score.
Analysis If we only preserve the first-order term by setting a as a very small value that near zero, InfoMax will degenerate into vanilla score-based scheme, that is, those samples with the highest scores will be selected. If we discard the first-order term by setting a as a very huge value, it would degenerate into the problem of finding coreset S D to minimize >i, K is to find subset with the minimum similarity within the set. This is variant k- k-Median problem (Lloyd, 1982; Tan et al., 2006; Peled & Mazumdar, 2004), that minimizing the (Sst(S, Eves ExeD/s / d( 8) if we set the distance measurement d( 8) K
In other words, it
of the classic
5

---

Published as a conference paper at ICLR 2025
3.2 SALABLE INFOMAX SOLVER
Solving the quadratic problem defined in Eq .2 directly can be extremely computationally burdensome and may even prove intractable since the budget size p is generally on the order of tens of thousands or even millions (Krähenbühl & Koltun, 2011; Larsson et al., 2018). Here, we propose a continuous relaxation of the problem, enabling the use of gradient-based solver for more efficient optimization. Firstly, we conduct convex relaxation on the feasible domain of the original optimization problem by relaxing the binary constraint X {0. 1JM to a continuous version X E [o, 1]N. Then, we derive solver based on the proximal gradient method (Baqué et al., 2016; Tan et al., 2021). This solver decomposes the original complex and non-convex continuous problem into a series of simple and convex sub-problems, see Appendix D for details. Each sub-problem has analytic solution that serves as the update rule of our InfoMax solver as follows:
X Softmax (p. I- I KX.)
(3)
where I E RN is the vectorized [2) that I, [22, and K E RNN
N is the similarity matrix. Softmax the operation to map the real-valued vector X to non-negative vector with the summation of 1, specifically, Softmax (X) = cxp(X)/ >; exp(X);. After 7 iterations, we get the final solution x", of which each item represents the probability that the corresponding sample should be selected. Finally, we take the coreset corresponding to the top-p largest values.
We have summarized InfoMax in Algorithm 1. First, the algorithm requires the input of the intra- sample information vector I RN and the pairwise feature similarity K RN*N, Then, will iteratively execute the iterative solver defined in Eq. (3) for T iterations. Next, we present additional design enhancements to further improve efficiency.
Efficiency Enhancement Technique Naively applying the InfoMax solver in Eq. (3) on the original dataset leads to a quadratic complexity of O(N²), where N is the number of samples to be processed. This complexity is rather high when dealing with large-scale datasets. Here, we introduce two practical techniques to boost the efficiency of InfoMax.
Dataset partition: Before executing the InfoMax algorithm, we divide the original dataset into d smaller random subsets and then conduct pruning on each subgroup independently, With this scheme, the complexity of the algorithm on each subset significantly reduced to ONN2/d/ /d²).. At the same time, pruning for each subset can be carried out simultaneously on multiple computing devices, further reducing the time consumption.
Sparsification:
Since the information overlap between distant samples is minimal, we further improve computational efficiency by sparsifying the similarity matrix K, retaining only the top & values as non-zero and setting the rest to zero. Specifically, we only take into account the similarity between each sample and its k nearest neighbors (for instance, & = 5). As a result, K has only Nk non-zero elements. With the sparsification technique, the complexity of the algorithm on each subset is significantly reduced to (N
Note that the partition factor d and the sparsification rate k are two hyperparameters that determine the trade-off between efficiency and performance. experiments (Sec. 4.4), we also study the effects of these two hyperparameters.
3.3 How DOES INFOMAX FIND THE MOST INFORMATIVE CORESET?
We explain why InfoMax can find the most informative coreset from the perspective of information theory. First, we restate the information maximization formulation of data pruning defined in Eq. (1), S* arg D. I(S), where p is the size of the target coreset. The I(S) measures the set-level information the candidate subset S, specifically,
((S) = I(21) (22/21) + +... (pp.) I(=1)
Σ 2
(I,., Ex-1)
(4)
where (2,,..., 2p} S are all samples from the set. Note that this equation always holds regardless of the order of samples. The iatra- information I(21) of the sample 2 could be instantiated by various sample-wise scores (Sorscher et al., 2022; Cody Coleman al., 2019; Tan et al., 2023;

---

Published as a conference paper at ICLR 2025
Paul et al., 2018). Regarding the conditional gain term Z), refer to the recent progress in submodular information measures (SIM) Wei et al. (2015); Kaushal et al. (2021); Kothawade et al. (2021), which present several instantiations for the submodular conditional gain. Here, we select the simplest yet effective instantiation, Graph Cut conditional gain (GCCG), T(=x] 21, 2k =k-1) = f(zx)) - 2X Eics K 2,, It measures the dissimilarity between the sample 24 and all conditional samples, Specifically, K measures the similarity between the sample i and the sample k. A is an undetermined coefficient and is a hyperparameter in the system. The submodular function f maps from ground set to a real value, and we can simply set it with f(z) I(2). Hence, we have the following instantiation for the conditional gain term: 21. 2k-1) [(x) 2X Eic* K By substituting it into the Eq. (4), we can instantiate and reformulate the set-level information as: T(S)- EeED [(z) 2X E.E. II# 2.s. We introduce a set of binary variables X E (0. 1) where
N D is the size of the whole training set. In the selection procedure, X; 1 indicates the sample 2 was selected, otherwise it was pruned. By the problem definition in Eq. (1) and the set-level information formulation Eq. (4), we can transform the original information maximization problem in Eq. (1) into the following combinatorial optimization problem,
max (()) X. p
Σ K., X.X., 2ASED
(5)
XE(O.1) IXI-P 2ED
where À is a flexible hyperparameter to be determined in Eq. (4). Hence, if we set a p-1 then we obtain the quadratic programming problem as defined in InfoMax. Therefore, we have proved that under the premise of using the instantiation of Graph-cut conditional gain (GCCG) for the conditional gain term, solving the data pruning problem in Eq. (1) to find the most informative subset is equivalent to solving the quadratic problem defined in Eq. (2). See Appendix D.2 for the proof.
4 EXPERIMENTS
We carried out extensive experiments on three tasks. namely image classification, multi molti-moddlity pretraining, and instruction tuning for Large Language Models (LLMs), to investigate the performance of our InfoMax. Subsequently, we conducted ablation studies to explore the component design within InfoMax. Each result of InfoMax is the average of five independent runs. The standard deviation corresponding to each result of InfoMax is less than 0.85.
4.1 IMAGE CLASSIFICATION
The image classification task encompasses experiments on three datasets, namely CIFAR-10. CIFAR- 100 (Krizhevsky, 2009), and Imagenet- IK (Russakovsky et al., 2015). Following coreset selection, we will train model on the chosen subset to examine its performance as the performance of the coreset. The model employed here is ResNet- 18 for CIFAR and ResNet-34 for ImageNet. For InfoMax, the dataset partition scheme is not employed herein as the dataset scale is not large. The sparse-rate A is set to 5, the pairwise term weight a is set as 0.3, and the iteration T is set as 20. Regarding the detailed experimental settings, please refer the Appendix.
For the supervised setting, we compare InfoMax with several baselines: 1) Random selection of examples. 2) Entropy (Cody Coleman al., 2019) of the model's prediction. 3) Forgetting (Toneva et al., 2018) score for each example i.e., the number of times a model predicts the example incorrectly after having predicted correctly in the previous epoch. 4) EL2N (Paul al., 2018), the L2 norm of error vectors. 5) Moderate coresets (Xia et al., 2023) that selects samples at about the median distance from the class center, 6) CCS (CCS) divides range of difficulty scores into equal- qqiel-sized bins and randomly samples from each bin. 7) D2- -P-rrnnnng (Maharana et al., 2023) selects samples by performing message passing on the data graph. The scoring model for each method a ResNet model trained on the target dataset. K-center (Sener & Savarese, 2017) the standard geometry-basee coreset method.
For the Unsupervised setting, we select the following baseline: 1) SSP (Sorscher et al., 2022) that uses self-supervised embeddings to compute k-means clusters and treats samples at a farther distance from the cluster center more important, 2) CCS over SSP scores, and 3) D2- over SSP scores coreset selection. The unsupervised feature used here is from the officially public DINO- ViT-Base- V2 model (Oquab et al., 2023).
7

---

Published as a conference paper at ICLR 2025
Table 1: A comparative analysis of the performance ((Acc@) 1)) of InfoMax on image classification datasets with ResNet-18 18 (on CIFAR) and Re (on ImageNet). The best results are bolded.
Dataset
CIFAR-10
CIFAR-100
Image-Net-IK
Selection
100%
70%
50%
10%
100%
70%
50%
10%
100%
70%
30%
10%
Random
95.5
91.3
93.4
90.9
88.0
79.0
78.7
74.6
71.1
65.3
57.4
41.8
73.1
72.2
70.3
66.7
62.5
52.3
Entropy
94.8
92.9
90.1
84.1
72.1
74.7
68.9
60.3
49.6
35.0
72.3
70.8
64.0
55.8
39,0
92.5
69.5
47.2
71.4
Forgetting
95.7
94.9
88.1
73.8
46.3
76.0
68.1
49.3
30.3
20.6
72.6
70.9
66.5
62.9
52.3
E1.2N
95.4
94.8
89.2
78.6
30.3
75.6
68.1
47.2
24.8
11.8
72.2
67.2
48.8
31.2
12.9
SSP
93.1
66.4
27.9
52.7
20.5
Moderate
93.9
92.6
90.5
57.3
$1.0
74.6
71.1
65.3
$8.5
45.5
72.0
70.3
65.9
61.3
52.1
CCS (unsupervised)
95.2
93.4
90.5
85.5
80.6
76.4
71.8
61.2
37.5
25.4
71.6
69.5
62.1
47.4
30.2
CCS
95.0
74.4
70.5
57.3
D²-Pruning (unsupervisod)
94.4
94.2
87.6
86.1
83,9
77.8
70.0
66.6
62.3
51.7
72.6
694
66.1
60.9
52.5
D DP-Pruning
95.7
94.9
93.3
91,4
87.1
78.2
75.9
70.5
65.2
56,9
72.
71.8
68.1
65.9
55.6
InfoMax (unsupervised)
56.8
InfoMax
95.5
91.7
94.1
92.7
89.1
79.0
76.7
71.5
67.9
58.7
73.3
72.8
69.4
66.5
59.0
Table I presents the results on three image classification datasets comparing the performance (ac- curacy) of InfoMax with several baselines. Firstly, let's focus on the performance in the supervised setting. On CIFAR-10, at various pruning rates (30% 90%), InfoMax outperforms other methods in terms of accuracy in most settings. For instance, at a 90% pruning rate, InfoMax achieves an accuracy of 89.1, outperforming other methods (such as D²-Pruning) by 2.0% in accuracy. On CIFAR-100 and ImageNet, InfoMax has higher accuracy values compared to other methods across different pruning rates. Moreover, this advantage becomes more pronounced under a high pruming ratio. For example, at a 90% pruning rate, InfoMax surpasses the second-rande CCS by 1.4% and 1.7% the two datasets respectively, and outperforms the third-ranked Då-Pruning by 1.8% and 3.4% on the two datasets respectively. Next, we must highlight the performance of InfoMax under the unsupervised setting. The performance of InfoMax in the unsupervised setting steadily exceeds that of other schemes in the unsupervised setting. Even under the setting of a high pruning ratio, it can approach and even outperform most supervised schemes. For example, rate, the unsupervised InfoMax on CIFAR-10 10 lags only 0.1 % in performance compared with the supervised D2 DP-Pruning, and the unsupervised InfoMax on ImageNet leads the supervised D -Pruning by performance advantage of 1.2%. It showcases that InfoMax exhibits superior performance and notable effectiveness across different settings.
90% pruning
4.2 MULTI-MODALITY
For multi-modality pretraining tasks, we conducted experiments on the popular vision-language dataset CC12M (Changpinyo et al., 2021), which contains 12 million image-text pairs from the Internet, for CLIP-like vision-language pre-training (Radford et al., 2021). A common practice (Schuhmann et al., 2022; Gadre et al., 2024) for selecting coreset from VL datasets is to use the pre-trained CLIP model (Radford et al., 2021) (021)(u)) to score each image-text pair, where a higher CLIP score indicates better image-text aligmment. Hence,
set this as the most basic baseline. termed, 1) CLIP score. Additionally, we also select the following bascline: 2) Clustering + CLIP: Since using only the score for screening is based on the matching degree of images and texts it is difficult to reflect the redundancy degree of samples. Consequently, some schemes (Li et al., 2022: 2023) combine Clustering and CLIP scores, that is, first clustering in the feature space and then selecting a portion of samples with high scores within each cluster. 3) Moderate coreset (Xia et al., 2023) over CLIP scores: Selecting those samples with the median score since the highly-scored samples may be the too-easy samples. 4) CCS over CLIP scores: By following the implementation in Maharana et al. (2023), divides a range of CLIP scores into equal-sized bins and randomly samples from each bin. 5) DD DD*-Pruning over CL.IP scores: It constructs a sample graph by using the CLIP score as the node value and using image features from the CLIP vision encoder to calculate the edge value (similarity).
For our InfoMax approach, we also designate the CLIP score as the intra-sample information measurement. For the pairwise similarity, we employ the inner product between the features of samples from the CLIP vision encoder. The dataset partition factor d is set to 10; that is, we randomly partition the 12M data into 10 subsets each of size 1.2M. The sparse-rate is set at 5, the pairwise term weight is set 0.3, and the iteration T is set to 20. After data selection, perform CLIP pre- pre-trrining on the coreset and evaluate the trained model on four downstream tasks: Zero-shot ImageNet- K Classification, Linear Probing ImageNet- K Classification, Image-to-Tees Text
8

---

Published as a conference paper at ICLR 2025
shat image Nen-
FR:4/30K
Liveoeprch Imagrket-le
Cuaazee
CLP icor
CU" store
10-
Cledd
-30-
DCB
20-
187%
63%
40%
382%
67%
45%
D00%
20%
13%
100%
60%
205
10%
Sulation
Salccion
Figure 5: Experimental results of coreset selection on CC12M (Changpinyo et al., 2021) for multi- modality vision-language pre-training on CLIP-VVT-B//3 (Radford et al., 2021) model.
(12T) Flickr30K (Plummer et al., 2015) Retrieval and Text-to-Image (T21) Flickr30K Retrieval. For the detailed experimental settings and results along with standard errors, please refer to the Appendix.
The experimental results are presented in Table 5. InfoMax consistently outperforms all baselines across all selection ratios. This leading advantage is particularly prominent when the selection ratio is relatively small. For instance, when selection ratios- 10%, InfoMax surpasses DP-Pruning by 1.5%, 1.3%, 2.7%, and 1.0% respectively on these four downstream tasks, and exceeds the widely-used selection method of Clustering + CLIP by 2.1%, 1.9%, 4.0%, and 1.8% respectively on the four tasks. This demonstrates the superiority and effectiveness of InfoMax in different scenarios and under various selection conditions, highlighting its significance data-centric research areas.
4.3 INSTRUCTION TUNING FOR LARGE LANGUAGE MODELS
Recently, some works (Xia et al., 2024; Maharana et al., 2023) have demonstrated significant redundancy in language datasets. For instance, LESS (Xia et al., 2024) reduced the size of instruction tuning datasets to merely 5% of their original amount. The core of LESS is the influence score (Pruthi et al., 2020; Tan et al., 2023), which gauges how particular data point impacts the performance of the learned model on the validation set.
Following (Xia et al., 2024), we also conduct coreset selection on a mixed dataset containing data from FLAN-V2 (Longpre et al., 2023), COT (Wei et al., 2022), DOLLY (Conover et al., 2023), -SEN-ASSIT (Köpf et al., 2023). The mixed training set, consisting of approximately 270K data points, exhibits wide variation in its format and the underlying reasoning tasks. After the selection process, we conduct LoRA- ARR--inetunnng (Hu et al., 2021) on the selected coreset for A--7B model (Touvron et al., 2023). For D2-Pruning (Maharana et al., 2023) and our InfoMax, employ the LESS score as the satrle measurement and utilize the gradient of the sample as the sample's 's feature, Additionally, we select Moderate (Xia et al., 2023) over LESS and CCS (Zheng et al., 2022) over LESS as the baselines.
NNW (5 shot Accuracy!
TYDIGA (1 shot F1 Score)
BBH 13 shot Exact Match
32
42
Ransom
Random
Rundorn LESC
LTSC
Madarata
42-
LESC
Madorii
25
50
Figure 6: A comparative analysis of the performance of InfoMax on instruction tuning datasets for LLaMA2-7B (Touvron et al., 2023) model.
After the training process, we assess our method using three widely recognized benchmarks: 1) MMLU (Hendrycks et al., 2020) offers a diverse range of knowledge domains for evaluation, covering 57 knowledge areas, like mathematies, computer science, and others. 2) TYDIQA (Clark et al., 2020) is a multilingual question-anss answer dataset that includes 9 kinds of languages. 3) BBH (Suzgun et al., 2023) encompasses 27 arduous tasks to assess whether the model can handle complex reasoning
9

---

Published as a conference paper at ICLR 2025
situations. The experimental results are presented in Figure 6. InfoMax, our proposed method, demonstrates significant performance advantages over other competing methods, particularly at small selection rates ( 5%). On MMLU, it shows notable improvements compared to the second- ranked method, P For example, at the 95% and 99% pruning rates, InfoMax achieves an accuracy of 51.8 and 50.2, compared to DD-Pruning's performance, an improvement of about 2% in performance. Similar trends are observed on TYDIQA, InfoMax outperforms LESS (the second- ranked approach) at 90%, 95%, and 99% pruning rates, with differences ranging from about 1.1 to 1.5 percentage points. For the BBH dataset, the superiority of InfoMax is even more pronounced. At 95% pruning rate, it has an accuracy of 47.2 compared to D2- D2-Pruning's 41.7, substantial improvement of about 5.5 percentage points. Overall, InfoMax consistently shows better results, highlighting its effectiveness even when large portion of the data is pruned.
4.4 ANALYSIS DISCUSSION
Here, we have investigated four hyperparameters in InfoMax: the dataset partition rate d, the sparse rate k. the weight a of the pairwise term, and the iteration for InfoMax. In Appendix Sec. B, we will study the time cost and generalization ability of InfoMax (a) Partition strategy. The experiments on CC3M/CC12M (Changpinyo et al., 2021) in Figure 7(a) studies the effect of the dataset partition strategy. It suggests that decrease in the partitioned subset size inevitably reduces the performance of the coreset. This decline exhibits a significant downward tendency when the size of each subset is less than approximately IM. Consequently, when dealing with relatively large-scale datasets, we recommend ensuring that the size of each subset is at least IM samples. (b) Sparse rate k: The sparse rate represents the size of the neighborhood of each sample. For details, see Sec. 3.2. A smaller & implies more 0 items in the similarity matrix K. In Figure 7(b), we found that the performance of InfoMax is relatively robust to the choice of k. Hence, we set k 5 for better efficiency in all experiments.
(c) Pairwise weight at The pairwise weight determines the tradeoff between the first-order term and the second-order term in the formulation of InfoMax as in Eq.2. As shown in Figure 7(c), it is observed that the optimal range of a varies for different selection ratios. However, all these ranges yield satisfactory results when a is around 0.3 to 4. Consequently, recommend setting
0.3. (d) Iteration T: Figure 7(d) reveals that the performance of InfoMax rises with the increase in the iteration number T. Nevertheless, this gain tends to saturate after T > 20. Hence, suggest setting T 20 to balance good performance and efficiency.
- Sdedio rii 65
750
35.0
Accuracy
Accurasy
32.5
Accuracy
0C12M OCSM
260
30.0
20
15
350
25:0-
Sclectina
22.5
22.5
200
150
200
0.301
0.1
20
150
Subset Size (M)
Iteration
(a)
(d)
Figure 7: Ablation study for hyperparameters in InfoMax: the dataset partition strategy (the size of each subset), the sparse rate k. the weight a of the pairwise term, and the iteration number T of the InfoMax solver. Experiments in (a) are conducted on CC3M & CC12M (Changpinyo et al., 202 1). All experiments in (b,c,d) are conducted on CC12M. The selected model is CIIP-ViT-B/32 (Radford et al., 2021). The reported metric focuses on the accuracy of the coreset-trained the zero-shot ImageNet-1K classification tasks.
on
CONCLUSION
This paper has presented InfoMax, a novel and effective data pruning method. InfoMax is formulated as a quadratic optimization problem to maximize the sample-wise informativeness and minimize the redundancy of selected samples. Furthermore, sparsification techniques and dataset partitioning strategies is introduced to ensure scalability, enabling it to handle datasets with millions of samples within tens of minutes. Overall, InfoMax shows great potential and effectiveness in the field of data pruning to improve the efficiency and performance of data processing in various applications.
efficient gradient-based solver along with
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work has been supported in part by the Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422, 17212923), Theme-based Research (Grant No. T45-701/22-R), and the Shenzhen Science and Technology Inno- vation Commission (SGDX202205301 1 1405040). Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.

---

Published as a conference paper at ICLR 2025
REFERENCES
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020.
Pierre Baqué, Timur Bagautdinov, François Fleuret, and Pascal Fua. Principled parallel mean-ficd inference for discrete random fields. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5848-58555 2016.
Tom B Brown. Language models are few-shot learners. 2020.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunet: white-box deep network from the principle of maximizing rate reduction. The Journal of Machine Learning Research, 23(1):4907-5009, 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: benchmark for information nnfornation-seeking typologically diverse languages. Transactions of the Association for Computational Linguistics, 2020.
seeking question answering in
Adam Coates and Andrew Y. Ng. Learning feature representations with k-means. In Neural Networks: Tricks of the Trade Second Edition, pp. 561-580. 2012.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovee, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference Learning Representations, 2019.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free Dolly: Introducing the world's first truly open instruction-tuned LLM, 2023.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.
Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Proceedings of the forty-third annual ACM symposium on Theory of computing, pp. 569-578. ACM, 2011.
Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constant-size coresets for k-means, PCA and projective clustering. In Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposiun on Discrete Algorithms, pp. 1434 1453. SIAM, 2013.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets, Advances in Neural Information Processing Systems, 36, 2024.
Sariel Har-Peled and Soham Mazumdar. coresets for k-means and k-median clustering. In Proceedings of the Thirty-Sixih Annual ACM Symposium on Theory of Computing, pp. 291-300, 2004.
Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-mmdian clustering. In 36th Annual ACM Symposium on Theory of Computing, pp. 291-300, 2004.
Sariel Har-Peled, Dan Roth, and Dav Zimak. Maximum margin coresets for active and noise tolerant learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, pp. 836-841, 2007.
12

---

Published as a conference paper at ICLR 2025
Muyang He, Shuo Yang. Tiejun Huang, and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty. In Computer Vision and Pattern Recognition Workshops, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
arXiv:2106.09685, 2021.
Wenyu Jiang, Hao Cheng, MingCai Chen, Chongjun Wang, and Hongxin Wei. DOS: Diverse outlier sampling for out-of-distribune detection. In The Twelfth International Conference on Learning Representations, 2024.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Reprexentations, 2019.
Vishal Kaushal, Suraj Kothawade, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer. Prism: A unified framework of parameterized submodular information measures for targeted data subset selection and summarization. 2021.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp- 1464-5474, PMLR, 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. pp. 4026-4604026,, 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. OpenAssistant converstions-democraaiiggg large language model alignment. 2023.
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning in realistic scenarios. volume 34, pp. 18685-18697, 2021.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems, 24, 2011.
Kristof Meding, Luca M. Schulze Buschoff, Robert Geirhos, and Felix A. Wichmann. Trivial or dichotomous data difficulty masks model differences (on imagenet and beyond). In International Conference on Learning Representations, 2022.
impossible- pupposible-ddicootmoouu
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Måns Larsson, Anurag Arnab, Fredrik Kahl, Shuai Zheng, and Philip Torr. projected gradient descent method for crf inference allowing end-to-end training of arbitrary pairwise potentials. In Marcello Pelillo and Edwin Hancock (eds.), Energy Minimization Methods in Computer Vision and Pattern Recognition, 2018.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICMI., 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Stuart P. Lloyd. Least squares quantization in PCM. IEEE Trans. Information Theory, 28(2):129-133 1982.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688, 2023.
13

---

Published as a conference paper at ICLR 2025
Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity & difficulty in data pruning. 2023.
Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, and Ari S. Morcos. Sieve: Multimodal Dataset Pruning Using Image Captioning Models. In Computer Vision and Pattern Recognition (CVPR), 2024.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR, 2020.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang-Was Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. In Advances in Neural Information Processing Systems, 2018.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Garima Pruthi, Frederick Liu, Sundararajan Mukund, and Satyen Kale. Estimating training data influence by tracing gradient descent. arXiv preprint arXiv:2002.08488 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. pp. 8748-8763. 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. pp. 10684-10695, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision. 115:211-252, 2015.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. volume 35, pp. 25278-25294 2022.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. 2017.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances Neural Information Processing Systems, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thoon thouugthttt can solve them. In Findings of the Association for Computational Linguistics: ACI. 2023, pp. 13003-13051, 2023.
14

---

Published as a conference paper at ICLR 2025
H.-R. Tan, C. Wang. S.-T. Wu, T.-Q. Wang, X.-Y. Zhang, and C.-L. Liu. Proxy graph matching with proximal matching networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35 (11):9808-985) 98188-9881111111 2021.
Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang, Fan Wang, and Xiaojuan Qi. Data pruning via moving-one-sampll-ouu. In Advances in neural information processing systems, 2023.
Pang-Ning Tan, Michael Steinbach, Vipin Kumar, et al. Cluster analysis: basic concepts and algorithms. Introduction data mining, 8:487-568, 2006.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307 2307.0988, 2023.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of tf-houhtt prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International conference on machine learning, pp. 1954-1963, PMLR, 2015.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.0453 2024.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In International Conference Learning Representations, 2023.
Shuo Yang, Zeke Xie, Hanyu Peng. Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. In International Conference on Learning Representations, 2023.
Shuo Yang. Zhe Cao, Sheng Guo, Ruiheng Zhang, Ping Luo, Shengping Zhang, and Liqiang Nie. Mind the boundary: Coreset selection via reconstructing the decision boundary. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 35948-55960, 2024.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding reduction. Advances in Neural Information Processing Systems, 33:9422-9434, 2020.
Yu Yu, Shahram Khadivi, and Jia Xu. Can data diversity enhance learning generalization? In Proceedings of the 29th international conference on computational linguistics, pp. 4933-4945, 2022.
Haizhong Zheng. Rui Liu, Fan Lai, and Atul Prakash. Coverage- centric coreset selection for high pruning rates. 2022.
15

---

Published as a conference paper at ICLR 2025
BROADER IMPACT
This paper presents a novel and effective data pruning algorithm to advance the deep learning area. There are some potential positive societal effects, such as helping people better understand the role of data to develop more robust deep learning systems and possibly even be used to reduce training and storage costs, Additionally, with the explosive growth of data, similar data-centric deep learning schemes can also be easily applied to inhumane surveillance or scenarios that violate data privacy and data copyright. Therefore, we believe that strict legislation needed to constrain the occurrence of these.
LIMITATIONS
This paper proposes an efficient and high-performance coreset selection scheme. However, due to the limitations of experimental equipment, the maximum scale of the experiments in this paper only reaches data scale of 12M. In the future, will consider investing more in renting purchasing more computing equipment and exploring the performance boundaries of InfoMax on a larger scale (at the billion level) dataset.
Table 2: Comprehensive overview of the notational convention.
Notation
Description
D
The training set, and the size of the training set [D] N. A subset from D.
D
S* D
The optimal coreset.
The coreset budget and the target coreset size. A training data.
T(2)
The information of sample
T(S)
The set-level information of the
I(z]...)
The conditional information gain of sample z. The set-level information of the set [......, žn.
I(21, Zn)
((2;8)
The mutual information between sample and
IERN
The information vector, where each item measures the informativeness of the corresponding sample, that is, 1, I(=). The similarity matrix measures the similarity between samples. The similarity between sample and sample s.
Ke KERNNN
K,,
k
The sparse rate of K. The dataset partitioning rate. The pairwise weight in the objective of InfoMax, see Eq. (2). The hyper-parameter in the Graph-Cut Conditional Gain measurement (Kothawade al., 2021). see Eq. (15).
d
A
X
The binary variable in Eq (before continuous relaxation).
X.
The variable item corresponding to the sample z.
X E RN
The iterant variable /-th iteration (after continuous relaxation).
T
The maximum iteration number of the InfoMax solver.
16

---

Published as a conference paper at ICLR 2025
Algorithm 1: InfoMax Coreset Selection.
1: Input: A dataset D with N samples, the division coefficient d, the target coresct size budget p.
2: Initialization: Set the coreset as a null-set S* Ø: Divide the dataset D into d random subsets
[Di,... DaJ, where each subset is size of N/d.
3: Initialization: Uniformly initialize the initial guess X 11
4: for dl....dd do
5: // //jjective construction: supervised or unsupervised mode.
6: Calculate the intra-information information vector and the similarity matrix K:
7: // Iterativelly perform the InfoMax solver.
8:
for {0, ... T' 1} do
1.-I-2po- 9: X Softmax p I 2pa KXx):
10: end for
11: Append the samples corresponding to the top-k largest item in X into S*
12: end for
13: Output: The InfoMax coreset S*
A EXPERIMENTAL SETTINGS
A.1 IMAGE CLASSIICATION
We utilized Pytorch Paszke et al. (2017) to implement our method. Our experiments were con- ducted on server equipped with 8 Tesla-V100 GPUs. Additionally, nyper-parameters and experimental settings for training before and after dataset pruning. To ensure fairness, made that the number of iterations the selected subset and the full set was the same by following Zheng et al. (2022); Maharana et al. (2023). For CIFAR-100, we utilize the SGD optimizer with weight-decay Tiny Imngem,,Net, we use the SGD optimizer with weight-decay set to 5e-4, a learning rate of 0.3, and batch size of 64. For ImageNet-IK we use the SGD optimizer with weight-decay set le-4, warmup for 5 epochs, a learning rate of 0.4, and a batch size of 256. Regarding data augmentation, we solely adopt RandomResizedCrop and Pandomiorizontullliip for all experiments.
maintained identical
5e-4, learning rate of 0.1, and batch size of 128. For
A.2 VISION-LANGUAGGE PRETRAINING
For coreset selection the vision-language dataset CC12M (Changpinyo al., 2021), all experiments are conducted on servers with a totally of 16 NVIDIA V100 GPUs. The selected model is CLIP model (Radford et al., 2021). We follow the settings provided in the original paper. Specifically, the CLIP model Radford et al. (2021) is trained for 32 epochs with AdamW optimizer, weight decay 0.2, and batch size of 2048. After warmup epoch, the learning rate gradually decreases from le-4 following the cosine strategy.
Zero-shot ImageNet classification. The CLIP model has two encoders, one for text and one for images. During the zero-shot classification process, text descriptions corresponding to the ImageNet classes are formulated. For example, if class "dog" exists, a text description like "a picture of dog" might be created. These text descriptions are encoded by the text encoder of CLIP to obtain text embeddings. At the same time, the images from the ImageNet dataset are encoded by the image encoder of CLIP to get image embeddings. Then, the similarity between each image embedding and all the text embeddings (representing different classes) is calculated. The image is classified into the class whose text embedding has the highest similarity the image embedding.
Linear Prob. This is a technique used to evaluate and analyze the performance a pre-trained model. For the CLIP model, linear probing involves adding linear layer top of the pre-trained CLIP model and then training only this linear layer while keeping the rest of the CLIP model's 's parameters fixed.
Image-Texa Text Retrieval. This is a task where the goal is to find the most relevant text description for a given image or find the most relevant image for given text description. Let use the Image-to-Text
17

---

Published as a conference paper at ICLR 2025
Retrieval example. The image is encoded using the vision encoder. This results in an image embedding that represents the visual features of the image. Then, text documents are also encoded (using the text encoder) to obtain their respective text embeddings. The similarity between the image embedding and all the text embeddings is computed. The text with the highest similarity score is retrieved as the relevant description for the image.
A.3 INSTRUCTION TUNING
The specific settings for LoRA fine-tuning are as follows: the Lora-rank is 64, b precision used, the number of epochs is 4, the tora-targetmmodules include q-proj, k-proj, v-proj. o-proj. the learning rate is 1e- the batch size 8, the gradient accumulation steps is 16, and the AdamW optimizer is used. This experiments is conducted on server with 8 A A100 GPUs. As for the calculation of the LESS score. please refer to (Xia et al., 2024) for details. Before computing the score, it performs LoRA fine-tuning the LLM on the training dataset and then retains the randomly pooojeett-- LoRA gradient corresponding to each sample. The LESS score reflects how well the gradient of a training sample consistent with the gradient of the target dataset are concerned about.
B FURTHER ANALYSIS
B. GENERALIZATION ABILITY TEST
Here, we demonstrate the generalization ability of InfoMax. In the test for cross-model generalization ability, we observe that InfoMax's 's coreset exhibits superior generalization ability compared to other coresets. Experimental results are reported in Table 3. InfoMax achieves the best performance the cross-model generalization ability test. Notably, when using unsupervised scores (SSP (Sorscher et al., 2022)) and unsupervised features from DINO (Oquab et al., 2023), InfoMax has better cross-model generalization ability compared the supervised version. This also indicates the significance of using unsupervised base models to extract features during data selection.
Table 3:
Cross model generalization ability test, including the setting of ResNet to SENet, and the setting of ResNet to EfficientNet-BO. The experiments are conducted on CIFAR- 10 (Krizhevsky, 2009). InfoMax (unsupervised) uses the unsupervised scores (SSP (Sorscher al., 2022)) and unsupervised features from DINO (Oquab et al., 2023), while all remaining methods are based on the EI.2N score (Paul al., 2018) and the feature extractor trained on CIFAR-10.
Dataset
ResNet
ResNet SENet
ResNet ENel-B0
Selection Rate
50%
30%
20%
50%
30%
20%
50%
30%
20%
Random
93.4
90.9
88.0
93.8
92.9
88.8
90.0
87.2
84.6
Moderate
92.6
90.6
87.3
91.4
88.3
85.7
87.1
86.9
82.2
CCS
95.0
93.0
91.0
92.7
89.4
88.1
90.5
86.7
84.1
D Prrninnn
93.3
91.4
87.1
94.3
91.4
87.1
93.3
91.4
87.1
InfoMax
94.1
92.7
89.1
InfoMax (Unsupervised)
93.6
92.2
90.1
94.4
93.8
90.5
93.1
90.8
88.4
Furthermore, assess the cross-setting generalization ability by varying the score (Forgetting (Toneva et al., 2018) and Margin (Har-Peled et al., 2007)) and the type of feature (unsupervised DINO features (Oquab al., 2023), unsupervised VQGAN features (Esser et al., 2020)). We observe that regardless of the configurations, InfoMax can achieve remarkably superior results compared to score-based approaches that only utilize score schemes that conduct K-Median Coreset solely with features. This thoroughly demonstrates the cross- generalization ability of InfoMax.
B.2 TIME COST TEST
We study the time cost of InfoMax in Table 5. There are two main stages for InfoMax, the first one the objective construction stage, including inferencing on all data and calculating the similarity matrix K and calculating the sample-wist score I, and the second stage is the optimizing stage, which iteratively rumning the solver defined in Eq. (3). is easy to find that although the computational efficiency of our method is slower than that of score-based schemes, it is still faster than D2_Pruning
18

---

Published as a conference paper at ICLR 2025
Table 4: Cross setting generalization ability test. The experiments are conducted on CIFAR-I0 (Krizhevsky, 2009). The pruning ratio is 10%. VQGAN features are obtained from the VQGAN model (Esser et al., 2020).
Method
Accuracy
Forgetting (Score-based) (Toneva et al. 2018)
46,3
Margin (Score based) (Har-Peled et 2007))
Supervised feature (K- Median diversity based)
38.9
DINO feature (K-Median diversity-based) (Oquab et 2023)
VQGAN feature (K-Modian diversity-basec) (Esser ct al,, 2020)
28.2
InfoMax: forgetting supervised feature
89.4
InfoMax; margin supervised feature
88.0
InfoMax: margin DINO feature
85,4
InfoMax: margin VQGAN feature
(Maharana et al., 2023). This is because InfoMax does not have greedy selection process per-sample basis. For CC12M (Changpinyo et al., 2021), we divide the dataset into 10 subsets, each with size of approximately 1.2 million. We run InfoMax in multiple threads two server nodes with eight GPUs each to process these subsets. The overall time consumption is approximately 37 minutes. Note that this time is significantly longer than that for processing 1 million ImageNet data because multiple processes occupying the common disk I/O slow down the time efficiency. is not difficult observe that the main sources of the aforementioned time consumption are the first stage, including the inference process on all the data and the construction process of the similarity matrix K with the k-NN algorithm. The block of disk IO by multiple operations will further drag down the efficiency. Therefore, using more advanced K-NN algorithm and a more advanced disk can significantly improve efficiency.
Table 5: A study on the time cost of InfoMax and two selected competitors, D2-Pruning (Maharana al., 2023) and score-based method (Entropy (Cody Coleman et al., 2019)), for I million image data (the training set of ImageNet- IK (Russakovsky et al., 2015)).
Name
Time cost (min)
Stage- InfoMax
Stage 2 of InfoMax
1.7
Overall of InfoMax
16.1
D Pruning (Maharana al., 2023)
23 (Maharana et al., 2023)
Score based method (entropy (Cody Coleman et al., 2019))
4.1
C RELATED WORKS
Score-based methods. The score-based techniques are the most popular data selection approaches. The EI.2N score (Paul et al., 2018) measures the data difficulty by computing the average of the f2-norm eтror vector from a set of networks. GraNd (Paul et al., 2018) measures the importance by calculating the expectation of the gradient norm. The Forgetting score (Toneva et al., 2018) counts how many times a model changes its prediction from correct to incorrect for each example during the training process. Memorization (Vitaly Feldman & Chiyuan Zhang, 2020) assigns score each example based on how much its presence or absence in the training set affects the model's ability predict correctly. Diverse ensembles (Kristof Meding al., 2022) gave a score each sample based on how many models in a group misclassified it. (Sorscher et al., 2022) proposed to use the distance between the sample and its corresponding cluster center as the importance score. Influence score (Tan et al., 2023; Xia et al., 2024) measures the sample-wis wise leave-one-oue retraining influence on the model's 's performance. The score-based approach often suffers from performance problems in application, especially when the pruning ratio is large. Some recent works have tried to solve this problem through various methods, for example, Moderate (Xia et al., 2023) suggested selecting data points with scores close to the score median. Note that Moderate (Xia et al., 2023) can use any selection criterion, such as EL.NN score (Paul et al., 2018), as a basis. Dyn-Unc He et al. (2024) proposed an efficient imcertainty-basee score with awareness of training dynamics. Some related works also use sample-wise scores (Radford et al., 2021; Mahmoud et al., 2024) to reflect the quality of multi-mon modality data.
19

---

Published as a conference paper at ICLR 2025
Diversity-based (Geometry-basedd methods. Traditionally, diversity-based coreset schemes are very classic computer science problem (Lloyd, 1982; Tan et al., 2006; Coates & Ng, 2012; Har-Peled & Mazumdar, 2004; Feldman & Langberg, 2011; Feldman et al., 2013; Jiang et al., 2024). These schemes aim to find a subset of data points that maximizes the diversity among the selected elements. Sener & Savarese (2017) applied greedy k-center choose the coreset with good data coverage. Yu et al. (2020); Chan et al. (2022) proposed to use the coding rate to model measure the diversity. Yu al. (2022) formulates the problem of finding the most diverse subset into the problem of maximizing the dispersion or convex hull volume. In addition, some works proposed to prune data from the perspective of submodule theory (Wei et al., 2015; Kaushal et al., 2021; Kothawade et al., 2021) and linear programming (Yang et al., 2023) to ensure diversity.
Hybrid methods. Solely using the diversity- method alone can hardly perform satisfactorily because they do not consider the intra-sample information. Recently, some hybrid works have also attempted introduce that score-based scheme into the diversity-drite performance. CCS (Zheng et al., 2022) sets different sampling ratios for samples with different scores to enhance data coverage and balance both easy and hard samples. Yang et al. (2024) introduces reconstructing the classification boundary framework of CCS. BADGE (Jordan T Ash et al., 2019) is a diversity- active learning that clusters the gradient embeddings of the current model using k-means and selects subset from each cluster. DP-Pruning (Maharana et al., 2023) views data pruning as a node selection problem based on Message-Passing on graph, where the intra-sample information is utilized as the node values on the sample graph. One of the core steps of D2-Pruning is the Inverse Message Passing operation, which iteratively performs a greedy sample selection step. In each iteration, will select the sample with the highest score from all unselected candidates and then reduce the score of samples in the neighborhood to guarantee that highly redundant parts are not selected in the subsequent iterations. Among the above methods, DP-Pruning is currently the most advanced solution terms of performance. However, due to the heuristic or greedy nature of the algorithm, the result obtained by using is often suboptimal, check Figure I for details.
pipelines to achieve better
the original dataset
goal and brings it into the
selection method in
D PROOF
D.1 DERIVATION OF THE INFOMAX SOLVER
Firstly, we restate the quadratic optimization problem of InfoMax:
max (=(0,
Σ xx) - a Σ K,X,X S.L.
Σ x; p:
(6)
Solving the quadratic problem defined in Eq .2 directly can be extremely computationally burdensome and may even prove intractable since the budget size p is generally on the order of tens of thousands or even millions (Krähenbühl & Koltun, 2011; Larsson et al., 2018). 8). Continuous relaxation simplifies the problem by relaxing some of the discrete constraints continuous ones, reducing the complexity of the search space and making the problem more amenable to efficient optimization algorithms, such as the gradient-based methods (Krähenbühl & Koltun, 2011; Larsson al., 2018). Here, we also introduce continuous relaxation of the problem, enabling the use of a gradient-based solver for more efficient optimization:
Σ Xx1(z)) a- Σ K..XXXs, 2,
S.L. Σx 2&D
max
X 2DD
According to (Krähenbühl & Koltun, 2011: Larsson et al., 2018; Baqué et al., 2016; Tan et al.. 2021), the continuous (complex and non-convex) problem in Eq. (7) could be optimized by solving series of the following (convex) sub-problems:
X arg
min
+ Xh x XT 11-2oKx))
1 X ( +
(8)
XER EeeD X.=p
where X is the solution of the I-th sub-problem, (I- [I-2oKX' is the gradient of the objective
in Eq. (7) at X We introduce the convex entropy function h(.)) controlled by factor À the
20

---

Published as a conference paper at ICLR 2025
regularization term. A large À value makes the problem easier solve (Krähenbühl & Koltun, 201 Larsson et al., 8018; Baqué et al., 2016; Tan et al., 2021), but it may deviate from the original problem. The proximal operator ( X²) is optional regularization term to prevent the solution difference
between the two iterations is too large. Following tradition (Krähenbühl & Koltun, 201 1; Larsson
et al., 2018; Baqué et al., 2016; Tan et al., 2021), we use D(X,X') X log X X log X which
p
P p
is the Kullback-Leibler divergence measure the discrepancy between X non-negativity of X and the property that the sum is a fixed value p. has probabilistic meaning. Each element of it represents the probability that each sample is selected. If we differentiate this convex sub sub-problem, the optimal solution is identified by solving the following equation:
and X Note that due the g
0 - (1- +
1 X p
log
- -
1 ( logí
X* ) X
x
s.t. X*
Σ
x
p
2DD
This has an analytic solution to (Krähenbühl & Koltun, 2011; I; Larsson et al., 2018; Baqué et al., 2016; Tan et al., 2021),
3p
X x
ABp
X exp
(log (I 2xxX 1 AB p 1 X
(10)
AB
X X X* Σ x
(11)
Since the third term in Eq. 10 is constant, the solution could be formulated as the following form:
3p A
X
x = Softmax
2aKX*) (1-2xxx))
(log
x²))
(12)
1 A3
P
The mapping by the exponential function exp followed by the summation normalization is just equivalent to the Softmax operator which is widely used in deep learning. By just setting À = 1 and setting B cc, we have the most simplified iterative solution:
X Softmax (p- 2p Sip.aKX))
(13)
Note that the convergence rate of this solver is quite fast. In particular, the norm the iterant difference [X++1 X converges at rate of ((+) according to (Baqué et al., 2016; Tan al., 2021).
D.2 PROD: HOw DOES INFOMAX FIND THE MOST INFORMATIVE CORESET?
We explain why InfoMax can find the most informative coresct from the perspective of information theory. First, we restate the information maximization formulation of data pruning defined in Eq. (1), S* arg ma I(S), where p is the size of the target coreset. The T(S) measures the set-level information of the candidate subset S, specifically,
T(S) = 1(21) + ((22/11) +...
(14)
T(>1) + Σ [(Ex/21, ((2el)),..,21111)) 2k-11 2
where (z1, 2p} S are all samples from the set. Note that this equation always holds regardless of the order of samples. The intra-sample information [(21) of the sample 21 could be instantiated by various sample-wise scores (Sorscher al., 2022; Cody Coleman al., 2019; Tan al., 2023; Paul et al., 2018). Regarding the conditional gain term [2 Z), we refer to the recent progress in submodular information measures (SIM) Wei et al. (2015); Kaushal et al. (2021); Kothawade et al. (2021), which present several instantiations for the submodular conditional gain. Here, we select the simplest yet effective instantiation, Graph Cut conditional gain (GCCG),
(2x/22 26-1) = f((x) - 22 KES..
(15)
ick
measures the dissimilarity between the sample 24 and all conditional samples. Specifically, K measures the similarity between the sample and the sample k. À is an undetermined coefficient
21

---

Published as a conference paper at ICLR 2025
and is hyperparameter in the system. The submodular function f maps from ground set to a real value, and we can simply set it with f(z) = [(2). Hence, we have the following instantiation for
the conditional gain term: [2x][2 (1./2,... ...: 2 (=x.) - 22 Eies K By substituting it into the Eq. (14), we can instantiate and reformulate the set-level information as:
I(S) = Σ -12)-221P0 22
Σ K..
(16)
We introduce a set of binary variables X E {0. where N [D] is the size of the whole training set. In the selection procedure, X= = indicates the sample was selected, otherwise it was pruned. By the problem definition in Eq. (1) and the set-level information formulation Eq. (14), we can transform the original information maximization problem in Eq. (1) into the following combinatorial optimization problem.
max Xc (0.1)) 1X--
Σ I X.I(S) 8 D :63
Σ
||
x ((.))
Ryee upp
Σ
x
Σ [(=) 22 Σ Ks.e).
Σ
y-1 X,I: 166
IX:: Xx; 21-
Σ
y-2 X,X,K,,,
I Xx,
spadas K
151
Σ
X,I: Xx,
Σ X X - x,,)
(7)
1p-1
2A
Σ
x, X,X,K,
II x
Σ
X 2-- - Σ x.y)
2 gen
sp-2
p-2
I,
Xx;
Σ Xsy)
DCD
I6i
3-3
p-3
21
Σ
X.X.K,,
II (p-
Σ x.²)
Since X is binary, hence,
max
I1
X-(o,1)
|
SCD
1
Σ
X.I.
2-2
II
(p-
Σ x,,)
2cD
147
p-3
p-3
21
Σ
X,X.K,,a
|1 Xx;
(p-
Σ x.;)
DD
151
j-1
X,L
X
- p.
(p-2)
X,I. 1|x x,
p
D
s.ap...Ppp 2cD
1s:
2p. A
Σ
X.X.K. aa
p-3
II
Xx, 2(p 3)
1
Σ
X,X, K..
p-3
X
pl
161
xdadai 2DD
151
p-3 X.X.K,,, Xx
Σ
X.I.
X.
Σ
2
2p-
(18)
By applying a similar reduction process to the rest variables, we have:
Σ X((()
max Xc(o.1)
[Xi=p
SCD :c8
7 - 1)!
Σ X.L. - (p-2) Σ X.X.
(19)
mD
IN! [(=)- X,
A(p 2)!
Σ K. x.x.x,X.)).
(p 1)!
where (p)! is a factorial function of p. and X is hyperparameter to be determined in Eq. (4), And we have (p-2)! -1. 13! p-I. Hence, if we use to indicate A(p-2]! we obtain the quadratic programming [p-1)]
22

---

Published as a conference paper at ICLR 2025
problem defined in InfoMax. Therefore, have proved that under the premise of using the instantiation of Graph-cut conditional gain (GCCG) for the conditional gain term, solving the data pruning problem in Eq. (1) to find the most informative subset is equivalent to solving the quadratic problem defined Eq. (2).
23


Reviews:
[{"review_id": "t5QDvPuxkO", "reviewer": "Reviewer_J8Fq", "summary": "This paper introduces InfoMax, a novel data pruning method designed to maximize the information content of selected samples while minimizing overlap. The authors formulate this objective as a discrete quadratic programming problem, which they then relax and solve using an efficient gradient-based approach. Experimental results demonstrate the substantial effectiveness of InfoMax, underscoring its potential in data-centric applications such as image classification, multi-modal pretraining, and instruction tuning for large language models.", "questions": "The authors argue that $D^2$-Pruning may result in suboptimal solutions due to its greedy selection process. However, the relaxation applied to the quadratic optimization problem in Eq. 7 is not proven to produce solutions consistent with the original formulation, which could also result in suboptimal solutions. Given this, what factors contribute to the improved performance of your method compared to $D^2$-Pruning?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an elegant and well-formulated approach to data pruning, with a solid theoretical foundation that supports its design.\n2. The authors conduct a diverse set of experiments, including pretraining vision-language and fine-tuning LLM, further strengthening the validation of the method.\n3. The performance of InfoMax is impressive, achieving high accuracy in various applications and outperforming existing state-of-the-art methods in many cases.", "weaknesses": "1. Some notations in the paper are unclear. For example, the symbols \\( P \\) on line 163 and \\( z_n \\) on line 1104 lack sufficient explanation. Furthermore, the variable \\( X_t \\) in lines 1054 to 1067 should be bolded for consistency.\n2. The motivation behind InfoMax is not entirely novel, as the concepts of diversity and importance (information) have been previously discussed in [1, 2].\n3. The paper does not include comparisons with some relevant baseline methods, such as geometry-based methods.\n\n[1] \"D2 pruning: Message passing for balancing diversity & difficulty in data pruning\", ICLR 2024.\n[2] \"Dos: Diverse outlier sampling for out-of-distribution detection,\" ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYAWtGyvQL", "reviewer": "Reviewer_nGro", "summary": "This work presents a novel approach to data pruning, that aims to maximise the core set information through the simultaneous optimisation of individual sample information and overlapping information between samples. To solve this equality the authors present this as a discrete quadratic programming problem, presenting an efficient gradient based solver to improve the scalability of the method. The resulting core set’s are tested in a variety of settings to demonstrate performance and information preservation, where the authors demonstrate significant improvements over prior methods. Furthermore, the authors demonstrate their findings across a variety of tasks, and datasets to present generalisation of the method. A full sensitivity analysis is presented alongside all key details for reproducibility, providing readers with the necessary information to apply or replicate the findings.", "questions": "1. Have the authors considered testing the method under noisy data distributions? If not, why not?\n2. Where do you see the future of this work? You mention larger scale experimentation, but from a methodological perspective where are the obvious gaps / limitations?\n- Most questions are posed in the weaknesses section to improve clarity of the question.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Structure and Clarity:**\n- The work is well organised and presented clearly defining the problem statement hypothesis of the work. The core narrative and all technical contributions are written in a clear and concise manner, guiding most readers well to fully understand the contributions.\n- Most of the key concepts discussed are presented in the form of visualisations, or figures which help justify the narrative, and provide evidential basis of the investigations.\n\n**Method, hypothesis, findings, and rationale:**\n- The method is well justified with Figure 2 presenting clear empirical evidence to claims of prior method weaknesses and properties. This figure alone does a lot of the heavy lifting in providing strong rationale behind the decision making which defines the proposed method. \n- Method description itself is sensible, clearly presented and interpretable. The method section is further supported with proofs and significant empirical findings justifying decisions made.\n- The findings demonstrate the proposed method is highly performant when compared to the selected benchmark methods and on the selected tasks.\n- Broader impact statement is provided and some limitations are addressed.\n\n**Reproducibility:**\n- All details are presented for full reproduction in the text including optimisation, hyperparameterisation, datasets, and architectural settings. The algorithm presented in the appendix is a nice addition to support reproducibility, and the \n- Descriptions of empirical setups are present and clear to follow, \n\n**Experimental results:**\n- Generally, the authors do a good job covering a variety of empirical evaluations to test their method, presenting results on a variety of datasets for different tasks. \n- The sensitivity analysis is a nice addition, presenting the trade-off between the first and second order terms. While also having a secondary effect of demonstrating the robustness of the method to hyperparamter changes. \n- While additional experimentation would improve the work (see weaknesses) the core empirical evaluation does a good job of conveying the core message and rationale of the work", "weaknesses": "**Empirical Comparisons**\n- How does the method perform when compared to other data pruning methods not included in this work such as Sieve And Dyn-Unc. If there is a strong reason for not including these works then please correct me on this point. \n- The results in table 1 could be considered misleading with incorrect bolding of top results. For 70% cifar10 d2 is performing better, yet, infomax is highlighted. I assume this is a simple mistake.\n- Computational compression between methods is performed but only at a small scale, against d2 and entropy. The addition to all analysed works would further support the statement that InfoMax is computationally efficient while being performant. It is hard from this small scale evaluation the trade-off between these points. \n- The addition of a more explicit ablation would be nice to have. How does changing the instantiation of the first order term I(z) effect the performance for example?\n\n**Data Generalisation of method:**\n- The authors method employs pre-trained networks to produce representations on which the method is applied (shown in Figure 3). However, from my understanding, the learnt representations themselves are produced from learning . This leads to two distinct weaknesses.\n- The first is that the authors do not evaluate the performance and generalisation of the method to produce core sets when applying a different dataset. Thus, how does the method perform if the core feature extractor itself has not been trained on the data in question. \n- Secondly, this leads to a distinct contradiction of the work if it is require to re-train this network per dataset, thus meaning full datasets will have to be trained on before pruning can occur. Thus, leading to the desired computational cost decrease being lost.\n- Furthermore, the choice of SSL method could be highly impactful here, it has been shown that some methods akin to MSN and DINO are both optimising to preserve maximal information across samples than say predictive methods such as BYOL. Therefore, have the authors considered how different pertained models to produce features may interact with the criteria for pruning proposed?\n\n**Further analysis on the selected core set:**\n- Does the coreset exhibit the properties you are aiming to persevere? While the empirical evaluation provide strong evidence that the method does produce a notion that the coreset is indeed informative, are there any other analyses that could be performed to provide further evidence that information redundancy is improved?\n- While I am happy to be corrected on the above point, it does feel that some simpler and more explicit analyses could provide interesting introspective of the selected coreset prompting insights for future works.\n\n**Minor:**\n- Paragraph starting at line 81 could be moved to preliminaries, its place in the introduction does not flow naturally.\n- Error’s are not reported on empirical results, while you state that the SD lies within 0.85, this is considerable in some comparisons, therefore these should be given.\n- The paper is marginally over the page limit, but this is likely due to a formatting error which can easily be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PCBg4NkKYv", "reviewer": "Reviewer_HVmb", "summary": "The article introduces a novel data pruning method called InfoMax, which is also known as coreset selection, designed to maximize the information content of selected samples while minimizing redundancy. The proposal of the InfoMax algorithm maximizes overall information by considering both individual contributions and information overlap of samples. The development of an efficient gradient-based solver is enhanced by sparsification techniques and dataset partitioning strategies, enabling InfoMax to scale to large-scale datasets. Extensive experiments demonstrating InfoMax's superior performance across various data pruning tasks, including image classification, vision-language pre-training, and instruction tuning for large language models.", "questions": "- What is the set-level information of the candidate subset of \\(D^2\\) Pruning? What is its difference from InfoMax?\n- Why can \\(D^2\\) Pruning perform better than other hybrid approaches from a more intuitive perspective? I understand the content of Section3.3 but I hope the authors could convert the Section 3.3 into a more high-level and insightful motivation, which could be placed in Section 1 for more readers to get the key insight.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- InfoMax can effectively handle datasets with millions of samples within tens of minutes through sparsification techniques and dataset partitioning strategies.\n- InfoMax shows better performance compared to existing methods, especially under high pruning ratios.\n- InfoMax exhibits strong generalization capabilities across different datasets and tasks, including cross-model and cross-setting generalization.", "weaknesses": "- The Introduction in this paper lacks a high-level insight of the InfoMax to explain why it could work better than \\(D^2\\) Pruning, which makes the reader difficult to understand the motivation of the proposed pruning algorithm intuitively, for example, what is the more intuitive motivation of the proposed work to maintain a proper balance between importance and diversity?\n- In line#243, K_{z,s} should be inter-sample redundancy instead of intra-sample redundancy.\n- The symbolic sign used in the method lacks clarity and conciseness. For example, I and X are repeated frequently with different and dazzling superscripts and subscripts.\n- The length of the main text disobeyed the strict limits of 10 pages, because in the current typesetting the Conclusion section belongs to the main text.\n- In my opinion, the contribution and novelty of this work is limited due to its start points similar to \\(D^2\\) Pruning, except the only scalable solver (step 4 in Figure 4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RPoaUOJOxJ", "reviewer": "Reviewer_tXt5", "summary": "The paper introduces InfoMax, a novel data pruning method aimed at maximizing information content while minimizing redundancy in selected samples. It measures sample information through importance scores and quantifies redundancy using pairwise sample similarities. The coreset selection problem is formalized as a discrete quadratic programming task. An efficient gradient-based solver is proposed, along with sparsification techniques and dataset partitioning strategies, to scale InfoMax to large datasets. The significance lies in its ability to enhance model training efficiency and data storage without compromising performance.", "questions": "see the Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.) The paper is well-organized, with clear explanations of complex concepts, making it accessible to a broad readers.\n2.) InfoMax offers a new perspective on data pruning by focusing on information maximization, formalizing the problem as a quadratic programming task and offering a clear explanation of the underlying information theory.\n3.) Extensive experiments across diverse datasets and tasks validate the effectiveness of InfoMax, showing consistent improvements over existing methods.", "weaknesses": "1.) The method's reliance on calculating pairwise similarities and the construction of the similarity matrix may become computationally intensive and could be further optimized.\n2.) The performance of InfoMax is sensitive to hyperparameters like the partition rate, sparse rate, and pairwise weight, which may require careful tuning for different datasets.\n3.) The paper could provide more analysis on the risk of overfitting when using the gradient-based solver, especially with high pruning ratios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cnee3DzslC", "reviewer": "Reviewer_kwDs", "summary": "A novel method, InfoMax, is proposed for data pruning and core set selection. The motivation of the method is finding a subset of samples that maximizes overall information by simultaneously considering each sample’s information contribution and the information overlap among them. The authors formulate the core set selection as a discrete quadratic programming (DQP) problem with equality constraints that specify the desired number of selected samples. And a robust gradient-based solver is proposed to address scalability. Extensive experiments conduct the best performance and consistently outperforms the state-of-the-art schemes in a series of tasks.", "questions": "The selection of the four hyper-parameters depends on the specific dataset or task? If it depends on the specific dataset, how should the corresponding hyper-parameters be determined when faced with a new dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed InfoMax designed to maximize overall information by accounting for each sample’s individual contribution while reducing information overlap, with a simultaneous focus on maintaining diversity and importance. And the proposed efficient gradient-based solver makes InfoMax scale to large-scale datasets. The proposed method brings performance enhancements in a series of different tasks.", "weaknesses": "Some typo: a) 95.7 is best result in CIFAR-10 70% setting from Tab. 1.  b) 51.8 should not be bolded in MMUL 10% setting from Tab. 3.\nThe ablation study for 4 hyper-parameters only conducted in multi-modality pre-training task. For classification and instruction tuning tasks, what kind of impact do the 4 parameters have on the final results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "luGZzOFNsY", "reviewer": "Reviewer_2PGZ", "summary": "This paper targets at designing a new coreset selection method to enhance the informativeness of the coreset. Specifically, the authors formalize the coreset selection problem as a discrete quadratic programming task. They adopt an objective of indivisual sample contribution minus the redundancies introduced by similar samples within the coreset. An efficient gradient-based solver is introduced to enhance the efficiency for the implementation. The method is proven to be effective on multiple coreset selection benchmarks including both vision and language tasks.", "questions": "1. The authors adopt Dinov2 to extract embeddings for the unsupervised scenario. Will the model choice matter? \n2. The authors apply softmax on $\\mathbf{X}$. Will this operation have large influence on the results? How about simple normalize? I cannot see the exact motivation of adopting a softmax here.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed informax solver is interesting, and can be a solution for coreset selection optimization. \n2. The proposed method achieves state-of-the-art performance on multiple benchmarks including both vision and language data. \n3. The presentation of the paper is professional.", "weaknesses": "1. The framework of the proposed Infomax method is reasonable. However, the actual implementation simply adopts the existing submodular and graph cut methods. And there is no comparison with graph cut in the experiment section. Can the authors give more detailed explanation what is the difference of the proposed method from graph cut? Will the proposed gradient-based solver improve the performance over the original graph cut?\n2. The authors propose an efficiency enhancement technique, where the original dataset is divided into several subsets, and only the similarity between neighbors is calculated.\n    - Have the authors compare the efficiency enhancement technique with some existing efficient similarity calculation techniques, e.g., faiss?\n    - The technique is not only applicable to the proposed Infomax method, but also to previous coreset selection methods. The authors claim that the proposed method achieves faster selection compared with D$^2$-Pruning. The comparison seems a little bit unfair. \n3. While efficiency is a major advantage of the proposed method, the authors only provide one group of time cost comparison. As the computation time will be affected by the coreset size, more comparison is expected under different coreset sizes. Futhermore, there is also expected to be the time comparison w/ or w/o each proposed efficient module. \n4. In section 4.4(a), the text part says partition rate d, but in figure 5(a) it says subset size. Although they refer to the same thing, it would still be better to keep these two terms consistent. \n5. The original paper surpasses the limit of 10 pages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)