<figure><img src="image_1.png" /></figure>

HAL
open science

# Data Pruning in Machine Learning Using Influence Function

Mohammad Mahdi Danesh Pajouh

<figure><img src="image_2.png" /></figure>

## To cite this version:

Mohammad Mahdi Danesh Pajouh. Data Pruning in Machine Learning Using Influence Function. 2025. hal-04872248

HAL Id: hal-04872248

https://hal.science/hal-04872248v1

Preprint submitted on 16 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Data Pruning in Machine Learning Using Influence Function

Mohammad Mahdi Danesh Pajouh<fnref n="1" />

School of Mathematics, Statistics and Computer Science of University of Tehran

## Abstract

Data pruning is an essential technique in machine learning, enabling efficient dataset management without compromising model accuracy. This study explores the potential of influence functions in designing novel data pruning algorithms. Two methods are introduced: the first combines data selection and dataset distillation by merging similar data points in feature space, while the second removes low-influence redundant examples. The proposed methods aim to reduce computational costs while maintaining generalization performance. Empirical results demonstrate the effectiveness of the second method, achieving significant dataset size reductions across multiple benchmarks, including CIFAR-10, Cats vs Dogs, and MNIST, with minimal to no loss in accuracy. This paper advances the mathematical foundations of data pruning and highlights the potential for broader adoption of these techniques in real-world applications.

<footnote n="1">¹ mmdaneshpajouh@ut.ac.ir</footnote>
# 1. Introduction

In machine learning, the common perception is that increasing the amount of data reduces test error, as demonstrated by the study [1], showing a power-law relationship between data quantity and error reduction. This principle applies not only to data size but also to model parameters and computational resources. However, not all data points are equally valuable. Some examples—due to noise, mislabeling, or measurement errors—can act as adversarial elements, steering the model away from the natural learning trajectory of clean datasets and ultimately increasing the final model error. In addition, some data points are redundant, meaning that the model could achieve optimal performance without squandering precious resources on these points. Consequently, it becomes essential to remove or manage these detrimental or redundant data points. In this work we focus on the latter application of data pruning i.e. removing redundant data points.

While larger datasets generally enhance model performance, practical constraints such as storage capacity, energy consumption, computational costs, and training time necessitate reducing dataset size. This raises the question of how to balance data volume with quality. Although the power-law relationship suggests that test error $L \approx P^{-v}$ decreases with increasing data size $P$, the small value of $v$ means that significant data expansion is required for even marginal improvements in performance. This makes the case for effective data pruning methods that retain critical information while discarding irrelevant or harmful data.

The influence function, a classical tool in robust statistics introduced in paper [2], offers a mathematical framework to address this challenge. Influence functions analyze how infinitesimal changes in a data point's weight affect model parameters, providing closed-form solutions for quantifying the impact of various modifications to training data. Despite their extensive use in statistics, influence functions have seen limited application in machine learning. They have, however, been successfully employed in understanding model behavior, debugging, detecting mislabeled data, and generating adversarial training examples.

This paper builds on the foundational capabilities of influence functions to introduce two novel data pruning methods. The first method synthesizes new data points by merging similar examples, leveraging influence functions to quantify the impact of the merging process. The second method removes redundant, low-influence data points, effectively
balancing dataset size and information retention. Both methods aim to address the dual goals of computational efficiency and robust model performance.

By combining theoretical rigor with practical experimentation, this work aims to contribute to the evolving field of data pruning, providing both a foundation for future research and a practical tool for real-world machine learning tasks. Empirical results demonstrate that our second method, which removes redundant, low-influence data points, achieves significant dataset reductions across CIFAR-10, Cats vs Dogs, and MNIST datasets with minimal or no accuracy loss. This work highlights the critical importance of both theoretical rigor and practical implementation in advancing the field of data pruning.

## 2. Related Works

Data pruning methods can be broadly categorized into data selection and dataset distillation approaches, each addressing the challenge of optimizing datasets in distinct ways.

**Data Selection Methods:** Data selection methods aim to identify and retain the most informative subset of data points while discarding redundant or noisy examples. These methods leverage the concept that not all data points contribute equally to model training. Some data points, often referred to as "easy" examples, are well-represented in the feature space and require minimal effort for the model to learn. Conversely, "hard" examples provide unique information that enhances generalization but may also introduce noise if too challenging. Effective data selection balances the retention of easy and hard examples to optimize model performance. Some of the main approaches are:

* **Forgetting Scores:** Forgetting score introduced in paper [3] measure how frequently a data point is forgotten during training. Forgettable examples, which oscillate between being correctly and incorrectly predicted, are often more crucial for improving model generalization. However, examples with excessively high forgetting scores may indicate noisy or mislabeled data, making them less valuable. By identifying and retaining moderately forgettable examples, data selection methods can reduce redundancy while preserving critical information.
* **Gradient Norm and Error Norm:** These metrics both introduced in [4] evaluate the importance of data points by analyzing their gradients or error magnitudes during training. Points with smaller gradients or lower errors are deemed less impactful and are candidates for removal. These methods are computationally efficient and have shown success in various classification tasks.

* **Influence Functions:** Influence functions which were adopted for dataset pruning in machine learning by the authors of [5], quantify the impact of individual data points on the model's parameters and predictions. By estimating how model parameters would change if a specific data point were removed, influence functions enable the identification of high-impact examples. By calculating influence value for all data points and forming a discrete optimization problem, they remove the largest possible subset of data that doesn't affect the generalization capabilities of the model. This approach is grounded in a well-defined mathematical framework, making it highly reliable for data selection tasks. Influence functions are particularly effective in detecting outliers, mislabeled data, and critical training examples that significantly affect model performance.

* **Decision Boundary:** Samples near the decision boundary are inherently harder to classify and provide valuable information for model training. While the exact decision boundary is often unknown, authors of [6] approximates it in the feature space by slightly perturbing samples until the model's predicted label changes. Data requiring minimal perturbation to alter their labels are identified as being closer to the decision boundary, making them candidates for preservation as critical examples.

* **Memorization:** The authors of [7] introduce a "memorization criterion" to evaluate the importance of individual data points. This criterion measures the increase in the likelihood of a correct label when a sample is included in the dataset compared to when it is excluded. Larger increases indicate that the sample must be "memorized" by the model, as other data points in the dataset are insufficient to represent it adequately. Such data points are retained to ensure the model's generalization and accuracy.

* **Error-Based method:** The paper [8] presents an error-based pruning technique for removing redundant data. Initially, the dataset is randomly split into a training set and an unused set (initially empty). The training set is further divided into training and validation subsets (80% and 20%, respectively) in each iteration. Data points in the validation set with the smallest prediction error are removed, while the remaining examples are returned to the training set. This process is repeated iteratively to eliminate redundant or less informative examples, optimizing the dataset for efficiency.
**Dataset Distillation Methods:** Dataset distillation techniques synthesize a smaller, highly informative dataset from the original data. Unlike data selection, the distilled dataset need not consist of actual examples from the original set. Key approaches include:

* **Gradient Matching:** This method used by [9], generates synthetic data by matching the gradients of a model trained on the distilled data to those trained on the original data. This ensures the distilled dataset encapsulates the critical information for training.

* **Distribution Matching:** In this method utilized by [10], the goal is to create a synthetic dataset whose data distribution closely approximates the original dataset's distribution.

* **Training Trajectory Alignment:** This approach proposed by [11], aligns the training trajectory of models trained on distilled data with those trained on the original data by trying to minimize the distance between the resulting parameters of the two models. It has shown promise in preserving generalization performance while significantly reducing dataset size.

These methods offer complementary benefits. Data selection is computationally less intensive and retains real data samples, making it suitable for applications requiring interpretable data. In contrast, dataset distillation provides greater flexibility by synthesizing new data, enabling additional applications like privacy preservation and transfer learning. Our proposed methods lean more towards data selection methods using influence value as a metric to weigh the importance of each example.

## 3. Methodology

This section presents two novel data pruning methods designed to optimize dataset size without compromising model accuracy. Both methods leverage influence functions and feature similarity, with differing strategies for handling redundant or low-impact data points. The pruning is done in the context of a supervised classification task.
1. **Data Pruning by Merging Data:** This algorithm builds upon the works of [5, 12] and operates at the intersection of data selection and dataset distillation. Its goal is to merge similar data points to create new, synthesized examples that retain the essential characteristics of the original dataset while reducing redundancy. The steps are as follows:

Assume we have dataset $D = \{z_1, z_2, ..., z_n\}$ of size n. The algorithm below demonstrates how data pruning is done.

1. For each class if two samples $z_i, z_j$ are close enough in feature space, choose them for merging. Call this pair $m_i$.

2. Create artificial sample $a_i$ by merging $z_i$ and $z_j$.

3. Initialize S as an empty set.

4. Calculate the Influence Function for removing $m_i$ and adding the sample $a_i$ and insert it into S as $s_i$.

5. Initialize $W \in \{0,1\}^n$

6. Form the following discrete optimization problem and solve it:

$$ \begin{array}{ll} \text{maximize} & \displaystyle\sum_{i=1}^{n} W_i \\ \text{subject to} & ||W^T S||_2 \le \rho \\ & W \in \{0,1\}^n \end{array} $$

7. For $i = 1$ to $n$ if $w_i = 1$, then remove the pair $m_i$ and add $a_i$ to the dataset.

In order to start the process of pruning, we first need to find pairs of samples that are close to each other in the feature space. To do this, we first calculate the embedding of each data point using a convolutional autoencoder and if two samples are close enough in feature space then we form $m_i$. More concretely, $m_i$ is formed only if $||z_i - z_j||_2 \le \sigma$ in which $\sigma$ is a hyperparameter. Here we may choose the closest data point to $z_i$ or the first data point which satisfies the above inequality. We then merge the similar data points.
**Merging Methods:** Multiple strategies are employed for merging similar data points:

* **Simple Averaging:** Compute the mean of feature vectors from both data points.

* **Weighted Averaging:** Assign weights based on the influence values to generate the merged data point.

* **Hybrid Images:** In this approach, the merging process involves transforming both data points into the frequency domain. Low-pass filters are applied to one data point's Fourier domain, while high-pass filters are applied to the other's. The results are combined to produce a hybrid image. From a distance, this image resembles the low-pass filtered data, while close up, it appears similar to the high-pass filtered one. Neural networks tend to prioritize learning low-frequency components, so the image with the greater influence value is used for the low-pass filter.

* **Decoders:** Using a pre-trained convolutional autoencoder, the images are reduced to a 784-dimensional representation, significantly compressing the original dimensions. To merge data, the mean of these encoded representations is calculated and passed through the decoder to reconstruct a high-quality, merged image.

Afterwards, we create an empty array (later when the array is formed we will look at it as a vector of values) to hold the values coming from Influence Function.

To calculate influence value, consider a sample z is weighted by $\epsilon$ in training. The optimized parameters of the model can be written as:

$$ \theta_{\delta,z} = \operatorname{argmin}_{\theta} \frac{1}{n} \sum_{z_i \in D} L(z_i, \theta) + \epsilon L(z, \theta) \quad (1) $$

Putting $\epsilon = -\frac{1}{n}$ is the same as removing the sample z. Now the influence of weighting z on parameters $\theta$ is:

$$ I_{\text{param}}(z) = \frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon} \Big|_{\epsilon=0} = -H_{\hat{\theta}}^{-1} \nabla_{\theta} L(z, \hat{\theta}) \quad (2) $$
In which $H_{\hat{\theta}} = \frac{1}{n} \sum_{z_i \in D} \nabla_{\hat{\theta}}^2 L(z_i, \hat{\theta})$ is the Hessian matrix and by assumption positive definite. For proof refer to [5].

We can linearly approximate the change in parameters due to removing z by computing:

$$ \theta_z - \hat{\theta} = -\frac{1}{n} I_{\text{params}}(z) \qquad (3) $$

Now by using chain rule we find the influence of weighting z on loss (which is a function of $\hat{\theta}$) at a test point. By definition we have:

$$ \begin{aligned} I_{\text{param,loss}}(z, z_{\text{test}}) &= \frac{dL(z_{\text{test}}, \hat{\theta}_{\epsilon,z})}{d\epsilon} \Big|_{\epsilon=0} \\ &= \nabla_{\theta}L(z_{\text{test}}, \hat{\theta})^T \frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon} \Big|_{\epsilon=0} \qquad (4) \\ &= -\nabla_{\theta}L(z_{\text{test}}, \hat{\theta})^T H_{\hat{\theta}}^{-1} \nabla_{\theta}L(z, \hat{\theta}) \end{aligned} $$

Now assume that instead of weighting the sample z we perturb it a little. For a training point z define $z_{\delta} = (X + \delta, y)$ in which X is the vector of features. Let $\hat{\theta}_{z_{\delta},-z}$ be the optimized parameters of a model trained on the training samples with $z_{\delta}$ in place of z. To approximate its affect define:

$$ \hat{\theta}_{\epsilon,z_{\delta},-z} = \operatorname*{argmin}_{\theta} \frac{1}{n} \sum_{z_i \in D} L(z_i, \theta) + \epsilon L(z_{\delta}, \theta) - \epsilon L(z, \theta) \qquad (5) $$

Which means we are moving $\epsilon$ mass from z to $z_{\delta}$. By similar calculations we have:

$$ \begin{aligned} I_{\text{pert}}(z) &= \frac{d\hat{\theta}_{\epsilon,z_{\delta},-z}}{d\epsilon} \Big|_{\epsilon=0} = I_{\text{params}(z_{\delta})} - I_{\text{params}(z)} \\ &= -H_{\hat{\theta}}^{-1}(\nabla_{\theta}L(z_{\delta}, \hat{\theta}) - \nabla_{\theta}L(z, \hat{\theta})) \qquad (6) \\ &\approx -H_{\hat{\theta}}^{-1}(\nabla_{x}\nabla_{\theta}L(z, \hat{\theta}))\delta \end{aligned} $$

The last equality comes from [12]. As a result we have

$$ \theta_{z_{\delta},-z} - \hat{\theta} = \frac{1}{n} H_{\hat{\theta}}^{-1}(\nabla_{x}\nabla_{\theta}L(z, \hat{\theta}))\delta \qquad (7) $$
Now by differentiating w.r.t $\delta$ and utilizing chain rule we have:

$$ 
\begin{aligned}
I_{pert,loss}(z, z_{test}) &= \nabla_{\delta} L(z_{test}, \hat{\theta}_{z_{\delta}, -z}) \Big|_{\delta=0} \\
&= -\nabla_{\theta} L(z_{test}, \hat{\theta})^T H_{\hat{\theta}}^{-1} (\nabla_x \nabla_{\theta} L(z, \hat{\theta}))
\end{aligned}
\quad (8) $$

Using the equation above we can approximate the effect of perturbing $z \to z_{\delta}$ on the loss which is:

$$ I_{pert,loss}(z, z_{test})\delta $$

Now to tailor everything to our problem we first note that this is our optimized parameters:

$$ \theta_{\delta,z_{\delta},-z_i,-z_j} = \operatorname{argmin}_{\theta} \frac{1}{n} \sum_{z_i \in D} L(z_i, \theta) + \epsilon L(z_{\delta}, \theta) - \epsilon L(z_i, \theta) - \epsilon L(z_j, \theta) \quad (9) $$

What we did here is that we represented merging $z_i, z_j$ by removing $z_j$ and perturbing $z_i \to z_{i\delta}$ i.e. instead of removing both data points and adding their merged value as artificial data point, we removed $z_j$ and moved the other one to the merged value.

Analogous to previous calculations we have:

$$ 
\begin{aligned}
I_{tot}(z_i, z_j) &= \frac{d\hat{\theta}_{\epsilon,z_{\delta},-z_i,-z_j}}{d\epsilon} \Big|_{\epsilon=0} = (I_{params}(z_{\delta}) - I_{params}(z_i)) - I_{params}(z_j) \\
&\approx -H_{\hat{\theta}}^{-1}[\nabla_x \nabla_{\theta} L(z_i, \hat{\theta})\delta + \nabla_{\theta} L(z_j, \hat{\theta})]
\end{aligned}
\quad (10) $$

So we can say the parameter change due to removing $z_i, z_j$ and adding $z_{\delta}$ is:

$$ \theta_{z_{\delta},-z_i,-z_j} - \hat{\theta} = \frac{1}{n} H_{\hat{\theta}}^{-1} [\nabla_x \nabla_{\theta} L(z, \hat{\theta}) \delta + \nabla_{\theta} L(z, \hat{\theta})] \quad (11) $$

Now to examine the influence of both actions (perturbing $z_i$ to $z_{\delta}$ and removing $z_j$) on the loss function we have:

$$ 
\begin{aligned}
I_{tot,loss} &= I_{params,loss} + I_{perturb,loss} \\
&= -\nabla_{\theta} L(z_{test}, \hat{\theta})^T H_{\hat{\theta}}^{-1} \nabla_{\theta} L(z, \hat{\theta}) - \nabla_{\theta} L(z_{test}, \hat{\theta})^T H_{\hat{\theta}}^{-1} (\nabla_x \nabla_{\theta} L(z, \hat{\theta})) \quad (12) \\
&= -\nabla_{\theta} L(z_{test}, \hat{\theta})^T H_{\hat{\theta}}^{-1} [\nabla_{\theta} L(z, \hat{\theta}) + \nabla_x \nabla_{\theta} L(z, \hat{\theta})]
\end{aligned}
$$
## Generalization Analysis

In this section which was inspired by [5], we try to find an upper bound for the loss in generalization caused by data pruning. The expected test loss over the data distribution $P(D)$ is $E_{z_{test} \sim P(D)} L(z_{test}, \theta)$. The difference in generalization due to removing part of the dataset in the process of merging –for which an upper bound is going to be found- is:

$$|L(\hat{\theta}_{-\hat{D},D_{\delta}}) - L(\hat{\theta})|$$

We have the following result:

**Theorem:** Assume that the original dataset is $D$ of size $n$, the pruned dataset is $\hat{D} = \{z_i\}_{i=1}^m$ and the artificial data is in $D_{\delta}$. If we have

$$\begin{gathered} ||W^T S||_2 \le \rho \ \& \\ ||z_i - z_j||_2 \le \sigma \text{ for each } i, j \end{gathered}$$

then the upper bound of the difference in generalization is:

$$O(\rho + \frac{m}{n^2} + \frac{\sqrt{m\sigma}}{n} + \sigma^2)$$

## Proof:

By first-order Taylor approximation we have:

$$L(z_{test}, f(\hat{\theta}_{\epsilon, -\hat{D}, D_{\delta}})) = L(z_{test}, f(\hat{\theta})) + \frac{dL(z_{test}, f(\hat{\theta}))}{d\epsilon}\epsilon + \frac{dL(z_{test}, f(\hat{\theta}))}{d\delta}\delta + O(||\epsilon||^2 + ||\epsilon\delta|| + ||\delta||^2)$$

We can omit the last term because $||\epsilon||^2 = \frac{m}{n^2}$ is very small for many datasets. $||\epsilon\sigma||$ is also small since $||z_i - z_j||_2 \le \sigma$ and thus $\delta \le \sigma$. We usually choose a small $\sigma$ because we want to merge close data points. $||\delta||^2$ is also very small for the same reason. Now we have:
$$
\begin{align*}
& \rightarrow L(\hat{\theta}_{-\hat{D}, \hat{D}_{\delta}}) - L(\hat{\theta}) \approx E_{z_{test} \sim P(D)}[I_{params,loss}(z, z_{test})\epsilon + I_{pert,loss}(z, z_{test})\delta] \\
&= \sum_{z_i \in \hat{D}} -\frac{1}{n} E_{z_{test} \sim P(D)}[I_{params,loss}(z_{test}, z_i)] + \sum_{z_i \in \hat{D}_{\delta}} \delta_i E_{z_{test} \sim P(D)}[I_{pert,loss}(z_{test}, z_i)] \\
&= -\frac{1}{n} E_{z_{test} \sim P(D)}[-\nabla_{\theta} L(z_{test}, \hat{\theta})^T][\sum_{z_i \in \hat{D}} H_{\hat{\theta}}^{-1} \nabla_{\theta} L(z, \hat{\theta})] \\
&\quad + E_{z_{test} \sim P(D)}[-\nabla_{\theta} L(z_{test}, \hat{\theta})^T][\sum_{z_i \in \hat{D}_{\delta}} \delta_i H_{\hat{\theta}}^{-1} (\nabla_x \nabla_{\theta} L(z, \hat{\theta}))] \\
&\le \frac{1}{n} \|\nabla_{\theta} L(\hat{\theta})\|_2 \left\|\sum_{z_i \in \hat{D}} I_{params}(z_i)\right\|_2 + \left\|\nabla_{\theta} L(\hat{\theta})\right\|_2 \left\|\sum_{z_i \in \hat{D}_{\delta}} I_{pert}(z_i)\right\|_2 \\
&\le \frac{\rho}{n} + \rho + \epsilon^2 + \epsilon\delta + \delta^2 \\
&\le \frac{\rho}{n} + \rho + \frac{m}{n^2} + \frac{\sqrt{m\sigma}}{n} + \sigma^2
\end{align*}
$$

The first inequality is Cauchy-Schwarz inequality. The second inequality comes from the fact that
$$ \left\| \sum_{z_i \in \hat{D}} I_{\text{params}}(z_i) \right\|_2 \le I_{\text{tot}}, \left\| \sum_{z_i \in \hat{D}_{\delta}} I_{\text{pert}}(z_i) \right\|_2 \le I_{\text{tot}} \text{ and } I_{\text{tot}} \le \rho \text{ thus each term is smaller than } \rho. $$
This concludes the proof. □

**2. Data Pruning by removing pairwise redundant samples:** The second algorithm operates by finding similar data points in feature space and removing the less influential point. Assume we have dataset $D = \{z_1, z_2, ..., z_n\}$ of size n.

The algorithm below demonstrates how data pruning is done.

1. For each class if two samples $z_i, z_j$ are close enough in feature space, choose them for next stage.

2. Calculate influence value for both data points.

3. Remove the data point with smaller influence value only if its influence value is below a threshold T.
In the above algorithm, like the previous method we first find similar data points by comparing their embedding which is obtained from a pretrained autoencoder. Afterwards, for each pair of similar data points we calculate influence value using the formula below:

$$I_{\text{param}}(z) = \frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon} \Big|_{\epsilon=0} = -H_{\hat{\theta}}^{-1}\nabla_{\theta}L(z, \hat{\theta})$$

Note that it doesn't have to be pairs of data points. However, comparing more data points to each other adds to computation overhead. We can also cluster data points and only select the most influential data point of each cluster, however, for simplicity we chose to do pairwise comparison.

A data point with a smaller influence value plays a lesser role in adjusting weights during the training process and therefore has a smaller impact on the final result when removed. However, it should also be noted that both data points in a pair may have high influence values. Consequently, in addition to comparing the influence values of the two data points, the smaller value is also compared against a threshold T, which is a hyperparameter of the method. If this value is smaller than the threshold, the corresponding data point is removed from the dataset.

# 4. Experiments and Results

## Selected Datasets

Since many applications of this pruning method are for image datasets, the datasets used for evaluation include *Cats vs Dogs* [13], CIFAR10 [14] and MNIST [15]. The *Cats vs Dogs* dataset consists of two classes: cats and dogs, and is commonly used in competitions. This dataset contains 25,000 images. The CIFAR10 dataset consists of 10 classes, each containing 6,000 images. The MNIST dataset comprises handwritten digits from 0 to 9, totaling 60,000 training images and 10,000 testing images. The reason for using *Cats vs Dogs* and MNIST datasets is to complement the primary CIFAR10 dataset. Initially, the algorithm was designed and tested on CIFAR10, and later tested further on the *Cats vs Dogs* and MNIST datasets to ensure it works on various datasets.
## Libraries Used in Implementation

The main library used for implementation is TensorFlow. It was utilized to build neural networks and calculate gradients using the gradient function. Furthermore, CIFAR10, *Cats vs Dogs*, and MNIST datasets can easily be loaded as prebuilt datasets from this library. For influence function computation, the *InfluenceAI* library [16] which is built on TensorFlow, was employed. The *puLP* library [17] was used to solve the discrete optimization problem.

## Network Architecture

The base network is ResNet50 with ImageNet weights. The last fully connected layer is removed, and two fully connected layers are added at the end, with 32 and 10 neurons for CIFAR10 and MNIST, and 32 and 2 neurons for *Cats vs Dogs*. Similar to the work in [5], for simplicity in influence function calculations, the influence of removing data points is evaluated only on the weights of the last layer. The Hessian approximation uses 1,000 training samples. For the encoder, the autoencoder network comprises three convolutional layers and two fully connected layers. The decoder includes four convolutional layers. The autoencoder model was trained using 8,000 images. For MNIST, the input image dimensions are 56×56 pixels with three color channels. The input image dimensions for the other two datasets are 224×224 pixels. Seventy percent of the data is used for training, and the remainder is used for test accuracy evaluation.

## Implementation Details

Due to limited random access memory (RAM) and storage, as well as the time-intensive nature of the training process, the code was divided into multiple branches and sub-branches. Each sub-branch splits the dataset into smaller sections, which are loaded into memory for training. Finally, the results of each sub-branch are saved for future use.

## Autoencoder Results

Since the encoding process is shared between both algorithms, the autoencoder results are presented before detailing the algorithm outcomes. Multiple architectures were tested for this section. Initially, only fully connected layers were used, but their results were too poor to be usable. Convolutional layers with various filters were then tested, leading to the current structure, which produces acceptable reconstructed images that retain the general characteristics of the
original images while reducing the size of the encoded representation by 192-fold. Some of these images are shown in Figure 1. For simplicity, the encoded representation is also visualized as an image.

<figure><img src="image_4.png" /><figcaption>Figure 1 original, code and reconstructed image by autoencoder</figcaption></figure>
## Results of the First Algorithm

First, we analyze and evaluate the first algorithm. The results showed that the model could not effectively learn from the merged images. Specifically, the accuracy of a model trained solely on merged images was no better than random chance. Similarly, a model trained on both merged and remaining images also failed to achieve satisfactory accuracy, as the synthesized dataset did not serve as a good representation of the original dataset. Testing various merging methods did not yield significant improvements, leading to the proposal of the second algorithm.

Two factors may be the primary reasons for the limitations of this algorithm in its current state.

1. Finding Similar Data Points: A critical step in this method is identifying similar data points. There are various approaches to obtaining appropriate representations of data in the feature space, some of which rely on transformer-based autoencoders or other complex architectures. Due to limited computational resources, simpler methods were employed to extract data representations. Improving this step—by enhancing the identification of similar data—might make this pruning algorithm more effective.

2. Weakness in Merging Results: Although various merging strategies were tested, the final merged results were suboptimal. It seems that replacing the discussed merging methods with optimization techniques similar to dataset distillation could significantly improve the algorithm's outcomes, although this would require more computational resources.

Thus, while this algorithm did not achieve the desired results, the issue likely lies not in the core methodology itself but in the tools and techniques used in its implementation, which were constrained by computational limitations.

## Results of the Second Algorithm

In the second algorithm, similar to the first one, data points that are close to each other are identified by examining the distance between their codes. If $C_1, C_2$ are the code vectors of the first and second images, respectively, with a length of $n$, the distance is calculated using the following formula:

$$ \sqrt{\sum_{i=0}^{n} (C_{1i} - C_{2i})^2} $$
Next, for each of the two data points, the influence value is calculated. The one with the smaller influence value is selected for removal, provided that its influence value is smaller than the threshold T, which is one of the hyperparameters of the method. The new dataset is then formed by removing these data points.

The algorithm includes two hyperparameters $\sigma, T$. The parameter $\sigma$ determines how close two images need to be in feature space, and the threshold $T$ specifies the maximum allowable influence value for a data point to be removed. By adjusting these two hyperparameters, different pruning percentages can be achieved. For example, for the CIFAR10 dataset, the following hyperparameter values result in 40% pruning of the dataset:

$$ \sigma = 0.15, T = 50 $$

Using this algorithm with the specified settings, out of 35,000 training images, 21,073 images remained, meaning 40% of the dataset was pruned. The accuracy of the model trained on the original dataset was approximately 90.65%, and with 40% of the dataset removed using this algorithm, the accuracy dropped to 89.59%. This demonstrates that the algorithm effectively retained important data points in the dataset while removing redundant, less impactful data points.

For the Cats vs Dogs dataset, the hyperparameter values used were:

$$ \sigma = 0.15, T = 15 $$

Using the redundant low-influence data removal method, 9,128 images remained out of the initial 16,283, meaning 44% of the dataset was pruned. The model's accuracy on the full dataset was 97.69%, while its accuracy on the pruned dataset was 97.39%. This demonstrates that the algorithm performed effectively, retaining important data points while removing redundant, less impactful ones.

For the MNIST dataset, the hyperparameter values used were:

$$ \sigma = 0.85, T = 1.5 $$
Using this algorithm, 41,950 images remained out of the initial 60,000, meaning approximately 30% of the dataset was pruned. The model's accuracy on the full dataset was 98.35%, which, after pruning, not only did not decrease but improved to 99.07%.

<figure><img src="image_5.png" /><figcaption>Figure 2 accuracy comparison between models trained on original vs the pruned dataset on CIFAR10, Cats vs Dogs & MNIST</figcaption></figure>

## Summary of Results

Table 1 summarizes the results, comparing the ResNet50 model accuracy on the full and pruned datasets across three datasets:

<table><thead><tr><th rowspan="2">Dataset</th><th>Accuracy on Pruned Dataset</th><th>Accuracy on Full Dataset</th><th>Data Count After Pruning</th><th>Full Dataset Count</th></tr><tr><th>(%)</th><th>(%)</th><th></th><th></th></tr></thead><tbody><tr><td>CIFAR-10</td><td>89.59</td><td>90.65</td><td>21,073</td><td>35,000</td></tr><tr><td>Cats vs Dogs</td><td>97.39</td><td>97.69</td><td>9,128</td><td>16,283</td></tr><tr><td>MNIST</td><td>99.07</td><td>98.35</td><td>41,950</td><td>60,000</td></tr></tbody></table>
## Comparison with Other Methods

Table 2 compares the accuracy drop after pruning 40% of the data using our method and other approaches:

<table><thead><tr><td>Method</td><td>Accuracy Drop After 40% Pruning (%)</td></tr></thead><tbody><tr><td>Forgetting Score</td><td>1.6</td></tr><tr><td>Influence Function with Optimization</td><td>1.2</td></tr><tr><td>Influence Function</td><td>3.3</td></tr><tr><td>Gradient Norm</td><td>3.2</td></tr><tr><td>Error Norm</td><td>1.5</td></tr><tr><td>Random Pruning</td><td>3.3</td></tr><tr><td>Our Method</td><td>1.06</td></tr></tbody></table>

It is also worth noting that our method was executed with only three epochs, whereas other approaches in the literature often use a greater number of epochs. Additionally, unlike other methods, our initialization was not performed with random weights.

## Additional Experiments with CIFAR-10

Another set of experiments were conducted using the CIFAR-10 dataset resized to dimensions of 56×56. This experiment is significant because ImageNet weights are based on 224×224 image dimensions, making transfer learning more challenging. The goal was to examine whether accuracy loss after pruning differed in this scenario.

In Figure 3, the accuracy versus pruning percentage for CIFAR-10 is shown. The results indicate that accuracy decreases almost linearly. After pruning 42% of the dataset, the accuracy dropped by only 2.55%, which still outperformed many other methods.
<figure><img src="image_6.png" /><figcaption>Figure 3 accuracy-pruning ratio chart on CIFAR10 dataset</figcaption></figure>

## 5. Conclusion

Broadly speaking, data pruning methods were divided into two categories: data selection and dataset distillation. Data selection methods rank data points using a specific criterion and then remove those with low scores. Essentially, in these methods, the pruned dataset is a subset of the original dataset. On the other hand, dataset distillation methods synthesize new data with high informational value using various approaches. The output of these methods is a synthesized dataset that fundamentally differs from the original dataset.

Subsequently, two novel data pruning methods were proposed. The first method, which merges similar data points, lies at the intersection of data selection and dataset distillation. The essence of this method involves identifying similar data points in the feature space and merging them if the influence value of the merging process is below a specified threshold. Merging was performed using methods such as simple and weighted averaging, hybrid images, and decoders. This method was analyzed in detail, and an upper bound for generalization loss was provided. Although this method was mathematically robust, it did not achieve satisfactory accuracy in experiments. This is likely due to the low informational value of the merged data points.
The second method, which removes low-influence similar data points, is also based on the mathematics of influence functions. Similar to the first method, it first identifies similar data points. Then, for each pair of similar data, the data point with the smaller influence value is removed, provided that its influence value is below a specified threshold. The results demonstrated the effectiveness of this method, as it successfully pruned about 40% of the dataset without any loss of accuracy. The good performance of this method highlights the substantial amount of redundant, semantically similar information in existing datasets.

# 6. Future Directions

Although numerous data pruning methods exist today, it is evident that we are still at the early stages of development. While it must be acknowledged that current methods perform very well in preserving model performance, their significant computational demands pose a challenge to widespread adoption.

A major limitation of many data pruning methods is the lack of mathematical guarantees for the final quality of the pruned dataset's generalization. While the results are promising, it is difficult to predict the extent of generalization loss if the pruned dataset is used with a model different from the one used for pruning, or if the pruning percentage is increased or decreased. Solving this issue requires not only advancing the underlying mathematical theories but also conducting extensive experiments to empirically approximate the generalization performance of pruned datasets under various conditions.

Another challenge is the lack of familiarity with data pruning within the computer science community. The prevailing belief remains that more data equals better performance, and pruned datasets have yet to gain widespread acceptance. The next step in this field is to build trust within the computer science community and expand the use of pruned datasets. One potential avenue is developing unsupervised pruning methods, which would make the output datasets less dependent on specific tasks. This means that pruned datasets could be used in a variety of applications, such as classification, object detection, and clustering.
Acknowledgment

This research was conducted as part of a master's thesis at the University of Tehran under the supervision of Dr. Hedieh Sajedi. Additionally, the authors are grateful to the School of Mathematics, Statistics, and Computer Science at the University of Tehran for providing the resources and environment necessary for the completion of this study.

During the preparation of this work the authors used ChatGPT in order to translate and summarize sections of the source thesis. After using this tool, the authors reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.

References

1. Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., & Morcos, A. (2022). Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35, 19523-19536.

2. Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., and Stahel, W. A. Robust Statistics: The Approach Based on Influence Functions. Wiley, 1986

3. Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. ICLR, 2019b.
4. Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34, 2021.

5. Yang, S., Xie, Z., Peng, H., Xu, M., Sun, M., & Li, P. (2023). Dataset pruning: Reducing training data by examining generalization influence. ICLR 2023.

6. Guo, C., Zhao, B., & Bai, Y. (2022). DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning. *International Conference on Database and Expert Systems Applications*.

7. Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Adv. Neural Inf. Process. Syst., 33:2881–2891, 2020

8. Li, K., Persaud, D., Choudhary, K. et al. Exploiting redundancy in large materials datasets for efficient machine learning with less data. Nat Commun 14, 7283 (2023).

9. B. Zhao, K. R. Mopuri, and H. Bilen, “Dataset condensation with gradient matching.” ICLR, vol. 1, no. 2, p. 3, 2021.
10. B. Zhao and H. Bilen, "Dataset condensation with distribution matching," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.

11. G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu, "Dataset distillation by matching training trajectories," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.

12. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885–1894. PMLR, 2017a.

13. Microsoft. (n.d.). *Microsoft Research - Cat and Dog Dataset*. Retrieved from
[https://www.microsoft.com/en-us/download/details.aspx?id=54765](https://www.microsoft.com/en-us/download/details.aspx?id=54765)

14. Krizhevsky, A. (2009). *Learning multiple layers of features from tiny images* [Dataset].
Canadian Institute for Advanced Research. Retrieved from
[https://www.cs.toronto.edu/~kriz/cifar.html](https://www.cs.toronto.edu/~kriz/cifar.html)

15. LeCun, Y., Cortes, C., & Burges, C. J. C. (2010). *The MNIST database of handwritten digits* [Dataset]. Retrieved from [http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/).
16. Picard, A. M., Hervier, L., Fel, T., & Vigouroux, D. (2023, November). *Influenciæ: A library for tracing the influence back to the data points* [Preprint]. HAL. Retrieved from https://hal.science/hal-04284178

17. Roy, J. S., Mitchell, S. A., & Peschiera, F. (n.d.). *PuLP: A linear programming toolkit for Python* [Software]. Retrieved from https://pypi.org/project/PuLP/