SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Retaining Beneficial Information from Detrimental Data for Deep Neural Network Repair
Long-Kai Huang $^{1}$ , Peilin Zhao $^{1}$ , Junzhou Huang $^{2}$ , Sinno Jialin Pan $^{3}$
Tencent AI Lab
2The University of Texas at Arlington
<sup>3</sup>The Chinese University of Hong Kong
hlongkai@gmail.com, masonzhao@tencent.com, jzhuang@uta.edu, sinnopan@cuhk.edu.hk
The performance of deep learning models heavily relies on the quality of the training data. Inadequacies in the training data, such as corrupt input or noisy labels, can lead to the failure of model generalization. Recent studies propose repairing the model by identifying the training samples that contribute to the failure and removing their influence from the model. However, it is important to note that the identified data may contain both beneficial and detrimental information. Simply erasing the information of the identified data from the model can have a negative impact on its performance, especially when accurate data is mistakenly identified as detrimental and removed. To overcome this challenge, we propose a novel approach that leverages the knowledge obtained from a retained clean set. Concretely, Our method first identifies harmful data by utilizing the clean set, then separates the beneficial and detrimental information within the identified data. Finally, we utilize the extracted beneficial information to enhance the model's performance. Through empirical evaluations, we demonstrate that our method outperforms baseline approaches in both identifying harmful data and rectifying model failures. Particularly in scenarios where identification is challenging and a significant amount of benign data is involved, our method improves performance while the baselines deteriorate due to the erroneous removal of beneficial information.
The effectiveness of deep neural networks (DNNs) is greatly influenced by the quantity and quality of the training data from which they are trained. However, obtaining a large amount of high-quality data can be costly. An alternative approach is to acquire training data from sources such as the Web [2], external data suppliers, or crowdsourcing [14]. Although this allows for the collection of a substantial volume of data, the quality of the acquired data cannot be guaranteed. Various issues can arise, including annotation errors [3], data corruption [13], or even deliberate adversarial manipulation [39] by malicious providers. Consequently, a portion of the collected data may have a different distribution than the majority of the data [21]. In this work, we refer to these two categories of data as "detrimental" and "target" data, respectively. The presence of detrimental data in the training set can significantly hinder the performance of a DNN [1]. While recent research has proposed methods to mitigate specific issues like label noise [31, 5, 11], adversarial attacks [28, 39], or data shifts during model training [21, 32], it remains challenging to train a model that generalizes well to target data when the specific cause of the detrimental data is unknown.
It is important to note that in practical scenarios, a DNN model must undergo a quality assurance test before it can be deployed. During this evaluation process, the performance of the trained DNN model is usually assessed using a small reserved set of data known as clean data. This clean data is expected
to have the same distribution as the target data. However, since the training process involves the use of detrimental data, the trained DNN model may produce inaccurate predictions on the clean data, leading to a failure in the quality assurance test. In such cases, it becomes necessary to repair the DNN model to successfully pass the quality assurance test.
Recent approaches to model repair focus on two main steps. Firstly, they aim to identify the detrimental data instances within the original training dataset with a set of clean data, a.k.a, test data. Once identified, these instances can be categorized for further action. Secondly, the model is updated by eliminating the identified detrimental data instances. This repair process helps in improving the model's performance and ensuring that it meets the desired quality standards for deployment. Notable works in this area include studies by Tanno et al.[33] and Kong et al.[23].
In contrast to previous approaches that focus on eliminating the entire detrimental dataset to learn a repaired DNN model, our approach is motivated by the belief that the detrimental data may contain some valuable underlying information. Once this information can be extracted, it can be leveraged to train a more generalized model alongside the target data. Drawing inspiration from domain adaptation techniques [35, 30], we propose a framework that aligns the identified detrimental data with the clean data. By aligning these datasets, we can utilize the aligned detrimental data to repair the DNN model and enhance its generalization ability, all without the need for complete retraining.
To elaborate, our proposed framework begins with an approach rooted in generalization theory to identify a potential set of detrimental data with the assistance of a clean data set, while simultaneously updating the model by mitigating the influence of the detrimental data during training. Subsequently, we introduce an energy-based algorithm [18] to align the identified detrimental data with the clean data. Through this alignment process, we selectively filter out harmful information that could adversely impact the model's generalization performance, while retaining the beneficial information that can bolster the model's overall generalization capabilities. By adopting this approach, we aim to strike a balance between effectively utilizing the beneficial aspects of the detrimental data and minimizing its negative impact on the model's performance. Our proposed framework offers a lightweight alternative to retraining the model from scratch, enabling the model to learn from both the target and aligned detrimental data, thereby improving its ability to generalize effectively.
It is crucial to acknowledge that identifying a perfect detrimental dataset from the training data is an arduous task, if not an impossible one. Previous approaches may discard useful information if certain target data instances are mistakenly identified as detrimental. In our proposed framework, we mitigate this issue by employing an energy-based alignment algorithm to reduce the divergence between the aligned detrimental data and target distributions. In essence, the alignment process serves as a corrective mechanism for potential errors introduced during the detrimental data identification step. By aligning the identified detrimental dataset with the clean data, our framework can rectify misidentifications and ensure that the model benefits from the valuable information contained within the misidentified instances. By leveraging the alignment process, our framework achieves a higher level of resilience, ensuring that the model's generalization performance remains intact, even in the presence of potential misidentifications. This robustness sets our proposed approach apart from previous methods and reinforces its effectiveness in dealing with the challenges associated with identifying and utilizing detrimental data.
Summary of contribution: We introduce a novel framework that acknowledges the potential value of detrimental data in enhancing the training of a more generalized model alongside target data. Our framework aligns the identified detrimental data with clean data, enabling us to leverage this aligned data to repair the DNN model and improve its generalization ability without the need for complete retraining. This approach strikes a balance between utilizing the beneficial aspects of detrimental data and minimizing its negative impact. By rectifying potential misidentifications through alignment, our framework maximizes the utilization of both target and aligned detrimental data, resulting in a more robust and resilient approach compared to previous methods for addressing the challenges associated with utilizing detrimental data. We conducted extensive experiments on real-world datasets to validate the effectiveness of our proposed framework, demonstrating its superiority over existing methods in identifying harmful data and repairing the model to enhance generalization.
Problem Settings: Assume our goal is to maximize model performance on some target data distribution  $\mathcal{Z}$  and we have trained a prediction model  $\hat{\theta}$  by minimizing the empirical risk  $\mathcal{L}(\mathcal{D};\theta)$  over a training set  $\mathcal{D} = \{z_i\}_{i=1}^N$  consisting of  $N$  data pairs  $z_i = (x_i, y_i)$ . While most of the training data are i.i.d. samples from the same distribution as the target data, a small set of training data is corrupted and assumed to be sampled from a detrimental data distribution  $\mathcal{Z}_c$ . We denote the detrimental data in the training set by  $\mathcal{D}_c$  and the remaining by  $\mathcal{D}_l$ . Notably, the existence of detrimental data is unknown during data collection and model training.
Due to the existence of detrimental data, the model absorbs unexpected adverse information which adversely affects its generalization ability, resulting in inaccurate predictions. In this paper, we focus on developing a model repair algorithm that aims to improve performance by removing the detrimental information from the data that causes prediction failures. To facilitate model repair, we assume the availability of a small set  $\mathcal{D}_r$  of clean data that is reserved for validating the model's success or failure and remains concealed from the model during training. And we identify and remove the detrimental information in the data by leveraging the crucial information from these clean data.
Similar to [33], we formulate the process of model repairment as two steps:
- Step 1: Cause Identification. Given a reserved set  $\mathcal{D}_r$  from the target distribution, identify a set of detrimental data points in the training set  $\mathcal{D}$  that contribute to the prediction failures for data in  $\mathcal{D}_r$ .
- Step 2: Model Treatment. Given the identified failure cause set, denoted by  $\hat{D}_c$ , refine the model to effectively correct the prediction failures in the target distribution while preserving the performance on remaining
2.1 Step 1: Cause Identification
In order to detect the detrimental data points in the training set, it is necessary to first define a measure that quantifies the extent to which a subset of training data contributes to the failed predictions in the reserved set  $\mathcal{D}_r$ . Then, based on this measure, we can screen out the data points with high failure contributions as the identified data.
Before introducing the measure, it is important to note that the data from detrimental dataset  $\mathcal{D}_c$  lead to generalization failure because they introduce a mismatch between train and test distribution. This is aligned with the generalization theory for mixture distributions [17, 38] which asserts that the degree of divergence between the two distributions impacts the generalization error bound. Additionally, the theory demonstrates that an increase in the amount of data from the target distribution during training reduces the generalization bound and decreases the loss of data from the same distribution. Besides, increasing the amount of target distribution data also enables the learned model to fit more closely to the target distribution and less to the detrimental distribution, thereby increasing the loss of detrimental data in  $\mathcal{D}_c$ . Since the reserved set  $\mathcal{D}_r$  is drawn from the target distribution, we can train the target-distribution-enhanced model, denoted  $\hat{\phi}$  on the combined set of  $\mathcal{D}$  and  $\mathcal{D}_r$ , and identify the data from  $\mathcal{D}_c$  based on the change of loss between the original trained model  $\hat{\theta}$  and the enhanced model  $\hat{\phi}$ . The measure is defined as
$$
s (z; \hat {\theta}, \hat {\phi}) = \mathcal {L} (z; \hat {\phi}) - \mathcal {L} (z; \hat {\theta}). \tag {1}
$$
By estimating  $s(z; \hat{\theta}, \hat{\phi})$  for each data point  $z \in \mathcal{D}$ , we can identify the data points responsible for the failure, which have a positive value of  $s(z; \hat{\theta}, \hat{\phi})$ . If we know the size of the cause set, we can identify the cause data by selecting the top  $|\mathcal{D}_c|$  data points with the largest  $s(z; \hat{\theta}, \hat{\phi})$  values. We denote the identified data set by  $\hat{\mathcal{D}}_c$ , and the remaining data set by  $\hat{\mathcal{D}}_l$ .
2.1.1 Obtaining the Enhanced Model  $\hat{\phi}$
Considering the large scale of data, training a new model  $\hat{\phi}$  from scratch is computationally prohibitive. Instead, a practical solution is to approximate the model trained on the combined dataset of  $\mathcal{D}$  and  $\mathcal{D}_r$  by performing fine-tuning or continual learning on the well-trained  $\hat{\theta}$  from  $\mathcal{D}$  using the data in  $\mathcal{D}_r$ . However, due to over-parameterization of the network and the small size of the reserved set  $\mathcal{D}_r$ , a fine-tuned model may simply memorize the data in  $\mathcal{D}_r$  [41], resulting in a negligible impact on the
Figure 1: The framework of the proposed method. Best view in color: beneficial and detrimental information are indicated in green and red.
loss of the data in the original training set  $\mathcal{D}$ . This makes the increase of loss  $s(z; \hat{\theta}, \hat{\phi})$  ineffective in identifying the detrimental data. On the other hand, a fine-tuned model may completely overfit  $\mathcal{D}_r$  while forgetting the knowledge acquired from  $\mathcal{D}$  [27], which can cause it to fail in approximating  $\hat{\phi}$ .
Given that different layers in a neural network capture distinct patterns [40], selectively updating a subset of layers while freezing the remaining layers can update the detrimental knowledge from  $\mathcal{D}_c$  to effective knowledge in  $\mathcal{D}_r$ , while preserving the acquired knowledge from  $\mathcal{D}$ . After updating the conflict knowledge from  $\mathcal{D}_c$ , the new model may result in a more significant increase in loss on  $\mathcal{D}_c$ . Recent studies [27, 15] support the layer-selective update by observing that selectively fine-tuning a subset of layers improves the fine-tuning generalization for different distribution shifts towards the target distribution. And we empirically validate the effectiveness of layer-selective update in acquiring the knowledge from  $\mathcal{D}_r$  and improving the identification accuracy, as presented in Appendix A.4.
To enable layer-selective update, a criterion for layer selection is required. Note that neural networks have a tendency to minimize loss by learning data patterns in the early epochs, but may end up memorizing the data in the later epochs [4, 9]. Therefore, we assume that the reduction in loss after the first epoch can serve as an indicator of the ability of the selected layers to learn data patterns. Based on this assumption, we estimate the loss after performing a one-epoch update for each subset of layers and select the subset with the smallest loss for further updates. Subsequently, the selected subset is updated while the remaining layers are kept frozen. As the reserved set is small, the criterion for layer selection via loss estimation after a one-epoch update is computationally efficient. It should be noted that in practice, we consider only a pre-defined set of layer subsets, rather than all possible combinations of subsets.
Connection to existing works: EWC-influence, as proposed in [33], utilized the Bayesian view of continual learning to identify the cause data. Specifically, it defined the identification metric as the drop in log-likelihood between a model trained on the combined dataset  $\mathcal{D}$  and  $\mathcal{D}_f$ , which is a subset of  $\mathcal{D}_r$  containing examples that are mispredicted by  $\hat{\theta}$ , and the current model  $\hat{\theta}$  trained on  $\mathcal{D}$ . By using negative log-likelihood as the loss function, replacing  $\mathcal{D}_r$  with  $\mathcal{D}_f$ , and applying the continual learning algorithm Elastic Weight Consolidation (EWC) [19] to update all layers, our proposed identification measure recovers the EWC-influence introduced in [33].
Both EWC-influence and the proposed method measure the contribution of a training point to the prediction failure by examining the change in the loss. However, the two measures are derived from different purposes. EWC-influence aims to correct the prediction in  $\mathcal{D}_f$  of using the gain log-likelihood improvement estimated by deleting the data from  $\mathcal{D}$ , whereas the proposed method aims to improve the generalization to the data from the same distribution of  $\mathcal{D}_r$  by eliminating detrimental information from  $\mathcal{D}$ . Consequently, the proposed method proposes to measure the contribution to failure by assessing the conflict between  $\mathcal{D}$  and  $\mathcal{D}_r$  and update the model using all data in  $\mathcal{D}_r$ .
Linear influence [20, 36, 23] is another effective metric to identify the training points that are most responsible for certain predictions. As analyzed in [33], the linear influence is the same as EWC-influence with the exception that it updates the model via a single update of natural gradient descent. Consequently, our proposed method can recover the linear influence in a similar manner.
2.2 Step 2: Model Treatment
In step 1, we identify a partition of the training data  $\hat{\mathcal{D}}_c$  that may contain harmful information leading to a suboptimal performance to the test distribution  $\mathcal{Z}$ . In this section, we present the second step of our method, which involves refining the model using the identified cause set  $\hat{\mathcal{D}}_c$  and the trained model  $\hat{\theta}$  in conjunction with the enhanced model  $\hat{\phi}$ . A straightforward solution for repairing the model is to eliminate the harmful information in  $\hat{\mathcal{D}}_c$  by removing the identified data from the training set by retraining the model using the remaining data [12] or performing machine unlearning [29] on the identified data [33]. However, identifying a perfect detrimental dataset from the training data is difficult. Simply erasing the information of identified data can negatively impact the model's performance, especially when accurate data is mistakenly identified as detrimental and removed. Furthermore, even the detrimental data contain useful information. For example, an incorrectly annotated image contains a clean input. Recall that the generalization failure is caused by the mismatching between the detrimental distribution and target distribution. Inspired by the domain adaptation framework [35, 30], we propose to align the detrimental data with the target distribution, which can reduce the divergence between the training distribution and the target distribution. The aligning process can rectify the detrimental information in the data, making it possible to reuse the aligned data toward better performance. A theoretical analysis of the following Gaussian mean estimation task supports the potential of this approach to improve performance.
Gaussian mean estimation task: Consider the task where the objective is to estimate the mean of a Gaussian distribution  $\mathcal{U} = \mathcal{N}(\mu, \sigma^2)$  based on a training set  $\mathcal{D}$  consisting of  $n$  training data  $\{x_i\}_{i=1}^n$  sampled from the distribution  $\mathcal{U}$  as well as  $m$  training data  $\{x_i\}_{i=n+1}^{n+m}$  sampled from another Gaussian distribution  $\mathcal{U}' = \mathcal{N}(\mu', \sigma^2)$ , where  $\mu$  and  $\mu'$  are the means of the two Gaussian distributions and  $\sigma^2$  denotes the variance. We define the loss function as  $\mathcal{L}(x; \theta) = (\theta - x)^2$ . In this example, we assume  $n$  is sufficiently large than  $m$  and the distribution divergence between  $\mathcal{U}$  and  $\mathcal{U}'$ , measured by  $|\mu - \mu'|$ , is sufficiently large, i.e.,  $|\mu - \mu'| > \frac{m+n}{m} \alpha(\sqrt{1/n} - \sqrt{1/(n+m)})$ , such that the training data from  $\mathcal{U}'$  hurt the generalization performance of the model.
The optimal solution is  $\theta^{*} = \mu$ . The empirical risk minimization solution is obtained by minimizing  $\mathcal{L}(\mathcal{D};\theta) = \frac{1}{n + m}\sum_{i = 1}^{n + m}(\theta -x_i)^2$  as  $\hat{\theta} = \frac{1}{m + n}\sum_{i = 1}^{n + m}x_{i}$ . We measure the generalization error as the difference between the empirical solution  $\hat{\theta}$  and the optimal solution  $\theta^{*}$ . Using Gaussian tail bounds [34], with probability at least  $1 - p$  for some probability, the generalization error is upper bounded as
$$
\left| \hat {\theta} - \theta^ {*} \right| \leq \alpha \sqrt {1 / (n + m)} + \frac {m}{m + n} \left| \mu - \mu^ {\prime} \right| := B, \tag {2}
$$
where  $\alpha = \sigma \sqrt{2\log(2 / p)}$  is some constant.
Assume we have identified the failure cause set as  $\{x_{i}\}_{i = n + 1}^{n + m}$ . Directly removing the cause set and retraining the model achieves a generalization error bound as  $B_{\mathrm{remove}} = \alpha \sqrt{1 / n} < B$ .
Proposition 2.1. In the Gaussian mean estimation task, there exists a set of alignment functions:  $h(x) = x - \mu' + \tilde{\mu}$  where  $|\mu - \tilde{\mu}| < |\mu - \mu'|$  such that the optimal empirical solution on the combined set of  $\{x_i\}_{i=1}^n$  and  $\{h(x_i)\}_{i=n+1}^{n+m}$ , i.e.,  $\tilde{\theta} = \frac{1}{m+n}\left(\sum_{i=1}^n x_i + \sum_{i=n+1}^{n+m} h(x_i)\right)$ , achieves a generalization bound  $B_{\text{align}} < B_{\text{remove}}$ .
The detailed proofs for the bound can be found in Appendix B. The alignment function  $h(x) = x - \mu' + \tilde{\mu}$  aligns the data in  $\mathcal{U}'$  with  $\mathcal{U}$ , removing the detrimental information  $\mu'$  and enhancing the beneficial information  $\tilde{\mu}$ . Upon alignment, the resulting data set,  $\{h(x_i)\}_{i=n+1}^{n+m}$ , approximates the target distribution closely, and utilizing it in training will enhance generalization performance.
We extend the idea of alignment-based model treatment from the Gaussian mean estimation task to general tasks. Concretely, we propose to perform alignment on the identified cause data to obtain the aligned data using an Energy-Based Model (EBM) [25], building upon the energy-based alignment proposed in [18]. EBMs represent a probability density  $p(z)$  for  $z \in \mathbb{R}^d$  as  $p_{\theta}(z) = \frac{\exp(-E_{\theta}(z))}{Z(\theta)}$ , where  $E_{\theta}(z)$  is the energy function which takes  $z$  as input and returns a scalar as energy, and  $Z(\theta) = \int_z \exp(-E_{\theta}(z))$  is the normalizing constant. To align a given data point  $z'$  with  $p_{\theta}(z)$ , we iteratively adapt  $z'$  to minimize the energy function [18] as
$$
z ^ {\prime} = z ^ {\prime} - \eta \nabla_ {z ^ {\prime}} E _ {\theta} (z ^ {\prime}),
$$
Algorithm 1 Beneficial Information Retaining (BIR) Model Repair
1: Input: training data  $\mathcal{D}$ , reserved data  $\mathcal{D}_r$ ; trained model  $\hat{\theta}$ , a pre-defined candidate set of layer subsets.  
# Step 1: Cause Identification
2: Estimate the loss after a one-epoch update of  $\hat{\theta}$  on  $\mathcal{D}_r$  for each layer subset in the pre-defined set while fixing the remaining layers
3: Select the corresponding layer subset to be updated with the smallest loss
4: Obtain  $\hat{\phi}$  by updating the selected layers of model  $\hat{\theta}$  on  $\mathcal{D}_r$
5: Estimate  $s(z;\hat{\theta},\hat{\phi})$  for each data  $z\in \mathcal{D}$
6: Identify the failure cause set as  $\hat{\mathcal{D}}_c\gets \{z\in \mathcal{D}:s(z;\hat{\theta},\hat{\phi}) > 0\}$  # Step 2: Model Treatment
7: Perform data alignment for data in  $\hat{D}_c$  and obtain the aligned cause set  $\hat{D}_c$
8: Repair the model by updating the selected layers of model  $\hat{\theta}$  on  $\hat{\mathcal{D}}_c$