SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Performance with different selective layers updated: Identification AUC and Treatment Accuracy(%) with the best result highlighted in bold. The corresponding performance of the selected layers update is highlighted in gray. The backbone of the classifier is LeNet.
Dataset	MNIST-Label	MNIST-Input	MNIST-Adv
Loss after one-epoch update	Identification AUC	Treatment Accuracy(%)	Loss after one-epoch update	Identification AUC	Treatment Accuracy(%)	Loss after one-epoch update	Identification AUC	Treatment Accuracy(%)
Layer 1	1.0795	0.91	86.02	0.2503	0.71	96.16	0.2418	0.61	96.54
Layer 2	0.8967	0.96	92.56	0.2641	0.61	96.14	0.2201	0.57	96.40
Layer 3	0.8145	0.98	95.36	0.4268	0.60	95.92	0.3458	0.60	96.28
Layer 4	0.9419	0.96	94.42	0.3027	0.52	95.78	0.2682	0.56	96.34
Layer 5	1.0725	0.96	93.14	0.2751	0.46	95.52	0.2289	0.51	96.18
Layer 1-2	0.9193	0.96	92.36	0.2799	0.58	96.14	0.2121	0.70	96.44
Layer 1-3	0.7360	0.99	95.60	0.4243	0.70	96.12	0.3311	0.73	96.46
Layer 1-5	0.9407	0.99	95.50	0.6009	0.69	95.34	0.3618	0.66	96.28
Table 7: Performance with different selective layers updated: Identification AUC and Treatment Accuracy(%) with the best result highlighted in bold. The corresponding performance of the selected layers update is highlighted in gray. The backbone of the classifier is ResNet20.
Dataset	CIFAR10-Label	CIFAR10-Input	CIFAR10-Adv
Loss after one-epoch update	Identification AUC	Treatment Accuracy(%)	Loss after one-epoch update	Identification AUC	Treatment Accuracy(%)	Loss after one-epoch update	Identification AUC	Treatment Accuracy(%)
Layer 1	1.0763	0.56	74.82	0.6490	0.68	85.52	0.6438	0.60	87.64
Layer 2	1.0520	0.57	75.00	0.5865	0.84	87.96	0.6058	0.69	86.60
Layer 3	1.0038	0.68	76.92	0.6489	0.70	86.46	0.6035	0.65	86.58
Layer 4	0.7813	0.87	83.98	0.6445	0.74	86.78	0.5936	0.68	87.94
Layer 5	0.9988	0.60	75.62	0.6572	0.69	87.46	0.6199	0.63	87.34
Layer 1-2	1.0213	0.63	74.82	0.6479	0.80	85.92	0.6292	0.69	86.58
Layer 1-3	1.0383	0.66	76.04	0.6309	0.76	86.08	0.6080	0.65	86.72
Layer 1-5	0.7947	0.86	83.70	0.6642	0.72	87.22	0.6298	0.71	86.58
In this section, we introduce the settings in training. Specifically, the training set is partitioned into  $90\%$  training data and  $10\%$  validation data. The base model is trained using the Adam optimizer with a learning rate of 1e-3 and a batch size of 64 for all datasets. And the model is trained for a maximum of 100 epochs, and early stopping after 5 epochs of no progress on the validation data.
To obtain the enhanced model, we update the model on all data in the reserved set using the Adam optimizer with a learning rate of 1e-3 and a batch size of 16. Training is stopped after no progress on the training loss. The update layers are selected based on the training loss estimated after a one-epoch update, i.e., an update going through all reserved data. Once the enhanced model is obtained, we estimate the score in Eqn. (1) for all training data and identify the cause data by selecting the top  $\tilde{K}$  data points with the highest  $s(z; \hat{\theta}, \hat{\phi})$  values. Here,  $\tilde{K}$  is equal to the size of the ground-truth cause set, i.e.,  $|\mathcal{D}_c|$ . For the baseline identification methods, the same number of data points, i.e.  $\tilde{K}$  are selected as the cause data.
During model treatment, we first perform energy-based alignment on the identified data using the enhanced model. Specifically, the data input is updated using Eqn. (4) with  $\eta = 1.0$  and the label is updated using Eqn. (5). We then update the model using the aligned data and  $\tilde{K}$  additional data points that are randomly sampled from the remaining training set. The purpose of the additional data from the remaining set is to alleviate bias toward the identified data. It is worth noting that all baseline methods perform model treatment using both identified data and randomly sampled remaining data. In the proposed method, we initialize the model with  $\hat{\theta}$  and optimize it using the Adam optimizer with a learning rate of 1e-3. For the corrupted MNIST datasets and CIFAR-10 datasets, we set the batch size to 16 and 64, respectively. We terminate the training process when there is no further decrease in the training loss.
A.4 Extra Experiment: The Effect of Layer-Selective Update
In this section, we investigate the effect of the layer-selective update strategy on cause identification and model treatment. Performances are presented in Tabs. 6 and 7, where the AUC between the cause score of the training data calculated by Eqn. (1) and the ground-truth label indicating whether the data is from  $\mathcal{D}_c$  is used to measure identification performance, and the accuracy of the test set of the repaired model with different layers updated is used to evaluate model treatment performance. The
Table 8: Number of identified cause data on corrupted MNIST and CIFAR-10 datasets using different strategies.
MNIST-Label	MNIST-Input	MNIST-Adv	CIFAR10-Label	CIFAR10-Input	CIFAR10-Adv
ground-truth amount	432	432	432	7200	5400	7200
threshold = 0	1115	1789	1919	23955	27449	30457
threshold = log(1.1)	469	122	198	8745	5076	6003
threshold = log(1.05)	487	157	251	9952	6362	7686
threshold = log(1.01)	531	268	393	12836	9716	11849
Table 9: The repair performance on corrupted MNIST datasets under different cause set sizes using different identified strategies. The size of the identified cause set is available in Tab. 8. The network is LeNet.
MNIST-Label	MNIST-Input	MNIST-Adv
All	Clean	Noisy	All	Clean	Noisy	All	Clean	Noisy
ground-truth amount	95.60	95.93	95.10	96.16	96.20	96.10	96.44	96.47	96.40
threshold = 0	95.84	96.30	95.15	96.32	96.13	96.60	96.48	96.63	96.25
threshold = log(1.1)	95.64	95.77	95.45	96.22	96.13	96.35	96.40	96.57	96.15
threshold = log(1.05)	95.66	96.20	94.85	96.28	96.07	96.60	96.30	96.43	96.10
threshold = log(1.01)	95.56	96.07	94.80	96.14	95.97	96.40	96.34	96.53	96.05
Table 10: The repair performance on corrupted CIFAR-10 datasets under different cause set sizes using different identified strategies. The size of the identified cause set is available in Tab. 8. The network is ResNet50.
CIFAR10-Label	CIFAR10-Input	CIFAR10-Adv
All	Clean	Noisy	All	Clean	Noisy	All	Clean	Noisy
ground-truth amount	83.02	91.73	69.95	88.64	91.80	83.90	90.26	93.20	85.85
threshold = 0	83.26	91.33	71.15	88.90	92.20	83.55	90.14	93.43	85.20
threshold = log(1.1)	83.00	91.50	70.25	88.50	91.63	83.80	90.20	93.30	85.55
threshold = log(1.05)	83.14	91.83	70.10	88.62	91.70	84.00	90.12	93.03	85.75
threshold = log(1.01)	82.98	91.57	70.10	88.66	92.07	83.55	89.86	93.07	85.05
results indicate that updating all layers is suboptimal for both identification and repair performance in most cases. Thus, a layer-selective update is necessary to achieve better performance. We suggest selecting the layers for update based on the loss after one-epoch update on the reserved set. The proposed method achieves the best identification performance in 4/6 cases and the best treatment performance in all datasets except MNIST-Adv, where it is only  $0.1\%$  less accurate than the best one. These results highlight the effectiveness of the layer-selective update in enhancing the proposed method's performance.
A.5 Extra Experiment: The Effect of Identified Cause Set Size
In our experiments, we assume knowledge of the ground-truth size of the cause set, denoted as  $N_{C}$ , and identify the cause data by selecting the top  $N_{C}$  data points with the highest score estimated by Eqn. (1). However, in cases where the number of corrupted data examples is uncertain, selecting an appropriate threshold becomes challenging. To address this issue, a practical solution is to apply a fixed threshold on the score (s) computed using Eqn. (1) as a heuristic measure. As a straightforward solution, setting the threshold to 0 identifies all training data with an increased loss, but this often results in identifying more data than the actual number of corrupted points, as demonstrated in our empirical studies presented in Tab. 8. Our investigations indicate that a threshold between  $\log(1.01)$  and  $\log(1.1)$  effectively filters out identified data close in size to the ground truth. Thus, we recommend employing a threshold of  $\log(1.05)$  for practical applications.
Importantly, we emphasize that the repair performance of our proposed BIR method is robust to the size of the identified cause set, as evidenced by the experimental results in Tabs. 9 and 10. Therefore, while identification using thresholds may not be perfect, the repair performance of BIR is hardly affected.
A.6 Extra Experiment: The Effect of Clean Reserved Set Size
In this section, we investigate the impact of varying the amount of reserved clean data on the performance of the proposed model repair method when applied to three corrupted MNIST datasets. The identification and repair performance is reported in Tab. 11. Our empirical analysis reveals three key findings: firstly, with a mere 30 clean samples, our approach resulted in enhanced performance on noisy classes, albeit with a lower identification AUC; secondly, as the number of clean samples increased from 30 to 300, both identification AUC and overall performance demonstrated a marked
Table 11: The repair performance on corrupted MNIST datasets with different sizes of preserved clean data. The network is LeNet.
Size of preserved clean data	MNIST-Label	MNIST-Input	MNIST-Adv
Identification AUC	ALL	Clean	Noisy	Identification AUC	ALL	Clean	Noisy	Identification AUC	ALL	Clean	Noisy
30	0.76	83.67	95.99	69.70	0.68	96.48	97.07	95.60	0.57	96.36	97.07	95.30
100	0.98	92.82	93.66	91.55	0.70	96.24	96.60	95.70	0.63	96.46	96.90	95.80
200	0.99	95.24	95.83	94.35	0.70	96.34	96.47	96.15	0.68	96.26	96.50	95.90
300	0.99	95.60	95.93	95.10	0.71	96.16	96.20	96.10	0.70	96.44	96.47	96.40
improvement; and thirdly, our experiments demonstrated that 100 clean samples were sufficient for achieving effective model repair across these datasets.
B Omitted Technical Details in Sec. 2
C The Gaussian Mean Estimation Task
Definition C.1. (Gaussian Mean Estimation): Given a training set  $\mathcal{D}$  consisting of  $n$  training data  $\{x_i\}_{i=1}^n$  sampled from the distribution  $\mathcal{U} = \mathcal{N}(\mu, \sigma^2)$  as well as  $m$  training data  $\{x_i\}_{i=n+1}^{n+m}$  sampled from another Gaussian distribution  $\mathcal{U}' = \mathcal{N}(\mu', \sigma^2)$ , where  $\mu$  and  $\mu'$  are the means of the two Gaussian distributions and  $\sigma^2$  denotes the variance, the objective is to estimate the mean of the target distribution  $\mathcal{U}$ . The loss function is defined as  $\mathcal{L}(x; \theta) = (\theta - x)^2$ .
Lemma C.2. The empirical solution of Gaussian Mean Estimation is  $\hat{\theta} = \frac{1}{m + n}\sum_{i = 1}^{n + m}x_i$ . With probability at least  $1 - p$  for some probability, the generalization error is upper bounded as
$$
\left| \hat {\theta} - \theta^ {*} \right| \leq \alpha \sqrt {1 / (n + m)} + \frac {m}{m + n} \left| \mu - \mu^ {\prime} \right| = B, \tag {6}
$$
where  $\theta^{*} = \mu$  is the optimal solution and  $\alpha = \sigma \sqrt{2\log(2 / p)}$  is some constant.
Proof. Since  $\{x_{i}\}_{i = n + 1}^{n + m}$  are sampled from  $\mathcal{U}'$ ,  $\{x_{i} - \mu' + \mu\}_{i = n + 1}^{n + m}$  are sampled from  $\mathcal{U}$ . Using Gaussian tail bounds [34], we have that it holds with probability  $1 - p$  that
$$
\left| \frac {1}{m + n} \left(\sum_ {i = 1} ^ {n} x _ {i} + \sum_ {i = n + 1} ^ {n + m} \left(x _ {i} - \mu^ {\prime} + \mu\right)\right) - \mu \right| \leq \sigma \sqrt {\frac {2 \log (2 / p)}{m + n}} = \alpha \sqrt {1 / (n + m)}
$$
$$
\begin{array}{l} | \hat {\theta} - \theta^ {*} | = \left| \frac {1}{m + n} \left(\sum_ {i = 1} ^ {n} x _ {i} + \sum_ {i = n + 1} ^ {n + m} x _ {i}\right) - \mu \right| \\ = \left| \frac {1}{m + n} \left(\sum_ {i = 1} ^ {n} x _ {i} + \sum_ {i = n + 1} ^ {n + m} \left(x _ {i} - \mu^ {\prime} + \mu\right)\right) - \mu - \frac {m}{n + m} \left(\mu - \mu^ {\prime}\right) \right| \tag {7} \\ \leq \left| \frac {1}{m + n} \left(\sum_ {i = 1} ^ {n} x _ {i} + \sum_ {i = n + 1} ^ {n + m} \left(x _ {i} - \mu^ {\prime} + \mu\right)\right) - \mu \right| + \frac {m}{n + m} | \mu - \mu^ {\prime} | \\ \leq \alpha \sqrt {1 / (n + m)} + \frac {m}{n + m} | \mu - \mu^ {\prime} | = B. \\ \end{array}
$$
Remark: Note that in cases where the value of  $m$  decreases to zero, indicating the absence of any data from the detrimental distribution  $\mathcal{U}'$  in the training set, the upper bound is then given by
$$
B _ {\mathrm {r e m o v e}} = \alpha \sqrt {1 / n}.
$$
If the distribution divergence between  $\mathcal{U}$  and  $\mathcal{U}'$ , measured by  $|\mu - \mu'|$ , satisfies  $|\mu - \mu'| > \frac{m + n}{m}\alpha (\sqrt{1 / n} - \sqrt{1 / (n + m)})$ , then  $B_{\mathrm{remove}} < B$ . This implies that including data from  $\mathcal{U}'$  in the training set negatively affects performance, and removing these data can improve the performance, given that the divergence is sufficiently large.
Proposition C.3. (Recap of Proposition 2.1) In the Gaussian mean estimation task, there exists a set of alignment functions:  $h(x) = x - \mu' + \tilde{\mu}$  where  $|\mu - \tilde{\mu}| < \langle |\mu - \mu'| \rangle$  such that the optimal empirical solution on the combined set of  $\{x_i\}_{i=1}^n$  and  $\{h(x_i)\}_{i=n+1}^{n+m}$ , i.e.,  $\tilde{\theta} = \frac{1}{m+n}\left(\sum_{i=1}^n x_i + \sum_{i=n+1}^{n+m} h(x_i)\right)$ , achieves a generalization bound  $B_{\text{align}} < B_{\text{remove}}$ .
Proof. For any alignment function  $h(x)$ , similar to Eqn. (7), we have
$$
\begin{array}{l} | \tilde {\theta} - \theta^ {*} | = \left| \frac {1}{m + n} \left(\sum_ {i = 1} ^ {n} x _ {i} + \sum_ {i = n + 1} ^ {n + m} h (x _ {i})\right) - \mu \right| \\ = \left| \frac {1}{m + n} \left(\sum_ {i = 1} ^ {n} x _ {i} + \sum_ {i = n + 1} ^ {n + m} \left(x _ {i} - \mu^ {\prime} + \tilde {\mu} - \tilde {\mu} + \mu\right)\right) - \mu - \frac {m}{n + m} (\mu - \tilde {\mu}) \right| \tag {8} \\ \leq \left| \frac {1}{m + n} \left(\sum_ {i = 1} ^ {n} x _ {i} + \sum_ {i = n + 1} ^ {n + m} \left(x _ {i} - \mu^ {\prime} + \mu\right)\right) - \mu \right| + \frac {m}{n + m} | \mu - \tilde {\mu} | \\ \leq \alpha \sqrt {1 / (n + m)} + \frac {m}{n + m} | \mu - \tilde {\mu} | = B _ {\text {a l i g n}}. \\ \end{array}
$$
If  $|\mu - \tilde{\mu}| < \langle \mu - \mu' \rangle$  such that  $|\mu - \tilde{\mu}|$  satisfies
$$
| \mu - \tilde {\mu} | <   \frac {m + n}{m} \alpha (\sqrt {1 / n} - \sqrt {1 / (n + m)}) <   | \mu - \mu^ {\prime} |,
$$
$$
\begin{array}{l} B _ {\mathrm {a l i g n}} = \alpha \sqrt {1 / (n + m)} + \frac {m}{n + m} | \mu - \tilde {\mu} | \\ <   \alpha \sqrt {1 / (n + m)} + \frac {m}{n + m} \frac {m + n}{m} \alpha (\sqrt {1 / n} - \sqrt {1 / (n + m)}) \tag {9} \\ = \alpha \sqrt {1 / n} = B _ {\text {r e m o v e}}. \\ \end{array}
$$
Remark: The alignment function aligns the data in  $\mathcal{U}'$  with that in  $\mathcal{U}$ . If the alignment is effective, that is  $|\mu - \tilde{\mu}| < \frac{m + n}{m}\alpha (\sqrt{1 / n} - \sqrt{1 / (n + m)})$ , it improves performance over the results of data removal. The degree of improvement is positively correlated with the quality of alignment, as quantified by the magnitude of  $|\mu - \tilde{\mu}|$ .
C.1 Cause Identification in the Gaussian Mean Estimation Task
In the Gaussian mean estimation task, we assume that all the data from the detrimental distribution  $\mathcal{U}'$  are identified before model treatment. In this section, we verify the effectiveness of the metric proposed in Eqn. (1) in identifying the cause data in the training set for this task. Specifically, the metric estimates the contribution of failure by measuring the change in loss between the original trained model and an enhanced model, which is trained with a larger sample size  $n$  from  $\mathcal{U}$ .
In the Gaussian mean estimation task, the expected loss of the data from  $\mathcal{U}$  is
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} (x _ {i}; \hat {\theta}) \right] = \mathbb {E} \left[ \mathbb {E} \frac {1}{n} \sum_ {i = 1} ^ {n} \left| x _ {i} - \frac {1}{m + n} \sum_ {j = 1} ^ {n + m} x _ {j} \right| ^ {2} \right] \\ = \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| x _ {i} - \frac {1}{m + n} \left(n \frac {1}{n} \sum_ {j = 1} ^ {n} x _ {j} + m \frac {1}{m} \sum_ {j = n + 1} ^ {n + m} x _ {j}\right) \right| ^ {2} \right] \\ = \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| x _ {i} - \frac {1}{n} \sum_ {j = 1} ^ {n} x _ {j} + \frac {m}{m + n} \left(\frac {1}{n} \sum_ {j = 1} ^ {n} x _ {j} - \frac {1}{m} \sum_ {j = n + 1} ^ {n + m} x _ {j}\right) \right| ^ {2} \right] \\ = \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left| x _ {i} - \frac {1}{n} \sum_ {j = 1} ^ {n} x _ {j} \right| ^ {2} \right] + \mathbb {E} \left[ \frac {m ^ {2}}{(m + n) ^ {2}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} x _ {j} - \frac {1}{m} \sum_ {j = n + 1} ^ {n + m} x _ {j} \right| ^ {2} \right] \\ = \sigma^ {2} + \frac {m ^ {2}}{(m + n) ^ {2}} \left(\mu - \mu^ {\prime}\right) ^ {2}. \tag {10} \\ \end{array}
$$
The forth equality uses the fact that  $\mathbb{E}|\xi -\mathbb{E}\xi +\varepsilon |^2 = \mathbb{E}|\xi -\mathbb{E}\xi |^2 +\varepsilon^2 -2\mathbb{E}(\xi -\mathbb{E}\xi)\varepsilon = \mathbb{E}|\xi -\mathbb{E}\xi |^2 +\varepsilon^2$  and the fifth equality uses the fact that  $\frac{1}{n}\sum_{i = 1}^{n}\left|x_i - \frac{1}{n}\sum_{j = 1}^{n}x_j\right|^2,\frac{1}{n}\sum_{j = 1}^{n}x_j$  and  $\frac{1}{m}\sum_{j = n + 1}^{n + m}x_{j}$  are unbiased estimation of  $\sigma^2,\mu$  and  $\mu^\prime$ , respectively. Similarly, the expected loss of the data from  $\mathcal{U}'$  is
$$
\mathbb {E} \left[ \frac {1}{m} \sum_ {i = n + 1} ^ {n + m} \mathcal {L} \left(x _ {i}; \hat {\theta}\right) \right] = \sigma^ {2} + \frac {n ^ {2}}{(m + n) ^ {2}} \left(\mu - \mu^ {\prime}\right) ^ {2}.
$$
If we introduce additional  $\Delta n$  data to train the enhanced model  $\hat{\phi}$ , similar to Eqn. (10), we have
$$
\mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} (x _ {i}; \hat {\phi}) \right] = \sigma^ {2} + \frac {m ^ {2}}{(m + n + \Delta n) ^ {2}} (\mu - \mu^ {\prime}) ^ {2},
$$
$$
\mathbb {E} \left[ \frac {1}{m} \sum_ {i = n + 1} ^ {n + m} \mathcal {L} (x _ {i}; \hat {\phi}) \right] = \sigma^ {2} + \frac {(n + \Delta n) ^ {2}}{(m + n + \Delta n) ^ {2}} (\mu - \mu^ {\prime}) ^ {2}.
$$
$$
\mathbb {E} _ {x _ {i} \in \mathcal {U}} s (x _ {i}; \hat {\theta}, \hat {\phi}) = \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \Big (\mathcal {L} (x _ {i}; \hat {\phi}) - \mathcal {L} (x _ {i}; \hat {\theta}) \Big) \right] = \left(\frac {m ^ {2}}{(m + n + \Delta n) ^ {2}} - \frac {m ^ {2}}{(m + n) ^ {2}}\right) (\mu - \mu^ {\prime}) ^ {2} <   0.
$$
$$
\mathbb {E} _ {x _ {i} \in \mathcal {U} ^ {\prime}} s (x _ {i}; \hat {\theta}, \hat {\phi}) = \mathbb {E} \left[ \frac {1}{m} \sum_ {i = n + 1} ^ {n + m} \Big (\mathcal {L} (x _ {i}; \hat {\phi}) - \mathcal {L} (x _ {i}; \hat {\theta}) \Big) \right] = \left(\frac {(n + \Delta n) ^ {2}}{(m + n + \Delta n) ^ {2}} - \frac {n ^ {2}}{(m + n) ^ {2}}\right) (\mu - \mu^ {\prime}) ^ {2} > 0.
$$
Therefore, the detrimental data can be identified by the positive value of  $s(x_{i};\hat{\theta},\hat{\phi})$
Furthermore, this metric can be utilized to identify harmful data, even if they are not part of the training dataset. In the Gaussian mean estimation example, similar to Eqn. (7), we can obtain the generalization error bound to the data in  $\mathcal{U}'$  as
$$
\left| \tilde {\theta} - u ^ {\prime} \right| \leq \alpha \sqrt {1 / (n + m)} + \frac {n}{n + m} \left| \mu - \mu^ {\prime} \right| \tag {11}
$$
As the number of training samples from  $\mathcal{U}$  increases, both terms in the generalization error bound for  $\mathcal{U}$  (Eqn. (7)) decrease. In contrast, for  $\mathcal{U}'$ , the first term  $\alpha \sqrt{1 / (n + m)}$  decreases, while the second
term  $\frac{n}{n + m} |\mu -\mu '|$  increases. Consequently, the generalization error bound for  $\mathcal{U}$  decreases more significantly than that for  $\mathcal{U}'$ . Given that the loss is related to the generalization bound, an increase in the number of training samples from  $\mathcal{U}$  leads to a rise in the loss of most samples from  $\mathcal{U}'$ , while causing a decline in the loss of most samples from  $\mathcal{U}$ . And we can utilize the metric in Eqn. (1) to identify any detrimental data.