A unified method to revoke the private data of patients in intelligent healthcare with audit to forget

Domain: earth_science
Published: 2023-10-06
Editorial outcome: accept

Abstract
Revoking personal private data is one of the basic human rights. However, such right is often overlooked or infringed upon due to the increasing collection and use of patient data for model training. In order to secure patients’ right to be forgotten, we proposed a solution by using auditing to guide the forgetting process, where auditing means determining whether a dataset has been used to train the model and forgetting requires the information of a query dataset to be forgotten from the target model. We unified these two tasks by introducing an approach called knowledge purification. To implement our solution, we developed an audit to forget software (AFS), which is able to evaluate and revoke patients’ private data from pre-trained deep learning models. Here, we show the usability of AFS and its application potential in real-world intelligent healthcare to enhance privacy protection and data revocation rights.

Introduction
Revoking personal private data is one of the basic human rights, which has already been sheltered by privacy-preserving regulations like The General Data Protection Regulation (GDPR)1, The Health Insurance Portability and Accountability Act of 1996 (HIPAA)2, and the California Consumer Privacy Act3 since twentieth century. With those regulations, users are allowed to request the deletion of their own data for privacy concerns and to secure their own “right to be forgotten”. However, with the development of data science, machine learning (ML) and deep learning (DL) techniques, this basic right is usually neglected or violated. For example, it has been observed that patients’ genetic markers were leaked from ML methods for genetic data processing4,5 while the patients were unaware of that. When users realize the existence of such risks, they may request their own data to be deleted to protect their privacy6. Meanwhile, those aforementioned regulations will force involved third parties to take actions immediately. According to the requirements of those regulations, not only the previously authorized data by individuals need to be deleted immediately from hosts’ storage systems but also the associated information should be removed from DL models trained with those data, because DL models could memorize sensitive information of training data and thus expose individual’s privacy under risk7–11.

Nowadays, healthcare is one of the most promising areas for the deployment of artificial intelligent (AI) systems as so-called intelligent healthcare. ML and DL-based computer-aided diagnosis (CAD) systems in intelligent healthcare accelerate the diagnosis of various diseases and achieve even better results than doctors, such as tumor detection12,13, retinal fundus imaging14, detection and segmentation of COVID-19 lung infections15,16 and so on. However, as more and more patients’ data are being collected and used for model training in intelligent healthcare, their privacy is exposed to high risk. Therefore, intelligent healthcare is a sector where technology must meet the law, regulations, and privacy principles to ensure that the innovation is for the common good17. To obey those privacy-preserving regulations, methods to revoke personal private data from pre-trained DL models are necessary.

Deleting the stored personal data is simple, whereas forgetting individuals’ private information from pre-trained DL models could be difficult as we could not fully measure the contribution of individual data on the training process of DL models due to the stochasticity of training18. Besides, due to the incremental nature of training, the model update brought by one sample would affect the model performance on samples followed, thus making it difficult to unlearn18. Finally, catastrophic unlearning might happen and the unlearned model will perform worse than the model retrained on the remaining dataset19. In general, the process to forget data from a pre-trained DL model could be divided into two steps. Firstly, the unlearning process (forgetting) is performed on a given pre-trained DL model to forget the target data with different techniques and a new DL model will be generated. Secondly, an evaluation of the new model (auditing) against different metrics will be performed to prove that the model has forgotten the target data. These two processes should be repeated until the new model passes the evaluation. In simple terms, there are two commonly acknowledged sub-tasks, which could also be stated in the reverse order: auditing and forgetting, as a two-player game. Auditing requires auditors to precisely evaluate whether the data of certain patients were used to train the target DL model. Once the data of certain patients is confirmed to be used to train the target DL model by auditing, forgetting requires the removal of learnt information of certain patients’ data from the target DL model, which is also called machine unlearning, while auditing could act as the verification of machine unlearning18.

In order to achieve forgetting, existing unlearning methods could be classified into three major classes, including model-agnostic methods, model-intrinsic methods and data-driven methods20. Model-agnostic methods refer to algorithms or frameworks that can be used for different DL models, including differential privacy18,21–23, certified removal24–26, statistical query learning6, decremental learning27, knowledge adaptation28,29 and parameter sampling30. Model-intrinsic approaches are those methods designed for specific types of models, such as for softmax classifiers31, linear models32, tree-based models33 and Bayesian models19. Data-driven approaches focus on the data itself, including data partitioning18, data augmentation34–36 and other unlearning strategies based on data influence37. However, most of them are theoretical studies and do not provide open-source codes. Besides, most methods have their specific application scenarios and acknowledged limitations and few of them focused on the application in real-world intelligent healthcare. Among the three methods, model-agnostic methods might have the strongest application prospects, as they can be applied to different models, and SISA, short for Sharded, Isolated, Sliced, and Aggregated training, is the most classic and well-known method in the community38. As the state-of-the-art method, Goel et al.39 proposed Catastrophic Forgetting-k (CF-k) and Exact Unlearning-k (EU-k) to unlearn information from deep learning models. CF-k means to fine tune the last k layers of the original model on D r and freeze other layers, while EU-k means to retrain the last k layers of the original model from scratch on D r and freeze other layers, where the D r stands for the retain dataset.

When forgetting is accomplished, auditing is the next necessary step to verify it. Different metrics have been proposed to audit the membership of the query dataset, including accuracy, completeness6, unlearn time, relearn time, retrain time, layer-wise distance, activation distance, JS-divergence, membership inference40,41, ZRF score28, epistemic uncertainty42 and model inversion attack7. In recent studies, membership inference-based metrics were frequently utilized to determine whether or not any information about the samples to be forgotten was retained in the model in intelligent healthcare41. A black-box setting was shared by the membership inference attack (MIA) to calculate the probability of a single data point being a member of the training dataset D. Based on this individual-level MIA, Liu et al.40 and Yangsibo et al.41 focused on a more challenging task: audit the membership of a set of data points. The ensembled membership auditing (EMA)41 was proposed as the state-of-the-art method to verify whether a query dataset is memorized by a pre-trained DL model, which is also a benchmark metric in machine unlearning. However, due to the black box property of DL models, efficient and accurate auditing is still challenging and an under-studied topic. Moreover, researchers have tended to treat auditing and forgetting as separate tasks, ignoring the fact that the two can be linked up associatively to work as a self-consistent mechanism.

Here, we proposed a solution by using auditing to guide the forgetting process in a negative feedback manner. We unified the two tasks by introducing knowledge purification (KP), an approach to selectively transfer the needed knowledge to forget the target information instead of simply transferring all information like knowledge distillation (KD)43. On the basis of KP, we have developed a user-friendly and open-source audit to forget software (AFS) (Fig. 1), which can be easily used to revoke patients’ private data from DL models in intelligent healthcare with KP (Fig. 2). To demonstrate the generality of AFS, we applied it to four tasks based on four datasets, including the MNIST dataset, the PathMNIST dataset, the COVIDx dataset, and the ASD dataset, with different data sizes and various architectures of deep learning networks (Fig. 3). Our results demonstrate the usability of AFS and its application potential in real-world intelligent healthcare to enhance privacy protection and data revocation rights.

Results
AFS audits private datasets stably and robustly
To evaluate the robustness of auditing by AFS, we used it to audit query datasets with different sizes, various purity (k percent of the query dataset was overlapped with the training dataset) and the different sizes of calibration dataset (the size ranged from 100 to 5000) (“Method” and Fig. 4a). For each sample in the query dataset, AFS calculates three metrics for the membership inference, including correctness, confidence and negative entropy (“Method”). As shown in Figs. 4b and S1, all three metrics showed different distributions for QO (query dataset overlapped with the training dataset) and QNO (query dataset disjoint with the training dataset), indicating the dataset-wise divergence of metrics between samples in the training dataset and samples disjoint with the training dataset. Given the differences observed in the three metrics, we could distinguish between QO and QNO by calculating the p value and utilizing it as the sole audit metric for forgetting (Method). Finally, by integrating these three metrics, AFS reports a p value to evaluate whether or not a query dataset has been used to train the target DL model. The large p values indicate the higher probability that the query dataset was used in training.

When the size of the query dataset and the calibration dataset varied, AFS could still efficiently distinguish QO and QNO (Fig. 4c, d). Compared to QO, AFS reported a much smaller p value for QNO, indicating a weak membership (a small probability that the query dataset has been used to train the target DL model), thus allowing users to judge whether the query dataset was used to train the target DL model. Meanwhile, when the size of the dataset increased from 1 to 2000, AFS discriminated QO and QNO more confidently as there was a more significant divergence of the p values, which was not affected by the size of the calibration dataset. To further understand the effect of the purity of the query dataset in auditing, we mixed some samples from the training dataset to QNO, thus the new query dataset was labeled as QM (partial data overlapped with the training dataset). The percentage of data overlapped with the training dataset in QM was denoted by k = number of data overlapped with training dataset. As shown in Fig. 4e, AFS showed a decreasing p value of QM trend when k decreased, meaning that the query dataset was less likely to be used to train the target DL model. The distinctiveness of the p value on the ASD dataset is due to the small size of the ASD dataset. k = 0 was shown in Fig. 4d as QNO, which occurred when the query dataset size was 100 for ASD and 2000 for MNIST, PathMNIST, and COVIDx. Similarly, k = 1 was also displayed in Fig. 4d as QO, which occurred when the query dataset size was 100 for ASD and 2000 for MNIST, PathMNIST, and COVIDx. In conclusion, these results indicate the robustness of AFS in determining whether the query data has been used to train the target DL model.

AFS forgets the information of query dataset, maintains perfect usability and generates smaller model
Once the prior knowledge that a dataset has been used to train the target DL model is confirmed with auditing, AFS could be used for forgetting, to remove the information of the dataset from the pre-trained DL model. To comprehensively show the ability of AFS in removing information against the model performance, we compared nine methods, including (1) training the teacher model with a complete training dataset (Independent teacher), (2) retraining the student model with a complete training dataset (Independent student), (3) retraining the teacher model with k ∈ {0.25, 0.5, 0.75} percentage of the complete training dataset excluding the data to be forgotten (Independent teacher with k ∈ {0.25, 0.5, 0.75}), (4) retraining the student model with k ∈ {0.25, 0.5, 0.75} percentage of the complete training dataset excluding the data to be forgotten (Independent Student with k ∈ {0.25, 0.5, 0.75}), (5) retraining the model of the corresponding shard with SISA, (6) fine-tuning the last layers of the model with CF-k, (7) retraining the last layers of the model from scratch with EU-k, (8) AFS, and (9) training the student model with AFS without the guidance of auditing (AFS w/o Audit), as an ablation study of AFS. Both AFS w/o Audit and AFS were also conducted with varied k ∈ {0.25, 0.5, 0.75}. For both Independent teacher and Independent student methods trained with the complete training dataset, QF 100 and QF 1000 were included in the training dataset, while these two query datasets were excluded from the training dataset when k ∈ {0.25, 0.5, 0.75}.

Taking the MNIST dataset as an example, for models trained with each method, except for auditing on QO and QNO, we further audited the membership of two datasets designed to be forgotten (a small query dataset QF 100 and a large query dataset QF 1000) to assess the ability of different methods in forgetting the query dataset. As shown in Table 1, regardless of the model trained based on which method, AFS could effectively distinguish between QO and QNO, and the divergence in auditing two query datasets was enlarged as the size of the query dataset increased.

As shown in Table 2, AFS perfectly predicted the membership of QF 100 and QF 1000 on both models from Independent teacher and Independent student methods as both query datasets were included in the training dataset. Since both query datasets were disjoint with the partial training dataset when k ∈ {0.25, 0.5, 0.75}, thus auditing on the model trained with Independent teacher and Independent student with k ∈ {0.25, 0.5, 0.75} weakly denied the membership of QF 100 (P QF100,k=0.75 = 1.57E−1, P QF100,k=0.5 = 1.56E−1, P QF100,k=0.25 = 8.17E−2 for Independent teacher and P QF100,k=0.75 = 4.36E−2, P QF100,k=0.5 = 6.91E−3, P QF100,k=0.25 = 6.91E−3 for Independent student) and QF 1000 (P QF1000,k=0.75 = 1.05E−8, P QF1000,k=0.5 = 2.71E−11, P QF1000,k=0.25 = 2.80E−18 for Independent teacher and P QF1000,k=0.75 = 5.26E−12, P QF1000,k=0.5 = 2.34E−15, P QF1000,k=0.25 = 2.90E−19 for Independent student). However, since only the partial training dataset was used when k ∈ {0.25, 0.5, 0.75}, the retrained models with Independent student and Independent student only learnt the information of the partial training dataset and lost the information from the remaining data in the complete training dataset, thus resulting in the significant drop of model performance compared to either the Independent student or the Independent teacher trained with the complete training dataset. The same conclusion could be drawn with SISA (The training data were divided into 10 shards. Then, QF was removed from the shard where the QF was located, and the model of the corresponding shard was re-trained to re-aggregate the final model). Meanwhile, due to the unique design, SISA requires storing 10 model parameters simultaneously, resulting in greater storage consumption than AFS. Although CF-k and EU-k achieved good accuracy and F1-score, the audit results showed that only fine-tuning or retraining the last few layers of the original model is not enough to forget the information of the query data.

To rescue the information lost due to the usage of partial training samples and further increase the model performance, AFS could use only a partial training dataset (k ∈ {0.25, 0.5, 0.75}) to transfer the knowledge from the Independent teacher pre-trained with the complete training dataset. As shown in Table 2, the model trained with AFS provided higher accuracy and F1-score compared to the Independent student trained with partial training dataset (k ∈ {0.25, 0.5, 0.75}) and together with a better forgetting performance (much smaller auditing score on QF 100 and QF 1000), as AFS used auditing as feedback for forgetting and could forget not only the query samples but also other samples with similar features.

We also applied AFS on the 9-classes classification of hematoxylin and eosin-stained histological images from the PathMNIST dataset with CNN. As shown in Table 3, AFS could still distinguish QO and QNO from the PathMNIST dataset. The divergence of auditing between QO and QNO was more significant than that on the MNIST dataset. With the requirement to forget both query datasets (QF 100 and QF 1000), the model trained with AFS outperformed on forgetting information (P QF100,k=0.75 = 2.25E−5, P QF100,k=0.5 = 2.87E−6, P QF100,k=0.25 = 3.32E−7, P QF1000,k=0.75 = 2.05E−41, P QF1000,k=0.5 = 4.75E−35, P QF1000,k=0.25 = 1.84E−56) while learnt more information from the Independent teacher model trained with a complete training dataset. In summary, AFS could effectively forget the information of the query dataset from the target DL model. Since KP was integrated into AFS, it could generate a smaller DL model, which masters knowledge from the larger teacher model by using only a partial training dataset (k = 0.5 could achieve a good balance between forgetting and model performance), without the need to retrain the larger model with the complete training dataset. Compared to retraining the student model, the model trained with AFS showed even better performance in forgetting the information while maintaining better model performance (accuracy and F1-score) as it learnt the knowledge from the model trained with the complete training dataset. As shown by the ablation study in Tables 2 and 4, compared to AFS w/o Audit, the audit-guided AFS could forget the information more significantly but with an acceptable cost in decreasing the model performance (accuracy and F1-score).

Apply AFS to forget medical images
To show the versatility of AFS, we applied it to the classification of pneumonia and normal with chest X-ray images from the COVIDx dataset with ResNet, which is a classic task in medical image analysis. As shown in Fig. 5a, on both query datasets (QF 100 and QF 1000), AFS could effectively forget the information of the query dataset, while generating the new model with much less number of parameters as shown in Fig. 5b. Surprisingly, the model generated by AFS showed even better accuracy than the Independent teacher trained with the complete dataset and the Independent student trained with the partial training dataset. This result not only indicated that AFS could effectively transfer the knowledge from the teacher model to the student model but also suggested that the student model with simpler architecture could even perform better than the teacher model with KP in the AFS due to the reduction of model parameters and purification of knowledge in some real-world cases. Meanwhile, the results also showed that the student model trained by the AFS achieved better generalization ability than the model obtained by using the exact same model structure and training data using only the hard-target training method.

Apply AFS to forget electrical health records
To further prove the generalizability of AFS in both the auditing and forgetting, we applied AFS to predicting early autism spectrum disorder (ASD) traits of toddlers, which contains sensitive information about patients, such as the age, gender and the family gene trait. That information was stored as electrical health records (EHR). As shown in Fig. 5a, similar to previous results on other datasets, AFS effectively removed the information of both query datasets from the pre-trained DL model. Since the size of the ASD dataset was quite small, we adopted two smaller query datasets (QF 50 and QF 100) to be forgotten. Compared to the models trained with other methods, the model trained with AFS successfully forgot the information of both QF 50 (P QF50,k=0.75 = 0.08, P QF50,k=0.55 = 0.08, P QF50,k=0.25 = 0.156) and QF 100 (P QF100,k=0.75 = 0.004, P QF100,k=0.55 = 0.007, P QF100,k=0.25 = 0.007) without affecting the model utility significantly (Acc AFS,k=0.75 = 0.98, Acc AFS,k=0.5 = 0.98, Acc AFS,k=0.25 = 0.98).

Methods
The overall framework of AFS
AFS is a unified method to revoke patients’ private data by using auditing to guide the forgetting process in a negative feedback manner (Fig. 1). To audit the membership of the query dataset, AFS takes a pre-trained DL model and the query dataset as inputs, and determines whether the query dataset has been used for training the target DL model. This function was re-implemented based on EMA41, a published MIA-based method to evaluate the membership of a query dataset. Our re-implementation allows quicker and easier usage of auditing by introducing parallel computing in each epoch, which suggests a significant acceleration for a complete forgetting process and could be more attractive to institutions to forget larger-scaled data (Fig. S2).

To forget the query dataset from a DL model, AFS takes the pre-trained DL model and the query dataset to be forgotten as inputs, in which the query dataset has been used to train the DL model. To effectively forget the information of the query dataset from the pre-trained DL model, an idea is to transfer the information of the remaining dataset except for the query dataset from the pre-trained model to a new model. Therefore, we designed a mechanism called knowledge purification (KP) by using auditing to guide the forgetting process to exclude the information of the query dataset while transferring the remaining information by incorporating the auditing loss into the training process (Fig. 2). With KP integrated, AFS could generate a new model, in which the information of the target dataset should be forgotten under the guidance of auditing.

To provide an applicable solution, we implemented AFS as open-source software that provides a user-friendly entry point allowing users to use both functions with only one command. To demonstrate the generality of AFS, we applied it to four tasks based on four datasets, including the MNIST dataset, the PathMNIST dataset, the COVIDx dataset and the ASD dataset, which have different data sizes (Fig. 3) and various architectures of deep learning networks.

Dataset preparation
We used four public datasets that were commonly acknowledged in the machine learning and intelligent healthcare field to demonstrate the versatility of AFS. For the benchmark experiment, we applied AFS on MNIST44 and PathMNIST45 from the MedMNIST46 dataset. The MNIST dataset contains 60,000 training images and 10,000 testing images of handwritten digits with size 28 × 28 and labeled from 0 to 9. PathMNIST contains 100,000 nonoverlapping image patches from hematoxylin and eosin-stained histological images and 7180 image patches from different clinical centers. In total, 9 types of tissues are involved in the PathMNIST dataset, including adipose, background, debris, lymphocytes, mucus, smooth muscle, normal colon mucosa, cancer-associated stroma, and COAD epithelium. All images in PathMNIST were 224 × 224 (0.5 µm px−1) and were normalized with the Macenko method47. For the application of AFS in intelligent healthcare, we used the COVIDx48 dataset, which contains 13,975 chest X-ray (CXR) images across 13,870 patient cases, and the autism spectrum disorder (ASD) dataset for toddlers49, which contains 20 features of 1054 samples to be utilized for determining influential autistic traits and improving the classification of ASD cases.

For each dataset, we further sampled partial data as the training dataset, the testing dataset, and the calibration dataset as below:

MNIST. We randomly sampled 10,000 images as the training dataset and 10,000 images as the testing dataset. We also randomly sampled 100, 1000, 2000, and 5000 images that are disjoint with the training dataset as four calibration datasets to illustrate the effect of the calibration dataset of varied sizes on auditing and forgetting.

PathMNIST. We randomly sampled 10,000 images as the training dataset and 5000 images as the testing dataset. We also randomly sampled 1000 images that are disjoint with the training dataset as the calibration dataset.

COVIDx. We randomly sampled 5000 images as the training dataset and 1000 images as the testing dataset. We also randomly sampled 1000 images that are disjoint with the training dataset as the calibration dataset.

ASD. We randomly sampled 500 images as the training dataset and 100 images as the testing dataset. We also randomly sampled 100 images that are disjoint with the training dataset as the calibration dataset.

For all four datasets, we randomly sampled partial data from the training dataset with percentage k from {0.25, 0.5, 0.75} as the training dataset for knowledge distillation (KD) and AFS.

In addition, we prepared query datasets with different sizes N from {1, 10, 100, 500, 1000, 2000}. A query dataset that completely overlapped with the training dataset is labeled as QO, while the query dataset that is completely disjoint with the training dataset is labeled QNO. To further understand the effect of the purity of the query dataset, we also prepared the query dataset called QM with a k percentage of the query dataset to be overlapped with the training dataset. Finally, for the query dataset designed to be forgotten, we labeled it as QF. QO, QNO, QM, and QF were all sampled randomly from the completed dataset and all reported values were the average of 5 replicate experiments.

Deep learning models and experiment setup
To present the generalizability of AFS toward various DL models, we adopted different architectures for each of the four tasks, including the multilayer perception50 (MLP), the convolutional neural network (CNN)51 and ResNet52. There were a large DL model and a small DL model for each task, where the large model refers to the original pre-trained model and the small model is the new model generated by AFS.

For the MNIST dataset, we used MLP with 671,754 parameters as the teacher model and 155,658 parameters as the student model to achieve the 10-class classification task.

For the PathMNIST dataset, we adopted CNN with 21,285,698 parameters as the teacher model and 11,177,538 parameters as the student network for the 9-class classification task.

For the COVIDx dataset, we took ResNet34 with 21,285,698 parameters as the teacher model and ResNet18 with 11,177,538 parameters as the student network to achieve the binary classification of healthy people and patients.

For the ASD dataset, we used the MLP with 3586 parameters as the teacher model and the MLP with 898 parameters as the student model for the binary classification of autism in toddlers.

During model training, the number of epochs was fixed to 50, the learning rate was set to 1e–5 and the Adam optimizer was used. A workstation with 252 GB RAM, 112 CPU cores and 2 Nvidia V100 GPUs were adopted for all experiments. The AFS method was developed based on Python 3.7, PyTorch 1.9.1 and CUDA 11.4. A detailed list of dependencies could be found in our code availability.

Audit the membership of query dataset
EMA41 is designed as a 2-step process. In the first step, the best threshold for each metric is selected to optimize (TPR(t) + TNR(t))/2 based on the calibration dataset as shown in Algorithm 1. Once the thresholds for all metrics are selected, the membership of each sample in the query dataset will be confirmed as at least one metric is larger than the corresponding threshold. In total, three metrics, including correctness53, confidence54,55, and entropy56,57, were adopted to further calculate the p value, which is the key audit metric in AFS as proposed in the previous work41,58. The details for calculating correctness, confidence, and entropy are as below:

Correctness: the target model is trained to predict correctly on training data and may not generalize well on test data. Thus, we can define the correctness as in Eq. (1):

I correctness (F, (x, y)) = 1{argmaxi F(x) i = y} (1)

where F is the deep learning model, x is the input data, F(x) is the output logits, y is the label, and 1 is the indicator function.

Confidence: the target model is usually more confident in predictions on training data, but less confident in test data. Thus, we can define confidence as in Eq. (2):

I confidence (F, (x, y)) = 1{F(x) y ≥ τ y} (2)

where F is the deep learning model, x is the input data, F(x) is the output logits, y is the label, F(x) y is the output logits for label y, τ y is a threshold for the logit for label y, and 1 is the indicator function.

Entropy: the target model is trained by minimizing the prediction loss over training data and usually has a larger prediction entropy on a test sample. Thus, we can define entropy as in Eq. (3):

I entropy (F, (x, y)) = 1(∑ i − F(x) i log F(x) i ≤ τ^ y) (3)

where F is the deep learning model, x is the input data, F(x) is the output logits, y is the label, F(x) i is the output logits for class i, τ^ y is a threshold, and 1 is the indicator function.

Once the membership of all samples in the query dataset is confirmed in the previous step, the query dataset will be further evaluated to determine whether the query dataset has been used to train the target pre-trained DL model. A two-sample statistical test is adopted to evaluate the query dataset based on the sample-wise membership and an all-one vector. The p value of the two-sample statistical test is used as the output of auditing. Given a user-defined threshold α, if p < α, then users could conclude that the query dataset was not used for training the target DL model. EMA was re-implemented and integrated into AFS to allow easy and fast auditing.

Audit-guided forgetting of query dataset with AFS
Forgetting aims to remove the remembered information of the query dataset from the target DL model. Similar to knowledge distillation (KD), a teacher-student paradigm was also adopted in AFS, but with an additional requirement to selectively forget information associated with the data we want to forget. Thus, we designed an approach called knowledge purification (KP), meaning purifying the knowledge in the teacher model (the original pre-trained model), discarding the information related to the data that needed to be forgotten and transferring the purified information into the student model (the new model). AFS unified auditing and forgetting into a circular process to effectively enhance the unlearning in a negative feedback manner.

As shown in Fig. 1, during each epoch of training, the training data will be fed into both the teacher model and the student model, while the data to be forgotten will be audited on the student model. Our main goal is to transfer the knowledge from the teacher model to the student model while forcing the student model to reject the information associated with data to be forgotten. In order to achieve that, we added the audit loss into the total loss, thus allowing the student model to accept partial knowledge from the teacher model and achieve KP as shown in Algorithm 2.

Evaluation metrices
Since all four tasks are either multi-classes classification tasks or binary classification tasks, we adopted the accuracy and F1-score as the evaluation metrics as in Eqs. (4) and (5):

Accuracy = (TP + TN) / (TP + TN + FP + FN) (4)

F1-score = (2TP) / (2TP + FP + FN) (5)

where TP represents true positives, TN stands for true negatives, FN represents false negatives and FP stands for false positives.

To evaluate the membership of the query dataset, the p value of the two-sample statistical test was used as mentioned previously.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "SUMMARY: The paper focuses on developing tools and methods to support the right to be forgotten in\nthe digital world within a scenario where patients’ private data is used in ML-based intelligent\nhealthcare solutions. The paper proposes a unified solution that can audit the use of patients’ private\ndata and ensure that the target model forgets the personal data when requested.\nFor auditing, given a DL model and query dataset used for training, the developed AFS solution uses a\nquery dataset to be audited and determine if this dataset has been used in training. AFS use an existing\nsolution called Ensembled Membership Auditing (EMA), which is based on Membership Inference\nAttacks (MIA). The authors claim that their re-implemented version of EMA as part of AFP is quicker and\neasier for auditing use (not experimentally evaluated/compared in the paper).\nFor forgetting, the authors implement a variant of the knowledge distillation approach within their AFS\nsolution. Knowledge distillation generally uses a larger and more complex teacher model to train a\nsmaller model that can achieve a similar convergence. This is achieved by establishing a correspondence\nbetween intermediate outputs of teacher and student models. The proposed knowledge purification\nprocess builds over the knowledge distillation process and eliminates correspondence between teacher\nand student networks due to forgotten data queries. Hence, the overall distillation process also ensures\nthat a specific part of training data is forgotten. However, with teacher-student settings, theoretical\nsearch space gets smaller with smaller convergence space. This may mean faster model drift.\nDeveloped AFS solution has been evaluated using 4 datasets (MNIST, PathMNIST, COVIDx, ASD).\nNoteworthy results: The paper focuses on building a solution that brings the right to forgetting the\npatients’ private data used in ML-based intelligent healthcare solutions. The overall solution is a\ncombination of two components: Auditing and Forgetting. Auditing is based on existing solutions.\nThe authors evaluate metrics (P-Value, Correctness, Confidence and Entropy) and show that these metrics\ncan be used effectively in auditing. Forgetting is an incremental solution adapting the well-established\nknowledge distillation approach. The overall solution is implemented and tested on four datasets.\nOverall, novelty is very limited and not significant. But, a decent engineering and development effort is\ninvolved in evaluating and building a combined solution.\nSignificance to the field and related fields: The problem that authors engage in is a significant problem in\nthe health domain and domains where ML-based approaches are used along with data with individuals.\nForgetting is a critical requirement in most regulations, but there needs to be a well-established solution\nto be used with complex deep-learning models, especially in commercial settings. However, the solution\nworks based on knowledge distillation (model compression – Figure 5 B). So, after each data forget\noperation, the model will be compressed significantly. That means the theoretical search space gets\nsmaller with a smaller convergence space - i.e., there is some loss (not evaluated in the paper). The\nquestion is, how many times can an intelligent health solution repeat this iteratively? After each\nforgetting, the model will be compressed substantially in a lossy way. So, the proposed solution is\npractically worthy for probably 1 or 2 forget operations. One cannot say that the original large model\nwas kept so that cumulative data was forgotten at each forget process because that means that the\nhealth institute is not actually forgetting the patient information and keeping a version all the time.\nComparison to the established literature: Literature is growing around forgetting, as covered by the\npaper, including model-agnostic methods, model-intrinsic methods and data-driven methods. However,\nthe paper needs to convincingly explain why a knowledge distillation (model compression) based\napproach is required for real-time intelligent healthcare solutions. Why do the authors compare the\nsolution only with knowledge distillation approach (teacher and student/compressed models) instead of\nthe existing forgetting solution listed/cited in the paper?\nConclusions and claims: The paper illustrates the proposed and developed AFS as a generalized solution\n(different datasets and deep learning networks) to deep-learning-based systems starting from intelligent\nhealthcare. However, the overall solution and evaluation are far from this conclusion and claim.\n- Auditing: The paper proposes four metrics but mainly evaluates and uses p-value as the auditing\nmetrics. If the p-value is sufficient, why does the paper need other metrics? Otherwise, why does the\npaper have some limited review of the other three metrics?\n- Auditing: There is no clarity around how QO (query dataset overlapped with the training dataset) and\nQNO (query dataset disjoint with the training dataset) are sampled. The way they are sampled can\nchange results substantially. It is possible to engineer a QNO to support the claim.\n- Auditing (Figure 4B results) Are these results valid for all datasets and structures?\n- Auditing (Figure 4E results) Why are the ASD P-value results so distinctive? The graphs do not show\nPercentage (k) at zero, which should correspond to QNO. Is there any specific reason?\n- Auditing: More clarity is required to conclude that the results in Figure 4 show the robustness of AFS in\ndetermining that the targeted data is forgotten.\nFlaws in data analysis, interpretation and conclusions: Why do authors compare AFS with the\nindependent student (k) instead of the independent teacher with data to be forgotten is removed from\nthe training set? Independent teacher would form a better ground truth. Additionally, the independent\nstudent (k=0.25) is performing very close and better in some cases. Why does the paper drop it and\nfocus on the independent student (k=0.5) for the rest of the paper (Tables 3 and 4)? F1-Score is\nproviding very close results. What is the reasoning behind the conclusion that AFS performs substantially\nbetter? Moreover:\n- Auditing/Forgetting: COVIDx results show a quite different trend. What is the reason for it?\n- Auditing/Forgetting (Tables 1, 2, 3 and 4): The proposed solution shows its effect for sufficiently large\nquery sets (500 and above). However, is it really practical to reach those sizes in a real-life scenario\nwhere a single individual withdraws consent and asks for data removal?\n- Auditing/Forgetting: The conclusion that “AFS could help institutions and companies to efficiently\niterate their models to forget individual information at the model level” is not agreeable. AFS can be\nused for a small amount of iteration as the model gets compressed at each forget operation. If a hospital\nreceives multiple forget requests spread over a long period, those requests cannot be handled using this\napproach.\nMethodology: There are some flaws in the methodology. First, using a distillation scheme for forgetting\nmay solve the problem for one iteration but create much bigger problems later on. The paper also needs\nto utilize the introduced metrics more effectively to support the claims. Comparison is tailored to\nsupport the claim but is far from convincing. The approach should be compared with the existing\nforgetting solution or at least the teacher model (using a dataset where individual data is removed) as\nthe ground truth.\nReproducibility: The authors provide links to code and dataset to reproduce the solutions.\nEditing:\n- Paper needs some language improvements, for example long sentences make it hard to read:\n“However, with the development of data science, machine learning and deep learning techniques, this\nright is usually neglected or violated as more and more patients’ data are being collected and used for\nmodel training, especially in intelligent healthcare, thus making intelligent healthcare a sector where\ntechnology must meet the law, regulations, and privacy principles to ensure that the innovation is for\nthe common good.”\n-The abstract and introduction do not communicate the contribution and its significance correctly. The\nreader needs to go through several sections to understand the exact approach.\n-Figure 1 is not showing the proper flow and confuses the reader. The left side of the figure illustrates\nthe high-level iterative flow of the solution, while the right side is about how forgetting and auditing work\ntogether. Those two parts together becomes quite confusing.\n- What are the numerical values in Table 1, p-values?", "reviewer_2": "Summary of the paper:\nThis paper presents a novel unified method AFS, which unifies forgetting and auditing tasks through\nknowledge purification (KP), allowing auditing to guide the forgetting process in a negative feedback\nmanner. AAFS is tested on four different datasets (MNIST, PathMNIST, COVIDx, and ASD) with various\ndeep learning architectures (MLP, CNN, ResNet), demonstrating its applicability and potential for real-\nworld applications in intelligent healthcare. Additionally, accuracy and F1-score are used as evaluation\nmetrics for multi-class and binary classification tasks. The p-value of the two-sample statistical test is\nused to evaluate the membership of the query dataset.\nStrengths:\nAFS addresses the need for better protection of people's privacy and the right to revoke their data,\nparticularly in the context of intelligent healthcare.\nThe method is model-agnostic and open-source, making it broadly applicable to different deep learning\nmodels and tasks.\nThe paper provides a comprehensive evaluation of AFS on various datasets and tasks, including medical\nimages and EHR.\nWeaknesses:\nThe current version of AFS has not been tested on large-scale models and datasets commonly found in\nreal-world production environments.\nThe study relies on a limited set of auditing metrics and only classification problems, which could be\nfurther expanded.\nDeveloping methods for individual-level auditing may be crucial for enhancing the applicability of AFS in\nreal-world scenarios.\nMain Reviews, Suggestions, and Limitations:\nThe paper omits the full name of AFS, but it can be inferred that AFS likely stands for Auditing and\nForgetting System; mentioning this in the abstract or introduction would be beneficial.\nIt would be helpful for the authors to further clarify the differences and novelty of AFS compared to\nother methods.\nTo better understand AFS's applicability and effectiveness in real-world production environments,\nscaling it to larger models and more data is essential.\nAlthough the authors focus on four classification tasks, it would be interesting to explore the idea of\nexpanding AFS to regression tasks, which may present challenges in incorporating additional auditing\nmetrics.\nDespite the aforementioned limitations, this reviewer still believes that the proposed AFS method\ncontributes significantly to enhancing privacy protection and data revocation rights. The paper can be\npublishable if the listed limitations are addressed."}, "author_response": "Authors’ Response to Reviews of\nNCOMMS-23-05640-T\nAudit to Forget: A Unified Method to Revoke Patients’ Pri-\nvate Data in Intelligent Healthcare\nJuexiao Zhou1,2,#, Haoyang Li1,2,#, Xingyu Liao1,2, Bin Zhang1,2, Wenjia He1,2, Zhongxiao Li1,2, Longxi\nZhou1,2, Xin Gao1,2,∗\n1Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division,\nKing Abdullah University of Science and Technology (KAUST), Thuwal 23955-6900, Kingdom of Saudi\nArabia\n2Computational Bioscience Research Center, King Abdullah University of Science and Technology, Thuwal\n23955-6900, Kingdom of Saudi Arabia\n∗Corresponding author\n#Equal contribution\nRC: Reviewers’ Comment, AR: Authors’ Response, □ Manuscript Text\n1. General Response\nAR: We really appreciate the editor [redacted] and the Reviewer’s comments as well as constructive sugges-\ntions and have revised the manuscript accordingly. We are pleased to see that Reviewer 2 agrees that\nour manuscript has the potential for publication with major revisions and agrees that our method\ncontributes significantly to enhancing privacy protection and data revocation rights. We are also happy\nto see that both reviewers agree that the problem we are working on is significant in the health domain.\nTherefore, we carefully considered all comments and made necessary changes to improve the quality and\nimpact of our work. We believe that we have addressed the concerns raised by the reviewers to the best of\nour ability by either adding new results or clarifying misunderstandings and are confident that our research makes\na valuable contribution to the field. We have highlighted the edited places in the manuscript with a yellow\nhighlight to make the revised portions of the manuscript clear to the reviewers.\n1\n2. Response to Reviewer 1’s Comments\n2.1. Summary\nRC: The paper focuses on developing tools and methods to support the right to be forgotten in the digital world\nwithin a scenario where patients’ private data is used in ML-based intelligent healthcare solutions. The\npaper proposes a unified solution that can audit the use of patients’ private data and ensure that the target\nmodel forgets the personal data when requested. For auditing, given a DL model and query dataset used\nfor training, the developed AFS solution uses a query dataset to be audited and determine if this dataset has\nbeen used in training. AFS use an existing solution called Ensembled Membership Auditing (EMA), which\nis based on Membership Inference Attacks (MIA). The authors claim that their re-implemented version of\nEMA as part of AFS is quicker and easier for auditing use (not experimentally evaluated/compared in the\npaper).\nAR: We would like to express our sincere appreciation to the reviewer for taking the time to review our manuscript\nand providing us with constructive feedback. We understand the significance of valuable feedback and\nare grateful for the reviewer’s thoughtful comments and suggestions. We are particularly grateful for the\nreviewer’s feedback regarding the lack of experimental evaluation or comparison in our paper regarding the\nclaim that our re-implemented version of EMA as part of AFS is quicker and easier for auditing use.\nWe acknowledge that our manuscript does not include an experimental evaluation or comparison of our\nre-implemented version of EMA. In the previous version of our manuscript, we made the claim based on\nour experience and observations during the development and implementation of the AFS method, as well\nas feedback from auditors who have used our method. Once again, we appreciate the reviewer’s valuable\nfeedback and are grateful for the opportunity to improve our manuscript.\nTo address the concern raised by the reviewer, we have measured the time required for AFS and EMA for\nauditing and have provided a detailed comparison of the two approaches as below.\nSuppose we have a pre-trained model best_model.pth and a query dataset with label query_100, we can\nsimply audit with AFS in one command by providing two arguments as below:\npython afs.py audit --model2audit best_model.pth --query_label query_100\nHowever, EMA does not provide such arguments, users need to modify their codes accordingly (https:\n//github.com/Hazelsuko07/EMA).\nMeanwhile, AFS uses parallel computing to accelerate the calculations during auditing. As shown in\ndataset. Meanwhile, as the query dataset grows in size, the time difference between AFS and EMA increases.\nMeanwhile, we must call Audit in every training round in practice, thus the Audit method offered by AFS\nproves more beneficial in terms of overall time requirements.\nWe have updated the manuscript accordingly on Page 3 and added the figure to the supplementary file. Here\nare the modifications made to the main text:\nTo audit the membership of the query dataset, AFS takes a pre-trained DL model and the query dataset\nas inputs, and determines whether the query dataset has been used for training the target DL model.\nThis function was re-implemented based on EMA, a published MIA-based method to evaluate the\nmembership of a query dataset. Our re-implementation allows quicker and easier usage of auditing by\nintroducing parallel computing. (Section 2.4 and Supplementary Figure 1).\nseconds.\nRC: For forgetting, the authors implement a variant of the knowledge distillation approach within their AFS\nsolution. Knowledge distillation generally uses a larger and more complex teacher model to train a\nsmaller model that can achieve a similar convergence. This is achieved by establishing a correspondence\n4\nbetween intermediate outputs of teacher and student models. The proposed knowledge purification process\nbuilds over the knowledge distillation process and eliminates correspondence between teacher and student\nnetworks due to forgotten data queries. Hence, the overall distillation process also ensures that a specific\npart of training data is forgotten. However, with teacher-student settings, theoretical search space gets\nsmaller with smaller convergence space. This may mean faster model drift. Developed AFS solution has\nbeen evaluated using 4 datasets (MNIST, PathMNIST, COVIDx, ASD).\nAR: Thank the reviewer for the valuable feedback. We understand the concern regarding faster model drift in a\nteacher-student setting, where the theoretical search space may decrease with a smaller convergence space.\nHowever, we would like to point out that when using AFS, the size of the student model can be manually\nadjusted, allowing us to tailor the model size according to specific requirements. Our KP algorithm differs\nfrom KD in two significant ways. Firstly, we need to selectively transfer information, which has not been\npreviously investigated. Secondly, our KP does not necessarily demand that the student model is smaller than\nthe teacher model. If our student model already satisfies the size requirement, we can allow both the teacher\nmodel and the student model to be the same size.\nAs the reviewer suggested, we can start with a smaller student model size than the teacher model in the initial\nrounds for compression purposes. But rather than continuously generating smaller student models in the\nforgetting process, we can maintain the same size for both the student and teacher models once the student\nmodel reaches a certain size, which is small enough to meet our needs. At this point, knowledge purification\nonly forgets information and transfers the remaining knowledge without compressing the model. To address\nthis concern, we have added experiments later in the response letter to further illustrate this point.\nWe have also updated the Discussion section of the manuscript accordingly on Page 11. Here are the\nmodifications made to the main text:\nIn practice, the size of the student model could be manually adjusted when applying AFS to meet\nspecific requirements. In the initial stages, we could make the student model smaller than the teacher\nmodel to achieve model compression while forgetting private information. However, instead of continu-\nously generating smaller and smaller student models as knowledge is forgotten, we could maintain the\nsize of the student model once it reaches a certain threshold (e.g., small enough to meet our needs). In\ndoing so, AFS would focus solely on forgetting knowledge and transferring the remaining knowledge,\nwithout the need for further model compression.\nRC: Noteworthy results: The paper focuses on building a solution that brings the right to forgetting the patients’\nprivate data used in ML-based intelligent healthcare solutions. The overall solution is a combination of\n5\ntwo components: Auditing and Forgetting. Auditing is based on existing solutions. The authors evaluate\nmetrics (P-Value, Correctness, Confidence and Entropy) and show that these metrics can be used effectively\nin auditing. Forgetting is an incremental solution adapting the well-established knowledge distillation\napproach. The overall solution is implemented and tested on four datasets. Overall, novelty is very limited\nand not significant. But, a decent engineering and development effort is involved in evaluating and building\na combined solution.\nAR: We would like to express our gratitude to the reviewer for the valuable feedback on our work. We appreciate\nthe reviewer’s recognition of our efforts in engineering and development.\nWe believe that our study presents a novel contribution to the field for several reasons. First, our research\naddresses a unique gap in the literature by integrating Audit and Forget into a unified workflow for the\nfirst time, rather than treating them as separate components. We have shown that Audit and Forget are\nalgorithmically linked, and our approach provides a novel solution to this problem. Moreover, our study\nintroduces the concept of knowledge purification, which could be a promising avenue for future research\ninstead of a simple incremental solution. Our KP algorithm differs from KD in two significant ways. Firstly,\nwe need to selectively transfer information, which has not been previously investigated. Secondly, our KP\ndoes not necessarily demand that the student model is smaller than the teacher model.\nWe acknowledge the reviewer’s concern about the novelty of our work, but we believe that our unique research\nquestion and methodology justify its contribution to the field. We thank the reviewer once again for the\nthoughtful comments and suggestions.\nWe have updated the Introduction section of the manuscript accordingly on Page 2. Here are the modifications\nmade to the main text:\nHere, we proposed a novel solution by using auditing to guide the forgetting process in a negative\nfeedback manner. We unified the two tasks by introducing knowledge purification (KP), a new\napproach to selectively transfer the needed knowledge to forget the target information instead of simply\ntransferring all information like knowledge distillation (KD). On the basis of KP, we have developed\na user-friendly and open-source audit to forget software (AFS), which can be easily used to revoke\npatients’ private data from DL models in intelligent healthcare. To demonstrate the generality of\nAFS, we applied it to four tasks based on four datasets, including the MNIST dataset, the PathMNIST\ndataset, the COVIDx dataset, and the ASD dataset, with different data sizes and various architectures\nof deep learning networks. Our results demonstrate the usability of AFS and its application potential\nin real-world intelligent healthcare to enhance privacy protection and data revocation rights. To our\nknowledge, AFS is the first unified method of auditing and forgetting that could effectively forget the\ninformation of the target query dataset from the pre-trained DL model with the guidance of auditing.\nAFS could generate a smaller model, which requires much less time and GPU memory during the\ninference, by training with a partial training dataset (∼50%) with our novel KP approach.\nRC: Significance to the field and related fields: The problem that authors engage in is a significant problem in\nthe health domain and domains where ML-based approaches are used along with data with individuals.\nForgetting is a critical requirement in most regulations, but there needs to be a well-established solution to\nbe used with complex deep-learning models, especially in commercial settings. However, the solution works\nbased on knowledge distillation (model compression – Figure 5 B). So, after each data forget operation,\nthe model will be compressed significantly. That means the theoretical search space gets smaller with a\nsmaller convergence space - i.e., there is some loss (not evaluated in the paper). The question is, how many\ntimes can an intelligent health solution repeat this iteratively? After each forgetting, the model will be\ncompressed substantially in a lossy way. So, the proposed solution is practically worthy for probably 1 or\n2 forget operations. One cannot say that the original large model was kept so that cumulative data was\nforgotten at each forget process because that means that the health institute is not actually forgetting the\npatient information and keeping a version all the time.\nAR: Thank the reviewer for the comments. We really appreciate the reviewer’s recognition of the importance\nof our research. However, we believe there may have been a misunderstanding regarding the impact of\nthe forgetting process on model compression and the search space. We apologize for the confusion and\nmisunderstanding caused by our unclear description.\nAs we have explained previously, the size of the student model can be manually adjusted, allowing us to\nadapt it to our needs. Instead of generating progressively smaller student models as we forget data, we can\nmaintain the size of the student model once it reaches a certain size that is small enough for our requirements.\nTherefore, we respectfully disagree with ‘So, after each data forget operation, the model will be compressed\nsignificantly.’ Because at this point, knowledge purification could only forget knowledge and transfer the\nremaining knowledge, without compressing the model further.\nTo further address this concern, we designed an experiment to continuously forget a batch of data from\nthe model as described below. Our results show that while there was some compression of the model, the\nreduction in search space was not significant, and the model was still able to converge to a similar accuracy\nas before the forgetting process. We hope this clarifies our approach to this concern, and we appreciate the\nopportunity to address it in more detail as below.\nTo begin, we trained the teacher model with the maximum number of parameters (671,754) using the complete\n7\ntraining set containing 10,000 samples. Additionally, we created five continuous query datasets (QF_1, QF_2,\nQF_3, QF_4, QF_5), each consisting of 500 samples (5%), which overlapped with the training dataset but\nwere independent of each other. Next, we utilized AFS to continuously forget the information of the five\nquery datasets from the previous model. For instance, we applied AFS to forget QF_1 from the original\nteacher model, resulting in a small model with 155,658 parameters. Since the model size was satisfactory\n(small enough), we proceeded to forget QF_2 from the small model, generating a new model with the same\nsize. The results are shown in Response Table 1. Based on the results, we are confident that AFS can be used\nfor continuous forgetting.\nWe also acknowledge that as the model forgets more and more data (From QF1 to QF5), its performance\ndeteriorates as it retains less knowledge of the initial training data. However, this is usually not an issue in\npractice, as companies often possess training sets consisting of millions, or even tens of millions, of data.\nModels trained on such extensive datasets are unlikely to experience a significant impact on their performance\neven if we need to forget a small fraction of the data (eg. 0.1%). Therefore, we can permit a certain degree of\nconsecutive forgetting with AFS.\nModel name Accuracy F1-score QF1 QF2 QF3 QF4 QF5 number of parameters\nOriginal teacher model 0.9644 0.9911 1 1 1 1 1 671,754\nModel from AFS with QF1 forgotten 0.9326 0.9844 7.20e-10 8.13e-02 1.58e-01 8.65e-01 7.94e-01 155,658\nModel from AFS with additional QF2 forgotten 0.9301 0.9835 4.06e-10 5.70e-15 2.53e-02 1.09e-02 9.24e-03 155,658\nModel from AFS with additional QF3 forgotten 0.9251 0.9821 7.38e-09 7.09e-12 2.18e-20 5.75e-04 6.55e-03 155,658\nModel from AFS with additional QF4 forgotten 0.9237 0.9853 9.29e-09 3.12e-09 3.46e-14 2.20e-12 7.69e-04 155,658\nModel from AFS with additional QF5 forgotten 0.9253 0.9830 2.93e-07 2.16e-08 8.91e-11 1.27e-09 2.71e-16 155,658\nTable 1: Continuously forgetting with AFS\nRC: Comparison to the established literature: Literature is growing around forgetting, as covered by the\npaper, including model-agnostic methods, model-intrinsic methods and data-driven methods. However, the\npaper needs to convincingly explain why a knowledge distillation (model compression) based approach is\nrequired for real-time intelligent healthcare solutions. Why do the authors compare the solution only with\nknowledge distillation approach (teacher and student/compressed models) instead of the existing forgetting\nsolution listed/cited in the paper?\nAR: We appreciate the reviewer’s suggestion to compare our proposed approach with other existing methods in the\nfield. We did not compare more methods in the previous manuscript due to the fact that a lot of the existing\nresearch is purely theoretical and lacks practical implementation through usable code. Furthermore, many\n8\nof these theoretical approaches have acknowledged limitations. Still, we acknowledge the importance of\ncomparing other methods.\nTo address this concern, we have implemented and added new experiments to compare AFS with the most\nclassical and famous solution in the community, named SISA (Sharded, Isolated, Sliced, and Aggregated\ntraining). SISA aims to reduce the computational cost of the retraining process by employing a data\npartitioning technique as shown in Response Figure 2. For SISA, data is divided into shards, which are\nthemselves divided into slices. One constituent model is trained on each shard by presenting it with\nincrementally many slices and saving its parameters before the training set is augmented with a new slice.\nWhen data needs to be unlearned, only one of the constituent models whose shards contain the point to be\nunlearned needs to be retrained. Retraining can start from the last parameter values saved before including\nthe slice containing the data point to be unlearned. However, the main drawback of SISA is that we need\nmore space to keep the models corresponding to each shard. Our new experimental results indicate that\nAFS outperforms SISA in terms of forgetting, model size reduction, inference speed, GPU memory, and\nelimination of the need to store models for each shard.\nFigure 2: Illustration of SISA.\nWe also compared AFS to two state-of-the-art methods named Catastrophic Forgetting-k (CF-k) and Exact\nr\nlayers, while EU-k means to retrain the last k layers of the original model from scratch on D and freeze\nr\nother layers, where the D stands for the retain dataset.\nr\nWe have extended the introduction accordingly on Page 2. Here are the modifications made to the main text:\nHowever, most of them are theoretical studies and do not provide open-source codes. Besides, most\nmethods have their specific application scenarios and acknowledged limitations and few of them focused\non the application in real-world intelligent healthcare. Among the three methods, model-agnostic\nmethods might have the strongest application prospects, as they can be applied to different models, and\nSISA, short for Sharded, Isolated, Sliced, and Aggregated training, is the most classic and well-known\nmethod in the community. As the state-of-the-art method, Goel et al. proposed Catastrophic Forget-\nting-k (CF-k) and Exact Unlearning-k (EU-k) to unlearn information from deep learning models. CF-k\nmeans to finetune the last k layers of the original model on D and freeze other layers, while EU-k\nr\nmeans to retrain the last k layers of the original model from scratch on D\nr\nand freeze other layers,\nwhere the D stands for the retain dataset.\nr\nWe have also extended the manuscript accordingly on Page 7 to list all methods we have compared. Here are\nthe modifications made to the main text:\nOnce the prior knowledge that a dataset has been used to train the target DL model is confirmed\nwith auditing, AFS could be used for forgetting, to remove the information of the dataset from the\npre-trained DL model. To comprehensively show the ability of AFS in removing information against\nthe model performance, we compared nine methods, including 1) training the teacher model with\na complete training dataset (Independent teacher), 2) retraining the student model with a complete\ntraining dataset (Independent student), 3) retraining the teacher model with k ∈ {0.25, 0.5, 0.75}\npercentage of the complete training dataset excluding the data to be forgotten (Independent teacher\nwith k ∈ {0.25, 0.5, 0.75}), 4) retraining the student model with k ∈ {0.25, 0.5, 0.75} percentage\nof the complete training dataset excluding the data to be forgotten (Independent Student with k ∈\n{0.25, 0.5, 0.75}), 5) retraining the model of the corresponding shard with SISA, 6) fine-tuning the\nlast layers of the model with CF-k, 7) retraining the last layers of the model from scratch with EU-k,\n8) AFS, and 9) training the student model with AFS without the guidance of auditing (AFS w/o\nAudit), as an ablation study of AFS. Both AFS w/o Audit and AFS were also conducted with varied\nk ∈ {0.25, 0.5, 0.75}. For both Independent teacher and Independent student methods trained with the\n10\ncomplete training dataset, QF and QF were included in the training dataset, while these two\n100 1000\nquery datasets were excluded from the training dataset when k ∈ {0.25, 0.5, 0.75}.\nWe have also updated Tables 1∼6 (Response Tables 2∼7) and Figure 5 (Response Figure 3) in the main text\naccordingly as below.\nMethods\n1 10 100 500 1000 2000 1 10 100 500 1000 2000\nIndependent teacher 1 1 1 1 1 1 1 7.32e-01 7.39e-02 2.43e-05 1.06e-08 4.68e-18\nIndependent student 1 1 1 3.43e-01 4.22e-01 8.61e-02 1 6.96e-01 4.50e-02 2.74e-06 4.51e-11 8.58e-22\nIndependent teacher (k=0.75) 1 1 1 1 1 1 1 5.62e-01 8.84e-02 2.38e-05 3.93e-09 1.87e-19\nIndependent student (k=0.75) 1 1 1 6.80e-01 1.67e-01 7.87e-02 1 5.27e-01 2.28e-02 7.50e-07 4.61e-14 3.10e-27\nAFS w/o Audit (k=0.75) 1 1 1 8.64e-01 5.91e-01 5.26e-01 1 5.98e-01 1.02e-01 6.49e-06 4.15e-11 1.28e-20\nAFS (k=0.75) 1 1 8.64e-01 1 8.64e-01 4.54e-01 1 6.96e-01 2.94e-02 1.45e-06 9.06e-13 3.79e-24\nIndependent teacher (k=0.5) 1 1 1 1 1 1 1 6.96e-01 1.56e-02 1.84e-06 1.67e-12 4.81e-26\nIndependent student (k=0.5) 1 1 8.64e-01 5.59e-01 2.39e-01 2.49e-02 1 8.13e-01 7.75e-02 6.44e-08 1.63e-15 4.15e-30\nAFS w/o Audit (k=0.5) 1 1 1 8.64e-01 8.64e-01 8.64e-01 1 5.98e-01 1.32e-01 5.94e-06 2.43e-12 5.73e-23\nAFS (k=0.5) 1 1 1 8.64e-01 1 7.27e-01 1 6.68e-01 2.88e-02 9.25e-07 3.70e-13 3.10e-27\nIndependent teacher (k=0.25) 1 1 1 1 1 1 1 8.66e-01 5.27e-02 6.05e-06 3.24e-18 9.75e-34\nIndependent student (k=0.25) 1 1 1 8.64e-01 8.64e-01 3.17e-01 1 5.98e-01 1.04e-02 6.40e-10 3.05e-20 1.22e-38\nAFS w/o Audit (k=0.25) 1 1 1 1 1 1 1 5.27e-01 4.47e-02 7.49e-07 2.18e-13 5.96e-28\nAFS (k=0.25) 1 1 1 1 1 1 1 6.96e-01 8.52e-03 3.85e-10 1.98e-20 1.29e-41\nSISA (10 shards) 1 1 1 1 1 1 1 7.08e-01 7.20e-02 4.61e-05 3.81e-09 6.34e-21\nCF-k (k=1) 1 1 1 1 1 1 1 7.32e-01 1.11e-01 7.64e-05 5.48e-08 2.32e-17\nEU-k (k=1) 1 1 1 8.66e-01 1 4.91e-01 1 5.98e-01 4.10e-02 1.29e-05 7.96e-11 2.90e-22\nTable 2: Comparison of AFS with other methods on auditing QO and QNO from the MNIST dataset with a\nvaried number of samples in the query dataset. The data in the table shows the results of auditing QO and\nQNO on models trained by different methods. The numerical values in the table represent p-values. A larger\nvalue indicates stronger membership.\nIndependent teacher 1 1 0.9622 0.9911\nIndependent student 1 1 0.9504 0.9911\nIndependent teacher (k=0.75) 1.57e-01 1.05e-08 0.9548 0.9915\nIndependent student (k=0.75) 4.36e-02 5.26e-12 0.9458 0.9880\nAFS w/o Audit (k=0.75) 3.19e-01 1.33e-06 0.9582 0.9884\nAFS (k=0.75) 1.08e-03 5.22e-23 0.9470 0.9889\nIndependent teacher (k=0.5) 1.56e-01 2.71e-11 0.9437 0.9898\nIndependent student (k=0.5) 6.91e-03 2.34e-15 0.9282 0.9848\nAFS w/o Audit (k=0.5) 1.57e-01 7.79e-07 0.9526 0.9884\nAFS (k=0.5) 1.27e-02 9.44e-22 0.9380 0.9866\nIndependent teacher (k=0.25) 1.56e-01 2.80e-18 0.9176 0.9871\nIndependent student (k=0.25) 6.91e-03 2.90e-19 0.9067 0.9875\nAFS w/o Audit (k=0.25) 1.57e-01 7.04e-10 0.9388 0.9875\nAFS (k=0.25) 5.79e-04 6.84e-33 0.9174 0.9853\nSISA (10 shards, QF excluded, k=0.99 for QF 100 and 0.9 for QF _1000) 3.18e-01 6.12e-09 0.9568 0.9911\nCF-k (QF excluded) 1 1 0.9608 0.9907\nEU-k (QF excluded) 1 1.41e-02 0.9504 0.9898\nTable 3: Comparison of AFS with other methods on forgetting QF and model performance with the MNIST\ndataset. QF is the small query dataset containing 100 samples and QF is the large query dataset\n100 1000\ncontaining 1000 samples. We present the p-values of auditing models trained with different methods on\nQF and QF and the model performance including the accuracy and F1-score.\n100 1000\nMethods\n1 10 100 500 1000 2000 1 10 100 500 1000 2000\nIndependent teacher 1 1 1 1 1 1 1 4.29e-01 6.29e-03 1.40e-13 4.91e-28 1.88e-60\nIndependent student 1 1 1 1 1 1 1 5.27e-01 5.11e-03 1.29e-10 3.38e-20 7.24e-42\nIndependent teacher (k=0.75) 1 1 1 1 1 1 1 5.62e-01 2.76e-03 4.22e-16 5.52e-27 1.62e-60\nIndependent student (k=0.75) 1 1 1 1 1 1 1 7.32e-01 1.89e-02 1.55e-08 5.74e-16 1.92e-30\nAFS w/o Audit (k=0.75) 1 1 5.26e-01 1.78e-01 2.54e-02 1.17e-03 1 2.95e-01 2.26e-02 9.23e-11 1.04e-19 8.21e-41\nAFS (k=0.75) 1 1 3.90e-01 3.21e-02 6.13e-04 7.19e-06 1 4.98e-01 2.34e-04 4.25e-16 3.95e-31 4.89e-65\nIndependent teacher (k=0.5) 1 1 1 1 1 1 1 5.62e-01 5.03e-03 1.50e-12 3.36e-22 2.39e-48\nIndependent student (k=0.5) 1 8.66e-01 1 8.64e-01 8.64e-01 5.58e-01 1 5.62e-01 4.02e-03 3.01e-10 1.91e-21 7.85e-40\nAFS w/o Audit (k=0.5) 1 1 5.11e-01 2.48e-01 1.24e-02 9.59e-04 1 6.26e-01 1.18e-02 2.99e-11 4.03e-21 2.51e-40\nAFS (k=0.5) 1 1 1.59e-01 3.45e-04 4.06e-06 1.85e-10 1 2.69e-01 9.17e-05 2.98e-16 3.84e-30 1.04e-62\nIndependent teacher (k=0.25) 1 1 1 1 1 1 1 3.31e-01 5.98e-04 2.84e-12 1.62e-28 6.34e-52\nIndependent student (k=0.25) 1 1 8.64e-01 6.95e-01 1.60e-01 4.54e-02 1 1.26e-01 1.24e-06 2.30e-31 2.88e-60 4.73e-109\nAFS w/o Audit (k=0.25) 1 8.66e-01 2.39e-01 3.37e-03 1.01e-05 9.15e-10 1 2.42e-01 3.64e-05 2.55e-24 3.01e-43 1.37e-90\nAFS (k=0.25) 1 8.66e-01 1.40e-01 2.71e-03 1.07e-06 1.63e-12 1 4.37e-01 7.07e-06 2.01e-27 4.45e-53 1.71e-101\nSISA (10 shards) 1 1 1 1 1 1 1 6.71e-01 6.58e-03 2.11e-12 9.05e-27 5.85e-49\nCF-k (k=1) 1 1 1 8.63e-01 8.63e-01 1 1 3.75e-01 7.28e-03 9.10e-14 1.07e-21 1.48e-42\nEU-k (k=1) 1 1 1 6.96e-01 7.38e-01 6.92e-01 1 4.73e-01 1.60e-03 4.66e-11 1.47e-23 1.48e-44\nTable 4: Comparison of AFS with other methods on auditing QO and QNO from the PathMNIST dataset with\na varied number of samples in the query dataset. The data in the table shows the results of auditing QO and\nQNO on models trained by different methods. The numerical values in the table represent p-values. A larger\nvalue indicates stronger membership.\nIndependent teacher 1 1 0.8538 0.9885\nIndependent student 1 1 0.8446 0.9836\nIndependent teacher (k=0.75) 1.08e-03 3.11e-30 0.8214 0.9796\nIndependent student (k=0.75) 2.35e-02 4.08e-15 0.8396 0.9555\nAFS w/o Audit (k=0.75) 1.08e-03 1.67e-22 0.8682 0.9777\nAFS (k=0.75) 2.25e-05 2.05e-41 0.8560 0.9605\nIndependent teacher (k=0.5) 3.74e-03 2.91e-23 0.7100 0.8314\nIndependent student (k=0.5) 6.91e-03 2.99e-21 0.7934 0.9533\nAFS w/o Audit (k=0.5) 3.74e-03 4.93e-18 0.8494 0.9697\nAFS (k=0.5) 2.87e-06 4.75e-35 0.8242 0.9575\nIndependent teacher (k=0.25) 3.74e-03 2.52e-26 0.7026 0.8282\nIndependent student (k=0.25) 1.58e-07 9.05e-57 0.7582 0.9287\nAFS w/o Audit (k=0.25) 3.32e-07 2.05e-41 0.7842 0.9406\nAFS (k=0.25) 3.32e-07 1.84e-56 0.7810 0.9385\nSISA (10 shards, only QF excluded, k=0.99 for QF 100 and 0.9 for QF _1000) 2.15e-03 6.73e-29 0.8501 0.9840\nCF-k (QF excluded) 1 1.69e-02 0.8506 0.9839\nEU-k (QF excluded) 2.35e-02 7.69e-16 0.8328 0.9596\nTable 5: Comparison of AFS with other methods on forgetting QF and model performance with the PathM-\nNIST dataset. QF is the small query dataset containing 100 samples and QF is the large query dataset\n100 1000\ncontaining 1000 samples. We present the p-values of auditing models trained with different methods on\nQF and QF and the model performance including the accuracy and F1-score.\n100 1000\nDataset SISA (10 shards) CF-k/EU-k\nmodel generated by AFS\nMNIST 438µs ± 1.28µs 439µs ± 1.41µs 439 µs ± 1.54µs 284µs ± 447ns\nPathMNIST 5.13ms ± 17µs 5.14ms ± 10µs 5.13ms ± 22µs 4.99ms ± 14.1µs\nCOVIDx 1.27s ± 355ms 1.28s ± 634ms 1.27s ± 500ms 661ms ± 9.98ms\nAutism 126µs ± 213ns 126µs ± 302ns 126µs ± 177ns 87.3µs ± 120ns\nTable 6: Time for inferring 100 samples with the model retrained sith SISA, the model retrained with\nCF-k/EU-k, the original model and the model generated by AFS.\nDataset CF-k/EU-k\n(10 shards) model generated by AFS\nMNIST 258 258 258 61\nPathMNIST 173 173 173 129\nCOVIDx 17,805 17,805 17,805 10,600\nAutism 2 2 2 1\nTable 7: GPU memory (MB) for inferring 100 samples with the model retrained sith SISA, the model retrained\nwith CF-k/EU-k, the original model and the model generated by AFS.\nquery dataset and a large query dataset (QF) and the accuracy of models trained with different methods,\nincluding Original (Independent teacher trained with the complete training dataset), SISA, Data Deletion\n(the Independent student model trained with partial training dataset and k = 0.5), AFS (w/o Audit) and\nAFS. B. The number of parameters for the original large model and the new small model generated by AFS.\nC. The qualitative evaluation of three methods, including Original (Independent teacher trained with the\ncomplete training dataset), SISA, CF-k, EU-k, Data Deletion (the Independent student model trained with\npartial training dataset and k = 0.5), and AFS on five dimensions (Ability to forget, accuracy, size of dataset\nneeded for training, size of the generated model and the efficiency of training). A larger value means a\nstronger ability to forget, higher model accuracy, a smaller size of dataset needed for training, a smaller size\nof the generated model, and better efficiency of training.\n18\n2.2. Conclusions and claims\nRC: The paper illustrates the proposed and developed AFS as a generalized solution (different datasets and\ndeep learning networks) to deep-learning-based systems starting from intelligent healthcare. However, the\noverall solution and evaluation are far from this conclusion and claim. Auditing: The paper proposes four\nmetrics but mainly evaluates and uses p-value as the auditing metrics. If the p-value is sufficient, why does\nthe paper need other metrics? Otherwise, why does the paper have some limited review of the other three\nmetrics?\nAR: We appreciate the feedback from the reviewer regarding our proposed solution and its evaluation. However, we\nrespectfully disagree with the reviewer’s assertion that our AFS solution and evaluation are not a generalized\nsolution. We designed our proposed AFS solution to be flexible and applicable to different datasets and deep\nlearning networks. To demonstrate the effectiveness and generalizability of our approach, we evaluated our\nsolution using four diverse datasets, namely MNIST, PathMNIST, COVIDx, and ASD. Additionally, we used\nmultiple deep-learning models to evaluate our solution, which suggests that our approach is not limited to\nspecific architectures.\nMeanwhile, we believe there may be a misunderstanding regarding the role of the P-value and the other\nauditing metrics due to our unclear writing. We apologize for the confusion and misunderstanding caused by\nour unclear description to the reviewer.\nAs pointed out by the reviewer, ‘The paper proposes four metrics but mainly evaluates and uses p-value as\nthe auditing metrics’. In fact, Correctness, Confidence, and Entropy are upstream metrics that contribute\nto calculating the P-value. Therefore, we primarily use the P-value as the auditing metric, as it is the most\ninformative and meaningful metric for our proposed auditing framework. The details of how these three\nmetrics are used to calculate the P-value are described in the main text. We hope this clarifies any confusion\nregarding our auditing metrics.\nWe have included more details about correctness, confidence, entropy, and their downstream metric, p-value,\non Page 5 of the main text. Here are the modifications made to the main text:\nEMA is designed as a 2-step process. In the first step, the best threshold for each metric is selected\nto optimize (TPR(t) + TNR(t))/2 based on the calibration dataset as shown in Algorithm 1. Once\nthe thresholds for all metrics are selected, the membership of each sample in the query dataset will\nbe confirmed as at least one metric is larger than the corresponding threshold. In total, three metrics,\nincluding correctness, confidence, and entropy, were adopted to further calculate the p-value, which is\nthe key audit metric in AFS as proposed in the previous work. The details for calculating correctness,\n19\nconfidence, and entropy are as below:\nCorrectness: The target model is trained to predict correctly on training data and may not generalize\nwell on test data. Thus we can define the correctness as:\nI (F, (x, y)) = 1{argmaxF (x) = y} (1)\ncorrectness i i\nwhere F is the deep learning model, x is the input data, F (x) is the output logits, y is the label, and 1\nis the indicator function.\nConfidence: The target model is usually more confident in predictions on training data, but less\nconfident in test data. Thus we can define confidence as:\nI (F, (x, y)) = 1{F (x) ≥ τ} (2)\nconfidence y y\nwhere F is the deep learning model, x is the input data, F (x) is the output logits, y is the label, F (x)\ny\nis the output logits for label y, τ is a threshold for the logit for label y, and 1 is the indicator function.\ny\nEntropy: The target model is trained by minimizing the prediction loss over training data and usually\nhas a larger prediction entropy on a test sample. Thus we can define entropy as:\nΣ\nI (F, (x, y)) = 1{− F (x)log(F (x)) ≤ τˆ} (3)\nentropy i i y\ni\nwhere F is the deep learning model, x is the input data, F (x) is the output logits, y is the label, F (x)\ni\nis the output logits for class i, τˆ is a threshold, and 1 is the indicator function.\ny\nOnce the membership of all samples in the query dataset is confirmed in the previous step, the query\ndataset will be further evaluated to determine whether the query dataset has been used to train the target\npre-trained DL model. A two-sample statistical test is adopted to evaluate the query dataset based on\nthe sample-wise membership and an all-one vector. The p-value of the two-sample statistical test is\nused as the output of auditing. Given a user-defined threshold α, if p < α, then users could conclude\nthat the query dataset was not used for training the target DL model. EMA was re-implemented and\nintegrated into AFS to allow easy and fast auditing.\nRC: Auditing: There is no clarity around how QO (query dataset overlapped with the training dataset) and\nQNO (query dataset disjoint with the training dataset) are sampled. The way they are sampled can change\n20\nresults substantially. It is possible to engineer a QNO to support the claim.\nAR: We appreciate the reviewer’s feedback on the sampling methodology of QO and QNO datasets. We acknowl-\nedge that we did not provide enough clarity on this matter in the original manuscript, and we apologize for the\nconfusion. We have updated the manuscript to provide a detailed explanation of the sampling methodology\nfor QO and QNO datasets.\nAs mentioned in the revised manuscript, QO and QNO datasets are randomly sampled, and all values presented\nin the paper are averaged based on multiple experiments with randomly sampled data. We understand the\nconcern that the way datasets are sampled can affect the results, but we have taken measures to mitigate\nthis risk by averaging on multiple experiments, using multiple datasets, and evaluating our solution on\nvarious deep-learning models. We believe that these steps make our results more robust and demonstrate the\neffectiveness of our approach. Once again, we thank the reviewer for their valuable feedback, and we have\nincorporated the suggestions in our manuscript to improve the quality of our research.\nWe also add more details regarding the sampling of different query datasets on Page 4. Here are the\nmodifications made to the main text:\nIn addition, we prepared query datasets with different sizes N from {1, 10, 100, 500, 1000, 2000}. A\nquery dataset that completely overlapped with the training dataset is labelled as QO, while the query\ndataset that is completely disjoint with the training dataset is labelled QNO. To further understand\nthe effect of the purity of the query dataset, we also prepared the query dataset called QM with a k\npercentage of the query dataset to be overlapped with the training dataset. Finally, for the query dataset\ndesigned to be forgotten, we labelled it as QF. QO, QNO, QM, and QF were all sampled randomly\nfrom the complete dataset and all reported values were the average of 5 replicate experiments.\nRC: Auditing (Figure 4B results) Are these results valid for all datasets and structures?\nAR: Yes, we present the result in Figure 4 and also added the figure to supplementary. We would like to reiterate\nthat the metrics of correctness, confidence, and entropy are utilized to calculate the p-value. In our experiments,\nwe observed that the QO and QNO datasets exhibit distinct distributions in these three metrics. Thus by\nleveraging these differences and calculating the p-value, we are able to differentiate between QO and QNO\nand use the p-value as the primary auditing metric for AFS.\nRC: Auditing (Figure 4E results) Why are the ASD P-value results so distinctive? The graphs do not show\nPercentage (k) at zero, which should correspond to QNO. Is there any specific reason?\nAR: We would like to thank the reviewer for the insightful question about the p-value results in Figure 4E for the\nASD dataset. The reason why the ASD p-value results are distinctive is due to the small size of the dataset.\nsamples for MNIST and PathMNIST, and 5000 samples for COVIDx (We even applied AFS to larger datasets\nand models, which could be found later in the response to Reviewer2).\nRegarding the percentage (k) at 0, we did not display it separately in Figure 4E because it was already shown\nin Figure 4D as QNO, when the query dataset size was 2000 for MNIST, PathMNIST, and COVIDx, and 100\nfor the ASD dataset. Similarly, the percentage (k) at 1 was already displayed in Figure 4D as QO when the\nquery dataset size was 2000 for MNIST, PathMNIST, and COVIDx, and 100 for the ASD dataset.\nWe appreciate the reviewer’s attention to detail, and we hope this clarifies any confusion. Meanwhile, we\nnotice the plot for ASD in Figure 4E in the previous manuscript was based on n=200 when we check the raw\ndata. We are sorry for the confusion. To make it consistent, we redraw the plot for ASD in Figure 4E and\nfixed the size of the query dataset to 100.\nWe also add more details regarding this concern on Page 6. Here are the modifications made to the main text:\nWhen the size of the query dataset and the calibration dataset varied, AFS could still efficiently\ndistinguish QO and QNO (Figure 4C and D). Compared to QO, AFS reported a much smaller p-\nvalue for QNO, indicating a weak membership (a small probability that the query dataset has been\nused to train the target DL model), thus allowing users to judge whether the query dataset was used\nto train the target DL model. Meanwhile, when the size of the dataset increased from 1 to 2000,\nAFS discriminated QO and QNO more confidently as there was a more significant divergence of\nthe p-values, which was not affected by the size of the calibration dataset. To further understand\nthe effect of the purity of the query dataset in auditing, we mixed some samples from the training\ndataset to QNO, thus the new query dataset was labelled as QM (partial data overlapped with the\ntraining dataset). The percentage of data overlapped with the training dataset in QM was denoted by\nk = number of data overlapped with training dataset . As shown in Figure 4E, AFS showed a decreasing\nsize of QM\np-value trend when k decreased, meaning that the query dataset was less likely to be used to train the\ntarget DL model. The distinctiveness of the p-value on the ASD dataset is due to the small size of the\nASD dataset. k = 0 was shown in Figure 4D as QNO, which occurred when the query dataset size was\n100 for ASD and 2000 for MNIST, PathMNIST, and COVIDx. Similarly, k = 1 was also displayed in\nFigure 4D as QO, which occurred when the query dataset size was 100 for ASD and 2000 for MNIST,\nPathMNIST, and COVIDx. In conclusion, these results indicate the robustness of AFS in determining\nwhether the query data has been used to train the target DL model.\nRC: Auditing: More clarity is required to conclude that the results in Figure 4 show the robustness of AFS in\ndetermining that the targeted data is forgotten.\nrequired to explain the robustness of AFS in determining that the targeted data is forgotten. To address this,\nwe have provided additional details in the manuscript to emphasize the consistency of our evaluation across\ndifferent datasets and deep-learning networks. Our proposed approach achieved a high p-value for query\ndatasets that were part of the training dataset and a small p-value for query datasets that were not overlapped\nwith the training dataset. We also included detailed explanations in the paper regarding the auditing process\nand the metrics used to evaluate the performance of AFS.\n2.3. Flaws in data analysis, interpretation and conclusions\nRC: Why do authors compare AFS with the independent student (k) instead of the independent teacher with\ndata to be forgotten is removed from the training set? Independent teacher would form a better ground\ntruth. Additionally, the independent student (k=0.25) is performing very close and better in some cases.\nWhy does the paper drop it and focus on the independent student (k=0.5) for the rest of the paper (Tables 3\nand 4)? F1-Score is providing very close results. What is the reasoning behind the conclusion that AFS\nperforms substantially better?\nAR: Thank the reviewer for the comment. We did not compare the ‘independent teacher with data to be forgotten\nis removed from the training set’ in our previous manuscript because we believe that the teacher model is\nlarger than the student model and therefore does not have a significant advantage in terms of GPU resources\nand inference time, which has been discussed in the manuscript (eg. Tables 5 and 6).\nStill, we appreciate the reviewer’s suggestion, and we have included the results for the independent teacher\nwith data to be forgotten removed from the training set in all tables (Tables 1∼4 (Response Tables 2∼5)),\nas Independent teacher (k=0.75 or 0.5 or 0.25). Although the independent teacher model is larger than the\nstudent model, we agree that it is still necessary to compare the performance of the two methods. We focused\non k=0.5 as we aimed to achieve a good trade-off between the amount of data needed for training, the average\nperformance on different datasets, and the ability to forget information. While the f1-score may provide\nsimilar results in some cases, when we consider those aforementioned indicators together, particularly in\nFigure 5 (Response Figure 3), we can conclude that AFS has the following strengths over other methods:\n1. AFS requires less data than training a model from scratch.\n2. AFS has a better ability to forget than simply deleting data or using SISA.\n3. AFS strikes a good trade-off between forgetting and model performance, as it achieves comparable accuracy\nwhile still being able to forget data.\n24\n4. AFS generates smaller models that are more efficient in computing resources and inference time, making it\nsuitable for deployment on resource-constrained devices.\nWe hope this clarifies the strengths of AFS and how it compares to other approaches for forgetting in deep\nlearning.\nRC: Auditing/Forgetting: COVIDx results show a quite different trend. What is the reason for it?\nAR: Thanks for the reviewer’s valuable comments. We appreciate the reviewer’s concern about the accuracy\ntrend of the COVIDx dataset in Figure 5, which is an interesting phenomenon worth discussing. We briefly\ndiscussed this phenomenon in the previous version of the manuscript in section 3.4. Upon comparing the\nCOVIDx dataset with the other three tasks, we found that the COVIDx dataset has two primary characteristics:\na large model size and high-dimensional data, which may be the potential causes of the different trend.\nWe have found that the student model trained by AFS has better generalization ability than the model obtained\nby using the exact same model structure and training data using only the hard-target training method. This\nis because knowledge distillation is known to improve student generalization, as discussed in Stanton S et\nal.’s paper \"Does knowledge distillation really work?\" (Advances in Neural Information Processing Systems,\n2021).\nWe also add more details regarding this concern on Page 10. Here are the modifications made to the main text:\nTo show the versatility of AFS, we applied it to the classification of pneumonia and normal with chest\nX-ray images from the COVIDx dataset with ResNet, which is a classic task in medical image analysis.\nAs shown in Figure 5A, on both query datasets (QF and QF ), AFS could effectively forget the\n100 1000\ninformation of the query dataset, while generating the new model with much less number of parameters\nas shown in Figure 5B. Surprisingly, the model generated by AFS showed even better accuracy than\nthe Independent teacher trained with the complete dataset and the Independent student trained with the\npartial training dataset. This result not only indicated that AFS could effectively transfer the knowledge\nfrom the teacher model to the student model but also suggested that the student model with simpler\narchitecture could even perform better than the teacher model with KP in AFS due to the reduction of\nmodel parameters and purification of knowledge in some real-world cases. Meanwhile, the results also\nshowed that the student model trained by the AFS achieved better generalization ability than the model\nobtained by using the exact same model structure and training data using only the hard-target training\nmethod.\nRC: Auditing/Forgetting (Tables 1, 2, 3 and 4): The proposed solution shows its effect for sufficiently large\nquery sets (500 and above). However, is it really practical to reach those sizes in a real-life scenario where\n25\na single individual withdraws consent and asks for data removal?\nAR: Thank the reviewer for the comments. As we have mentioned in the manuscript, AFS is not yet suitable for\nindividual-level forgetting due to its algorithm design. However, we have achieved good forgetting results on\na batch of query data. We do acknowledge the reviewer’s concern ‘However, is it really practical to reach\nthose sizes in a real-life scenario where a single individual withdraws consent and asks for data removal’.\nIn practical scenarios, large companies usually have millions of users or even trillions of users, so it is easy\nfor them to meet the required number of the query datasets. Additionally, when hospitals or companies receive\ncontinuous requests for data removal from individual patients, a possible simple but effective solution is to\ncollect and process the requests as a batch when the number of queries reaches the required threshold.\nWe also add more details regarding this concern on Page 12. Here are the modifications made to the main text:\nWith current laws that guarantee people the right to revoke their own data, AFS could help institutions\nand companies to efficiently iterate their models to forget individual information at the model level.\nHowever, there are still some shortcomings in the application of the current version of AFS in the\nproduction environment, which could be the main potential direction of research in the future. Firstly,\nthe models and data we tested in this study were still not large enough compared to the data in the\nreal production environment. Therefore, it is unknown whether scaling AFS to current large models\nwill cause new problems (eg. LLM like ChatGPT, which we are unable to further pursue). Secondly,\nthere are different approaches to audit, and thus we could add more metrics of auditing to AFS to\nguide the forgetting process in the future version. Meanwhile, due to the limitation of auditing, it is\nstill difficult to perform individual-level forgetting, as we need to compare the difference in statistical\ndistribution based on a fraction of data points, which could be the major possible improvement for the\nfuture version of AFS. Though AFS is not applicable for individual-level forgetting due to limitations\nof algorithm design, we can achieve favourable forgetting outcomes when operating on a batch of query\ndata. In real-world scenarios, companies with millions of users can easily meet the required number of\nthe query dataset. Furthermore, when hospitals or companies face continuous requests for forgetting\nindividual patient data, they can collect and store these requests, and perform the forgetting as a batch\nonce the amount of data reaches the required threshold. Finally, the current version of AFS could not\nbe applicable for regression tasks because the design of auditing metrics does not work for regression\ntasks and KP is limited to classification tasks only. Despite these limitations, we believe that AFS will\nmake a valuable contribution towards better protection of people’s privacy and the right to revoke data\nwith the rapid development of intelligent healthcare.\nRC: Auditing/Forgetting: The conclusion that “AFS could help institutions and companies to efficiently\niterate their models to forget individual information at the model level” is not agreeable. AFS can be\nused for a small amount of iteration as the model gets compressed at each forget operation. If a hospital\nreceives multiple forget requests spread over a long period, those requests cannot be handled using this\napproach.\nAR: We appreciate the reviewer’s feedback on the application of AFS in the real world. However, we respectfully\ndisagree with the reviewer’s concern. As we mentioned in our previous response, the size of the student\nmodel can be adjusted according to the actual requirements, and we can keep the size of the student model\nand the teacher model the same when the student model reaches a certain size. At this point, our knowledge\npurification only has the effect of forgetting knowledge and transferring the remaining knowledge, without\ncompressing the model. The additional results we provided earlier support our claim that AFS can help\ninstitutions and companies efficiently iterate their models to forget individual information at the model level\nconsecutively. Meanwhile, when the hospital receives multiple forget requests spread over a long period,\na possible simple but effective solution is to collect and process the requests as a batch when the number\nof queries reaches the required threshold. We believe that our approach can be adapted to meet real-world\nrequirements and challenges.\nOnce again, we appreciate the reviewer’s valuable feedback and are grateful for the opportunity to improve\nour manuscript.\n2.4. Methodology\nRC: There are some flaws in the methodology. First, using a distillation scheme for forgetting may solve the\nproblem for one iteration but create much bigger problems later on. The paper also needs to utilize the\nintroduced metrics more effectively to support the claims. Comparison is tailored to support the claim but\nis far from convincing. The approach should be compared with the existing forgetting solution or at least\nthe teacher model (using a dataset where individual data is removed) as the ground truth.\nAR: We appreciate the reviewer’s feedback on the methodology. We have taken the reviewer’s suggestion into\naccount and added the results of the teacher model, where the dataset with individual data removed was\nused. We have also included a comparison with SISA in the main text. We hope our previous responses have\naddressed this concern ‘using a distillation scheme for forgetting may solve the problem for one iteration but\ncreate much bigger problems later on.’\n2.5. Reproducibility\nRC: The authors provide links to code and dataset to reproduce the solutions.\nAR: We appreciate the reviewer’s valuable feedback and comments. Hope that our point-to-point response could\n27\naddress the reviewer’s concerns.\n2.6. Editing\nRC: Paper needs some language improvements, for example long sentences make it hard to read:\n“However, with the development of data science, machine learning and deep learning techniques, this\nright is usually neglected or violated as more and more patients’ data are being collected and used for\nmodel training, especially in intelligent healthcare, thus making intelligent healthcare a sector where\ntechnology must meet the law, regulations, and privacy principles to ensure that the innovation is for\nthe common good.”\nAR: We have modified as required and addressed other similar problems.\nFor example, we have rewritten the abstract as below:\nRevoking personal private data is one of the basic human rights, which has already been sheltered\nby several privacy-preserving laws in many countries. As data science, machine learning, and deep\nlearning techniques continue to advance, such right is often overlooked or infringed upon due to the\nincreasing collection and use of patient data for model training. The prevalence of such violations in\nintelligent healthcare highlights the need for technology to comply with laws, regulations, and privacy\nprinciples to ensure that innovation serves the common good. In order to secure patients’ right to\nbe forgotten, we proposed a novel solution by using auditing to guide the forgetting process, where\nauditing means determining whether a dataset has been used to train the model and forgetting requires\nthe information of a query dataset to be forgotten from the target model. We unified these two tasks by\nintroducing a new approach called knowledge purification. To implement our solution, we developed\nan audit to forget software (AFS), which is able to evaluate and revoke patients’ private data from\npre-trained deep learning models. To demonstrate the generality of AFS, we applied it to four tasks\nbased on four datasets, including the MNIST dataset, the PathMNIST dataset, the COVIDx dataset,\nand the ASD dataset, with different data sizes and various architectures of deep learning networks.\nOur results demonstrate the usability of AFS and its application potential in real-world intelligent\nhealthcare to enhance privacy protection and data revocation rights.. The software is publicly available\nat https://github.com/JoshuaChou2018/AFS.\nRC: The abstract and introduction do not communicate the contribution and its significance correctly. The\nreader needs to go through several sections to understand the exact approach.\nAR: We have updated the abstraction and introduction as required. We are very grateful for the reviewers’\nsuggestions.\nthe high-level iterative flow of the solution, while the right side is about how forgetting and auditing work\ntogether. Those two parts together becomes quite confusing.\nAR: We appreciate the feedback from the reviewer regarding Figure 1. We have added more explanation in the\ncaption of Figure 1 as suggested by the reviewer as below:\nAFS is a unified method to revoke patients’ private data in intelligent healthcare. The left side of the\nfigure illustrates the high-level iterative flow of AFS, while the right side illustrates the details of how\nforgetting and auditing work together. As shown on the left side, given a pre-trained DL model and\na query dataset (patients’ private data), AFS could audit and provide confidence whether the query\ndataset has been used to train the target DL model. When a dataset has been used to train the target DL\nmodel, AFS could effectively forget the information about the dataset from the target DL model with\nthe guidance of auditing. To achieve that, we proposed a novel method called knowledge purification\nas shown on the right side, which utilizes results from auditing as a new term in the loss function to\nforget information.\nRC: What are the numerical values in Table 1, p-values?\nAR: Yes, we have updated the captions of both Table 1 and Table 3 to reflect that the numerical values represent\np-values. Thank the reviewer for bringing this to our attention.\nComparison of AFS with other methods on auditing QO and QNO from the MNIST dataset with a\nvaried number of samples in the query dataset. The data in the table shows the results of auditing QO\nand QNO on models trained by different methods. The numerical values in the table represent p-values.\nA larger value indicates stronger membership.\n29\n3. Response to Reviewer 2’s Comments\n3.1. Summary\nRC: This paper presents a novel unified method AFS, which unifies forgetting and auditing tasks through\nknowledge purification (KP), allowing auditing to guide the forgetting process in a negative feedback\nmanner. AAFS is tested on four different datasets (MNIST, PathMNIST, COVIDx, and ASD) with various\ndeep learning architectures (MLP, CNN, ResNet), demonstrating its applicability and potential for real-\nworld applications in intelligent healthcare. Additionally, accuracy and F1-score are used as evaluation\nmetrics for multi-class and binary classification tasks. The p-value of the two-sample statistical test is used\nto evaluate the membership of the query dataset.\nAR: We would like to express our gratitude for the reviewer’s careful review of our manuscript. We recognize\nthe value of constructive criticism and are sincerely grateful for the reviewer’s insightful remarks and\nrecommendations.\n3.2. Strengths\nRC: AFS addresses the need for better protection of people’s privacy and the right to revoke their data,\nparticularly in the context of intelligent healthcare. The method is model-agnostic and open-source, making\nit broadly applicable to different deep learning models and tasks. The paper provides a comprehensive\nevaluation of AFS on various datasets and tasks, including medical images and EHR.\nAR: Thank the reviewer for acknowledging the importance and contribution of AFS to privacy protection and data\nrevocation rights, especially in the field of intelligent healthcare.\n3.3. Weaknesses\nRC: The current version of AFS has not been tested on large-scale models and datasets commonly found\nin real-world production environments. The study relies on a limited set of auditing metrics and only\nclassification problems, which could be further expanded. Developing methods for individual-level auditing\nmay be crucial for enhancing the applicability of AFS in real-world scenarios.\nAR: Thank the reviewer for the valuable feedback. We appreciate the reviewer’s acknowledgement of the\nlimitations of the current version of AFS, such as scalability and auditing metrics, as well as its focus\non classification problems. We agree with the reviewer’s concerns and have discussed these issues in the\ndiscussion section of our manuscript.\nconcerns or provided clarification to the best of our ability.\nHere are the modifications made to the discussion section of the main text:\nTo our knowledge, AFS is the first unified method of auditing and forgetting that could effectively\nforget the information of the target query dataset from the pre-trained DL model with the guidance of\nauditing. We designed AFS as a model-agnostic and open-source method that is applicable to different\nmodels. As shown in Figure 5C, AFS could generate a smaller model, which requires much less time\nand GPU memory during the inference (Tables 6 and 7), by training with a partial training dataset\n(∼50%) with our novel KP approach. Moreover, AFS could forget the information of the query dataset\nat the expense of an acceptable reduction in the model performance.\nOur experiments on four datasets showed that AFS was generalized for datasets of different sizes and\nforms, including medical images and EHR. Since deep learning models with different architectures\nwere applied to four tasks, we further demonstrated the broad applicability of AFS to common deep\nlearning models. In addition, our tasks include both binary classification and multiclassification tasks,\nwhich also suggested that AFS was applicable for tasks with multiple labels.\nIn practice, the size of the student model could be manually adjusted when applying AFS to meet\nspecific requirements. In the initial stages, we could make the student model smaller than the teacher\nmodel to achieve model compression while forgetting private information. However, instead of continu-\nously generating smaller and smaller student models as knowledge is forgotten, we could maintain the\nsize of the student model once it reaches a certain threshold (e.g., small enough to meet our needs). In\ndoing so, AFS would focus solely on forgetting knowledge and transferring the remaining knowledge,\nwithout the need for further model compression.\nWith current laws that guarantee people the right to revoke their own data, AFS could help institutions\nand companies to efficiently iterate their models to forget individual information at the model level.\nHowever, there are still some shortcomings in the application of the current version of AFS in the\nproduction environment, which could be the main potential direction of research in the future. Firstly,\nthe models and data we tested in this study were still not large enough compared to the data in the\nreal production environment. Therefore, it is unknown whether scaling AFS to current large models\nwill cause new problems (eg. LLM like ChatGPT, which we are unable to further pursue). Secondly,\nthere are different approaches to audit, and thus we could add more metrics of auditing to AFS to\nguide the forgetting process in the future version. Meanwhile, due to the limitation of auditing, it is\nstill difficult to perform individual-level forgetting, as we need to compare the difference in statistical\n31\ndistribution based on a fraction of data points, which could be the major possible improvement for the\nfuture version of AFS. Though AFS is not applicable for individual-level forgetting due to limitations\nof algorithm design, we can achieve favourable forgetting outcomes when operating on a batch of query\ndata. In real-world scenarios, companies with millions of users can easily meet the required number of\nthe query dataset. Furthermore, when hospitals or companies face continuous requests for forgetting\nindividual patient data, they can collect and store these requests, and perform the forgetting as a batch\nonce the amount of data reaches the required threshold. Finally, the current version of AFS could not\nbe applicable for regression tasks because the design of auditing metrics does not work for regression\ntasks and KP is limited to classification tasks only. Despite these limitations, we believe that AFS will\nmake a valuable contribution towards better protection of people’s privacy and the right to revoke data\nwith the rapid development of intelligent healthcare.\n3.4. Major concerns #1\nRC: The paper omits the full name of AFS, but it can be inferred that AFS likely stands for Auditing and\nForgetting System; mentioning this in the abstract or introduction would be beneficial.\nAR: Thank the reviewer for the valuable feedback. We appreciate the correction regarding the meaning of AFS.\nIt stands for Audit to Forget Software, and we have updated our abstract accordingly as below. We also\nappreciated the explanation of the Auditing and Forgetting System as it also sounded very fancy.\nRevoking personal private data is one of the basic human rights, which has already been sheltered\nby several privacy-preserving laws in many countries. As data science, machine learning, and deep\nlearning techniques continue to advance, such right is often overlooked or infringed upon due to the\nincreasing collection and use of patient data for model training. The prevalence of such violations in\nintelligent healthcare highlights the need for technology to comply with laws, regulations, and privacy\nprinciples to ensure that innovation serves the common good. In order to secure patients’ right to\nbe forgotten, we proposed a novel solution by using auditing to guide the forgetting process, where\nauditing means determining whether a dataset has been used to train the model and forgetting requires\nthe information of a query dataset to be forgotten from the target model. We unified these two tasks by\nintroducing a new approach called knowledge purification. To implement our solution, we developed\nan audit to forget software (AFS), which is able to evaluate and revoke patients’ private data from\npre-trained deep learning models. To demonstrate the generality of AFS, we applied it to four tasks\nbased on four datasets, including the MNIST dataset, the PathMNIST dataset, the COVIDx dataset,\nand the ASD dataset, with different data sizes and various architectures of deep learning networks.\nhealthcare to enhance privacy protection and data revocation rights.. The software is publicly available\nat https://github.com/JoshuaChou2018/AFS.\n3.5. Major concerns #2\nRC: It would be helpful for the authors to further clarify the differences and novelty of AFS compared to other\nmethods.\nAR: We would like to express our gratitude to the reviewer for the valuable feedback on our work. We believe\nthat our study presents a novel contribution to the field compared to other methods for several reasons. First,\nour research addresses a unique gap in the literature by integrating Audit and Forget into a unified workflow\nfor the first time, rather than treating them as separate components. We have shown that Audit and Forget\nare algorithmically linked, and our approach provides a novel solution to this problem. Moreover, our study\nintroduces the concept of knowledge purification, which could be a promising avenue for future research.\nWe can conclude that AFS has the following strengths over other methods:\n1. AFS requires less data than training a model from scratch.\n2. AFS has a better ability to forget than simply deleting data or using SISA.\n3. AFS strikes a good trade-off between forgetting and model performance, as it achieves comparable accuracy\nwhile still being able to forget data.\n4. AFS generates smaller models that are more efficient in computing resources and inference time, making it\nsuitable for deployment on resource-constrained devices.\nWe hope this clarifies the strengths of AFS and how it compares to other approaches for forgetting in deep\nlearning. We thank the reviewer once again for the thoughtful comments and suggestions.\nTo further address this concern, we have added new experiments to compare AFS with the most famous and\nstate-of-the-art solution in the community, named SISA (Sharded, Isolated, Sliced, and Aggregated training).\nSISA aims to reduce the computational cost of the retraining process by employing a data partitioning\ntechnique as shown in Response Figure 2. For SISA, data is divided into shards, which are themselves divided\ninto slices. One constituent model is trained on each shard by presenting it with incrementally many slices\nand saving its parameters before the training set is augmented with a new slice. When data needs to be\nunlearned, only one of the constituent models whose shards contain the point to be unlearned needs to be\nretrained. Retraining can start from the last parameter values saved before including the slice containing the\ndata point to be unlearned. However, the main drawback of SISA is that we need more space to keep the\n33\nmodels corresponding to each shard. Our new experimental results indicate that AFS outperforms SISA in\nterms of forgetting, model size reduction, inference speed, GPU memory, and elimination of the need to store\nmodels for each shard.\nWe also compared AFS to two state-of-the-art methods named Catastrophic Forgetting-k (CF-k) and Exact\nUnlearning-k (EU-k). CF-k means to finetune the last k layers of the original model on D and freeze other\nr\nlayers, while EU-k means to retrain the last k layers of the original model from scratch on D and freeze\nr\nother layers, where the D stands for the retain dataset.\nr\nWe have extended the introduction accordingly on Page 2. Here are the modifications made to the main text:\nHowever, most of them are theoretical studies and do not provide open-source codes. Besides, most\nmethods have their specific application scenarios and acknowledged limitations and few of them focused\non the application in real-world intelligent healthcare. Among the three methods, model-agnostic\nmethods might have the strongest application prospects, as they can be applied to different models, and\nSISA, short for Sharded, Isolated, Sliced, and Aggregated training, is the most classic and well-known\nmethod in the community. As the state-of-the-art method, Goel et al. proposed Catastrophic Forget-\nting-k (CF-k) and Exact Unlearning-k (EU-k) to unlearn information from deep learning models. CF-k\nmeans to finetune the last k layers of the original model on D and freeze other layers, while EU-k\nr\nmeans to retrain the last k layers of the original model from scratch on D\nr\nand freeze other layers,\nwhere the D stands for the retain dataset.\nr\nWe have also extended the manuscript accordingly on Page 7 to list all methods we have compared. Here are\nthe modifications made to the main text:\nOnce the prior knowledge that a dataset has been used to train the target DL model is confirmed\nwith auditing, AFS could be used for forgetting, to remove the information of the dataset from the\npre-trained DL model. To comprehensively show the ability of AFS in removing information against\nthe model performance, we compared nine methods, including 1) training the teacher model with\na complete training dataset (Independent teacher), 2) retraining the student model with a complete\ntraining dataset (Independent student), 3) retraining the teacher model with k ∈ {0.25, 0.5, 0.75}\npercentage of the complete training dataset excluding the data to be forgotten (Independent teacher\nwith k ∈ {0.25, 0.5, 0.75}), 4) retraining the student model with k ∈ {0.25, 0.5, 0.75} percentage\nof the complete training dataset excluding the data to be forgotten (Independent Student with k ∈\n{0.25, 0.5, 0.75}), 5) retraining the model of the corresponding shard with SISA, 6) fine-tuning the\nlast layers of the model with CF-k, 7) retraining the last layers of the model from scratch with EU-k,\n34\n8) AFS, and 9) training the student model with AFS without the guidance of auditing (AFS w/o\nAudit), as an ablation study of AFS. Both AFS w/o Audit and AFS were also conducted with varied\nk ∈ {0.25, 0.5, 0.75}. For both Independent teacher and Independent student methods trained with the\ncomplete training dataset, QF and QF were included in the training dataset, while these two\n100 1000\nquery datasets were excluded from the training dataset when k ∈ {0.25, 0.5, 0.75}.\nWe have updated Tables 1∼6 (Response Tables 2∼7 could be found in the responses to reviewer 1) and Figure\n5 (Response Figure 3) in the main text accordingly. We also have updated the Introduction section of the\nmanuscript accordingly on Page 2. Here are the modifications made to the main text:\nHere, we proposed a novel solution by using auditing to guide the forgetting process in a negative\nfeedback manner. We unified the two tasks by introducing knowledge purification (KP), a new\napproach to selectively transfer the needed knowledge to forget the target information instead of simply\ntransferring all information like knowledge distillation (KD). On the basis of KP, we have developed\na user-friendly and open-source audit to forget software (AFS), which can be easily used to revoke\npatients’ private data from DL models in intelligent healthcare. To demonstrate the generality of\nAFS, we applied it to four tasks based on four datasets, including the MNIST dataset, the PathMNIST\ndataset, the COVIDx dataset, and the ASD dataset, with different data sizes and various architectures\nof deep learning networks. Our results demonstrate the usability of AFS and its application potential\nin real-world intelligent healthcare to enhance privacy protection and data revocation rights. To our\nknowledge, AFS is the first unified method of auditing and forgetting that could effectively forget the\ninformation of the target query dataset from the pre-trained DL model with the guidance of auditing.\nAFS could generate a smaller model, which requires much less time and GPU memory during the\ninference, by training with a partial training dataset (∼50%) with our novel KP approach.\n3.6. Major concerns #3\nRC: To better understand AFS’s applicability and effectiveness in real-world production environments, scaling\nit to larger models and more data is essential.\nAR: We would like to express our gratitude to the reviewer for the valuable feedback on our research. We agree\nthat it is important to scale AFS to larger models and data sets to fully understand its applicability and\neffectiveness in real-world production environments. Though we are unable to perform experiments to scale\nAFS to the current famous large models such as LLM (large language model, eg. chatGPT), we still have\ntried our best to add a new experiment to address the reviewer’s concerns.\nWe have created an augmented MNIST dataset with 180,000 samples (three times the original MNIST\n35\ndataset, 18 times the training dataset we used in the previous manuscript) and used a much deeper MLP\nwith 129,105,930 parameters (around 192 times the original model) as the teacher model and 12,615,690\nparameters (around 81 times the original model) as the student model for the 10-class classification task. We\nalso prepared a query dataset with 5,000 samples. Based on the results in Response Table 8, we can prove\nthat it is feasible to apply AFS to deeper neural networks with more data as it demonstrates the ability to\nsimultaneously forget information of the query data and transfer the remaining information from the original\nmodel.\nMethods QF5000 Accuracy F1-score\nIndependent teacher 1 0.9785 0.9951\nIndependent student 1 0.9769 0.9942\nIndependent teacher (k=0.5) 1.27e-24 0.9701 0.9921\nIndependent student (k=0.5) 1.69e-26 0.9684 0.9903\nAFS w/o Audit (k=0.5) 6.33e-12 0.9743 0.9938\nAFS (k=0.5) 4.73e-41 0.9707 0.9929\nTable 8: Comparison of AFS with other methods on forgetting QF and model performance with the Augment-\nedMNIST dataset. We present the p-values of auditing models trained with different methods onQF and\n5000\nthe model performance including the accuracy and F1-score.\n3.7. Major concerns #4\nRC: Although the authors focus on four classification tasks, it would be interesting to explore the idea of\nexpanding AFS to regression tasks, which may present challenges in incorporating additional auditing\nmetrics.\nAR: We would like to express our utmost gratitude to the reviewer for providing valuable feedback on our\nmanuscript. The reviewer’s suggestions and comments have been extremely helpful in shaping our work.\nWe agree that extending AFS to regression tasks is a fascinating idea. However, we have encountered some\npractical limitations that prevent us from pursuing it further. Therefore, we regret to inform the reviewer that\nwe are unable to implement this suggestion due to the following two reasons.\n1. The design of auditing doesn’t work for regression tasks.\nAs mentioned in the manuscript, the auditing process needs to calculate three metrics, including correctness,\nconfidence, and entropy to further calculate the p-value, which will be used in the forgetting process as below.\ntasks, where we have soft logits and hard labels. In regression tasks, we have no theoretical support and can\nnot calculate these metrics, therefore, the auditing component, which is a critical part of AFS, cannot be\napplied.\nEMA is designed as a 2-step process. In the first step, the best threshold for each metric is selected\nto optimize (TPR(t) + TNR(t))/2 based on the calibration dataset as shown in Algorithm 1. Once\nthe thresholds for all metrics are selected, the membership of each sample in the query dataset will\nbe confirmed as at least one metric is larger than the corresponding threshold. In total, three metrics,\nincluding correctness, confidence, and entropy, were adopted to further calculate the p-value, which is\nthe key audit metric in AFS as proposed in the previous work. The details for calculating correctness,\nconfidence, and entropy are as below:\nCorrectness: The target model is trained to predict correctly on training data and may not generalize\nwell on test data. Thus we can define the correctness as:\nI (F, (x, y)) = 1{argmaxF (x) = y} (4)\ncorrectness i i\nwhere F is the deep learning model, x is the input data, F (x) is the output logits, y is the label, and 1\nis the indicator function.\nConfidence: The target model is usually more confident in predictions on training data, but less\nconfident in test data. Thus we can define confidence as:\nI (F, (x, y)) = 1{F (x) ≥ τ} (5)\nconfidence y y\nwhere F is the deep learning model, x is the input data, F (x) is the output logits, y is the label, F (x)\ny\nis the output logits for label y, τ is a threshold for the logit for label y, and 1 is the indicator function.\ny\nEntropy: The target model is trained by minimizing the prediction loss over training data and usually\nhas a larger prediction entropy on a test sample. Thus we can define entropy as:\nΣ\nI (F, (x, y)) = 1{− F (x)log(F (x)) ≤ τˆ} (6)\nentropy i i y\ni\nwhere F is the deep learning model, x is the input data, F (x) is the output logits, y is the label, F (x)\ni\n37\nis the output logits for class i, τˆ is a threshold, and 1 is the indicator function.\ny\nOnce the membership of all samples in the query dataset is confirmed in the previous step, the query\ndataset will be further evaluated to determine whether the query dataset has been used to train the target\npre-trained DL model. A two-sample statistical test is adopted to evaluate the query dataset based on\nthe sample-wise membership and an all-one vector. The p-value of the two-sample statistical test is\nused as the output of auditing. Given a user-defined threshold α, if p < α, then users could conclude\nthat the query dataset was not used for training the target DL model. EMA was re-implemented and\nintegrated into AFS to allow easy and fast auditing.\n2. Knowledge purification is limited to classification tasks.\nThe current version of the Knowledge Purification algorithm requires soft logits in classification tasks to\ncalculate the loss, which makes it difficult to be applied to regression tasks. Additionally, Knowledge\nDistillation is not specifically designed for regression tasks, which involve predicting continuous values.\nAs noted in the paper \"BAM! Born-Again Multi-Task Networks for Natural Language Understanding\" by\nStanford University and Google Brain, distillation may not work well for regression tasks as there is no\ndistribution over classes passed on by the teacher to aid learning.\nWhile we acknowledge the potential interest of applying AFS to regression tasks, it requires a redesign of\nboth the auditing and knowledge purification algorithms, which is beyond the scope of our current work.\nNonetheless, we deeply appreciate the reviewer’s efforts and will try our best to take the reviewer’s feedback\ninto consideration in our future research endeavours.\n3.8. Conclusion\nRC: Despite the aforementioned limitations, this reviewer still believes that the proposed AFS method contributes\nsignificantly to enhancing privacy protection and data revocation rights. The paper can be publishable if\nthe listed limitations are addressed.\nAR: Thank the reviewer for the positive assessment of our method. We appreciate the reviewer’s feedback and\nare glad that the reviewer finds our work valuable in enhancing privacy protection and data revocation rights.\nHope that our point-to-point response could address the reviewer’s concerns.\nReviewer #2 (Remarks to the Author):\nThanks for the update, revisions and well-detailed response letter.\n1- EMA vs AFS: Reviewer would like to note that AFS improvement is not algorithmic but computational\nthrough parallelism. The reviewer notes that this reduces the significance of the contribution. The\nreviewer advises authors to cover algorithmic changes necessary for parallelism as the contribution\npoint.\n2- Figure 1 of the response letter: The reviewer advises authors to highlight the significance of this\nseveral-second of time difference. E.g., 1 milisec can be life-threatening in real-time systems like\nautonomous vehicles but means nothing when loading a page in a browser.\n3- Authors state that \"We have shown that Audit and Forget are algorithmically linked, and our\napproach provides a novel solution to this problem.\" Reviewers would agree if achieving such a chaining\nrequires solving a significant problem rather than running one algorithm after another. Could authors\nclarify the difficulty and significance of conducting Audit and Forget, combined?\n4- The reviewer understands the author's point that subsequent teacher students training can be done\nin a way to keep the model at a similar size. However, what is the requirement for the model size\nreduction in the first iteration? Please clarify why the model needs to be distilled to a smaller size.\nSaying that we successfully forget, and as a side effect, we get a smaller model which fits in smaller\nmemory does not sound to be a good motivation.\n5- Authors state that \"but we believe that our unique research question and methodology justify its\ncontribution to the field\" Please elaborate further to show that this is sufficient for a prestigious venue.\n6- Table 1 suggests an accuracy drop in subsequent forgetting rounds (from 0.96 to 0.92 in the fifth\nround). What is the budget here? How do authors model the budget? Where exactly should the\norganization stop forgetting and re-train the model with data minus the forgotten records?\n7- Thanks. The reviewer acknowledges the additional comparisons and the significance of the work done\nto achieve these comparisons.\n8- The reviewer acknowledges the clarifications around metrics and how they are used to obtain p-value\ncomparison.\n9- \"large companies usually have millions of users or even trillions of users.\" The reviewer highlights that\nthere are approximately 8 billion people living on the Earth.\nReviewer #3 (Remarks to the Author):\nThe revision addressed some of the raised issues. But there are still remaining issues.\nThe paper says that the auditing process needs to calculate three metrics, correctness, confidence, and\nentropy, which cannot support the regression tasks. There are some confusions here and hope the\nauthors can clarify. The computation of correctness is decided by the indicator functions, output logits,\nand label. Suppose that we take F as the output, y as the true value, and the indicator function as the\nloss function, is it possible to extend the proposed method? Correspondingly, the computation of\nconfidence takes a similar strategy and has a threshold for error of (output-true value). Entropy can be\nregarded as one of the loss functions.\n2. Scalability\nMNIST may not be viewed as a large dataset. Although the proposed method might work well for small-\nsized datasets, it is still not clear how it performs on very large datasets or complex models in terms of\nits effectiveness and efficiency, and hence further exploration is needed.\nAuthors’ Response to Reviews of\nNCOMMS-23-05640A-Z\nAudit to Forget: A Unified Method to Revoke Patients’ Pri-\nvate Data in Intelligent Healthcare\nJuexiao Zhou1,2,#, Haoyang Li1,2,#, Xingyu Liao1,2, Bin Zhang1,2, Wenjia He1,2, Zhongxiao Li1,2, Longxi\nZhou1,2, Xin Gao1,2,∗\n1Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division,\nKing Abdullah University of Science and Technology (KAUST), Thuwal 23955-6900, Kingdom of Saudi\nArabia\n2Computational Bioscience Research Center, King Abdullah University of Science and Technology, Thuwal\n23955-6900, Kingdom of Saudi Arabia\n∗Corresponding author\n#Equal contribution\nRC: Reviewers’ Comment, AR: Authors’ Response, □ Manuscript Text\n1. General Response\nAR: We really appreciate the editor [redacted] and both reviewers’ valuable feedback and suggestions, which\nhave extremely helped us improve the quality and clarity of our work. Once again, we would like to\nexpress our sincere gratitude. We are pleased to see that we have addressed the majority of the concerns\nraised by both reviewers in the previous revision. Still, we also acknowledge that there is scope for us to\nfurther improve the manuscript at this stage. We are delighted to have the opportunity to elevate the\nmanuscript to the next level with the assistance and guidance of our esteemed reviewers. Therefore, we\ncarefully considered all comments and made necessary changes to further improve the quality and\nimpact of our work. We believe that we have addressed the concerns raised by the reviewers to the best of\nour ability by either adding new results or clarification. We showed the revised places in the previous\nrevision with blue underline and highlighted the newly edited places in the manuscript with a yellow\nhighlight to make the revised portions of the manuscript clear to the reviewers.\n1\n2. Response to Reviewer #2’s Comments\nRC: Thanks for the update, revisions and well-detailed response letter.\nAR: We would like to express our heartfelt gratitude to the reviewer for dedicating valuable time to review our\nmanuscript and for providing us with precious and constructive feedback. We recognize the importance of\nsuch feedback and sincerely appreciate the reviewer’s thoughtful comments and suggestions.\nRC: 1- EMA vs AFS: Reviewer would like to note that AFS improvement is not algorithmic but computational\nthrough parallelism. The reviewer notes that this reduces the significance of the contribution. The reviewer\nadvises authors to cover algorithmic changes necessary for parallelism as the contribution point.\nAR: We appreciate the reviewer’s comment regarding the difference between the Audit module in AFS and EMA.\nWe acknowledge the importance of clarifying this point to highlight the contribution of our work and to avoid\noverstating contributions.\nIn general, AFS consists of two modules, the first part is the Audit module developed on EMA and the\nsecond part is the Forget module (our knowledge purification algorithm). Though the Audit module in AFS is\ndeveloped on EMA, we have optimized the computation time by adding parallel computation to the code to\nachieve faster speed. At this level, we agree that our contribution is computational. However, to implement\nit, we have also made modifications to the algorithm, as shown in Response Algorithm 1 (Algorithm 1 in\nthe main text), and used some tricks in coding, such as using GPU accelerated matrix computation, so we\nhope the reviewer could agree that there is a degree of contribution to the algorithm as well. Though the\ncontribution to the Audit module of AFS is not as significant as our knowledge purification algorithm, the\nimproved Audit indeed solves practical problems for acceleration and we will further specify this point in the\nresponse to question 2.\nWe have updated the algorithm 1 in the main text as below:\nRequire: The calibration dataset D , the pre-trained DL model A and n different metrics (m , ..., m ) for\ncal 1 n\nmembership testing.\n1: procedure\n2: Split D cal into training dataset Dtrain and test dataset Dtest\ncal cal\n3: The calibration model is trained as f D c al ←\nA(D trai\nc\nn\na\n)\nl\n4: for m i ∈ {m 1 , ..., m n } do in parallel\n5: Compute metrics for training dataset as M train ← {m i (f D c al ,s |s ∈ D c t a r l a in)}\n6: Compute metrics for test dataset as M test ← c al ,s |s ∈ Dtest)}\n{m (f\n7:\ni D\nFind t i ∈ argmax t∈[M t rain , Mtest ] ( T P R(t)+2T NR(t) ), where TPR\nc\n(\na\nt\nl\n) =\nΣ\ns∈Dtrain 1{m i (s) ≥\ns∈D\ncal\nt\nΣr\n}\net\n/\nu\n|\nr\nD\nn\nct ar\nT\nla\nh\nin\ne\n|\nt h\nan\nre\nd\ns\nT\nh\nN\nold\nR\ns\n(\nt\nt1)\n,\n=\n... , t\nn\nfl or n\nt\nc\ne\nam\nst\ne\n1\nt\n{\nr\nm\nics\ni\n( s) < t}/|Dctaels t|\nRC: 2- Figure 1 of the response letter: The reviewer advises authors to highlight the significance of this several-\nsecond of time difference. E.g., 1 milisec can be life-threatening in real-time systems like autonomous\nvehicles but means nothing when loading a page in a browser.\nAR: We acknowledge the reviewer’s concerns regarding the significance of acceleration at the second level when\ncomparing the Audit module of AFS with EMA. We apologize for not explaining this aspect in our previous\nresponse.\nAs shown in Figure 1 (the same one from the previous response letter), the time difference indicated refers to\na single epoch in the training process. We explicitly mentioned in the manuscript that the Audit module is\nreferenced during each training epoch to facilitate information forgetting through knowledge purification. To\nforget, the model is trained with knowledge purification until convergence is reached. The total number of\nepochs required for convergence depends on factors such as the size of the data and the speed at which the\nmodel converges. For example, we need more than 1000 epochs for the MNIST case in the manuscript.\nLet’s suppose that we need 1000 epochs for a query dataset of 50,000 samples as shown in Figure 1, and\nconsidering the median time of 25 seconds for the Audit module in AFS and 12 seconds for EMA for single\nauditing, the total time difference would be approximately 1000 * (25 - 12) = 13,000 seconds (around 3.6\nhours). This difference in time is indeed significant, and the gap could be even larger when more epochs are\nneeded or when multiple query datasets need to be forgotten.\nThe acceleration demonstrated by the Audit module of AFS becomes particularly attractive for companies\ndealing with larger-scale data and larger deep learning models. Longer GPU computation times mean\n3\nincreased costs, making the ability to accelerate the forgetting process highly beneficial. By reducing the\ntime required for forgetting, AFS offers potential cost savings for companies by minimizing GPU usage and,\nsubsequently, the associated expenses.\nWe have updated the main text on Page 3 as below:\nTo audit the membership of the query dataset, AFS takes a pre-trained DL model and the query dataset\nas inputs, and determines whether the query dataset has been used for training the target DL model.\nThis function was re-implemented based on EMA, a published MIA-based method to evaluate the\nmembership of a query dataset. Our re-implementation allows quicker and easier usage of auditing by\nintroducing parallel computing in each epoch, which suggests a significant acceleration for a complete\nforgetting process and could be more attractive to institutions to forget larger-scale data. (Section 2.4\nand Supplementary Figure 1).\nseconds.\napproach provides a novel solution to this problem.\" Reviewers would agree if achieving such a chaining\nrequires solving a significant problem rather than running one algorithm after another. Could authors\nclarify the difficulty and significance of conducting Audit and Forget, combined?\nAR: We greatly value the reviewer’s feedback regarding the challenges and importance of combining Audit and\nForget processes. We acknowledge the concern and recognize the need to provide a more comprehensive\nexplanation of the unique advantages and potential synergies that arise from this combination. We will\nelaborate on these points below:\nFirst, without combining Audit and Forget algorithmically, the processes of Audit and Forget are conducted\nas separate steps. For example, to forget a query dataset from a pre-trained model, the following steps are\ntypically performed:\n1. Determine if the data is remembered by the model by calculating relevant metrics such as Audit.\n2. Employ specific algorithms (e.g., SISA, EU-k, CF-k as mentioned in our manuscript) to generate new\nmodels. However, these methods do not leverage the knowledge of whether the data is remembered by the\nmodel, resulting in uncertain performance in the forgetting process.\n3. Assess whether the data has been successfully forgotten by examining multiple models generated in the\nprevious step. If not, we repeat the previous process.\n4. Finally we choose the model with the best forgetting performance as the new model.\nOne limitation of the previous procedures is that the information regarding whether the data is remembered\nby the model is not fully utilized during the training phase, but only considered during the model selection\nphase. Thus the forgetting performance can be further improved as we could make full use of the information\nof “if the data is remembered by the model” in the training phase. By incorporating the knowledge of whether\nthe data is remembered by the model as one of the goal functions, we can enhance the directed forgetting\nprocess and improve the overall forgetting performance.\nTo achieve that, we integrate the information of “if the data is remembered by the model” into the forgetting\nprocess, specifically in the knowledge purification stage. By doing so, we utilize this information to update\nthe model during each step of forgetting, establishing an algorithmic link between the information and the\nmodel training process. The benefits of doing so are also demonstrated by comparing AFS to other methods\nin the manuscript. Our results clearly indicate that incorporating this information into the training process is\nmore effective for forgetting than solely relying on it for model selection.\ndata is remembered by the model is not fully utilized. In contrast, our approach establishes an algorithmic\nconnection between Audit and Forget, enabling forgetting to occur during the training process itself. Linking\nthe two algorithmically is not technically difficult in itself, but proposing this idea is innovative and effective.\nRC: 4- The reviewer understands the author’s point that subsequent teacher students training can be done\nin a way to keep the model at a similar size. However, what is the requirement for the model size\nreduction in the first iteration? Please clarify why the model needs to be distilled to a smaller size.\nSaying that we successfully forget, and as a side effect, we get a smaller model which fits in smaller\nmemory does not sound to be a good motivation.\nAR: We would like to express our gratitude to the reviewer for their valuable feedback. We highly appreciate the\ncomprehensive evaluation and acknowledge the concern raised regarding the motivation for reducing the\nmodel size in the initial iteration.\nModel compression is a widely utilized technique that holds significant potential for various real-life ap-\nplications. One common approach to model compression is knowledge distillation, which aims to transfer\nknowledge from a larger, more complex model to a smaller one. Our knowledge purification could also be a\ntechnique of model compression when we need to compress the model.\nFirstly, we answer the question of “why the model needs to be distilled to a smaller size”. The need to distil or\ncompress a model to a smaller size arises from practical considerations during the model deployment phase.\nWhile larger models often exhibit better learning capabilities and performance during the training phase,\ndeploying these larger models on other devices or providing cloud-based inference services can introduce\nchallenges related to computational resources and inference time. Therefore, during the model training phase,\nwe tend to use larger models. But in the model inference phase, we prefer to use a small model. Model\ncompression techniques offer valuable solutions by reducing the model size, thereby saving computational\nresources and reducing inference time. This compression process involves transforming the model from its\nlarger, resource-intensive training phase to a smaller, more streamlined form suitable for deployment.\nFor example, some current large language models have an enormous number of parameters, such as the\n176B BLOOM model that requires more than one hundred A100 GPUs to train and at least 6 A100 GPUs to\nhost. This poses challenges for most users who lack the necessary resources to deploy these models locally.\nDisregarding the most popular large language models, even the most common CNN models in medical\ndiagnostics face the same problem (to reduce inference time from hours or minutes to seconds). To address\nthis issue, extensive research has been conducted on leveraging knowledge distillation techniques to compress\nthese large models without compromising their effectiveness. The aim is to create smaller models that can\n7\nrun efficiently with reduced computational resources and faster inference times. In the medical field, in\nparticular, faster inference models are of greater value as the reviewer stated previously “1 milisec could be\nlife-threatening in the real-time system”. Therefore, generating a smaller model which fits in smaller memory\nand faster inference time indeed is a good motivation in the community. Meanwhile, previous studies have\nalso demonstrated that training a small model directly may not be as effective as training a large model and\nsubsequently compressing it to a smaller size. Hence, the practice of distilling models to a smaller size is\nprevalent in various real-life scenarios.\nIn addition to model compression, our knowledge distillation approach introduces an additional requirement:\nthe ability to forget specific information during the knowledge transfer process. As mentioned in the previous\nresponse, the KP algorithm has two application scenarios. The first scenario involves forgetting knowledge\nwhile compressing the model, while the second scenario focuses on achieving knowledge forgetting without\nmodel compression. To provide further clarification on this point, we have included Figure 2D (Figure 5D in\nthe main text) as below:\ngnitiduA\nrof\neula\nPv-\ntesataD\nyreuQ\nllamS\nsretemarap\nfo\n.oN\nOriginal\nModel\nNew\ngnitiduA\nrof\neula\nPv-\ntesataD\nyreuQ\negraL\nC\nsretemarap\nfo\n.oN\nModel\ntesataD\ntseT\nno\nycaruccA\nD Newly Acquired\nData\n...\nRetained\nDataset\nInitial Dataset Institution Trained Large Model New Dataset Institution Trained Large Model\nWith Model Compression Forget Request Model Compression\nForget Request w/o Forget Request w/o Forget Request\nNew Model New Model\n(For Deployment) (For Deployment)\nContinuous Continuous\nForget Request Forget Request\nFigure 2: Performance of forgetting using AFS on four datasets. A. The p-value of auditing on a small\nquery dataset and a large query dataset (QF) and the accuracy of models trained with different methods,\nincluding Original (Independent teacher trained with the complete training dataset), SISA, Data Deletion (the\nIndependent student model trained with partial training dataset and k = 0.5), AFS (w/o Audit) and AFS. B.\nThe number of parameters for the original large model and the new small model generated by AFS. C. The\nqualitative evaluation of three methods, including Original (Independent teacher trained with the complete\ntraining dataset), SISA, CF-k, EU-k, Data Deletion (the Independent student model trained with partial\ntraining dataset and k = 0.5), and AFS on five dimensions (Ability to forget, accuracy, size of dataset needed\nfor training, size of the generated model and the efficiency of training). A larger value means a stronger\nability to forget, higher model accuracy, a smaller size of dataset needed for training, a smaller size of the\ngenerated model, and better efficiency of training. D. Illustration of how to incorporate AFS into real-world\napplications.\nalgorithm can be effectively utilized in two distinct scenarios: forgetting prior to model compression and\nforgetting after obtaining a compressed model, which will be further explained in the response to question 6.\nThe requirement for model size reduction in the first iteration depends on the specific needs and constraints of\nthe application scenario. Whether model compression is necessary and to what extent it should be applied\nwill vary based on the target platform’s computational resource limitations and deployment requirements\nfor the student models. For example, deploying a model requiring 100GB of GPU memory to a platform\nwith only 8GB of GPU memory, we will need a significant level of model compression in the first round.\nThe exact ratio of model compression will depend on the practical limitations and constraints of the target\nplatform. It is important to note that there is no fixed standard or predefined threshold for model compression.\nInstead, the level of compression needed is determined by the specific circumstances and constraints of the\ndeployment scenario. Factors such as available computational resources, memory limitations, and other\npractical considerations will guide the decision-making process when determining the appropriate level of\nmodel compression in the first iteration.\nRC: 5- Authors state that \"but we believe that our unique research question and methodology justify its\ncontribution to the field\" Please elaborate further to show that this is sufficient for a prestigious venue.\nAR: We sincerely appreciate the reviewer’s feedback on the contribution of our method and the potential signifi-\ncance of knowledge purification as a research topic. Hope our responses to questions 3 and 4 and also in the\nresponse to question 6 later could address this concern, we will provide a brief summary here.\nFirstly, our approach leverages the information of whether the data is remembered by the model to achieve\ntargeted forgetting algorithmically, resulting in improved forgetting performance. By incorporating this infor-\nmation into the forgetting process, we can effectively forget specific data from trained models. Additionally,\nas the field of machine learning continues to advance, privacy concerns have become increasingly important.\nForgetting specific data from trained models has become a common and vital requirement, particularly in\ndomains such as the medical field. Protecting the right of users to revoke their private data is crucial, and\nthe concept of knowledge purification holds significant potential in addressing this need. Considering these\nfactors, knowledge purification as a novel concept has the potential to garner widespread attention and become\na subject of extensive research.\nRC: 6- Table 1 suggests an accuracy drop in subsequent forgetting rounds (from 0.96 to 0.92 in the fifth\nround). What is the budget here? How do authors model the budget? Where exactly should the organization\nstop forgetting and re-train the model with data minus the forgotten records?\ndrop and a potential guideline for when to stop forgetting and train a new model. This question is extremely\nvaluable and helpful. It is very important to further elaborate this concern.\nIndeed, it is widely acknowledged that while users may request institutions to forget their data, institutions\nthemselves are constantly collecting new data for the purpose of training new models. This is particularly\nprevalent in various industries where the continuous acquisition of new data is essential for training state-of-\nthe-art models and maintaining a competitive edge. Therefore, we added a new figure to the manuscript as\nshown in Response Figure 2 (Figure 5D in the main text) to further explain this point. The figure illustrates\nthe workflow of the institution in two distinct phases. In the first phase, the institution collects an initial\ndataset and trains a large model during the training phase. Following the training phase, the model may need\nto be compressed to meet specific requirements for deployment. As previously mentioned, the compression\nratio of the model depends on the actual demand and can vary accordingly.\nPatients’ requests for data forgetting may occur in two different phases. The first phase relates to requests\nmade before model compression, where patients request the institution to forget their data from the initial\ndataset. The second phase pertains to requests made after model compression, where patients seek to have\ntheir data forgotten from the compressed model. In both cases mentioned, the AFS algorithm can be employed\nto forget information from the model.\nIn the first case, AFS facilitates model compression while performing forgetting. This ensures that the\ncompressed model not only meets the requirements for deployment but also respects data privacy by removing\nunnecessary or sensitive information. In the second case, AFS is utilized to forget information from the\nmodel without involving model compression. This allows the institution to respond to data forget requests\nwhile retaining the compressed model’s structure and size. If additional forget requests arise (as referred to\nas continuous forgetting), the AFS algorithm can be repeatedly applied to forget the relevant information.\nHowever, it is crucial to note that this process cannot continue indefinitely. Each forgetting step leads to a\ncertain loss of model effectiveness, and the AFS algorithm relies on a certain percentage of raw data (e.g.,\nk=0.5) to facilitate information transfer.\nAs pointed out by the reviewer, “Table 1 suggests an accuracy drop in subsequent forgetting rounds (from\n0.96 to 0.92 in the fifth round)”, which happens when we have 5 consecutive rounds, each time forgetting\n5% of the data. However, it is important to consider the scale of forgetting requests in real-life scenarios.\nIn practical situations, organizations may have vast amounts of data, and a 5% forgetting request is already\nsubstantial and is not expected to occur frequently. In contrast, we would typically anticipate much smaller\nscale forgetting requests. For example, a forgetting request of 0.1% or even smaller may be more common.\nThe model’s performance is less affected, allowing organizations to accommodate forget requests without\nsignificant accuracy degradation.\nIndeed, the relationship between the decline in accuracy and the amount of forgotten data is not strictly\nlinear. This is due to the fact that, during the forgetting process, we still require a certain percentage of\ntraining data to facilitate the transfer of retained information. The accuracy decline is influenced by various\nfactors, including the unique information contained within the data to be forgotten and the degree of similarity\nbetween the data to be forgotten and the retained data.\nTherefore, to answer the question “What is the budget here?”, we may need to specify it according to the\nactual problem. We understand that budget means how much of the model performance degradation we can\naccept when we need to balance privacy and model effectiveness. The modelling of the budget should be\ncase-specific. Thus, the selection of budget needs to refer to the actual situation. For example, in critical\nmedical diagnosis applications, even a small decrease in accuracy, such as 0.5%, could have significant\nconsequences, potentially putting patients at risk if incorrect diagnoses are made. In such cases, maintaining a\nhigh level of model performance is crucial, and a stricter budget on accuracy degradation may be necessary to\nprioritize patient safety. On the other hand, in more general scenarios where some degree of fault tolerance is\npermissible, such as a preliminary diagnosis, organizations may be more willing to accept a certain percentage\nof performance decrease in order to protect the users’ right to revoke their data and ensure data privacy.\nBased on the response above, to answer “Where exactly should the organization stop forgetting and re-train\nthe model with data minus the forgotten records?”, two possible scenarios can be considered. The first\nscenario occurs when the number of forgetting requests becomes too high, leading to a significant degradation\nin the current model’s performance that exceeds the institution’s predefined budget. In such cases, it may be\nnecessary to stop the forgetting process and initiate the retraining of a new model. The second scenario arises\nwhen the institution introduces new data into the system. In this case, instead of continuing to forget specific\nrecords from the existing model, the institution can incorporate the retained data along with the new data to\ntrain a new model as shown in 2 (Main Figure 5D).\nWe have updated the discussion section of the main text as below:\nIn practice, the size of the student model could be manually adjusted when applying AFS to meet\nspecific requirements. In the initial stages, we could make the student model smaller than the teacher\nmodel to achieve model compression while forgetting private information. However, instead of continu-\nously generating smaller and smaller student models as knowledge is forgotten, we could maintain the\n12\nsize of the student model once it reaches a certain threshold (e.g., small enough to meet our needs). In\ndoing so, AFS would focus solely on forgetting knowledge and transferring the remaining knowledge,\nwithout the need for further model compression.\nAFS could be incorporated into the workflow of institutions as shown in Figure 2D. Patient’s requests\nfor data forgetting may occur in two different phases. The first phase relates to requests made before\nmodel compression, where patients request the institution to forget their data from the initial dataset.\nThe second phase pertains to requests made after model compression, where patients seek to have\ntheir data forgotten from the compressed model. In both cases, AFS could be employed to forget\ninformation from the model. In the first case, AFS facilitates model compression while performing\nforgetting, which ensures that the compressed model not only meets the requirements for deployment\nbut also respects data privacy by removing sensitive information. In the second case, AFS could be\nutilized to forget information from the model without involving model compression. This allows the\ninstitution to respond to data forget requests while retaining the compressed model’s structure and\nsize. The institution may stop forgetting and retrain a new model in two possible scenarios. The first\nscenario occurs when the number of forgetting requests becomes too high, leading to a significant\ndegradation in the current model’s performance that exceeds the institution’s predefined budget. In such\ncases, it may be necessary to stop the forgetting process and initiate the retraining of a new model. The\nsecond scenario arises when the institution introduces new data into the system. In this case, instead\nof continuing to forget specific records from the existing model, the institution can incorporate the\nretained data along with the new data to train a new model.\nRC: 7- Thanks. The reviewer acknowledges the additional comparisons and the significance of the work done\nto achieve these comparisons.\nAR: We greatly appreciate the reviewers’ valuable time and constructive suggestions. The feedback has been\ninstrumental in enhancing the quality and clarity of our work.\nRC: 8- The reviewer acknowledges the clarifications around metrics and how they are used to obtain p-value\ncomparison.\nAR: We are very grateful for the reviewer’s suggestion to clarify the manuscript.\nRC: 9- \"large companies usually have millions of users or even trillions of users.\" The reviewer highlights that\nthere are approximately 8 billion people living on the Earth.\nAR: Thank the reviewer for bringing this up, and we apologize that we misrepresented it in the previous response.\nWhat we were trying to say was “large companies usually have millions of users or even trillions of samples\n13\nfrom users”. In practical scenarios, a single user may provide several samples, which collectively contribute\nto the large-scale dataset used for training models.\n14\n3. Response to Reviewer #3’s Comments\nRC: The revision addressed some of the raised issues. But there are still remaining issues.\nAR: We would like to express our sincere appreciation to the reviewer for taking the valuable time to review our\nmanuscript and providing us with constructive feedback. We value the opportunity to improve our manuscript\nonce again and elevate our manuscript to the next level with the assistance and guidance of the esteemed\nreviewer.\nRC: 1 Generalization of the proposed method to regression task. The paper says that the auditing process\nneeds to calculate three metrics, correctness, confidence, and entropy, which cannot support the regression\ntasks. There are some confusions here and hope the authors can clarify. The computation of correctness\nis decided by the indicator functions, output logits, and label. Suppose that we take F as the output, y as\nthe true value, and the indicator function as the loss function, is it possible to extend the proposed method?\nCorrespondingly, the computation of confidence takes a similar strategy and has a threshold for error of\n(output-true value). Entropy can be regarded as one of the loss functions.\nAR: We would like to express our gratitude to the reviewer for the valuable suggestion. We appreciate the feedback\non incorporating correctness, confidence, and entropy directly into the loss function to extend the applicability\nof our method to regression tasks. To further elaborate on this point, we will show the practical difficulties in\ndetail below:\nLet’s suppose X to be all samples, x to be one input sample, F (x) to be the predicted vector and y to be the\nlabel vector (e.g. one-hot encoded label) and y is the label.\nSo, for classification tasks (let’s suppose 5 classes, labelled as 0-4), let’s look at a sample x from X and\nsuppose the prediction of x to be [0.1, 0.1, 0.6, 0.1, 0.1] the label y of x to be [0, 0, 1, 0, 0] and label y to be 2.\nHowever, for the regression task, let us look at a sample x again from x. Let’s suppose the prediction of x to\nbe [7.5], the label y to be [10.3] and the label y to be 10.3.\nCorrectness\nAs we defined in the manuscript, correctness is\nI (F, (x, y)) = 1{argmaxF (x) = y} (1)\ncorrectness i i\nFor the classification task, with 1{argmaxF (x) = y}, 1 means the label y of x has the highest prediction\ni i\nscore.\nthe definition to:\nI (F, (x, y)) = 1{F (x) = y} (2)\ncorrectness\nwhere 1{F (x) = y} means if the regression value is exactly the same as the label. If so, we say it is correct.\nWe appreciate the reviewer’s insight regarding the definition of correctness in the context of regression\nproblems. While predicting the exact same value as the label in regression tasks can be challenging, we\nacknowledge that a strict definition of correctness may lead to most samples being classified as 0 in terms\nof correctness. Therefore, correctness is not a good metric for regression tasks. Still, by incorporating the\ndefinition of confidence below, as suggested by the reviewer, we can potentially address this issue. The\nintroduction of an \"error\" term, which quantifies the deviation between the predicted value and the actual\nlabel, can provide a measure of confidence in the prediction.\nConfidence\nSimilarly, as we defined in the manuscript, confidence is\nI (F, (x, y)) = 1{F (x) ≥ τ } (3)\nconfidence y y\nFor the classification task, with 1{F (x) ≥ τ }, 1 means the prediction score for label y of x is larger than\ny y\nthe threshold τ we defined from the calibration dataset.\ny\nFor the regression task, with 1{F (x) ≥ τ }, 1 means the regression value of x is larger than the threshold\ny y\nτ we defined from the calibration dataset. Obviously, we can not state it the same way. Still, as suggested by\ny\nthe reviewer, we can calculate the error between the prediction score and the label, and modify the definition\nto:\nI (F, (x, y)) = 1{F (x) − y ≥ τ } (4)\nconfidence y y\nwhere with 1{F (x) − y ≥ τ }, 1 means the regression value of x minus the label y (the error) is larger than\ny y\nthe threshold τ we defined from the calibration dataset. Now, we need to see what is τ . However, as shown\ny y\nin Response Algorithm 2 (Algorithm 1 in the main text), the definition of τ does not work for the regression\ny\ntask, because we are unable to perform step 7. Thus we need to redesign the whole algorithm for inferring the\nthresholds.\nAs suggested by the reviewer, we could potentially re-design line 7 of the algorithm as Algorithm 3:\nRequire: The calibration dataset D , the pre-trained DL model A and n different metrics (m , ..., m ) for\ncal 1 n\nmembership testing.\n1: procedure\n2: Split D cal into training dataset Dtrain and test dataset Dtest\ncal cal\n3: The calibration model is trained as f D c al ←\nA(D trai\nc\nn\na\n)\nl\n4: for m i ∈ {m 1 , ..., m n } do in parallel\n5: Compute metrics for training dataset as M train ← {m i (f D c al ,s |s ∈ D c t a r l a in)}\n6: Compute metrics for test dataset as M test ← c al ,s |s ∈ Dtest)}\n{m (f\n7:\ni D\nFind t i ∈ argmax t∈[M t rain , Mtest ] ( T P R(t)+2T NR(t) ), where TPR\nc\n(\na\nt\nl\n) =\nΣ\ns∈Dtrain 1{m i (s) ≥\ns∈D\ncal\nt\nΣr\n}\net\n/\nu\n|\nr\nD\nn\nct ar\nT\nla\nh\nin\ne\n|\nt h\nan\nre\nd\ns\nT\nh\nN\nold\nR\ns\n(\nt\nt1)\n,\n=\n... , t\nn\nfl or n\nt\nc\ne\nam\nst\ne\n1\nt\n{\nr\nm\nics\ni\n( s) < t}/|Dctaels t|\nAlgorithm 3 Infer thresholds (Regression)\nRequire: The calibration dataset D , the pre-trained DL model A and n different metrics (m , ..., m ) for\ncal 1 n\nmembership testing.\n1: procedure\n2: Split D cal into training dataset Dtrain and test dataset Dtest\ncal cal\n3: The calibration model is trained as f D c al ←\nA(D trai\nc\nn\na\n)\nl\n4: for m i ∈ {m 1 , ..., m n } do in parallel\n5: Compute metrics for training dataset as M train ← {m i (f D c al ,s |s ∈ D c t a r l a in)}\n6: Compute metrics for test dataset as M test ← c al ,s |s ∈ Dtest)}\n{m (f\n7:\ni D\nFind t i ∈ argmax t∈[M t rain , Mtest ] ( T P R(t)+2T NR(t) ), where TPR\nc\n(\na\nt\nl\n) =\nΣ\ns∈Dtrain 1{m i (s) −\ns∈D\ncal\ny Σ\ns\n≤ t} T / h |D e c t ta h rla r i e n s | h a o n l d d s T t N1 , . R .., ( t tn) f o = r n ml etr\nt\nc\ne\nia\ns\nc\nt\ns 1 {m\ni\n(s) − y\ns\n> t}/|Dctaels t|\nreturn\nSimilarly, as we defined in the manuscript, entropy is\nΣ\nI (F, (x, y)) = 1{− F (x)log(F (x)) ≤ τˆ} (5)\nentropy i i y\ni\nSo, for classification tasks (let’s suppose 5 classes, labelled as 0-4), let’s look at a sample x from X and\nsuppose the prediction of x to be [0.1, 0.1, 0.6, 0.1, 0.1] the label y of x to be [0, 0, 1, 0, 0] and label y to be 2.\nCalculating the entropy across labels asΣ − F (x)log(F (x)) is meaningful.\ni i i\nHowever, for the regression task, let us look at a sample x again from x. Let’s suppose the prediction\nof x to be [7.5], the label y to be [10.3] and the label y to be 10.3. Now, calculating the entropy as\n− Σ F (x)log(F (x)) is meaningless, because we no longer have information across multiple labels.\ni i i\nSummary\nTherefore, the current algorithm design in the manuscript is not applicable to the regression task. Even though\nwe can make some modifications to make the definition potentially work on the regression problem, such\nas the confidence alone, it is important to note that this approach only addresses a specific aspect (Audit)\nand does not encompass the entire framework (AFS, Audit and Forget). As we explained in the previous\nresponse, finally, we still end up transferring information with knowledge purification. As mentioned in our\nprevious response, the knowledge purification process ultimately involves transferring information from the\nmodel. However, the regression task lacks the soft logits representation across multiple labels, which is a\ncrucial component for effective knowledge purification and knowledge distillation. Previous studies have\nshown the limitations of applying knowledge distillation to regression tasks, even when replacing the loss\non soft logits with the mean squared error (MSE) loss (https://github.com/google-research/\ngoogle-research/issues/911). To fully address this issue, it would require the development of a\nseparate framework specifically designed for regression problems, distinct from knowledge distillation or\nknowledge purification. However, the scope of the current manuscript does not encompass the exploration of\nsuch a regression-specific framework.\nWe appreciate the reviewer’s understanding of the limitations and applicability of the current algorithm design\nin the manuscript for regression tasks. Exploring novel frameworks and approaches specifically tailored to\nregression problems remains an important avenue for future research.\nWe also do recognize the limitation of our proposed method in the main text (Page 12, line 677) as below:\nditing metrics does not work for regression tasks and KP is limited to classification tasks only. Despite\nthese limitations, we believe that AFS will make a valuable contribution towards better protection of\npeople’s privacy and the right to revoke data with the rapid development of intelligent healthcare.\nRC: 2. Scalability. MNIST may not be viewed as a large dataset. Although the proposed method might work\nwell for small-sized datasets, it is still not clear how it performs on very large datasets or complex models\nin terms of its effectiveness and efficiency, and hence further exploration is needed.\nAR: Thank the reviewer for the valuable feedback. We understand the concern regarding the size of the dataset and\nmodel complexity. We will further explain the experiment in the previous response and add a new experiment\nwith a larger dataset and a more complex deep learning model.\nIn the previous response, we didn’t just use the original MNIST dataset but an augmented MNIST dataset\nwith 180,000 samples, which is three times the original MNIST dataset, and 18 times the training dataset we\nused in the previous manuscript. We also used a much deeper MLP with 129,105,930 parameters (around\n192 times the original model) as the teacher model and 12,615,690 parameters (around 81 times the original\nmodel) as the student model for the 10-class classification task. We also prepared a query dataset with 5,000\nsamples.\nStill, we have great respect for the concern of the reviewer. Therefore, to further address the issue, we\nprepared a much larger dataset named ImageNet that contains 14,197,122 samples, which is 236 times the\nMNIST dataset. To speed up model training, we used the down-sampled version containing 2,562,320 images\nwith a shape of 64x64 from 1000 classes, which still took us around 3 days to train the base model. We\nalso used a more complex network called VGG-19 with 143,667,240 parameters as the teacher model and\nVGG-11 with 132,863,336 parameters as the student model for the 1000-class classification task. We also\nprepared a query dataset with 5,000 samples. Based on the results in Response Table 1, it should be adequate\nto prove that it is feasible to apply AFS to more complex neural networks and large datasets.\nIndependent teacher 1 0.8421 0.9273\nIndependent student 1 0.8312 0.8901\nIndependent teacher (k=0.5) 1.88e-39 0.5056 0.6891\nIndependent student (k=0.5) 1.52e-57 0.4994 0.6758\nAFS (k=0.5) 8.13e-245 0.7903 0.8502\nTable 1: Comparison of AFS with other methods on forgetting QF and model performance with the ImageNet\ndataset. We present the p-values of auditing models trained with different methods on QF and the model\n5000\nperformance including the accuracy and F1-score.\nReviewer #2 (Remarks to the Author):\nI thank the authors for their thorough responses. There is no other concern from my side.\nReviewer #3 (Remarks to the Author):\nThis reviewer would like to thank the authors for the thorough explanation and the clarification on why\nthe current algorithm is not directly applicable to regression tasks. The absence of a soft logits\nrepresentation across multiple labels in regression tasks and the reliance of the method on this\nrepresentation indeed make it challenging to directly apply the proposed approach to such tasks.\nNevertheless, it is encouraging to see that the authors intend to investigate new frameworks and\nstrategies specifically designed for regression problems in their future research.\nAuthors’ Response to Reviews of\nNCOMMS-23-05640B\nAudit to Forget: A Unified Method to Revoke Patients’ Pri-\nvate Data in Intelligent Healthcare\nJuexiao Zhou1,2,#, Haoyang Li1,2,#, Xingyu Liao1,2, Bin Zhang1,2, Wenjia He1,2, Zhongxiao Li1,2, Longxi\nZhou1,2, Xin Gao1,2,∗\n1Computer Science Program, Computer, Electrical and Mathematical Sciences and Engineering Division,\nKing Abdullah University of Science and Technology (KAUST), Thuwal 23955-6900, Kingdom of Saudi\nArabia\n2Computational Bioscience Research Center, King Abdullah University of Science and Technology, Thuwal\n23955-6900, Kingdom of Saudi Arabia\n∗Corresponding author\n#Equal contribution\nRC: Reviewers’ Comment, AR: Authors’ Response, □ Manuscript Text\n1. General Response\nAR: We really appreciate the editor [redacted] and both reviewers’ valuable feedback and suggestions, which\nhave extremely helped us improve the quality and clarity of our work. We are pleased to see that both\nreviewers are satisfied with the previous revision.\n2. Response to Reviewer #2’s Comments\nRC: I thank the authors for their thorough responses. There is no other concern from my side.\nAR: We would like to express our heartfelt gratitude to the reviewer for dedicating valuable time to review our\nmanuscript and for providing us with precious and constructive feedback.\n1\n3. Response to Reviewer #3’s Comments\nRC: This reviewer would like to thank the authors for the thorough explanation and the clarification on why the\ncurrent algorithm is not directly applicable to regression tasks. The absence of a soft logits representation\nacross multiple labels in regression tasks and the reliance of the method on this representation indeed make\nit challenging to directly apply the proposed approach to such tasks. Nevertheless, it is encouraging to see\nthat the authors intend to investigate new frameworks and strategies specifically designed for regression\nproblems in their future research.\nAR: We would like to express our sincere appreciation to the reviewer for taking valuable time and providing us\nwith constructive feedback. We are very grateful to the reviewer for pointing out a valuable research direction,\nand we will focus our future work on how to design new solutions for the regression task.\n2"}]

Decision letter