Restricted Category Removal from Model Representations using Limited Data
Authors: Pratik Mazumder
Pravendra Singh
Mohammed Asad Karim
Venue: ICLR 2022 Submitted (2022)

Abstract: Deep learning models are trained on multiple categories jointly to solve several real-world problems. However, there can be cases where some of the classes may become restricted in the future and need to be excluded after the model has already been trained on them (Class-level Privacy). It can be due to privacy, ethical or legal concerns. A naive solution is to simply train the model from scratch on the complete training data while leaving out the training samples from the restricted classes (FDR - full data retraining). But this can be a very time-consuming process. Further, this approach will not work well if we no longer have access to the complete training data and instead only have access to very few training data. The objective of this work is to remove the information about the restricted classes from the network representations of all layers using limited data without affecting the prediction power of the model for the remaining classes. Simply fine-tuning the model on the limited available training data for the remaining classes will not be able to sufficiently remove the restricted class information, and aggressive fine-tuning on the limited data may also lead to overfitting. We propose a novel solution to achieve this objective that is significantly faster ($\sim200\times$ on ImageNet) than the naive solution. Specifically, we propose a novel technique for identifying the model parameters that are mainly relevant to the restricted classes. We also propose a novel technique that uses the limited training data of the restricted classes to remove the restricted class information from these parameters and uses the limited training data of the remaining classes to reuse these parameters for the remaining classes. The model obtained through our approach behaves as if it was never trained on the restricted classes and performs similar to FDR (which needs the complete training data). We also propose several baseline approaches and compare our approach with them in order to demonstrate its efficacy.

Under review as a conference paper at ICLR 2022
RESTRICTED CATEGORY REMOVAL FROM MODEL REPRESENTATIONS USING LIMITED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning models are trained on multiple categories jointly to solve several real-world problems. However, there can be cases where some of the classes may become restricted in the future and need to be excluded after the model has already been trained on them (Class-level Privacy). It can be due to privacy, ethical or legal concerns. A naive solution is to simply train the model from scratch the complete training data while leaving out the training samples from the restricted classes (FDR full data retraining). But this can be very time-consuming process. Further, this approach will not work well we no longer have access to the complete training data and instead only have access very few training data. The objective of this work is to remove the information about the restricted classes from the network representations of all layers using limited data without affecting the prediction power of the model for the remaining classes. Simply fine-tuning the model on the limited available training data for the remaining classes will not be able to sufficiently remove the restricted class information, and aggressive fine-tuning on the limited data may also lead to overfitting. We propose a novel solution achieve this objective that is significantly faster (~ 200 on ImageNet) than the naive solution. Specifically, we propose novel technique for identifying the model parameters that mainly relevant to the restricted classes. We also propose a novel technique that uses the limited training data of the restricted classes to remove the restricted class information from these parameters and uses the limited training data of the remaining classes to reuse these parameters for the remaining classes. The model obtained through our approach behaves if it never trained the restricted classes and performs similar to FDR (which needs the complete training data). We also propose several baseline approaches and compare our approach with them in order to demonstrate its efficacy.
INTRODUCTION
There are several real-world problems in which deep learning models have exceeded human performance. This has led to a wide deployment of deep learning models. Deep learning models generally train jointly on a number of categories/classes of data. However, the of some of these classes may get restricted in the future (restricted classes), and a model with the capability to identify these classes may violate legal/privacy concerns, e.g., company may legally prevent a deep learning model from having the capability to identify its seeyright-protected logo, patented products, and so on. Another example is a treatment prediction model that predicts the best treatment for a patient based on the disease. If one of the treatments for a disease is banned due to its side-effects or ethical concerns, the restricted treatment category has to be excluded from the trained model. Individuals and organizations are becoming increasingly aware of these issues leading to an increasing number of legal cases on privacy issues recent years. such situations, the model has to be stripped of its capability to identify these categories. This is difficult problem to solve, especially if the full training data is no longer available and only few training examples are available. We present "Restricted Category Removal from Model Representations with Limited Data" (RCRMR-LD) problem setting that simulates the above problem. In this paper, we propose to solve this problem in fast and efficient manner.
The objective of the RCRMR-LD problem is to remove the information regarding the restricted classes from the network representations of all layers using the limited training data available without

---

Under review as a conference paper at ICLR 2022
affecting the ability of the model to identify the remaining classes. If we have access the full training data, then we can simply exclude the restricted class examples from the training data and perform a full training of the model from scratch using the abundant data (FDR full data retraining). However, the RCRMR-LD problem setting is based on the scenario that the directive to exclude the restricted classes is received in the future after the model has already been trained the full data and now only a limited amount of training data is available to carry out this process. Simply training the network from scratch on only the limited training data of the remaining classes will result in severe overfitting and significantly affect the model performance (Baseline 2, as shown in Tables I
Another possible solution to this problem is to remove the weights of the fully-connected classification layer of the network corresponding to the excluded classes such that it can no longer classify the excluded classes. However, this approach suffers from serious problem. Since, in this approach, we only remove some of the weights of the classification layer and the rest of the model remains unchanged, it still contains the information required for recognizing the excluded classes. This information can be easily accessed through the features that the model extracts from the images. Therefore, we can use these features for performing classification. In this paper, we use a nearest prototype- -ased classifier to demonstrate that the model features still contain information regarding the restricted classes. Specifically, we use the model features of the examples from the limited training data to compute the average class prototype for each class and create nearest class prototype-basee classifier using them. Next, for any given test image, we extract its features using the model and then find the class prototype closest to the given test image. This nearest class prototype-biat classifier performs close to the original fully-connected classifier on the excluded classes as shown in Tables I (Baseline 1). Therefore, even after using this approach, the resulting model still contains information regarding the restricted classes. Another possible approach can be to apply the standard fine-tuning approach to the model using the limited available training data of the remaining classes (Baseline 8). However, fine-tuning the restricted class information from the model representations (see Tables fine-tuning on the limited training data may result in overfitting.
such limited training data is not able to sufficiently remove
6). and aggressive
Considering the problems faced by the naive approaches mentioned above, we propose a novel "Efficient Removal with Preservation" (ERwP) approach to address the RCRMR-LD problem. First, we propose a novel technique to identify the model parameters that are highly relevant to the restricted classes, and to the best of our knowledge, there no existing prior works for finding such class- specific relevant parameters. Next, we propose a novel technique that optimizes the model on the limited available training data in such a way that the restricted class information is discarded from the restricted class relevant parameters, and these parameters are reused for the remaining classes
To the best of our knowledge, this is the first work that addresses the RCRMR-LD problem. Therefore, we also propose several baseline approaches for this problem (see Sec. [1.2]. However, our proposed approach significantly outperforms all the proposed baseline approaches. Our proposed approach requires very few epochs to address the RCRMR-LD problem and is, therefore, very fast (~ 200 times faster than the full data retraining model for the ImageNet dataset) and efficient. The model obtained after applying our approach forgets the excluded classes to such extent that it behaves as though was never trained on examples from the excluded classes. The performance of our model is very similar to the full data retraining (FDR) model (see Sec. 8. Fig. 5). We also propose the performance metrics needed to evaluate the performance of any approach for the RCRMR-LD problem. We perform experiments several datasets to demonstrate the efficacy of our method.
PROBLEM SETTING
In this work, we present the restricted category removal from model representations with limited data (RCRMR-LD) problem setting, in which a deep learning model M, trained on a specific dataset has to be modified to exclude information regarding a set of restricted/excluded classes from all layers of the deep learning model without affecting its identification power for the remaining classes (see Fig. 1]. The classes that need to be excluded are referred to as the restricted/excluded classes. Let {CT, ..,,. C²²)) be the restricted/excludet classes, where N₂ refers to the number of excluded classes. The remaining classes of the dataset are the remaining/non-exte exxellude classes. Let {CTe, Cre..... ..... CiV. } be the non-excluded classes, where N,e refers to the number of remaining/non- excluded classes. AAdiitionall,, we only have access to limited amount of training data for the restricted classes and the remaining classes, for carrying out this process. Therefore, any approach for addressing this problem can only utilize this limited training data.
2

---

Under review as a conference paper at ICLR 2022
M
M
M
x
M
M
M
Figure 1: The RCRMR-LD problem setting aims to re- move the information regard- ing the restricted/excluded classes ((Ci,.., C%,) from all layers of trained model M. while preserving its pre- dictive power for the remain- ing classes ((CTe... ,.,,,.., C'V. }) using limited training data. The category removal (de- noted by red cross) has to take place at the classifier level (denoted squares for each output logit) and at the fea- uure/representation level (de- noted as a circle).
ERwP identifies those parameters the model that are highly relevant to the restricted classes. To ob- tain these parameters, ERwP modifies training images from a restricted class using data augmentation ff and per- forms backpropagation using the classification loss these training images. ERwP then studies the gradient update that each parameter receives in this process in order iden- tify the highly relevant param- eters for the restricted classes (denoted by dotted circles).
Figure 2:
Figure 3: ERwP only optimizes the restricted class relevant parameters in the model (denoted by dotted cir- cles). ERwP uses C L and Lea losses to remove the restricted class information from the model while preserving its performance on the remaining classes, L and L de- note the classification loss on the re- stricted class training examples and the remaining class training exam- ple, respectively. Lia denotes the knowledge distillation-bl based regu- larization loss that preserves the logits corresponding to only the re- maining classes for all the training examples.
RCRMR-LD PROBLEM IN REAL WORLD SCENARIOS
real-world scenario where our proposed RCRMR-LD problem can arise is federated learn- ing (MeMahan et al., 2017). In the federated learning setting, there are multiple collaborators that have part of the training data stored locally, and a model is trained collaboratively using these private data without sharing or collating the data due to privacy concerns. Suppose organization has part of the training data, and there are other collaborators that have other parts of the training data for the same classes. Organization using federated learning. After the model has been trained, few classes may become restricted in the future due to some ethical or privacy concerns, and these classes should be removed from the model. However, the other collaborators may not be available or may charge a huge amount of money for collaborating again to train a fresh model from scratch. In this case, organization A does not have access to the full training data of the non-excludeddrrmaiingg classes that it model from scratch in order to exclude the restricted classes information. This clearly shows that the RCRMR-LD problem is possible in federated learning.
collaboratively trains a model with other collaborators
to re-train
Another real-world scenario is the incremental learning setting (Rebuffi et al. 2017 Kemker & Kanan 2018), where the model receives training data in the form of sequentially arriving tasks. Hach task contains a new set of classes. During a training session t, the model receives the task for training and cannot access the full data of the previous tasks. Instead, the model has access very few exemplars of the classes in the previous tasks. Suppose before training a model on training session t, is noticed that some classes from previous task (< have be removed from the model since those classes have become restricted due to privacy or ethical concerns. In this case, only limited number of exemplars are available for these previous classes (restricted and remaining). This demonstrates that the RCRMR-LD problem is also possible in the incremental learning setting. We experimentally demonstrate in Sec. 8.3 how our approach can address the RCRMR-LD problem in the incremental learning setting.
3

---

Under review as a conference paper at ICLR 2022
4 PROPOSED METHOD
Let, B refer to a mini-batch (of size S) from the available limited training data, and B contains training datapoints from the restricted/excluded classes (f(2), Vf (299 vi), .... (2's V'S }}) and from the remaining/non-exte excludee classes (f(x7', 33 ((44s ) .... (I y }}.. Here, (4f, yf refers to training datapoint from the excluded classes where x; is an image. y; is the corresponding label and of E {CT, C2 .... CN. J. "03""97" refers to a training datapoint from the non-excluded classes where 23 is an image, 0j is the corresponding label and v E [CTT cere.cr ..... CN... }. Here, S. and CN..
Sne refer to the number of training examples in the mini-batch from the excluded and non con-excluded classes, respectively, such that S = S. + Sne- N₂ and Ne refer to the number of excluded and non-excluded classes, respectively. Let M refer to the deep learning model being trained using our approach and Mo is the original trained deep learning model.
In a trained model, some of the parameters may be highly relevant to the restricted classes, and the performance of the model on the restricted classes is mainly dependent on such highly relevant parameters. Therefore, in our approach, we focus removing the excluded class information from these restricted class relevant parameters. Since the model is trained on all the classes jointly, the parameters are shared across the different classes. Therefore identifying these class-specific relevant parameters is very difficult. Let us consider a model that is trained on color images of a class. If we now train it on grayscale images of the class, then the model has to learn identify these new images. In order to do so, the parameters relevant to that class will receive large gradient updates as compared to the other parameters (see Sec. 9.I.. We propose a novel approach for identifying the relevant parameters for the restricted classes using this idea. For each restricted class, we choose the training images belonging to that class from the limited available training data. Next, we apply grayscale data augmentation technique/transformatioo f to these images so that these images become different from the images that the original model was earlier trained on (assuming that the original model has not been trained grayscale images). We can also other data augmentation techniques that not seen during the training process of the original model and that do not change the class of the image (refer to Sec. 11.7 in the appendix). Next, we combine the predictions for each training image into a single average prediction and perform backpropagation. During the backpropagation, we study the gradients for all the parameters in each layer of the model. Accordingly, we select the parameters with the highest absolute gradient as the relevant parameters for the corresponding restricted class. Specifically, for given restricted class, we choose the minimum number of such parameters from each network layer such that pruning these parameters will result in the maximum degradation of model performance on that restricted class. We provide detailed description of the process for identifying the restricted class relevant parameters in Sec. [1.1 of the appendix. The combined set of the relevant parameters for all the excluded classes is referred to as the restricted/excluded class relevant parameters O Derrel (see Fig. 2) Please note that we use this process only to identify + Oezrel- and we do not update the model parameters during this step.
Pruning the relevant parameters for a restricted class can severely impact the performance of the model for that class (see Sec. D.T). However, this may also degrade the performance of the model on the non-excluded classes because the parameters are shared across multiple classes. Therefore, we cannot address the RCRMR-LD problem by pruning the relevant parameters of the excluded classes. Finetuning these parameters on the limited remaining class data will also not be able to sufficiently remove the restricted class information from the model (see Sec. 11.10 in appendix). Based on this, we propose address the RCRMR-LD problem by optimizing the relevant parameters of the restricted classes to remove the restricted class information from them and to reuse them for the remaining classes.
After identifying the restricted class relevant parameters, our ERwP approach uses a classification loss based on the cross-entropy loss function to optimize the restricted class relevant parameters of the model on each mini-batch (see Fig. 3. We know that the gradient ascent optimization algorithm can be used to maximize a loss function and encourage the model to perform badly on the given input. Therefore, we use the gradient ascent optimization on the classification loss for the limited restricted class training examples to remove the information regarding the restricted classes from Oearet- We achieve this by multiplying the classification loss for the augmented training examples from the excluded classes by a constant negative factor of -1. We also optimize Gexrel using the gradient descent optimization on the classification loss for the limited remaining class training example, in order to reuse these parameters for the remaining classes. We validate using this approach through various ablation experiments shown in Sec. 9.2 The classification loss for the examples from the
4

---

Under review as a conference paper at ICLR 2022
excluded and non-excluded classes and the overall classification loss for each mini-batch are defined as follows.
Se
C
E(yi y1²))
(1)
i=1
Sue
LBe
(2)
j=1
Le S S(C. + C(ee
Where, of" and vf refer the predicted class labels for xf and x.e respectively. ((.,-) refers to the cross-entropy loss function. L and D refer to the classification loss for the examples from the excluded and non-excluded classes in the mini-batch, respectively. Le refers the overall classification loss for each mini-batch.
Since all the network parameters were jointly trained all the classes (restricted and remaining), the restricted class relevant parameters also contain information relevant to the remaining classes, Applying the above process alone will still harm the model's predictive power for the non-excluded classes (as shown in Sec. 19.2 Table 3). This is because the gradient ascent optimization strategy will also erase some of the relevant information regarding the remaining classes. Further, applying the on the limited training examples of the remaining classes will lead to overfitting and will not be effective enough to fully preserve the model performance on the remaining classes. In order to ensure that the model's predictive power for the non-excluded classes does not change, we use knowledge distillation- baasllatio--baedd regularization loss. Knowledge distillation (Hinton et al. (014) ensures that the predictive power of the teacher network is replicated in the student network. In this problem setting, we want the final model to replicate the same predictive power of the original model for the remaining classes. Therefore, given any training example, regularization loss to ensure that the output logits produced by the model corresponding to only the non-excluded classes remain the same that produced by the original model. We apply the knowledge distillation loss to the limited training examples from both the excluded and remaining classes, to preserve the non-excluded class logits of the model for any input image. We validate this regularization loss through ablation experiments as shown in Table 3 We use the original model M. (before applying ERwP) as the teacher network and the current model M being processed by ERwP as the student network, for the knowledge distillation process. Please note that the optimization for this loss is also carried out only for the restricted class relevant parameters of the model. Let K D refer to the knowledge distillation loss function. It computes the Dullback-Lieblee (KL) divergence between the soft predictions of the teacher and the student networks and can be defined as follows:
the knowledge distillation-basedd
K D Pt) = (L(((ps)) (p:))
(4)
where, (.) refers to the softmax activation function that converts logit ag for each class into
eap" probability by comparing a, with logits other classes aj, i.e., (a;)) = Expp" K refers to the temperature (Hinton et al. 2014), K L refers to the KL-Divergence function. Ps, refer to the logits produced by the student network and the teacher network, respectively.
The knowledge distillation-basee regularization losses in our approach are defined as follows.
Lid
(5)
i=1
Sme
L
(6)
Lut (5 CTa)
(7)
Where, M [+)[[CE) and M.(#)(CTE] refer to the output logits corresponding the remaining classes produced by M and Mo. respectively. # can be either x or x Lid and Lya refer to
knowledge distillation-bas. regularization loss for the examples from the excluded and non-excluded classes, respectively. Led refers to the overall knowledge distillation-batt regularization loss for each mini-batch.
5

---

Under review as a conference paper at ICLR 2022
The total loss Lerup of our approach for each mini-batch is defined as follows.
Lerup Le + 3Lud
(8)
Where, 8 is a hyper-parameter that controls the contribution of the knowledge distillation-basel regularization loss. We use this loss for fine-tuning the model for very few epochs.
RELATED WORK
Pruning involves removing redundant and unimportant weights (Carreira Caareira-Perri Perpinán & Idelbayev 2018
Dong et al. 2017| Guo et 2016; Han
al.,
[201 2015app by Tung & Mori 2018 Zhang et (0.1) or
filters (He et al. 2019a 2018 2019b c Li et al. 2016) from deep learning model without affecting
the model performance. In contrast, our approach identifies class-specific important parameters, and
therefore, the pruning techniques cannot be applied in our approach. In the incremental learning
setting (Douillard et al. 2020, Hou et al. 2019; Tao et al. 2020 Yu et al. 2020 Liu et al. 2021), the objective is to preserve the predictive power of the model for previously seen classes while learning new set of classes. In contrast, our proposed RCRMR-LA problem setting involves removing the information regarding specific classes from the pre-traine ereitra model while preserving the capacity of the model to identify the remaining classes. Privacy-preservie
deep learning (Nan & Tao 2020 Louizos al., 2015 Edwards & Storkey 2015 Hamm| 2017) involves learning representations that incorporate features from the data relevant to the given task and ignore sensitive information (such as the identity of a person). In contrast, the objective of the RCRMR-LD problem setting, to achieve class-level privacy. i.e., if a class is declared as private/restricted, then all information about this class should be removed from the model trained on it, without affecting its ability to identify the remaining classes. The authors in (Ginart et al., 2019) propose an approach to delete individual data points from trained machine learning models like clustering model. In contrast, RCRMR-LD involves removing the information of set of classes from all layers of a deep learning model. Therefore, the approach proposed in (Ginart et al. 2019) cannot be applied to the RCRMR-LD problem setting.
BASELINES
We propose 9 baseline models for the RCRMR-LD problem and compare our proposed approach with them. The baseline involves deleting the weights of the fully-connected classification layer corresponding to the excluded classes. Baselines 2, 3, 4, 5 involve training the model on the limited training data of the remaining classes. Baselines 6, 7, 8, 9 involve fine-tuning the model on the available limited training data. Please refer to Sec. 11.2 in the appendix for details about the baselines.
7 PERFORMANCE METRICS
In the RCRMR-LD problem setting, we propose three performance metrics to validate the per- formance of any method: forgetting accuracy (FA,), forgetting prototype accuracy (FPAg), ), and constraint accuracy (CAne). The forgetting accuracy refers to the fully-connected classification layer accuracy of the model for the excluded classes. The forgetting prototype accuracy refers to the nearest class prototype- based classifier accuracy of the model for the excluded classes. CAne refers to the fully-connected classification layer accuracy of the model for the non-excluded classes.
In order to judge any approach on the basis of these metrics, we follow the following sequence. First, we analyze the constraint accuracy (CAga) of the model produced by the given approach to verify if the approach has preserved the prediction power of the model for the non-excluded classes. CAna of the model should be close to that of the original model. If this condition is not satisfied, then the approach is not suitable for this problem, and we need not analyze the other metrics. This is because if the constraint accuracy not maintained, then the overall usability of the model is hurt significantly. Next, we analyze the forgetting accuracy (F A.) of the model to verify if the excluded class information has been removed from the model the classifier level. FA, of the model should be as close to 0% as possible. Finally, we analyze the forgetting prototype accuracy (FPA.) of the model to verify if the excluded class information has been removed from the model at the feature level. FPA, of the model should be significantly less than that of the original model. However, the FPA, will not become zero since any trained model will learn to extract meaningful features, which will help the nearest class prato classifier to achieve some non-negligible accuracy on the excluded classes. Therefore, for better analysis of the level of forgetting of the excluded classes at the feature level, compare the FPA, of the model with the F FPA, of the FDR model. The

---

Under review as a conference paper at ICLR 2022
Table I: Experimental results on the CIFAR-100 dataset for the RCRMR-LD problem.
Methods
ResNet-20
ResNett-56
ResNet-164
F F.A.
CAre
F.e
FPAA
CAre
F.A.
FPA
CAre
Original
70.15%
65. 25% 67.066%
70.80%
68.65%
69.88%
79.00%
76.40%
76.30%
No Training Baseline 1 WD
0.005:
65.25% 69.888
0.005)
68.65%
72.44%
0.00%:
76.40%
77.01%
Full Train Schedule
Baseline 2- T--NNNG
0.005
22,20
31.55%
0.00%
20.20%
30.21
0.00%
33.05%
40.65%
Baseline TSLNRC-KD
0.00%
27.55% 40.81%
0.00%
22.50% 32.26%
0.00%
38.55%
74%
Buseline 4- TOLNRC
0.00%
50.85%
58.01%
0.00%
48.60%
57.81%
0.00%
51.55% 62.78%
Baseline TOLNRC-KD
0.00%
60.25%
67.85%
0.00%
51.25%
61.14%
0.00%
52.80%
63.75%
Only Fine-Tuning
Baseline 6 FOLMRCSC
24.25%
59.55%
64.03%
13.35%
60.25%
65.23%
15.40%
59.20%
71.06%
Buscline 7 FOLMRCSC-KD
13.50%
58.80% 63.79%
12.75%
64.95%
63.41%
16.75%
65.30% 68.61%
Buseline 8- FOLNRC
59.05%:
64.30%
68, 34%
90%
68, 45%
70.11%
77.35%
75.85%
75.95%
Baseline 9 FOLNRC-KD
57.99%
64.40%
68.40%
65 95%
68, 40%
70.01%
73.30%
73.55% 75.99%
ERwP (Ours)
0.00% 48.06% 66.84% 0.00%
47.84% 69.32% 0.74%
56.23% 75.65%
FDR model is a good candidate for this analysis since it has not been trained on the excluded classes (only trained on the complete dataset of the remaining classes), and it still achieves a non- noglneglig performance of the excluded classes (see Sec 1ec.8.1. However, it should be noted that this comparison is only for analysis and the comparison is not fair since the FDR model needs to train on the entire dataset (except the excluded classes).
8 EXPERIMENTS
We have reported the experimental results for the CIFAR-100 and ImageNet-Ik 1k datasets in this section. We have provided the results on the CUB-200 dataset in the appendix. Please refer the appendix for the details regarding the dataset and implementation.
8.1 CIFAR-100 RESULTS
We report the performance of different baselines and our proposed ERwP method on the RCRMR-LD problem using the CIFAR-100 dataset with different architectures in Table We observe that the baseline 1 (weight deletion) achieves high constraint accuracy CAne and 0% forgetting accuracy FA.. But its forgetting prototype accuracy FPA remains the same as the original model for all the three architectures, i.c., ResNet-20/566 2ON/Neet22///1166 164. Therefore, baseline I fails to remove the excluded class information from the model at the feature level. Baseline is not able to preserve the constraint accuracy CA even though it performs full training on the limited excluded class data. Baseline 3 achieves higher CAne than baseline 2, but the constraint accuracy is still too low. Baselines 4 and 5 demonstrate significantly better constraint accuracy than baseline 2 and 3, but their constraint accuracy is still significantly lower than the original model (except baseline 5 for ResNet-20). The baseline 5 with ResNet-20 maintains the constraint accuracy and achieves 0% forgetting accuracy FA, but its FPA. is still significantly high and, therefore, is unable to remove the excluded class information from the model the feature level. The fine-tuning based baselines and 7 are able to significantly reduce the forgetting accuracy FA, but their constraint accuracy CA drops significantly. The fine-tuning based baselines 8 and only finetune the model on the limited remaining class data and as a result they are not able to sufficiently reduce either the forgetting accuracy FA, or the forgetting prototype accuracy FPA-.
Our proposed ERwP approach achieves a constraint accuracy CAne that is very close to the original model for all three architectures. It achieves close 0% FAg. Further, it achieves a significantly lower FPA. than the original model. Specifically, the FPA, of our approach is significantly lower than that of the original model by absolute margins of 17% 20.81 1%, and 20.17% for the ResNet-20, ResNet- 56, and ResNet- 164 architectures, respectively. The FPA. for the FDR model is 44.20%, 45.40% and 51 51.85% for the ResNet-20, ResNet-56 and ResNet- 164 architectures, respectively. Therefore, the FPA, of our approach is close to that of the FDR model by absolute margins of 3.86%. 2.44% and 4.38% for the ResNet-20, ResNet-56 and ResNet- 164 architectures, respectively. Therefore, our ERwP approach makes the model behave similar to the FDR model even though it was trained on only limited data from the excluded and remaining classes. Further, our ERwP requires only 10 epochs to remove the excluded class information from the model. Since the available limited training
7

---

Under review as a conference paper at ICLR 2022
Table 2: Experimental results on ImageNet- 1k.
Model
Methods
Top-1
Top-5
CAca
CR
Original
69.76%
69.76%
89.58%
89.02%
Res-18
ERwP
0,28%
69.13%
1.01%
88.93%
Original
76.30%
76.11%
93.04%
92.84%
Res-50
ERwP
0.25%
75.45%
2.55%
92.39%
Original 72.38%
70.83%
91.28%
90.18%
Mob-V2
ERwP
0.17%
70.81%
0.81%
89.95%
Table 3: Significance of ERwP components.
C:
Cea
Fhe
FPA.
Clae
Figure 4: Ablation to validate our approach for identifying relevant model parameters for a ran- dom restricted class of CIFAR-100.
/
x x
66.30%
68. 19%
69.79%
0.00%
6.45% 24.40%
/
0.00%
47.84%
69.32%
Table 4: Performance of ERwP in the incremental learning setting using ResNet-18
Model
FAe
CA
Original Model ohtained after Session 4 [M4] M4 modified with ERwP (Ours)
$6.39%
58.32%
0.20%
59.93%
data is only 10% of the entire CIFAR-100 dataset, therefore, our ERwP approach is approximately 30 * 10 = 300 faster than the FDR method that trained on the full training data for 300 epochs.
8.2
IMAGENET RESULTS
Table reports reports the experimental results for different upproaches to RCRMR-LD problem over the ImageNet- lk dataset using the ResNet-18 18, ResNet-50 and MobileNet V2 architectures. Our proposed FRwP approach achieves a top-1 constraint accuracy CAne that is very close to that of the original model by absolute margins of 0.63%. 0.66% and 0.02% for the ResNet-18, ResNet-50 and MobileNet V2 architectures, respectively. It achieves close to 0% top-1 forgetting accuracy FAg for all the three architectures. Therefore, approach performs well even the large-scale ImageNet- 1k dataset. Further, our ERwP requires only 10 epochs to remove the excluded class information from the model. Since the available limited training data is only 5% of the entire CIFAR-100 dataset, therefore, ERwP approach is approximately 20 * 10 = 200 faster than the FDR method that is trained on the full data for 100 epochs.
8.3 RCRMR-LD PROBLEM IN INCREMENTAL LEARNING
In this section, experimentally demonstrate how the RCRMR-LD problem in the incremental learning setting is addressed using our proposed approach. We consider an incremental learning setting the CIFAR- 100 dataset in which each task contains 20 classes. We use the BIC (Wu et al.) 2019) method for incremental learning on this dataset. The exemplar memory size is fixed at 2000 as per the setting in (Wu et al., 2019). In this setting, there are 5 tasks. Let us assume that the model (M4) has already been trained on 4 tasks (80 classes), and we are in the fifth training session. Suppose, this stage, it is noticed that all the classes in the first task (20 classes) have become restricted and need to be removed before the model is trained on task 5. However, we only have a limited number of exemplars of the 80 classes seen till now, i.c., 2000/80 = 25 per class. We apply our proposed approach to the model obtained after training session 4, and the results are reported in Table The results indicate that our approach modified the model obtained after session 4, such that the forgetting accuracy of the restricted classes approaches 0% and the constraint accuracy of the remaining classes is not affected. In fact, the modified model behaves as if, it was never trained on the classes from task 1. We can now perform the incremental training of the modified model on task 5.
ABLATION STUDIES
9.1 RESTRICTED CLASS RELEVANT PARAMETERS
We perform ablation experiments to verify our approach identifying the highly relevant parameters for any restricted class. We perform these experiments the CIFAR-100 dataset with the ResNet-56 architecture and report the forgetting accuracy FAg for the randomly chosen excluded class. Please note that in this case, only the chosen class of CIFAR-100 100 is the restricted class and all the remaining
8

---

Under review as a conference paper at ICLR 2022
Table 5: Experimental results on the CIFAR-100
dataset using ResNet-56 for ERwP with different number of excluded classes. # R/E > no. of non- excluded classes / no. of excluded classes
# R/E Methods
ResNet-20
ResNet-56
PAg
CA
PAg
CPee
Original
18%
67.35%
69.98%
70.11%
60/40
FRwP 0.00% 67.03%%
0.00%
69.98%
Diipial
Original
67.83% 67.66 61.6%
69.60%
70.26%
70/30
ERwP 0.00% 67.25%
69.81%
Mat aM.
Original
70.15% 67 .6.06%
70.80%
69.88%
80/20
ERwP 0.00% 66.85%
0.00%
69.26%
Rumber of Specha
Original
67.90% 67. 66.6%%
68, 68%40%
70.24%
90/10
ERwP
0.00% 67. 67.26%%
0.00%
69.69%
Figure 5: Plot denoting the performance our proposed ERwP during the optimization process.
Original
66.20% 67.76%
67.000 10.000%
70.22%
95/5
ERwP 0.00% 67.55% 0.00% 69.639%
classes constitute the non-excluded classes. In order show the effectiveness of our approach, sort the absolute gradients of the parameters in the model (obtained through backpropagation for the excluded class augmented images) and choose set of high relevance and low relevance parameters. We then prune/zero out these parameters and record the forgetting accuracy. Fig. 4 demonstrates that as we zero out the high relevance parameters, the forgetting accuracy of the excluded class drops by huge margin. It also shows that as we zero out the low relevance parameters, there is only a minor change in the forgetting accuracy of the excluded class. This validates our approach for identifying the high relevant parameters for the restricted classes.
9.2 SIGNIFICANCE OF THE COMPONENTS OF THE PROPOSED ERwP APPROACH
We perform ablations on the CIFAR- dataset using the ResNet-56 model to study the significance of the C. L and Ltd components of our proposed ERwP approach. Table 3 indicates that optimizing the restricted class relevant parameters using only L cannot significantly remove the information regarding the restricted classes from the model. Applying Cee along with L significantly reduces the forgetting accuracy FAe and forgetting prototype accuracy FPAc but also significantly reduces the constraint accuracy CAne. Finally, applying the Lkd loss along with L and C significantly reduces FA, and FPA, while maintaining the constraint accuracy CAmo very close to that of the original model.
9.3 ABLATION ON THE NUMBER OF EXCLUDED CLASSES
We report the experimental results for our approach for different splits of excluded and remaining classes of the C1F dataset in Table 5 We observe that our ERwP performs well for all the splits for both the 2e and ResNet-56 architectures.
9.4 PERFORMANCE OF ERwP OVER TRAINING EPOCHS
We analyze the change in the performance of the model after every epoch of our proposed ERwP approach in Fig. [ We observe that as the training progresses the constraint accuracy is maintained close to that of the original model, the forgetting accuracy keeps dropping till it reaches 0% and the forgetting prototype accuracy keeps falling and comes closer to that of the FDR model.
10 CONCLUSION
In this paper, we present a "Restricted Category Removal from Model Representations with Lim- ited Data" problem in which the objective is to remove the information regarding a set of ex- cluded/restricted classes from a trained deep learning model without hurting its predictive power for the remaining classes. We propose several baseline approaches and also the performance metrics for this setting. First, we propose novel approach to identify the model parameters that are highly relevant to the restricted classes. Next, we propose a novel efficient approach that optimizes these model parameters in order to remove the restricted class information and re-use these parameters for the remaining classes. We experimentally show how our approach significantly outperforms all the proposed baselines and performs similar the full data retraining model.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Miguel A Carreira- Perpinnn and Yerlan Idelbayev. "learning contiingccmmressionn algorithms for neural net pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 8532-8541, 2018.
Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, pp. 4860 4874, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, 2020.
Harrison Edwards and Amos Storkey. Censoring representations with adversary. arXiv preprint arXiv:1511.0589, 2015.
Antonio Ginart, Melody Y Guan, Gregory Valiant, and James Zou. Making ai forget you: Data deletion in machine learning. arXiv preprint ( 2019.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns, Advances in Neural Information Processing Systems, 29:1379-1387, 2016.
Jihun Hamm. Minimax filter: Learning to preserve privacy from inference attacks. The Journal of Machine Learning Research, 18(1))4704 4734, 2017.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 15 10. 49, 201 5a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Y He. G Kang, X Dong, Y Fu, and Y Yang. Soft filter pruning for accelerating deep convolutional neural networks. In ICCA International Joint Conference on Artificial Intelligence, 2018.
Yang He, Xuanyi Dong, Guoliang Kang, Yanwei Fu, Chenggang Yan, and Yi Yang. Asymptotic soft filter pruning for deep convolutional neural networks. IEEE transactions on cybernetics, 50(8): 3594-3604, 2019a.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4340-4349, 2019b.
Yang He, Ping Liu, Linchao Zhu, and Yi Yang. Meta filter pruning to accelerate deep convolutional neural networks. arXiv preprint arXiv: 1900:9944.0330 039611 2019c.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop. goog lerogleeroups.. com/
2014. URL
pdf
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In CVPR, pp. 831-839, 2019.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In International Conference on Learning Representations, 2018. URL /RRLpptpps///peeeeeiee
net httt:/ooommiiddeomm 8J1XmE Rb
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2022
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.
Yaoyao Liu, Bernt Schicle, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. arXiv preprint arXiv: 15000000000000, 2015.
Brendan McMMaan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication--Eficiiee Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statisties, volume 54 of Proceedings of Machine Learning Research, pp. 1272-1282. PMLR. 20-22 Apr 2017. URL
mmlr www.s/pwssss....c v54 htt.///mmaaaannnc 7a html
Lihao Nan and Dacheng Tao. Variational approach for privacy funnel optimization on continuous data. Journal of Parallel and Distributed Computing, 137:17-25, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch, 2017.
Selvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 1-22, 2015.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee local- ization. In Proceedings of the IEEE International Conference Computer Vision, pp. 618 626, 2017.
Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. Topology-preserring class-incremental learning. In ECCV, 2020.
Frederick Tung and Greg Mori. Clip-q: Deep network compression learning by in-parallel pruning- quantization. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 7873-7882, 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset, 2011.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 372-382, 2019.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In CVPR, pp. 2020.
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. systematic dnn weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference Computer Vision (ECCV), pp. 184-199, 2018.
11

---

Under review as a conference paper at ICLR 2022
11 APPENDIX
11.1 PROCESS FOR SELECTING THE RESTRICTED CLASS RELEVANT PARAMETERS
First, we apply a data augmentation technique f. not used during training, to the images of the given restricted class. Next, we combine the predictions for these images and perform backpropagation. Finally, we select the parameters with the highest absolute gradient as the relevant parameters for the corresponding restricted class. Specifically, for a given restricted class, we choose the minimum number of such parameters from each network layer such that pruning these parameters will result in the maximum degradation of model performance on that restricted class. We use a process similar to the binary search for automatically selecting the parameters with the highest absolute gradient. We use an automated script that first creates a list of parameters in each layer, sorts them in descending order according to the gradient values, and checks if zeroing out the weights of the first 5% parameters from this list leads to near zero accuracy for that class. If not, then we select double the number of parameters chosen earlier and repeat this process. If the accuracy is near zero, repeat the process with half the number of parameters chosen earlier. Please note, this process is just for identifying the parameters relevant the restricted classes, and their weights are restored after this process. The combined set of the relevant parameters for all the excluded classes is referred to as the restricted/excluded class relevant parameters.
11.2 BASELINES
We propose several bascline models for the RCRMR-LD problem and compare our proposed approach with them. The bascline approaches are defined as follows:
several
Original model: refers to the original model that is trained on the complete training set containing all the training examples from both the excluded and non-excluded classes. It represents the model that has not been modified by any technique to remove the excluded class information.
Baseline 1- Weight Deletion (WD): It refers to the original model with modified fully- colnecco classification layer. Specifically, we remove the weights corresponding to the excluded classes in the fully-connected classification layer so that it cannot classify the excluded classes.
Baseline - Training from Scratch on Limited Non-Restricted Class data (TSLNRC): In this baseline, we train new model from scratch using the limited training examples of only the non- excluded classes. It uses the complete training schedule as the original model and only uses the classification loss for training the model.
Baseline 3- Training from Scratch on Limited Non-Restricted Class data with KD (TSLNRC- KD): This baseline is the same as baseline but in addition the classification loss, it also uses knowledge distillation loss to ensure that the non-excluded class logits of the model (student) match that of the original model (teacher).
Baseline 4 Training of Original model on Limited Non-Restricted Class data (TOLNRC): This bascline the same as bascline 2, but the model is initialized with the weights of the original model instead of randomly initializing it.
Baseline Training of Original model on Limited Ron-Restricted Class data with KD (TOLNRC-KDD) This baseline is the same as baseline 4, but in addition to the classification loss, it also uses knowledge distillation loss.
Baseline 6 Fine-tuning of Original model on Limited data after Mapping Restricted Classes to a Single Class (FOLMRCSC): this baseline approach, we first replace all the excluded class labels in the limited training data with new single excluded class label and then fine-tune the original model for a few epochs on the limited training data of both the excluded and remaining classes. In the case of the examples from the excluded classes, the model is trained to predict the new single excluded class. In the case of the examples from the remaining classes, the model is trained to predict the corresponding non-excluded classes.
Baseline Fine-tuning of Original model on Limited data after Mapping Restricted Classes to a Single Class with KD MRSCCKC)))))): This baseline is the same baseline 6, but in
12

---

Under review as a conference paper at ICLR 2022
addition to the classification loss, it also uses knowledge distillation loss ensure that the non- excluded class logits of the model (student) match that of the original model (teacher).
Baseline 8 Finne-tuinn of Original model on Limited Non-Restricted Class data (FOLNRC): In this baseline approach, we fine-tune the original model for a few epochs on the limited training data of non-excfuddedrrmmiinng classes. The model is trained to predict the corresponding excluded classes of the training examples.
Baseline 9 Fine-tuning of Original model Limited Non-Restricted Class data with KD (FOLNRC-KD); This baseline is the same as baseline 8, but in addition to the classification loss, it also uses knowledge distillation loss.
11.3 DATASETS
For the RCRMR-LD problem setting, we modify the CIFAR- 100 (Krizhevsky et al. 2009), CUB (Wah) et a 2011) and ImageNet-1k (Russakovsky et al.] 2015) datasets. In order to simulate the RCRMR- LD problem setting with limited training data, we choose the last 20 classes of the CIFAR-I00 dataset as the excluded classes and take only 10% of the training images of each class. Similarly, we choose the last 20 classes of the CUB dataset as the excluded classes with only 3 training images per class. For ImageNet- IK, choose the last 100 classes as the excluded classes with 5% of the training images to simulate the limited data available for this problem setting.
11.4 IMPLEMENTATION DETAILS
In this section, we provide all the details required to reproduce our experimental results. We use the ResNet-20 (He et al., 2016), ResNet-56, ResNet- 164 architectures for the experiments on the CIFAR- 100 dataset. We use the standard data augmentation methods of random cropping to a size of 32 x 32 (zero-padded on each side with four pixels before taking a random crop) and random horizontal flipping, which is standard practice for training model on CIFAR- 100. In order to obtain the original and FDR models for the CIFAR-I00 dataset, we train the network for 300 epochs with mini-batch size of 128 using the stochastic gradient descent optimizer with momentum 0.9 and weight decay 1e- - 4. We choose the initial learning rate as 0.1, and we decrease it by a factor of 5 after every 50 epochs. For the CIFAR-100 experiments with ER using the ResNet-20, ResNet-56. and ResNet- 164 architectures, we take learning rate= 1e - 4, 3 = 10 and optimize the network for 10 epochs. Since the available limited training data is only 10% of the entire CIFAR- 100 dataset, therefore, our ERwP approach is approximately 30- 10 = 300x faster than the FDR method.
For the experiments on the ImageNet dataset, we use the ResNet-18, 18, ResNet-50, and MobilcNet-V2 architectures. We use the standard data augmentation methods of random cropping to size of 224 224 and random horizontal flipping, which is a standard practice for training a model on ImageNet-Ik Ik. In order to obtain the original and FDR models for the ImageNet dataset, we train the network for 100 epochs with mini-batch size of 256 using the stochastic gradient descent optimizer with momentum 0.9 and weight decay le 4. We choose the initial learning rate as 0.1, and we decrease it by a factor of 10 after every 30 epochs. For evaluation, the validation images are subjected to center cropping of size 224 x 224. For the ImageNet- Ik experiments (5% training data) with ERwP using the ResNet-50 architecture, we optimize the network for 10 epochs with a learning rate of 9e 5 using / 200. For the ERwP experiments using the ResNet- 18 architecture, we optimize the network for 10 epochs using 3 200 with an initial learning rate of 1.1e - 4 and learning rate of 1.le 5 from the third epoch onward. In the case of the ERwP experiments with the MobileNet-V V2 architecture, we optimize the network for 10 epochs using 3 = 400 with an initial learning rate of 1.5e- 4 and learning rate of 1.5e- 5 from the third epoch onward. Since the available limited training data is only 5% of the entire ImageNet- 1k dataset, therefore, our ERwP approach is approximately 20 * 10 = 200x faster than the FDR method. For the experiments on the CUB-200 dataset, we use the ResNet-50 architecture pre-trained on the ImageNet dataset. In order to obtain the original and FDR models for the CUB-200 dataset, we train the network for 50 epochs with mini-batch size of 64 using the stochastic gradient descent optimizer with momentum 0.9 and weight decay le - 3. We choose the initial learning rate as le - 2, and we decrease it by a factor of 10 after epochs 30 and 40. For the CUB-200 experiments (3 images per class, i.c., 10% training data) with ERwP using the Re architecture, we optimize the network for 10 epochs with a learning rate of le using/ B 10. Since the available limited training data is only 10% of the
13

---

Under review as a conference paper at ICLR 2022
Table Experimental results on the CUB dataset with ResNet-50 architecture for the RCRMR-LD problem with 20 excluded classes using only 3 training images per class.
Methods
FPA.
CAca
Original
85.20%
84.69%
77.37%
Training Bascline 1- WD
0.00%
85.71% 77.64%
Full Train Schedule
Baseline 2- TSLNRC
0.00%
30.27%
27.56%
Bascline 3- TSLNRC-KD
0.00%
35.54%
31.66%
Baseline 4- TOLNRC
0.00%
60.37%
64.60%
Baseline 5 TOL-NR-KD
0.00%
68.37%
70.48%
Ouly Fine-Tuning
Baseline 6 FOLMRCSC
53,40%
77.38%
74.39%
Baseline 7 FOI MRCSC-KD
60.88%
81.12%
75.14%
Baseline 8 FOLNRC
84.86%
84.18%
76.85%
Baseline 9 1--FOLRCC-II
84.35%
85 20%
77.70%
ERwP (Ours)
0.77%
48.89%
75.45%
entire CUB-200 dataset, therefore, ERwP approach is approximately 5 10 50x faster than the FDR method.
In our proposed approach, use = for all the experiments. We a popular Pytorch implementation"] for performing knowledge distillation. We run all the experiments 3 times (using different random seeds) and report the average accuracy. We perform all the experiments using the Pytorch framework version 1.6.0 (Paszke et al. (2017) and Python 3.0. We use 4 GeForce GTX 1080 Ti graphics processing units for our experiments.
11.5 RCRMR-LD PROBLEM CUB-200 CLASSIFICATION
Table/6 the experimental results for different approaches to the RCRMR-LD problem over the CUB dataset using the ResNet-50 architecture. Our proposed ERwP approach achieves a constraint accuracy CAse that very close that of the original model even though we use only images per class for optimizing the model. It achieves close to 0% forgetting accuracy FAe and achieves a FPAp that significantly lower than that of the original model by an absolute margin of 35. 80%. Similar to the CIFAR- 100 experiments, our ERwP approach outperforms all the baseline approaches. Further, our ERwP requires only 10 epochs to remove the excluded class information from the model. Since the available limited training data is only 10% of the entire CUB dataset, therefore, our ERwP approach approximately * 50x faster than the FDR method that is trained on the full training data for 50 epochs.
11.6 ABLATION EXPERIMENTS FOR 3 AND K
We perform ablation experiments to identify the most suitable values for the hyper-parameters B and k. for our proposed ERwP. The ablation results Tables validate our choice hyper- parramerr values considering the forgetting accuracy and the constraint accuracy of the resulting model.
11.7 EFFECT OF DIFFERENT DATA AUGMENTATIONS ON THE IDENTIFICATION OF CLASS RELEVANT MODEL PARAMETERS
We perform experiments to verify our approach of identifying the highly relevant parameters for any restricted class using various augmentation techniques (grayscale, vertical flip, rotation, random affine augmentations). We chose the same restricted class CIFAR- 100 and used the ResNet-56 network for all the experiments. The results in Fig, 6 indicate that for all the compared data augmentations approaches, pruning/zeroing out the high relevance parameters obtained using our approach, results
14

---

Under review as a conference paper at ICLR 2022
Table 7: Experimental results on the CIFAR- Table Experimental results on the CIFAR- 100 dataset with ResNet-20 architecture for the RCRMR-LD problem with 20 excluded classes RCRMR-LD problem with 20 excluded classes using our proposed ERwP with different values using our proposed ERwP with different values of B.
100 dataset with ResNet-20 architecture for the
of K.
3
Methods
FAc
CApa
Methods
FAa
Cāgc
Original
70.15%
67.06%
Original
70.15%
67.06%
8
ERwP
0.00%
66.03%
1.0
ERwP
0.00%
66.05%
9
ERwP
0.00%
66. 66.23%
1.5
ERwP
0.00%
66.08%
10
ERwP
0.00%
66.79%
2.0
ERwP
0.00%
66.79%
11
ERwP
0.00%
66.58%
2.5
ERwP
0.00%
66.30%
12
FR
0.00%
65%15%
3.0
ERwP
0.00%
66.23%
High balerme
parseaterss
(b)
Figure 6: Ablation to validate our approach for identifying restricted class relevant model parameters using different augmentation techniques w.r.t. the same randomly chosen restricted class of CIFAR- 100. We use the ResNet-56 network for these experiments. The data augmentation techniques used (a) grayscale augmentation, (b) vertical flip augmentation, (c) rotation augmentation, (d) random affine augmentation. In each case, the figure shows the model performance when the low relevance and high relevance parameters obtained using our approach are zeroed out.
huge drop the forgetting accuracy of the excluded class. Further, zeroing out the low relevance parameters, has a minor impact on the forgetting accuracy of the excluded class.
11.8 ABLATION EXPERIMENTS ON THE RESTRICTED CLASS RELEVANT PARAMETER
We perform ablation experiments with ERwP to check if only 25% and 50% of the restricted class relevant parameters of each layer identified using our proposed procedure can be used for ERwP. We run each of these experiments for the same number of epochs for the CIFAR- 100 dataset and ResNet- 56 network. However, we observed that the final FPA. falls from 68.65% 60.35% and 53.7%, respectively, for 25% and 50% of restricted class relevant parameters of each layer as compared to 47.84% when using all the restricted class relevant parameters per layer identified using our approach. The good performance of our approach is more evident in light of the performance of the FDR model that achieves FPAg accuracy of 45.40%. We provide this result that the 47.84% FPA, accuracy is due to the generalization power of the model and not due to the restricted classes information in the model. This shows that our approach effectively identifies the class-relevant parameters of the model for a given class.
reference to demonstrate
11.9 PERFORMANCE OF ERwP OVER TRAINING EPOCHS
We analyze the change the performance the model after every epoch of our proposed ERwP approach in Fig. 7 for the CIFAR-100 dataset with 20 excluded classes using the ResNet-20 and ResNet-56 architectures. For both architectures, we observe that as the training progresses, ERwP maintains the constraint accuracy close that of the original model and forces the forgetting accuracy to drop to 0%. ERwP also forces the forgetting prototype accuracy to keep dropping and makes similar the FDR model.
15

---

Under review as a conference paper at ICLR 2022
ResNet-20
Res
Accuracy
67
ERwP CA
CRwP CAu
Original
Original
ERMP Me
ERWP FAc
20
ER (M
CRwP rIM,
FDR 1PM
FOR IFA
Number of Epochs
Number Epochs
(a)
(b)
Figure 7: Plots denoting the performance of our proposed ERwP during the optimization process for forgetting 20 excluded classes from CIFAR- 100 using a) ResNet-20 and b) ResNet-56 architectures.
Images of Remaining Classes
Images of Excluded Classes
Figure 8: Class activation maps ImageNet images from the excluded and non-excluded classes, for the original ResNet-50 (second row) and ResNet-50 after applying our proposed ERwP approach (third row). First row depicts the real images.
11.10 FINETUNING RESTRICTED CLASS RELEVANT PARAMETERS ON REMAINING CLASSES
In our experimental results, demonstrated how finetuning the model the limited training data of the non-excluded classes cannot sufficiently remove the excluded class information from the model. We also perform ablation experiment demonstrate that finetuning only the restricted class relevant parameters using the limited training data of the non-exclus non-excluded classes is also not effective in sufficiently removing the excluded class information from the model. We perform these experiments on the CIFAR- dataset with the ResNet-56 architecture. We observe the constraint accuracy CAne. forgetting accuracy FA, and the forgetting prototype accuracy FA, are almost the same even in this case. Therefore, finetuning only on the remaining class data cannot sufficiently remove the excluded class information from model representations.
11.11 EFFECT OF USING THE PROPOSED ERwP APPROACH WHEN THE ENTIRE DATASET IS AVAILABLE
We perform ablation experiments demonstrate the performance of our proposed ERwP approach when the entire training data is available. We perform these experiments on the CIFAR-100 100 dataset using ResNet-20 and ResNet-56. We observe experimentally that for both the ResNet-20 and ResNet-
16

---

Under review as a conference paper at ICLR 2022
56 experiments using ERwP, the forgetting accuracy FA, accuracy is 0% and the constraint accuracy CAne matches that of the original model. Further, the gap between the forgetting prototype accuracy FPA, of ERwP and the FDR model reduces from 3.86% (for limited data) 2.79% for ResNet-20. Similarly, the gap reduces from 2.44% (for limited data) to 1.65% for ResNet-56. However, ERwP requires only 2-3 epochs of optimization (~100-1550x faster than the FDR model) for achieving this performance when trained on the entire dataset. This makes it significantly faster than any approach that trains the entire dataset.
11.12 QUALITATIVE ANALYSIS
In order to analyze the effect of removing the excluded class information from the model using proposed ERwP approach, we study the class activation map visualizations (Selvaraju et al. 2017) of the model before and after applying ERwP. We observe in Fig. that for the images from the excluded classes, the model's s region of attention gets scattered after applying ERwP, unlike the images from the remaining classes.
17


Reviews:
[{"review_id": "wWo5aDISV3M", "reviewer": "Reviewer_Y25C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The problem RCRMR-LD seems interesting and practical, which addresses the specific class-level restriction by removing corresponding model representations. This setting also save time and computational resource for large scale datasets.\n\n2. Experiments in this paper are solid and convincing enough. They design 5 basic baselines and perform corresponding ablation study. Considering RCRMR-LD is a new problem, if there exists, comparing some related works will be better.\n\n################################################\n\nCons:\n\n1. From my point of view, the transformation $f$ plays a key role in identifying the parameters that are highly relevant to the\nrestricted classes. However, they seem only try the grayscale transformation and do not give more discussion about $f$. If the model is just trained by grayscale images, will this method fail? For natural language tasks, what transformation are you going to use? I suggest that the authors make more discussion and comparison of the various transformations.\n\n2. From Table 1, I find all the FPA$_e$ of ERwP are relatively high, indicating that the feature representations of the model still contain much restricted category information. Although they indeed remove restricted category from class-level, attackers still can use some model inversion techniques (e.g., [1]) to restore the restricted class data with few owned ones, leading to privacy leakage.\n\n3. Identifying those parameters that are relevant to the restricted classes through ERwP is still heuristic. I admit that ERwP seems to make sense, but some verifications about this claim need to be included.\n\n4. Except for \"Related work\", I do not find any references in this paper. At least in the \"Introduction\", you should cite some related works to support your claims.\n\n5. Colloquial expressions and grammar issues are common, thus the writting needs further improvement.\n\n#########################################################\n\nTypo:\n1. \"Baseline 4 - Training of Original model on Limited Non-Restricted Class data with (TOLNRC):\" -- missing words?\n\n2. \"$N_e$ and $N_r$ refer to the number of excluded classes, respectively.\" -- missing words?\n\n and so on...\n\n############################################################\n\nQuestions during rebuttal period:\n\nPlease address and clarify the cons above.\n\n##############################################################\n\nReferences:\n\n[1] Zhang Y, Jia R, Pei H, et al. The secret revealer: Generative model-inversion attacks against deep neural networks. CVPR, 2020.", "summary_of_the_paper": "This paper proposes a novel and practical problem called RCRMR-LD, aiming to removel restricted categories from model representations with limited data. They first give some direct solutions and analyze their weaknesses. Then, they propose their own solution to discard the restricted class information from the restricted class relevant parameters. Experiments verify that this approach not only performs similar to FDR but also is faster than it.", "summary_of_the_review": "The setting proposed by this paper is novel and practical. However, there exists some technical flaws that need to bu further solved.\n\nPlease see the \"Main Review\" for details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ian8ijttsIZ", "reviewer": "Reviewer_yBwg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n1. The paper studies an important problem of tackling with restricted classes.\n2. The presented approach displays an ability to remove restricted class information from model parameters.\n\nNegatives:\n1. The paper is not very clearly written, with concepts repeated several times and not clear description on some others that are mentioned below.\n2. While the problem is interesting indeed, the motivation for the proposed solution is not clearly presented. Instead of repeating the ideas, it would be helpful to have a few clear examples that illustrate the need to solve this problem, as well as a clear description of the behavior of the said approach. While an example about the company logo is stated, it would be helpful to have a few more clear examples from real-world settings to help the reader. One such example, a model trained to predict which treatment would be beneficial for the patient would need to be altered if the treatment cannot be offered in the future due to ethical or resource constraints. \n3. While empirical results on the CIFAR-100 and ImageNet-1K datasets seem promising, it would be helpful to study this in the real-world dataset. Issues such as generalizability due to distribution shifts in the future and fairness considerations when certain labels are dropped are potential directions. \n\nAdditional comments:\n1. The notation for the excluded and non-excluded classes is a bit confusing as $C_e, C_r$ can both mean excluded or restricted. I would suggest to change this. ", "summary_of_the_paper": "The paper tackles the problem of restricted class unavailability after a deep learning model has already been trained on such restricted classes and the aim is to remove any information pertaining to the restricted classes from the model parameters so that the model will not be able to correctly classify the restricted classes in the future. The approach presented includes identifying the model parameters that are most relevant to the restricted classes and removing the restricted class information from these parameters (gradient ascent) while ensuring that these parameters can still be used for accurately classifying other non-restricted classes. With the need to correctly assess the utility of the proposed approach, several baseline methods have been proposed. Empirical results on the CIFAR-100 and ImageNet-1K datasets illustrate how the proposed approach can be used. ", "summary_of_the_review": "The paper studies an important problem. However, there are some challenges with respect to the writing, motivation of the solution and potentially several important directions that can be addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dWK2pKdofdN", "reviewer": "Reviewer_pyXW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Introduction:\nThe paper is well written, and the evaluation is very detailed. It is an interesting idea to remove class information from the model with a limited amount of data. However, from the description in the paper, it is not clear why this is a real-world problem. It would be beneficial to rate the importance of this application if the authors had provided sources for such cases or a more detailed description of a specific scenario. \n\nMethod:\nThe re-training procedure of the model with only a limited amount of training data is very detailed. The description of the identification of the relevant parameters for the restricted classes is missing some details. For example, it is not defined what other transformation besides the grayscale transformation is used. If other transformations are optional, it would be good to know what type of transformations are used in the experiments. Furthermore, it is not clear how the parameters with the highest gradients are selected. Is a fixed threshold used? What is the minimum number of parameters of each layer that are selected? Is this a fixed number for each class? Does it depend on the number of excluded classes?\n\nEvaluation:\nThe evaluation is very detailed, with eight baseline methods to show the performance of the presented method. However, the results of the FDR model are not shown in Table 1, only mentioned in the text. Is there a reason for this? Adding the results of the FDR to Table 1 would be beneficial. It would be very interesting to see how the parameter selection influences the accuracy of the model. Unfortunately this is not part of the evaluation.\n", "summary_of_the_paper": "In this paper, the authors present a new method to remove information about specific classes from a trained model without reducing the performance of the remaining classes. After the information is removed, the model should not be able to identify the class anymore. Instead of retraining the complete model from scratch without the restricted classes, the presented method only needs a few examples of the restricted classes and the remaining classes. In terms of speed, the presented method is ~200 times faster on ImageNet than a new model training without the restricted classes. Furthermore, they present a method for identifying model parameters that are mainly relevant to the restricted classes. The evaluation of the model is performed on the CIFAR-100, ImageNet-1k, and the CUB-200 dataset. For a detailed comparison, eight baseline methods were designed and evaluated. An ablation study is performed on the class relevant parameters and the number of classes that are excluded. The presented method achieves an accuracy close to the original model on the remaining classes in terms of accuracy. Also, the forgetting prototype accuracy is close to the model trained only on the remaining classes.\n", "summary_of_the_review": "The presented method of re-training a model to forget a specific class is very interesting. However, the part for identifying the most relevant model parameter is missing some essential details. For example, how the parameters are selected (manually or automatic) or the number of selected parameters. This information is essential to understand the method. Moreover, the influence of the parameter selection method is not studied in the evaluation part.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QLjmJH0Xtls", "reviewer": "Reviewer_3V1Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a new learning setting of fine-tuning a pretrained model to forget some specific categories which is motivated by class-level privacy. The solution to this challenge is firstly detecting the most related model parameters that significantly affect model performance on restricted classes and then tuning on a small number of examples with the losses of desired classification capability. The proposed method is experimentally demonstrated effective than possible baselines.\n\nI mainly have the following concerns.\n1. The motivation of the new setting is not strong. In introduction, the class-level privacy is specified by violated privacy concerns and corresponding examples. However, they are not quite convincing to me, and I feel more practical instances are needed to clarify the significance of studying class-level privacy. In particular, in what situation there would be only a few training examples available when considering removing information of restricted class from model concerned with privacy? \n\n2. In related work, individual data deletion (Ginart et al 2019) is cited but not properly evaluated. Following the work of data deletion, I feel there also exists an important problem which is ignored. That is, making model forget some examples or some classes does not mean zero classification accuracy or random classification accuracy (i.e., 1/N). In data deletion work, Ginart et al tune the pretrained model by only compensating the impact of deleted samples instead of forcing model have large error on them. As a result, the tuned model turns out to be never seeing the deleted examples. This work obviously cannot guarantee it from the loss shown as Eq. 1. For example, a 3-way classifier on dog, cat, and leopard and leopard is the restricted class. It is predicted a classifier training on dog and cat only would intend classify leopard to cat because of their natural similarity. Thus, a careful clarification about this point is required in this paper, especially from the view of class-privacy.\n\n3. The process of identifying parameters related to restricted classes seems quite empirically, as a transformation component is needed from some prior knowledge. The authors have mentioned it for images. However, many data privacy related data are also tabular. In this case, how to apply a proper transformation? If this component is quite related to data format, any workaround for this issue?\n\n4. From Figure 3, KD is defined for remaining classes only, but the KD loss also includes restricted classes.\n\n5. It is interesting to see the model performance comparison with the original training in term of remaining classes only (also related to concern 2,  the model performance on the original raining data of remaining classes only may be a good reference point for evaluation), although original training data might be inaccessible in the proposed setting. \n", "summary_of_the_paper": "This paper proposes a new learning setting of fine-tuning a pretrained model to forget some specific categories which is motivated by class-level privacy. The solution to this challenge is firstly detecting the most related model parameters that significantly affect model performance on restricted classes and then tuning on a small number of examples with the losses of desired classification capability. The proposed method is experimentally demonstrated effective than possible baselines.", "summary_of_the_review": "The paper has a weak motivation for the new setting. The proposed method seems too heuristic and the evaluation for the new setting is not appropriate.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a technique to efficiently retrain a model when a small number of classes are required to be removed. 
Reviewers in general like the paper, but the key issue is motivation for the problem. The motivating examples in the rebuttal are not very good because a. authors do not provide any evidence that such situations are critical or commonplace, b. the data points that are available for retraining might be very biased. 
A more careful grounding of the work would be important to motivate the ICLR community and the ML community in general to further study this problem. But for now, unfortunately the paper does not seem ready for publication at ICLR.
Decision: Reject