Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack

Previous studies have verified that the functionality of black-box models can be1 stolen with full probability outputs . However , under the more practical hard-label2 setting , we observe that existing methods suffer from catastrophic performance3 degradation . We argue this is due to the lack of rich information in the probability4 prediction and the overfitting caused by hard labels . To this end , we propose a5 novel hard-label model stealing method termed black-box dissector , which consists6 of two erasing-based modules . One is a CAM-driven erasing strategy that is7 designed to increase the information capacity hidden in hard labels from the victim8 model . The other is a random-erasing-based self-knowledge distillation module9 that utilizes soft labels from the substitute model to mitigate overfitting . Extensive10 experiments on four widely-used datasets consistently demonstrate that our method11 outperforms state-of-the-art methods , with an improvement of at most 8.27 % . We12 also validate the effectiveness and practical potential of our method on real-world13 APIs and defense methods . Furthermore , our method promotes other downstream14 tasks , i.e. , transfer adversarial attacks.15 1 Introduction16 Machine learning models deployed on the cloud can serve users through the application program17 interfaces ( APIs ) to improve productivity . Since developing these cloud models is a product of18 intensive labor and monetary effort , these models are valuable intellectual property and AI companies19 try to keep them private . However , the exposure of the model ’ s predictions represents a significant20 risk as an adversary can leverage this information to steal the model ’ s functionality , a.k.a . model21 stealing attack [ 22 , 20 , 21 ] . With such an attack , adversaries are able to not only use the stolen model22 to make a profit , but also mount further adversarial attacks [ 34 , 29 ] . Besides , the model stealing23 attacks is a kind of black-box knowledge distillation which is a hot research topic . Studying various24 mechanisms of model stealing attack is of great interest both to AI companies and researchers.25 Previous methods [ 20 , 34 , 21 ] mainly assume the complete probability predictions of the victim26 model available , while the real-world APIs usually only return partial probability values ( top-k27 predictions ) or even the top-1 prediction ( i.e. , hard label ) . In this paper , we focus on the more28 challenging and realistic scenario , i.e. , the victim model only outputs the hard labels . However , under29 this setting , existing methods suffer from a significant performance degradation , even by 30.50 % ( as30 shown in the Fig . 1 ( a ) and the appendix Tab . I ) .31 To investigate the reason for the degradation , we evaluate the performance of attack methods with32 different numbers of prediction probability categories available and hard labels as in Fig . 1 ( b ) . With33 the observation that the performance degrades when the top-k information missing , we conclude34 that the top-k predictions are informative as it indicates the similarity of different categories or35 multiple objects in the picture , and previous attack methods suffer from such information obscured36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . by the top-1 prediction under the hard-label setting . It motivates us to re-mine this information by37 eliminating the top-1 prediction . Particularly , we design a novel CAM-based erasing method , which38 erases the important area on the pictures based on the substitute model ’ s top-1 class activation maps39 ( CAM ) [ 24 , 33 ] and queries the victim model for a new prediction . Note that we can dig out other40 class information in this sample if the new prediction changes . Otherwise , it proves that the substitute41 model pays attention to the wrong area . Then we can align the attention of the substitute and the42 victim model by learning clean samples and the corresponding erased samples simultaneously.43 ( a ) Hard label vs soft label ( c ) W/O self-KD ( d ) W/ self-KD ( b ) Performance with top-k available from the victim model and pseudo labels from the previous substitute model to train a new substitute69 model . Therefore , we can also consider the ensemble of the two models as the teacher in knowledge70 distillation . As in Fig . 1 ( d ) , such a module helps generalization and better performance.71 In summary , we propose a novel model stealing framework termed black-box dissector , which72 includes a CAM-driven erasing strategy and a RE-based self-KD module . Our method is orthogonal73 to previous approaches [ 20 , 21 ] and can be integrated with them . The experiments on four widely-74 used datasets demonstrate our method achieves 43.04 − 90.57 % test accuracy ( 47.60 − 91.37 % 75 agreement ) to the victim model , which is at most 8.27 % higher than the state of the art method.76 We also proved that our method can defeat popular defense methods and is effective for real-world77 APIs like services provided by Amazon Web Services ( AWS ) . Furthermore , our method promotes78 downstream tasks , i.e. , transfer adversarial attack , with 4.91 % − 16.20 % improvement.79 2 Background and Notions80 Model stealing attack is aim to find a substitute model f̂ : [ 0 , 1 ] d 7→ RN that performs as similarly81 as possible to the black-box victim model f : [ 0 , 1 ] d 7→ RN ( with only outputs accessed ) . Papernot82 et al . [ 22 ] first observed that online models could be stolen through multiple queries . After that , due83 to the practical threat to real-world APIs , several studies paid attention to this problem and proposed84 many attack algorithms.85 These algorithms consist of two stages : 1 ) constructing a transfer dataset DT ( step 1 in Fig . 2 ) and86 2 ) training a substitute model . The transfer dataset is constructed based on data synthesis or data87 selection and then feed into the victim model for labels . Methods based on data synthesis [ 34 , 14 , 2 ] 88 adopt the GAN-based models to generate a virtual dataset . And the substitute model and the GAN89 model are trained alternatively on this virtual dataset by querying the victim model iteratively . The90 data selection methods prepare an attack dataset as the data pool , and then sample the most informative91 data via machine learning algorithms , e.g. , reinforcement learning [ 20 ] or active learning strategy [ 21 ] ,92 uncertainty-based strategy [ 17 ] , k-Center strategy [ 25 ] , and DFAL strategy [ 5 ] . Considering that93 querying the victim model will be costly , the attacker usually sets a budget on the number of the94 queries , so the size of the transfer dataset should be limited as well . Previous methods assume the95 victim model returns a complete probability prediction f ( x ) , which is less practical.96 In this paper , we focus on a more practical scenario that is about hard-label φ ( f ( x ) ) setting , where φ97 is the truncation function used to truncate the information contained in the victim ’ s output and return98 the corresponding one-hot vector:99 φ ( f ( x ) ) i : = { 1 if i = arg maxn f ( x ) n ; 0 otherwise . ( 1 ) With the transfer dataset , the substitute model is optimized by minimizing a loss function L ( e.g.,100 cross-entropy loss function ) :101 { Ex∼DT [ L ( f ( x ) , f̂ ( x ) ) ] , for soft labels ; Ex∼DT [ L ( φ ( f ( x ) ) , f̂ ( x ) ) ] , for hard labels . ( 2 ) Knowledge distillation ( KD ) has been widely studied in machine learning [ 10 , 1 , 6 ] , which transfers102 the knowledge from a teacher model to a student model . Model stealing attacks can be regarded as a103 black-box KD problem where the victim model is the teacher with only outputs accessible and the104 substitute model is the student . The main reason for the success of KD is the valuable information105 that defines a rich similarity structure over the data in the probability prediction [ 10 ] . However,106 for the hard-label setting discussed in this paper , this valuable information is lost . Inspired by KD,107 our method tries to dig out the hidden information in the data and models , and then transfers more108 knowledge to the substitute model.109 The erasing-based method , e.g. , random erasing ( RE ) [ 32 , 3 ] , is currently one of the widely used110 data augmentation methods , which generates training images with various levels of occlusion , thereby111 reducing the risk of over-fitting and improving the robustness of the model . Our work is inspired112 by RE and designs a prior-driven erasing operation , which erases the area corresponding to the hard113 label to re-mine missing information.114 3 Method115 The overview of our proposed black-box dissector is shown in Fig . 2 . In addition to the conventional116 process ( i.e. , the transfer dataset DT constructing in step 1 and the substitute model training in the117 right ) , we introduce two key modules : a CAM-driven erasing strategy ( step 2.1 ) and a RE-based118 self-KD module ( step 2.2 ) .119 3.1 A CAM-driven erasing strategy120 Since the lack of class similarity infor-121 mation degrades the performance of122 previous methods under the hard-label123 setting , we try to re-dig out such hid-124 den information . Taking an example125 from the ILSVRC-2012 dataset for il-126 lustration as in Fig . 3 . Querying the127 CUBS200 trained victim model with128 this image , we get two classes with129 the highest confidence score : “ Anna130 hummingbird '' ( 0.1364 ) and “ Com-131 mon yellowthroat '' ( 0.1165 ) , and show132 their corresponding attention map in133 the first column of Fig . 3 . It is easy to134 conclude that two different attention135 regions response for different classes136 according to the attention map . When137 training the substitute model with the138 hard label “ Anna hummingbird '' and139 without the class similarity informa-140 tion , the model can not learn from the area related to the “ Common yellowthroat '' class , which means141 this area is wasted . To re-dig out the information about the “ Common yellowthroat '' class , we need to142 erase the impact of the “ Anna hummingbird '' class.143 To this end , a natural idea is to erase the response area corresponding to the hard label . Since the144 victim model is a black-box model , we use the substitute model to approximately calculate the145 attention map instead . If the attention map calculated by the substitute model is inaccurate and the146 victim model ’ s prediction on the erased image does not change , we can also align the attention map of147 two models by letting the substitute model learn the original image and the erased one simultaneously.148 The attention map is also a kind of supervision signal pushing two models to be similar [ 30 ] . To149 get the attention map , we utilize the Grad-CAM [ 24 ] in this paper . With the input image x ∈ [ 0 , 1 ] d150 and the trained DNN F : [ 0 , 1 ] d 7→ RN , we let αck denote the weight of class c corresponding to the151 k-th feature map , and calculate it as αck = 1 Z ∑ i ∑ j ∂F ( x ) c ∂Akij , where Z is the number of pixels in the152 feature map , F ( x ) c is the score of class c and Akij is the value of pixel at ( i , j ) in the k-th feature153 map . After obtaining the weights corresponding to all feature maps , the final attention map can be154 obtained as ScGrad−CAM = ReLU ( ∑ k α c kA k ) via weighted summation.155 To erase the corresponding area , inspired by [ 32 ] , we define a prior-driven erasing operation as156 ψ ( I , P ) , shown in Alg . 1 , which randomly erases a rectangle region in the image I with random157 values while the central position of the rectangle region is randomly selected following the prior158 probability P . The prior probability P is of the same size as the input image and is used to determine159 the probability of different pixels being erased . Here , we use the attention map from Grad-CAM as160 the prior . Let x ∈ [ 0 , 1 ] d denote the input image from the transfer set and Sarg max f̂ ( x ) Grad−CAM ( x , f̂ ) denote161 the attention map of the substitute model f̂ . This CAM-driven erasing operation can be represented:162 ψ ( x , S arg max f̂ ( x ) Grad−CAM ( x , f̂ ) ) . ( 3 ) We abbreviate it as ψ ( x , S ( x , f̂ ) ) . To alleviate the impact of inaccurate CAM caused by the difference163 between the substitute model and the victim one , for each image , we perform this operation N times164 ( ψi means the i-th erasing ) and select the one with the largest difference from the original label.165 Algorithm 1 : Prior-driven Erasing Operation ψ ( I , P ) Input : Input image I , prior probability P , image size W and H , area of image S , erasing area ratio range sl and sh , erasing aspect ratio range r1 and r2 . Output : Erased image I ′ . 1 Se ← Rand ( sl , sh ) × S , re ← Rand ( r1 , r2 ) 1 2 He ← √ Se × re/2 , We ← √ Se re /2 3 xe , ye sampled randomly according to P 4 Ie ← ( xe −We , ye −He , xe +We , ye +He ) 5 I ( Ie ) ← Rand ( 0 , 255 ) 6 I ′ ← I Such a data augment operation helps the erasing process to be more robust . We use the cross-entropy166 to calculate the difference between the new label and the original label , and we want to select the167 sample with the biggest difference . Formally , we define Π ( x ) as the function to select the most168 different variation of image x:169 Π ( x ) : = ψk ( x , S ( x , f̂ ) ) , where k : = arg max i∈ [ N ] − ∑ j φ ( f ( x ) ) j · log ( f̂ ( ψi ( x , S ( x , f̂ ) ) ) j ) = arg max i∈ [ N ] − log ( f̂ ( ψi ( x , S ( x , f̂ ) ) ) arg maxφ ( f ( x ) ) ) = arg min i∈ [ N ] f̂ ( ψi ( x , S ( x , f̂ ) ) ) arg maxφ ( f ( x ) ) . ( 4 ) Due to the limitation of the number of queries , we can not query the victim model for each erased170 image to obtain a new label . We continuously choose the erased image with the highest substitute ’ s171 confidence until reaching the budget . To measure the confidence of the model , we adopt the Maximum172 Softmax Probability ( MSP ) for its simplicity:173 arg max x∼DT MSP ( f̂ ( Π ( x ) ) ) = arg max x∼DT f̂ ( Π ( x ) ) arg max f̂ ( Π ( x ) ) , ( 5 ) where DT is the transfer set . The erased images selected in this way are most likely to change the174 prediction class . Then , we query the victim model to get these erased images ’ labels and construct175 an erased sample set DE . Note that when the victim model ’ s predictions on the erased images176 change , it means our erasing method does dig out other related class information in the sample . With177 the unchanged predictions , it points out the attentions of the substitute model and the victim are178 inconsistent . Though wrong attention areas erased , training with these samples benefits aligning the179 attentions of two models . As [ 30 ] stated , the attention alignment can help more powerful KD.180 3.2 A random-erasing-based self-KD module181 We also find that in training with limited hard-label OOD samples , the substitute model is likely182 to overfit the training set , which damages its generalization ability [ 15 , 31 ] . Therefore , based on183 the above erasing operation , we further design a simple RE-based self-KD method to improve the184 generalization ability of the substitute model.185 Formally , let x ∈ [ 0 , 1 ] d denote the unlabeled input image . We perform the erasing operation with a186 uniform prior U on it N times , and then average the substitute ’ s outputs on these erased images as187 the pseudo-label of the original image:188 yp ( x , f̂ ) = 1 N N∑ i=1 f̂ ( ψi ( x , U ) ) . ( 6 ) 1Rand ( a , b ) returns an evenly distributed random real number in the range of a to b. Algorithm 2 : Black-box Dissector Input : Unlabeled pool DU , victim model f , maximum number of queries Q . Output : Substitute model f̂ . 1 Initialize q ← 0 , DT ← ∅ , DE ← ∅ 2 while q < Q do 3 // Step 1 4 Select samples from DU according to budget and query f to updata DT 5 q = q + budget 6 L = ∑ x∈DT L ′ ( φ ( f ( x ) ) , f̂ ( x ) ) 7 f̂ ← update ( f̂ , L ) 8 // A CAM-driven erasing strategy ( step 2.1 ) 9 Erase samples in DT according to Eq . 4 10 Choose samples from erased samples according to Eq . 5 and budget 11 Query f to get labels and updata DE 12 L = ∑ x∈DT∪DE L ′ ( φ ( f ( x ) ) , f̂ ( x ) ) 13 f̂ ← update ( f̂ , L ) 14 q = q + budget 15 // A random-erasing-based self-KD ( step 2.2 ) 16 Select samples from DU 17 Get pseudo-labels according to Eq . 6 and construct a pseudo-label set DP 18 L = ∑ x∈DT∪DE L ′ ( φ ( f ( x ) ) , f̂ ( x ) ) +∑x∈DP L′ ( yp ( x , f̂ ) , f̂ ( x ) ) 19 f̂ ← update ( f̂ , L ) 20 end This is a type of consistency regularization , which enforces the model to have the same predictions189 for the perturbed images and enhances the generalization ability . With Eq.6 , we construct a new soft190 pseudo label set DP = { ( x , yp ( x , f̂ ) ) , . . . } .191 With the transfer set DT , the erased sample set DE , and the pseudo-label set DP , we train a new192 substitute model using the ensemble of the victim model and the previous substitute model as the193 teacher . Our final objective function is:194 minL = min [ ∑ x∈DT∪DE L′ ( φ ( f ( x ) ) , f̂ ( x ) ) + ∑ x∈DP L′ ( yp ( x , f̂ ) , f̂ ( x ) ) ] . ( 7 ) where L′ can be commonly used loss functions , e.g. , cross-entropy loss function.195 To sum up , we built our method on the conventional process of the model stealing attack ( step196 1 ) , and proposed a CAM-driven erasing strategy ( step 2.1 ) and a RE-based self-KD module ( step197 2.2 ) unified by a novel erasing method . The former strategy digs out missing information between198 classes and aligns the attention while the latter module helps to mitigate overfitting and enhance the199 generalization . We name the whole framework as black-box dissector and present the algorithm200 detail of it in Alg . 2.201 4 Experiment202 4.1 Experiment settings203 Victim model . The victim models we used ( ResNet-34 [ 9 ] ) are trained on four datasets , namely,204 CIFAR10 [ 16 ] , SVHN [ 19 ] , Caltech256 [ 7 ] , and CUBS200 [ 28 ] , and their test accuracy are 91.56 % ,205 96.45 % , 78.40 % , and 77.10 % , respectively . All models are trained using the SGD optimizer with206 momentum ( of 0.5 ) for 200 epochs with a base learning rate of 0.1 decayed by a factor of 0.1 every207 30 epochs . Following [ 20 , 21 , 34 ] , we use the same architecture for the substitute model and will208 analyze the impact of different architectures in the supplementary.209 Attack dataset . We use 1.2M images without labels from the ILSVRC-2012 challenge [ 23 ] as the210 attack dataset . In a real attack scenario , the attacker may use pictures collected from the Internet , and211 the ILSVRC-2012 dataset can simulate this scenario well . Note that we resize all images in the attack212 dataset to fit the size of the target datasets , which is similar to the existing setting [ 20 , 21 , 34 ] .213 Training process . We use the SGD optimizer with momentum ( of 0.9 ) for 200 epochs and a base214 learning rate of 0.02 × batchsize128 decayed by a factor of 0.1 every 60 epochs . The weight decay is215 set to 5 × 10−4 for small datasets ( CIFAR10 [ 16 ] and SVHN [ 19 ] ) and 0 for others . We set up a216 query sequence { 0.1K , 0.2K , 0.5K , 0.8K , 1K , 2K , 5K , 10K , 20K , 30K } as the iterative maximum217 query budget , and stop the sampling stage whenever reaching the budget at each iteration.218 Baselines and evaluation metric . We mainly compare our method with KnockoffNets [ 20 ] and219 ActiveThief [ 21 ] . Follow Jagielski et al . [ 12 ] , we mainly report the test accuracy ( Acc ) as the220 evaluation metric . We also report the Agreement metric proposed by Pal et al . [ 21 ] which counts how221 often the prediction of the substitute model is the same as the victim ’ s as a supplement.222 4.2 Experiment results223 We first report the performance of our method compared with previous methods . After that , we224 conduct ablation experiments to analyze the contribution of each module . Finally , we also analyze the225 performance of our method when encountering defense methods and real-world online APIs . More226 experiments ( e.g. , adversarial attack and overfitting analysis ) can be found in our supplementary.227 Effectiveness of our method . As in Tab . 1 , the test accuracy and agreement of our method are all228 better than the previous methods . We also plot the curves of the test accuracy versus the number of229 queries in Fig . 4 . The performance of our method consistently outperforms other methods throughout230 the process . Since our method does not conflict with the previous sample selection strategy , they231 can be used simultaneously to further improve the performance of these attacks . Here , we take232 the k-Center algorithm as an example . Note that , with or without the sample selection strategy,233 our method beats the previous methods by a large margin . Particularly , the test accuracies of our234 method are 4.85 % , 1.72 % , 3.88 % , and 8.27 % higher than the previous best method , respectively.235 And the agreement metric shares similar results . It is also interesting that it is less necessary to use236 the k-Center algorithm on datasets with a small number of classes ( i.e. , CIFAR10 and SVHN ) . While237 for the datasets with a large number of classes , the k-Center algorithm can make the selected samples238 better cover each class and improve the effectiveness of the method.239 Ability to evade the SOTA defense method . The SOTA perturbation-based defense method , adap-240 tive misinformation [ 13 ] , introduces an Out-Of-Distribution ( OOD ) detection module based on the241 maximum predicted value and punishes the OOD samples with a perturbed model f ′ ( · ; θ′ ) . The242 model f ′ ( · ; θ′ ) is trained with arg minθ′ E ( x , y ) [ − log ( 1− f ′ ( x ; θ′ ) y ) ] to minimize the probability of243 the correct class . Finally , the output will be:244 y′ = ( 1− α ) f ( x ; θ ) + ( α ) f ′ ( x ; θ′ ) , ( 8 ) where α = 1/ ( 1 + eν ( max f ( x ; θ ) −τ ) ) with a hyper-parameter ν is the coefficient to control how245 much correct results will be returned , and τ is the threshold used for OOD detection . The model246 returns incorrect predictions for the OOD samples without having much impact on the in-distribution247 samples.248 We choose four values of the threshold τ to compare the effects of our method with the previous249 methods . The threshold value of 0 means no defence . The result is shown in Tab . 2 . Compared250 with other methods , adaptive misinformation is almost invalid to our method . Furthermore , we find251 that if we remove the self-KD in our method , the performance is greatly reduced . We conclude that252 this is because adaptive misinformation adds noise labels to the substitute model ’ s training dataset,253 and self-KD can alleviate the overfitting of the substitute model to the training dataset , making this254 defence method not effective enough.255 Ablation study . To evaluate the contribution of different mod-256 ules in our method , we conduct the ablation study on CUBS200257 dataset and plot the results in Fig . 5 . If the CAM-driven erasing258 strategy is removed , the performance of our method will be259 greatly reduced , showing that it has an indispensable position260 in our method . We also give some visual examples in Fig . 7 to261 demonstrate that this strategy can help align the attention of two262 models . As depicted in the Fig . 7 , at the beginning time , the263 substitute model learns the wrong attention map . Along with264 the iterative training stages , the attention area of the substitute265 model tends to fit the victim model ’ s , which conforms to our266 intention . We further remove the self-KD module to evaluate267 its performance . It can be found from Fig . 1 and Fig . 5 that268 the self-KD can improve the generalization of our method and269 further improve the performance.270 Stealing functionality of a real-world API . We validate our271 method is applicable to real-world APIs . The AWS Marketplace272 is an online store that provides a variety of trained ML models273 for users . It can only be used in the form of a black-box setting.274 We choose a popular model ( waste classifier 2 ) as the victim275 model . We use ILSVRC-2012 dataset as the attack dataset and276 choose another small public waste classifier dataset 3 , contain-277 ing 2 , 527 images as the test dataset . As in Fig . 6 , the substitute278 model obtained by our method achieves 12.63 % and 7.32 % 279 improvements in test accuracy compared with two previous280 methods , which show our method has stronger practicality in281 the real world.282 2https : //amzn.to/3nFvA54 3https : //github.com/garythung/trashnet Transferability of adversarial samples . Though with the dominant performance on a wide range of283 tasks , deep neural networks are shown to be vulnerable to imperceptible perturbations , i.e. , adversarial284 examples [ 27 ] . Since the model stealing attack can obtain a functionally similar substitute model,285 some previous works ( e.g. , JBDA [ 22 ] , DaST [ 34 ] and ActiveThief [ 21 ] ) used this substitute model286 to generate adversarial samples and then performed the transferable adversarial attack on the victim287 model . We argue that a more similar substitute model leads to a more successful adversarial attacks.288 We test the transferability of adversarial samples on the test set of the CIFAR10 dataset . Keeping the289 architecture of the victim model as the ResNet-34 , we evaluate the attack success rate of adversarial290 samples generated from different substitute models ( i.e. , ResNet-34 , ResNet-18 , ResNet-50 [ 9 ] , VGG-291 16 [ 26 ] , DenseNet [ 11 ] ) . All adversarial samples are generated using Projected Gradient Descent292 ( PGD ) attack [ 18 ] with maximum L∞-norm of perturbations as 8/255 . As shown in Tab . 3 , the293 adversarial samples generated by our substitute models have stronger transferability in all substitute ’ s294 architectures . This again proves that our method is more practical in real-world scenarios.295 5 Conclusion296 We investigated the problem of model stealing attacks under the hard-label setting and pointed out297 why previous methods are not effective enough . We presented a new method , termed black-box298 dissector , which contains a CAM-driven erasing strategy and a RE-based self-KD module . We299 showed its superiority on four widely-used datasets and verified the effectiveness of our method300 with defense methods , real-world APIs , and the downstream adversarial attack . Though focusing301 on image data in this paper , our method is general for other tasks as long as the CAM and similar302 erasing method work , e.g. , synonym saliency words replacement for NLP tasks [ 4 ] . We believe our303 method can be easily extended to other fields and inspire future researchers . Model stealing attack304 poses a threat to the deployed machine learning models . We hope this work will draw attention to305 the protection of deployed models and furthermore shed more light on the attack mechanisms and306 prevention methods.307 References308 [ 1 ] Rohan Anil , Gabriel Pereyra , Alexandre Passos , Robert Ormandi , George E Dahl , and Geof-309 frey E Hinton . Large scale distributed neural network training through online distillation . arXiv310 preprint arXiv:1804.03235 , 2018.311 [ 2 ] Antonio Barbalau , Adrian Cosma , Radu Tudor Ionescu , and Marius Popescu . Black-box ripper:312 Copying black-box models using generative evolutionary algorithms . In NeurIPS , 2020.313 [ 3 ] Terrance DeVries and Graham W Taylor . Improved regularization of convolutional neural314 networks with cutout . arXiv preprint arXiv:1708.04552 , 2017.315 [ 4 ] Xinshuai Dong , Anh Tuan Luu , Rongrong Ji , and Hong Liu . Towards robustness against natural316 language word substitutions . In ICLR , 2021.317 [ 5 ] Melanie Ducoffe and Frederic Precioso . Adversarial active learning for deep networks : a margin318 based approach . In ICML , 2018.319 [ 6 ] Tommaso Furlanello , Zachary C Lipton , Michael Tschannen , Laurent Itti , and Anima Anandku-320 mar . Born again neural networks . In ICML , 2018.321 [ 7 ] Gregory Griffin , Alex Holub , and Pietro Perona . Caltech-256 object category dataset . 2007.322 [ 8 ] Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q Weinberger . On calibration of modern neural323 networks . In ICML , 2017.324 [ 9 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image325 recognition . In CVPR , 2016.326 [ 10 ] Geoffrey Hinton , Oriol Vinyals , and Jeff Dean . Distilling the knowledge in a neural network . In327 NIPS Deep Learning Workshop , 2015.328 [ 11 ] Gao Huang , Zhuang Liu , Laurens Van Der Maaten , and Kilian Q Weinberger . Densely connected329 convolutional networks . In CVPR , 2017.330 [ 12 ] Matthew Jagielski , Nicholas Carlini , David Berthelot , Alex Kurakin , and Nicolas Papernot.331 High accuracy and high fidelity extraction of neural networks . In 29th Usenix Security , 2020.332 [ 13 ] Sanjay Kariyappa and Moinuddin K Qureshi . Defending against model stealing attacks with333 adaptive misinformation . In CVPR , 2020.334 [ 14 ] Sanjay Kariyappa , Atul Prakash , and Moinuddin Qureshi . Maze : Data-free model stealing335 attack using zeroth-order gradient estimation . arXiv preprint arXiv:2005.03161 , 2020.336 [ 15 ] Kyungyul Kim , ByeongMoon Ji , Doyoung Yoon , and Sangheum Hwang . Self-knowledge337 distillation : A simple way for better generalization . arXiv preprint arXiv:2006.12000 , 2020.338 [ 16 ] Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images.339 2009.340 [ 17 ] David D Lewis and William A Gale . A sequential algorithm for training text classifiers . In341 SIGIR , 1994.342 [ 18 ] Aleksander Madry , Aleksandar Makelov , Ludwig Schmidt , Dimitris Tsipras , and Adrian Vladu.343 Towards deep learning models resistant to adversarial attacks . In ICLR , 2018.344 [ 19 ] Yuval Netzer , Tao Wang , Adam Coates , Alessandro Bissacco , Bo Wu , and Andrew Y Ng.345 Reading digits in natural images with unsupervised feature learning . In NIPS Workshop on346 Deep Learning and Unsupervised Feature Learning , 2011.347 [ 20 ] Tribhuvanesh Orekondy , Bernt Schiele , and Mario Fritz . Knockoff nets : Stealing functionality348 of black-box models . In CVPR , 2019.349 [ 21 ] Soham Pal , Yash Gupta , Aditya Shukla , Aditya Kanade , Shirish Shevade , and Vinod Ganapathy.350 Activethief : Model extraction using active learning and unannotated public data . In AAAI , 2020.351 [ 22 ] Nicolas Papernot , Patrick McDaniel , Ian Goodfellow , Somesh Jha , Z Berkay Celik , and Anan-352 thram Swami . Practical black-box attacks against machine learning . In ACM AsiACCS , 2017.353 [ 23 ] Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng354 Huang , Andrej Karpathy , Aditya Khosla , Michael Bernstein , et al . Imagenet large scale visual355 recognition challenge . IJCV , 2015.356 [ 24 ] Ramprasaath R Selvaraju , Michael Cogswell , Abhishek Das , Ramakrishna Vedantam , Devi357 Parikh , and Dhruv Batra . Grad-cam : Visual explanations from deep networks via gradient-based358 localization . In ICCV , 2017.359 [ 25 ] Ozan Sener and Silvio Savarese . Active learning for convolutional neural networks : A core-set360 approach . In ICLR , 2018.361 [ 26 ] Karen Simonyan and Andrew Zisserman . Very deep convolutional networks for large-scale362 image recognition . In ICLR , 2015.363 [ 27 ] Christian Szegedy , Wojciech Zaremba , Ilya Sutskever , Joan Bruna , Dumitru Erhan , Ian Good-364 fellow , and Rob Fergus . Intriguing properties of neural networks . In ICLR , 2014.365 [ 28 ] Catherine Wah , Steve Branson , Peter Welinder , Pietro Perona , and Serge Belongie . The366 caltech-ucsd birds-200-2011 dataset . 2011.367 [ 29 ] Jiancheng Yang , Yangzhou Jiang , Xiaoyang Huang , Bingbing Ni , and Chenglong Zhao . Learn-368 ing black-box attackers with transferable priors and query feedback . In NeurIPS , 2020.369 [ 30 ] Sergey Zagoruyko and Nikos Komodakis . Paying more attention to attention : Improving the370 performance of convolutional neural networks via attention transfer . In ICLR , 2017.371 [ 31 ] Chiyuan Zhang , Samy Bengio , Moritz Hardt , Benjamin Recht , and Oriol Vinyals . Understanding372 deep learning requires rethinking generalization . In ICLR , 2017.373 [ 32 ] Zhun Zhong , Liang Zheng , Guoliang Kang , Shaozi Li , and Yi Yang . Random erasing data374 augmentation . In AAAI , 2020.375 [ 33 ] Bolei Zhou , Aditya Khosla , Agata Lapedriza , Aude Oliva , and Antonio Torralba . Learning deep376 features for discriminative localization . In CVPR , 2016.377 [ 34 ] Mingyi Zhou , Jing Wu , Yipeng Liu , Shuaicheng Liu , and Ce Zhu . Dast : Data-free substitute378 training for adversarial attacks . In CVPR , 2020.379 Checklist380 1 . For all authors ... 381 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s382 contributions and scope ? [ Yes ] 383 ( b ) Did you describe the limitations of your work ? [ Yes ] See section 3384 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] See section385 5386 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to387 them ? [ Yes ] 388 2 . If you are including theoretical results ... 389 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 390 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 391 3 . If you ran experiments ... 392 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-393 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] See the394 supplemental material395 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they396 were chosen ) ? [ Yes ] See section 4.1397 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-398 ments multiple times ) ? [ Yes ] See Tab . 1399 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type400 of GPUs , internal cluster , or cloud provider ) ? [ No ] 401 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 402 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See section 4.1403 ( b ) Did you mention the license of the assets ? [ No ] 404 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ No ] 405 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re406 using/curating ? [ No ] 407 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable408 information or offensive content ? [ No ] 409 5 . If you used crowdsourcing or conducted research with human subjects ... 410 ( a ) Did you include the full text of instructions given to participants and screenshots , if411 applicable ? [ N/A ] 412 ( b ) Did you describe any potential participant risks , with links to Institutional Review413 Board ( IRB ) approvals , if applicable ? [ N/A ] 414 ( c ) Did you include the estimated hourly wage paid to participants and the total amount415 spent on participant compensation ? [ N/A ] 416

Summary: This paper proposes a new method for model stealing/knowledge distillation with hard label information only, based on a two-part modification of the standard model stealing pipeline. The standard pipeline feeds images to the target model, collects the corresponding labels, and trains a substitute model on the image-label pairs. First, the proposed method erases high-information patches from the images fed to the target model in order to collect additional (nearby) labels, providing extra information about the target model. Second, after training a first substitute model on the target images and their erased counterparts, the proposed method trains a second substitute model on both the original target data, the erased images and their labels, and "smoothed" predictions from the first substitute model in order to improve generalization.