Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack

Previous studies have verified that the functionality of black-box models can be1 stolen with full probability outputs . However , under the more practical hard-label2 setting , we observe that existing methods suffer from catastrophic performance3 degradation . We argue this is due to the lack of rich information in the probability4 prediction and the overfitting caused by hard labels . To this end , we propose a5 novel hard-label model stealing method termed black-box dissector , which consists6 of two erasing-based modules . One is a CAM-driven erasing strategy that is7 designed to increase the information capacity hidden in hard labels from the victim8 model . The other is a random-erasing-based self-knowledge distillation module9 that utilizes soft labels from the substitute model to mitigate overfitting . Extensive10 experiments on four widely-used datasets consistently demonstrate that our method11 outperforms state-of-the-art methods , with an improvement of at most 8.27 % . We12 also validate the effectiveness and practical potential of our method on real-world13 APIs and defense methods . Furthermore , our method promotes other downstream14 tasks , i.e. , transfer adversarial attacks.15 1 Introduction16 Machine learning models deployed on the cloud can serve users through the application program17 interfaces ( APIs ) to improve productivity . Since developing these cloud models is a product of18 intensive labor and monetary effort , these models are valuable intellectual property and AI companies19 try to keep them private . However , the exposure of the model ’ s predictions represents a significant20 risk as an adversary can leverage this information to steal the model ’ s functionality , a.k.a . model21 stealing attack [ 22 , 20 , 21 ] . With such an attack , adversaries are able to not only use the stolen model22 to make a profit , but also mount further adversarial attacks [ 34 , 29 ] . Besides , the model stealing23 attacks is a kind of black-box knowledge distillation which is a hot research topic . Studying various24 mechanisms of model stealing attack is of great interest both to AI companies and researchers.25 Previous methods [ 20 , 34 , 21 ] mainly assume the complete probability predictions of the victim26 model available , while the real-world APIs usually only return partial probability values ( top-k27 predictions ) or even the top-1 prediction ( i.e. , hard label ) . In this paper , we focus on the more28 challenging and realistic scenario , i.e. , the victim model only outputs the hard labels . However , under29 this setting , existing methods suffer from a significant performance degradation , even by 30.50 % ( as30 shown in the Fig . 1 ( a ) and the appendix Tab . I ) .31 To investigate the reason for the degradation , we evaluate the performance of attack methods with32 different numbers of prediction probability categories available and hard labels as in Fig . 1 ( b ) . With33 the observation that the performance degrades when the top-k information missing , we conclude34 that the top-k predictions are informative as it indicates the similarity of different categories or35 multiple objects in the picture , and previous attack methods suffer from such information obscured36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . by the top-1 prediction under the hard-label setting . It motivates us to re-mine this information by37 eliminating the top-1 prediction . Particularly , we design a novel CAM-based erasing method , which38 erases the important area on the pictures based on the substitute model ’ s top-1 class activation maps39 ( CAM ) [ 24 , 33 ] and queries the victim model for a new prediction . Note that we can dig out other40 class information in this sample if the new prediction changes . Otherwise , it proves that the substitute41 model pays attention to the wrong area . Then we can align the attention of the substitute and the42 victim model by learning clean samples and the corresponding erased samples simultaneously.43 ( a ) Hard label vs soft label ( c ) W/O self-KD ( d ) W/ self-KD ( b ) Performance with top-k available from the victim model and pseudo labels from the previous substitute model to train a new substitute69 model . Therefore , we can also consider the ensemble of the two models as the teacher in knowledge70 distillation . As in Fig . 1 ( d ) , such a module helps generalization and better performance.71 In summary , we propose a novel model stealing framework termed black-box dissector , which72 includes a CAM-driven erasing strategy and a RE-based self-KD module . Our method is orthogonal73 to previous approaches [ 20 , 21 ] and can be integrated with them . The experiments on four widely-74 used datasets demonstrate our method achieves 43.04 − 90.57 % test accuracy ( 47.60 − 91.37 % 75 agreement ) to the victim model , which is at most 8.27 % higher than the state of the art method.76 We also proved that our method can defeat popular defense methods and is effective for real-world77 APIs like services provided by Amazon Web Services ( AWS ) . Furthermore , our method promotes78 downstream tasks , i.e. , transfer adversarial attack , with 4.91 % − 16.20 % improvement.79 2 Background and Notions80 Model stealing attack is aim to find a substitute model f̂ : [ 0 , 1 ] d 7→ RN that performs as similarly81 as possible to the black-box victim model f : [ 0 , 1 ] d 7→ RN ( with only outputs accessed ) . Papernot82 et al . [ 22 ] first observed that online models could be stolen through multiple queries . After that , due83 to the practical threat to real-world APIs , several studies paid attention to this problem and proposed84 many attack algorithms.85 These algorithms consist of two stages : 1 ) constructing a transfer dataset DT ( step 1 in Fig . 2 ) and86 2 ) training a substitute model . The transfer dataset is constructed based on data synthesis or data87 selection and then feed into the victim model for labels . Methods based on data synthesis [ 34 , 14 , 2 ] 88 adopt the GAN-based models to generate a virtual dataset . And the substitute model and the GAN89 model are trained alternatively on this virtual dataset by querying the victim model iteratively . The90 data selection methods prepare an attack dataset as the data pool , and then sample the most informative91 data via machine learning algorithms , e.g. , reinforcement learning [ 20 ] or active learning strategy [ 21 ] ,92 uncertainty-based strategy [ 17 ] , k-Center strategy [ 25 ] , and DFAL strategy [ 5 ] . Considering that93 querying the victim model will be costly , the attacker usually sets a budget on the number of the94 queries , so the size of the transfer dataset should be limited as well . Previous methods assume the95 victim model returns a complete probability prediction f ( x ) , which is less practical.96 In this paper , we focus on a more practical scenario that is about hard-label φ ( f ( x ) ) setting , where φ97 is the truncation function used to truncate the information contained in the victim ’ s output and return98 the corresponding one-hot vector:99 φ ( f ( x ) ) i : = { 1 if i = arg maxn f ( x ) n ; 0 otherwise . ( 1 ) With the transfer dataset , the substitute model is optimized by minimizing a loss function L ( e.g.,100 cross-entropy loss function ) :101 { Ex∼DT [ L ( f ( x ) , f̂ ( x ) ) ] , for soft labels ; Ex∼DT [ L ( φ ( f ( x ) ) , f̂ ( x ) ) ] , for hard labels . ( 2 ) Knowledge distillation ( KD ) has been widely studied in machine learning [ 10 , 1 , 6 ] , which transfers102 the knowledge from a teacher model to a student model . Model stealing attacks can be regarded as a103 black-box KD problem where the victim model is the teacher with only outputs accessible and the104 substitute model is the student . The main reason for the success of KD is the valuable information105 that defines a rich similarity structure over the data in the probability prediction [ 10 ] . However,106 for the hard-label setting discussed in this paper , this valuable information is lost . Inspired by KD,107 our method tries to dig out the hidden information in the data and models , and then transfers more108 knowledge to the substitute model.109 The erasing-based method , e.g. , random erasing ( RE ) [ 32 , 3 ] , is currently one of the widely used110 data augmentation methods , which generates training images with various levels of occlusion , thereby111 reducing the risk of over-fitting and improving the robustness of the model . Our work is inspired112 by RE and designs a prior-driven erasing operation , which erases the area corresponding to the hard113 label to re-mine missing information.114 3 Method115 The overview of our proposed black-box dissector is shown in Fig . 2 . In addition to the conventional116 process ( i.e. , the transfer dataset DT constructing in step 1 and the substitute model training in the117 right ) , we introduce two key modules : a CAM-driven erasing strategy ( step 2.1 ) and a RE-based118 self-KD module ( step 2.2 ) .119 3.1 A CAM-driven erasing strategy120 Since the lack of class similarity infor-121 mation degrades the performance of122 previous methods under the hard-label123 setting , we try to re-dig out such hid-124 den information . Taking an example125 from the ILSVRC-2012 dataset for il-126 lustration as in Fig . 3 . Querying the127 CUBS200 trained victim model with128 this image , we get two classes with129 the highest confidence score : “ Anna130 hummingbird '' ( 0.1364 ) and “ Com-131 mon yellowthroat '' ( 0.1165 ) , and show132 their corresponding attention map in133 the first column of Fig . 3 . It is easy to134 conclude that two different attention135 regions response for different classes136 according to the attention map . When137 training the substitute model with the138 hard label “ Anna hummingbird '' and139 without the class similarity informa-140 tion , the model can not learn from the area related to the “ Common yellowthroat '' class , which means141 this area is wasted . To re-dig out the information about the “ Common yellowthroat '' class , we need to142 erase the impact of the “ Anna hummingbird '' class.143 To this end , a natural idea is to erase the response area corresponding to the hard label . Since the144 victim model is a black-box model , we use the substitute model to approximately calculate the145 attention map instead . If the attention map calculated by the substitute model is inaccurate and the146 victim model ’ s prediction on the erased image does not change , we can also align the attention map of147 two models by letting the substitute model learn the original image and the erased one simultaneously.148 The attention map is also a kind of supervision signal pushing two models to be similar [ 30 ] . To149 get the attention map , we utilize the Grad-CAM [ 24 ] in this paper . With the input image x ∈ [ 0 , 1 ] d150 and the trained DNN F : [ 0 , 1 ] d 7→ RN , we let αck denote the weight of class c corresponding to the151 k-th feature map , and calculate it as αck = 1 Z ∑ i ∑ j ∂F ( x ) c ∂Akij , where Z is the number of pixels in the152 feature map , F ( x ) c is the score of class c and Akij is the value of pixel at ( i , j ) in the k-th feature153 map . After obtaining the weights corresponding to all feature maps , the final attention map can be154 obtained as ScGrad−CAM = ReLU ( ∑ k α c kA k ) via weighted summation.155 To erase the corresponding area , inspired by [ 32 ] , we define a prior-driven erasing operation as156 ψ ( I , P ) , shown in Alg . 1 , which randomly erases a rectangle region in the image I with random157 values while the central position of the rectangle region is randomly selected following the prior158 probability P . The prior probability P is of the same size as the input image and is used to determine159 the probability of different pixels being erased . Here , we use the attention map from Grad-CAM as160 the prior . Let x ∈ [ 0 , 1 ] d denote the input image from the transfer set and Sarg max f̂ ( x ) Grad−CAM ( x , f̂ ) denote161 the attention map of the substitute model f̂ . This CAM-driven erasing operation can be represented:162 ψ ( x , S arg max f̂ ( x ) Grad−CAM ( x , f̂ ) ) . ( 3 ) We abbreviate it as ψ ( x , S ( x , f̂ ) ) . To alleviate the impact of inaccurate CAM caused by the difference163 between the substitute model and the victim one , for each image , we perform this operation N times164 ( ψi means the i-th erasing ) and select the one with the largest difference from the original label.165 Algorithm 1 : Prior-driven Erasing Operation ψ ( I , P ) Input : Input image I , prior probability P , image size W and H , area of image S , erasing area ratio range sl and sh , erasing aspect ratio range r1 and r2 . Output : Erased image I ′ . 1 Se ← Rand ( sl , sh ) × S , re ← Rand ( r1 , r2 ) 1 2 He ← √ Se × re/2 , We ← √ Se re /2 3 xe , ye sampled randomly according to P 4 Ie ← ( xe −We , ye −He , xe +We , ye +He ) 5 I ( Ie ) ← Rand ( 0 , 255 ) 6 I ′ ← I Such a data augment operation helps the erasing process to be more robust . We use the cross-entropy166 to calculate the difference between the new label and the original label , and we want to select the167 sample with the biggest difference . Formally , we define Π ( x ) as the function to select the most168 different variation of image x:169 Π ( x ) : = ψk ( x , S ( x , f̂ ) ) , where k : = arg max i∈ [ N ] − ∑ j φ ( f ( x ) ) j · log ( f̂ ( ψi ( x , S ( x , f̂ ) ) ) j ) = arg max i∈ [ N ] − log ( f̂ ( ψi ( x , S ( x , f̂ ) ) ) arg maxφ ( f ( x ) ) ) = arg min i∈ [ N ] f̂ ( ψi ( x , S ( x , f̂ ) ) ) arg maxφ ( f ( x ) ) . ( 4 ) Due to the limitation of the number of queries , we can not query the victim model for each erased170 image to obtain a new label . We continuously choose the erased image with the highest substitute ’ s171 confidence until reaching the budget . To measure the confidence of the model , we adopt the Maximum172 Softmax Probability ( MSP ) for its simplicity:173 arg max x∼DT MSP ( f̂ ( Π ( x ) ) ) = arg max x∼DT f̂ ( Π ( x ) ) arg max f̂ ( Π ( x ) ) , ( 5 ) where DT is the transfer set . The erased images selected in this way are most likely to change the174 prediction class . Then , we query the victim model to get these erased images ’ labels and construct175 an erased sample set DE . Note that when the victim model ’ s predictions on the erased images176 change , it means our erasing method does dig out other related class information in the sample . With177 the unchanged predictions , it points out the attentions of the substitute model and the victim are178 inconsistent . Though wrong attention areas erased , training with these samples benefits aligning the179 attentions of two models . As [ 30 ] stated , the attention alignment can help more powerful KD.180 3.2 A random-erasing-based self-KD module181 We also find that in training with limited hard-label OOD samples , the substitute model is likely182 to overfit the training set , which damages its generalization ability [ 15 , 31 ] . Therefore , based on183 the above erasing operation , we further design a simple RE-based self-KD method to improve the184 generalization ability of the substitute model.185 Formally , let x ∈ [ 0 , 1 ] d denote the unlabeled input image . We perform the erasing operation with a186 uniform prior U on it N times , and then average the substitute ’ s outputs on these erased images as187 the pseudo-label of the original image:188 yp ( x , f̂ ) = 1 N N∑ i=1 f̂ ( ψi ( x , U ) ) . ( 6 ) 1Rand ( a , b ) returns an evenly distributed random real number in the range of a to b. Algorithm 2 : Black-box Dissector Input : Unlabeled pool DU , victim model f , maximum number of queries Q . Output : Substitute model f̂ . 1 Initialize q ← 0 , DT ← ∅ , DE ← ∅ 2 while q < Q do 3 // Step 1 4 Select samples from DU according to budget and query f to updata DT 5 q = q + budget 6 L = ∑ x∈DT L ′ ( φ ( f ( x ) ) , f̂ ( x ) ) 7 f̂ ← update ( f̂ , L ) 8 // A CAM-driven erasing strategy ( step 2.1 ) 9 Erase samples in DT according to Eq . 4 10 Choose samples from erased samples according to Eq . 5 and budget 11 Query f to get labels and updata DE 12 L = ∑ x∈DT∪DE L ′ ( φ ( f ( x ) ) , f̂ ( x ) ) 13 f̂ ← update ( f̂ , L ) 14 q = q + budget 15 // A random-erasing-based self-KD ( step 2.2 ) 16 Select samples from DU 17 Get pseudo-labels according to Eq . 6 and construct a pseudo-label set DP 18 L = ∑ x∈DT∪DE L ′ ( φ ( f ( x ) ) , f̂ ( x ) ) +∑x∈DP L′ ( yp ( x , f̂ ) , f̂ ( x ) ) 19 f̂ ← update ( f̂ , L ) 20 end This is a type of consistency regularization , which enforces the model to have the same predictions189 for the perturbed images and enhances the generalization ability . With Eq.6 , we construct a new soft190 pseudo label set DP = { ( x , yp ( x , f̂ ) ) , . . . } .191 With the transfer set DT , the erased sample set DE , and the pseudo-label set DP , we train a new192 substitute model using the ensemble of the victim model and the previous substitute model as the193 teacher . Our final objective function is:194 minL = min [ ∑ x∈DT∪DE L′ ( φ ( f ( x ) ) , f̂ ( x ) ) + ∑ x∈DP L′ ( yp ( x , f̂ ) , f̂ ( x ) ) ] . ( 7 ) where L′ can be commonly used loss functions , e.g. , cross-entropy loss function.195 To sum up , we built our method on the conventional process of the model stealing attack ( step196 1 ) , and proposed a CAM-driven erasing strategy ( step 2.1 ) and a RE-based self-KD module ( step197 2.2 ) unified by a novel erasing method . The former strategy digs out missing information between198 classes and aligns the attention while the latter module helps to mitigate overfitting and enhance the199 generalization . We name the whole framework as black-box dissector and present the algorithm200 detail of it in Alg . 2.201 4 Experiment202 4.1 Experiment settings203 Victim model . The victim models we used ( ResNet-34 [ 9 ] ) are trained on four datasets , namely,204 CIFAR10 [ 16 ] , SVHN [ 19 ] , Caltech256 [ 7 ] , and CUBS200 [ 28 ] , and their test accuracy are 91.56 % ,205 96.45 % , 78.40 % , and 77.10 % , respectively . All models are trained using the SGD optimizer with206 momentum ( of 0.5 ) for 200 epochs with a base learning rate of 0.1 decayed by a factor of 0.1 every207 30 epochs . Following [ 20 , 21 , 34 ] , we use the same architecture for the substitute model and will208 analyze the impact of different architectures in the supplementary.209 Attack dataset . We use 1.2M images without labels from the ILSVRC-2012 challenge [ 23 ] as the210 attack dataset . In a real attack scenario , the attacker may use pictures collected from the Internet , and211 the ILSVRC-2012 dataset can simulate this scenario well . Note that we resize all images in the attack212 dataset to fit the size of the target datasets , which is similar to the existing setting [ 20 , 21 , 34 ] .213 Training process . We use the SGD optimizer with momentum ( of 0.9 ) for 200 epochs and a base214 learning rate of 0.02 × batchsize128 decayed by a factor of 0.1 every 60 epochs . The weight decay is215 set to 5 × 10−4 for small datasets ( CIFAR10 [ 16 ] and SVHN [ 19 ] ) and 0 for others . We set up a216 query sequence { 0.1K , 0.2K , 0.5K , 0.8K , 1K , 2K , 5K , 10K , 20K , 30K } as the iterative maximum217 query budget , and stop the sampling stage whenever reaching the budget at each iteration.218 Baselines and evaluation metric . We mainly compare our method with KnockoffNets [ 20 ] and219 ActiveThief [ 21 ] . Follow Jagielski et al . [ 12 ] , we mainly report the test accuracy ( Acc ) as the220 evaluation metric . We also report the Agreement metric proposed by Pal et al . [ 21 ] which counts how221 often the prediction of the substitute model is the same as the victim ’ s as a supplement.222 4.2 Experiment results223 We first report the performance of our method compared with previous methods . After that , we224 conduct ablation experiments to analyze the contribution of each module . Finally , we also analyze the225 performance of our method when encountering defense methods and real-world online APIs . More226 experiments ( e.g. , adversarial attack and overfitting analysis ) can be found in our supplementary.227 Effectiveness of our method . As in Tab . 1 , the test accuracy and agreement of our method are all228 better than the previous methods . We also plot the curves of the test accuracy versus the number of229 queries in Fig . 4 . The performance of our method consistently outperforms other methods throughout230 the process . Since our method does not conflict with the previous sample selection strategy , they231 can be used simultaneously to further improve the performance of these attacks . Here , we take232 the k-Center algorithm as an example . Note that , with or without the sample selection strategy,233 our method beats the previous methods by a large margin . Particularly , the test accuracies of our234 method are 4.85 % , 1.72 % , 3.88 % , and 8.27 % higher than the previous best method , respectively.235 And the agreement metric shares similar results . It is also interesting that it is less necessary to use236 the k-Center algorithm on datasets with a small number of classes ( i.e. , CIFAR10 and SVHN ) . While237 for the datasets with a large number of classes , the k-Center algorithm can make the selected samples238 better cover each class and improve the effectiveness of the method.239 Ability to evade the SOTA defense method . The SOTA perturbation-based defense method , adap-240 tive misinformation [ 13 ] , introduces an Out-Of-Distribution ( OOD ) detection module based on the241 maximum predicted value and punishes the OOD samples with a perturbed model f ′ ( · ; θ′ ) . The242 model f ′ ( · ; θ′ ) is trained with arg minθ′ E ( x , y ) [ − log ( 1− f ′ ( x ; θ′ ) y ) ] to minimize the probability of243 the correct class . Finally , the output will be:244 y′ = ( 1− α ) f ( x ; θ ) + ( α ) f ′ ( x ; θ′ ) , ( 8 ) where α = 1/ ( 1 + eν ( max f ( x ; θ ) −τ ) ) with a hyper-parameter ν is the coefficient to control how245 much correct results will be returned , and τ is the threshold used for OOD detection . The model246 returns incorrect predictions for the OOD samples without having much impact on the in-distribution247 samples.248 We choose four values of the threshold τ to compare the effects of our method with the previous249 methods . The threshold value of 0 means no defence . The result is shown in Tab . 2 . Compared250 with other methods , adaptive misinformation is almost invalid to our method . Furthermore , we find251 that if we remove the self-KD in our method , the performance is greatly reduced . We conclude that252 this is because adaptive misinformation adds noise labels to the substitute model ’ s training dataset,253 and self-KD can alleviate the overfitting of the substitute model to the training dataset , making this254 defence method not effective enough.255 Ablation study . To evaluate the contribution of different mod-256 ules in our method , we conduct the ablation study on CUBS200257 dataset and plot the results in Fig . 5 . If the CAM-driven erasing258 strategy is removed , the performance of our method will be259 greatly reduced , showing that it has an indispensable position260 in our method . We also give some visual examples in Fig . 7 to261 demonstrate that this strategy can help align the attention of two262 models . As depicted in the Fig . 7 , at the beginning time , the263 substitute model learns the wrong attention map . Along with264 the iterative training stages , the attention area of the substitute265 model tends to fit the victim model ’ s , which conforms to our266 intention . We further remove the self-KD module to evaluate267 its performance . It can be found from Fig . 1 and Fig . 5 that268 the self-KD can improve the generalization of our method and269 further improve the performance.270 Stealing functionality of a real-world API . We validate our271 method is applicable to real-world APIs . The AWS Marketplace272 is an online store that provides a variety of trained ML models273 for users . It can only be used in the form of a black-box setting.274 We choose a popular model ( waste classifier 2 ) as the victim275 model . We use ILSVRC-2012 dataset as the attack dataset and276 choose another small public waste classifier dataset 3 , contain-277 ing 2 , 527 images as the test dataset . As in Fig . 6 , the substitute278 model obtained by our method achieves 12.63 % and 7.32 % 279 improvements in test accuracy compared with two previous280 methods , which show our method has stronger practicality in281 the real world.282 2https : //amzn.to/3nFvA54 3https : //github.com/garythung/trashnet Transferability of adversarial samples . Though with the dominant performance on a wide range of283 tasks , deep neural networks are shown to be vulnerable to imperceptible perturbations , i.e. , adversarial284 examples [ 27 ] . Since the model stealing attack can obtain a functionally similar substitute model,285 some previous works ( e.g. , JBDA [ 22 ] , DaST [ 34 ] and ActiveThief [ 21 ] ) used this substitute model286 to generate adversarial samples and then performed the transferable adversarial attack on the victim287 model . We argue that a more similar substitute model leads to a more successful adversarial attacks.288 We test the transferability of adversarial samples on the test set of the CIFAR10 dataset . Keeping the289 architecture of the victim model as the ResNet-34 , we evaluate the attack success rate of adversarial290 samples generated from different substitute models ( i.e. , ResNet-34 , ResNet-18 , ResNet-50 [ 9 ] , VGG-291 16 [ 26 ] , DenseNet [ 11 ] ) . All adversarial samples are generated using Projected Gradient Descent292 ( PGD ) attack [ 18 ] with maximum L∞-norm of perturbations as 8/255 . As shown in Tab . 3 , the293 adversarial samples generated by our substitute models have stronger transferability in all substitute ’ s294 architectures . This again proves that our method is more practical in real-world scenarios.295 5 Conclusion296 We investigated the problem of model stealing attacks under the hard-label setting and pointed out297 why previous methods are not effective enough . We presented a new method , termed black-box298 dissector , which contains a CAM-driven erasing strategy and a RE-based self-KD module . We299 showed its superiority on four widely-used datasets and verified the effectiveness of our method300 with defense methods , real-world APIs , and the downstream adversarial attack . Though focusing301 on image data in this paper , our method is general for other tasks as long as the CAM and similar302 erasing method work , e.g. , synonym saliency words replacement for NLP tasks [ 4 ] . We believe our303 method can be easily extended to other fields and inspire future researchers . Model stealing attack304 poses a threat to the deployed machine learning models . We hope this work will draw attention to305 the protection of deployed models and furthermore shed more light on the attack mechanisms and306 prevention methods.307 References308 [ 1 ] Rohan Anil , Gabriel Pereyra , Alexandre Passos , Robert Ormandi , George E Dahl , and Geof-309 frey E Hinton . Large scale distributed neural network training through online distillation . arXiv310 preprint arXiv:1804.03235 , 2018.311 [ 2 ] Antonio Barbalau , Adrian Cosma , Radu Tudor Ionescu , and Marius Popescu . Black-box ripper:312 Copying black-box models using generative evolutionary algorithms . In NeurIPS , 2020.313 [ 3 ] Terrance DeVries and Graham W Taylor . Improved regularization of convolutional neural314 networks with cutout . arXiv preprint arXiv:1708.04552 , 2017.315 [ 4 ] Xinshuai Dong , Anh Tuan Luu , Rongrong Ji , and Hong Liu . Towards robustness against natural316 language word substitutions . In ICLR , 2021.317 [ 5 ] Melanie Ducoffe and Frederic Precioso . Adversarial active learning for deep networks : a margin318 based approach . In ICML , 2018.319 [ 6 ] Tommaso Furlanello , Zachary C Lipton , Michael Tschannen , Laurent Itti , and Anima Anandku-320 mar . Born again neural networks . In ICML , 2018.321 [ 7 ] Gregory Griffin , Alex Holub , and Pietro Perona . Caltech-256 object category dataset . 2007.322 [ 8 ] Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q Weinberger . On calibration of modern neural323 networks . In ICML , 2017.324 [ 9 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image325 recognition . In CVPR , 2016.326 [ 10 ] Geoffrey Hinton , Oriol Vinyals , and Jeff Dean . Distilling the knowledge in a neural network . In327 NIPS Deep Learning Workshop , 2015.328 [ 11 ] Gao Huang , Zhuang Liu , Laurens Van Der Maaten , and Kilian Q Weinberger . Densely connected329 convolutional networks . In CVPR , 2017.330 [ 12 ] Matthew Jagielski , Nicholas Carlini , David Berthelot , Alex Kurakin , and Nicolas Papernot.331 High accuracy and high fidelity extraction of neural networks . In 29th Usenix Security , 2020.332 [ 13 ] Sanjay Kariyappa and Moinuddin K Qureshi . Defending against model stealing attacks with333 adaptive misinformation . In CVPR , 2020.334 [ 14 ] Sanjay Kariyappa , Atul Prakash , and Moinuddin Qureshi . Maze : Data-free model stealing335 attack using zeroth-order gradient estimation . arXiv preprint arXiv:2005.03161 , 2020.336 [ 15 ] Kyungyul Kim , ByeongMoon Ji , Doyoung Yoon , and Sangheum Hwang . Self-knowledge337 distillation : A simple way for better generalization . arXiv preprint arXiv:2006.12000 , 2020.338 [ 16 ] Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images.339 2009.340 [ 17 ] David D Lewis and William A Gale . A sequential algorithm for training text classifiers . In341 SIGIR , 1994.342 [ 18 ] Aleksander Madry , Aleksandar Makelov , Ludwig Schmidt , Dimitris Tsipras , and Adrian Vladu.343 Towards deep learning models resistant to adversarial attacks . In ICLR , 2018.344 [ 19 ] Yuval Netzer , Tao Wang , Adam Coates , Alessandro Bissacco , Bo Wu , and Andrew Y Ng.345 Reading digits in natural images with unsupervised feature learning . In NIPS Workshop on346 Deep Learning and Unsupervised Feature Learning , 2011.347 [ 20 ] Tribhuvanesh Orekondy , Bernt Schiele , and Mario Fritz . Knockoff nets : Stealing functionality348 of black-box models . In CVPR , 2019.349 [ 21 ] Soham Pal , Yash Gupta , Aditya Shukla , Aditya Kanade , Shirish Shevade , and Vinod Ganapathy.350 Activethief : Model extraction using active learning and unannotated public data . In AAAI , 2020.351 [ 22 ] Nicolas Papernot , Patrick McDaniel , Ian Goodfellow , Somesh Jha , Z Berkay Celik , and Anan-352 thram Swami . Practical black-box attacks against machine learning . In ACM AsiACCS , 2017.353 [ 23 ] Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng354 Huang , Andrej Karpathy , Aditya Khosla , Michael Bernstein , et al . Imagenet large scale visual355 recognition challenge . IJCV , 2015.356 [ 24 ] Ramprasaath R Selvaraju , Michael Cogswell , Abhishek Das , Ramakrishna Vedantam , Devi357 Parikh , and Dhruv Batra . Grad-cam : Visual explanations from deep networks via gradient-based358 localization . In ICCV , 2017.359 [ 25 ] Ozan Sener and Silvio Savarese . Active learning for convolutional neural networks : A core-set360 approach . In ICLR , 2018.361 [ 26 ] Karen Simonyan and Andrew Zisserman . Very deep convolutional networks for large-scale362 image recognition . In ICLR , 2015.363 [ 27 ] Christian Szegedy , Wojciech Zaremba , Ilya Sutskever , Joan Bruna , Dumitru Erhan , Ian Good-364 fellow , and Rob Fergus . Intriguing properties of neural networks . In ICLR , 2014.365 [ 28 ] Catherine Wah , Steve Branson , Peter Welinder , Pietro Perona , and Serge Belongie . The366 caltech-ucsd birds-200-2011 dataset . 2011.367 [ 29 ] Jiancheng Yang , Yangzhou Jiang , Xiaoyang Huang , Bingbing Ni , and Chenglong Zhao . Learn-368 ing black-box attackers with transferable priors and query feedback . In NeurIPS , 2020.369 [ 30 ] Sergey Zagoruyko and Nikos Komodakis . Paying more attention to attention : Improving the370 performance of convolutional neural networks via attention transfer . In ICLR , 2017.371 [ 31 ] Chiyuan Zhang , Samy Bengio , Moritz Hardt , Benjamin Recht , and Oriol Vinyals . Understanding372 deep learning requires rethinking generalization . In ICLR , 2017.373 [ 32 ] Zhun Zhong , Liang Zheng , Guoliang Kang , Shaozi Li , and Yi Yang . Random erasing data374 augmentation . In AAAI , 2020.375 [ 33 ] Bolei Zhou , Aditya Khosla , Agata Lapedriza , Aude Oliva , and Antonio Torralba . Learning deep376 features for discriminative localization . In CVPR , 2016.377 [ 34 ] Mingyi Zhou , Jing Wu , Yipeng Liu , Shuaicheng Liu , and Ce Zhu . Dast : Data-free substitute378 training for adversarial attacks . In CVPR , 2020.379 Checklist380 1 . For all authors ... 381 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s382 contributions and scope ? [ Yes ] 383 ( b ) Did you describe the limitations of your work ? [ Yes ] See section 3384 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] See section385 5386 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to387 them ? [ Yes ] 388 2 . If you are including theoretical results ... 389 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 390 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 391 3 . If you ran experiments ... 392 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-393 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] See the394 supplemental material395 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they396 were chosen ) ? [ Yes ] See section 4.1397 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-398 ments multiple times ) ? [ Yes ] See Tab . 1399 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type400 of GPUs , internal cluster , or cloud provider ) ? [ No ] 401 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 402 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See section 4.1403 ( b ) Did you mention the license of the assets ? [ No ] 404 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ No ] 405 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re406 using/curating ? [ No ] 407 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable408 information or offensive content ? [ No ] 409 5 . If you used crowdsourcing or conducted research with human subjects ... 410 ( a ) Did you include the full text of instructions given to participants and screenshots , if411 applicable ? [ N/A ] 412 ( b ) Did you describe any potential participant risks , with links to Institutional Review413 Board ( IRB ) approvals , if applicable ? [ N/A ] 414 ( c ) Did you include the estimated hourly wage paid to participants and the total amount415 spent on participant compensation ? [ N/A ] 416

Summary: This paper proposes a new model stealing attack when the victim model only returns the class label to the queries instead of confidence or probability information. Prior attacks struggle in this setting to extract high-accuracy models. The proposes attack has two steps, first they query the model with a sample and probabilistically remove the input region using the GRAD-CAM of the substitute model. This aims to fit more information into a single query and simulates the prediction probabilities. Second, they use a self-knowledge distillation scheme on the substitute model to alleviate overfitting. Overall, their scheme achieves higher stolen accuracy than prior attacks, more resilient to a prior defense and allows the attacker to craft more transferable adversarial examples.