IDEAL: Query-Efficient Data-Free Learning from Black-Box Models
Authors: Jie Zhang
Chen Chen
Lingjuan Lyu
Venue: ICLR 2023 poster (2023)
Keywords: black-box model
knowledge distillation

Abstract: Knowledge Distillation (KD) is a typical method for training a lightweight student model with the help of a well-trained teacher model. 
However, most KD methods require access to either the teacher's training data or model parameter, which is unrealistic. To tackle this problem, recent works study KD under data-free and black-box settings. Nevertheless, these works require a large number of queries to the teacher model, which incurs significant monetary and computational costs. To address these problems, we propose a novel method called \emph{query-effIcient Data-free lEarning from blAck-box modeLs} (IDEAL), which aims to query-efficiently learn from black-box model APIs to train a good student without any real data.   In detail, IDEAL trains the student model in two stages: data generation and model distillation. Note that IDEAL does not require any query in the data generation stage and queries the teacher only once for each sample in the distillation stage. Extensive experiments on various real-world datasets show the effectiveness of the proposed IDEAL. For instance, IDEAL can improve the performance of the best baseline method DFME by 5.83\% on CIFAR10 dataset with only $0.02\times$ the query budget of DFME.

Published as a conference paper at ICLR 2023
IDEAL: QUERY-EFFICIENT DATA-FREE LEARNING FROM BLACK-BOX MODELS
Jie Zhang -f
Chen Chen" & Lingjuan Lyu
Zhejjang University
Sony AI
{zj zhangjie ez edu.cn
[ChenA. Lingjuan. / @sony com
ABSTRACT
Knowledge Distillation (KD) is a typical method for training a lightweight student model with the help of well-trained teacher model. However, most KD methods require access to either the teacher's training data or model parameter, which is unrealistic. To tackle this problem, recent works study KD under data-free and black-box settings. Nevertheless, these works require a large number of queries to the teacher model, which incurs significant monetary and computational costs. To address these problems, we propose novel method called query-effccien Data- free !Earning from blAck-box modeLs (IDEAL), which aims to query-efficientyy learn from black-box model APIs to train good student without any real data. In detail, IDEAL trains the student model two stages: data generation and model distillation. Note that IDEAL does not require any query in the data generation stage and queries the teacher only once for each sample in the distillation stage. Extensive experiments the proposed IDEAL. For instance, IDEAL can improve the performance the best baseline method DFME by 5.83% on CIFAR10 dataset with only 0.02 the query budget of DFME.
various real-world datasets show the effectiveness of
INTRODUCTION
Knowledge Distillation (KD) has emerged popular paradigm for model compression and knowl- edge transfer Gou et al. 2021). The goal of KD to train a lightweight student model with the help of well-trained teacher model. Then, the lightweight student model can be easily deployed to resource-limited edge devices such as mobile phones. In recent years, KD has attracted significant attention from various research communities, e.g., computer vision Wang (2021) ; Passalis et al.
2020); Hou et al. (2020); Li et al. (2020), natural language processing Hinton et al. (2015): Mun al.
2018 Nakashole Flauger (217)) Zhou et (2020b), and recommendation systems Kang et al.
2020); Wang et al. (2021a); K weon et al. (2021), Shen et al. (202I).
However, most KD methods are based on several unrealistic assumptions: (1) users can directly access teacher's training data; (2) the teacher model is considered as white-box model, i.e., model parameters and structure information can be fully utilized. For example, to facilitate the training process, FitNets Romero et al. (2015) uses not only the original training data, but also the output information from the teacher's intermediate layers. However, in real- world applications, model is usually provided by third party. Thus the teacher's 's training data is usually not public and unable to access. In fact, the teacher model mostly trained by big companies with extensive amounts of data and plenty of computation resources, which is the core competitiveness of compa- nies. As a result, the specific parameters and structural information of the teacher model are never exposed in the real world. Consequently, accessing the teacher model teacher's training data render these KD methods impractical in reality.
teacher
To solve these problems, some studies Truong al. 2021bb;; Fang al. (2021a) attempt learn from a black-box teacher model without any real data, i.e., data-free black-box KD. These
'Equal contribution.
I Work done during internship at Sony AL. 1 "Corrsponndigg author.

---

Published as a conference paper at ICLR 2023
Table 1: An empirical study of previous methods with limited number of queries (we set the query
budget Q = 25K for MNIST, Q = 250K for CIFARIO, and Q 2M for CIFARION 100.) in various scenarios. We also adopt CMI Fang et al. (2021b) for hard-label scenarios and name it "CMI""
Method
training data white hox / black hox
lagits hard label
MNIST
CIFARIO
CIFARI00
Normal KD et al 2015 Fangerr
V
white-box whiite-box
logits
98.91%
94.34%
76.87%
lopits
98.20%
92.22%
74.47%
CMI
white-box
hard-label
86.53%
76. 17%
63.45%
DFME Triong al. 2021a
black-box
lopits
68.268
51.28%
39.12%
ZSDBBS Wang 2021
X
black box
hard label
37.33%
32.18%
14.28%
methods do not need to access the private data and can train the student model with the class prob- abilities returned by the teacher model. However, in real-world scenarios, the pre-trained model on the remote server may only provide APIs for inference purpose (e.g., commercial cloud services), these APIs usually return the top-1 class (i.e., hard label) of the given queries. For example, Google BigQuery provides APIs for several applications. Such APIs only return a category index for each sample instead of the class probabilities. Moreover, these APIs usually charge for each query to the teacher model, and thus budget should be considered in the process of query. Nevertheless, previous methods methods/Truong al. al. (2021a)) [Wang (2021): (Zhou al. (2020a) require large number of queries to the teacher model, which is costly and impractical. Hence, training high-performance student model with small number of queries is still an unsolved problem.
(aang((021)
In this paper, we consider a more practical and challenging setting: (1) the teacher's training data not accessible, i.e., data-free; (2) the parameter of the teacher model is not accessible, i.e., black- box; (3) the teacher model only returns category index for each sample, i.e. hard-label; and (4) the number of queries is limited, i.e., query-efficint. To better understand the difficulty of this setting, we report the top-1 test accuracy of student models under different scenarios with a limited query budge in Table
As shown in Table 1 we have some valuable observations: (1) In white-box scenarios, data-free KD can achieve satisfied performance, but when the model API restricted to only hard labels, CMI Fang et al. (202Tb) suffers from serious performance degradation. It indicates that logits can provide more information for training, while hard labels are more difficult; (2) With the same num- ber of queries, the performance of these methods dramatically decrease under the black-box sce- narios. Furthermore, the performance of data-free black-box KD with hard labels is only 14.28% on CIFAR10 dataset, which is close to random guess (10%). Consequently, in this paper, focus primarily on how to query-efficiently train a good student model from black-box models with hard labels, which is very practical but challenging.
For this purpose, we propose novel method called query-effcienl Data-free lEarning from blAck- box modeLs (IDEAL), which trains the student model with two stages; a data generation stage and a model distillation stage. Instead of utilizing the teacher model (as in previous methods Truong ct al. (2021b). we propose to adopt the student model to train the generator in the first stage, which can solve the hard-label issue and largely reduce the number of queries to the teacher model. In the second stage, we train a student model that has similar predictions as the teacher model on the synthetic samples. As a result, IDEAL requires much less query budget than previous methods, which saves lot of money and becomes more practical in reality.
In summary, our main contributions include:
New Problem: We focus on how to -uerr-efficiently train a good student model from black- box models with only hard labels. To the best of our knowledge, our setting is the most practical and challenging to date.
More Efficient: We propose a novel method called IDEAL, which does not require any query in the data generation stage and queries the teacher only once for each sample in the distillation stage. Thus IDEAL. can train a high-performance student with a small number of queries.
'https//clooud g.wgglc.com/bigg........o
The detailed settings be found in Section 4.1
2

---

Published as a conference paper at ICLR 2023
SOTA Results: Extensive experiments various real-world datasets demonstrate the ef- ficacy of our proposed IDEAL. For instance, IDEAL can improve the performance of the best baseline method (DFME) by 33.46% on MNIST dataset.
2 RELATED WORKS
2.1 WHITE- BOX DATA FREE KNOWLEDGE DISTILLATION
Recent advances in data-free knowledge distillation have enabled the compression of large neural networks into smaller networks without using any real data Micaelli & Storkey (2019); Fang et al.
(2021b) Yin al. (2020); Chen et al. (2019): [hhrdwajj al. (2019); Haroush al. (020): Yoo et al. (211): Zhang et al. Nevertheless, all of these approaches require access to the white-box teacher model and the logits (or probabilities) calculated by the teacher model, which not always possible in realistic scenarios. For example, according to these approaches |ang et al. (2021b); Yin ict al. (2020;; Chen et al. (2019); Ye et al. (2020); Xu et al. (2020), the pre-trained teacher model is regarded as discriminator, and then the generator is adversarially trained. As the teacher model is only accessible as a black-bor box model, it is not possible to propagate gradients in this manner. Furthermore, the widely used KL divergence is inapplicable,
the logits of the teacher model are not accessible. Additionally, some works utilize the specific structural information of the white- box model, which also violate the black-box rules. For example, DeepIn Yin et al. (2020) used the running average statistics stored in the BatchNorm layers, while DAFL et al. (2019) proposed use features extracted by convolution filters. As a result of the above irrationality, these methods perform poorly in our settings. In more practical and challenging setting we considered in this work: data-free knowledge distillation from black-bos models with only hard labels.
2.2 DISTILLATION--ASEH BLACK-BOX ATTACKS
Previous studies have explored several ways to attack black-box models without real-world train- ing data, which can be roughly divided into transfer-based adversarial attack Zhou et al. (020a);
Yu & Sun (2022); ; Wang et al. (202b)) and data-free model extraction attack Truong et al. 2021a);
Kariyappa et al. (021). Essentially, these methods are based on data-free black-box model dis- tillation. Actually, they are designed to train substitute models in black-box situations to attack the victim model. While these methods are suitable for black-box scenarios, they mainly rely on score-based teacher which outputs class probabilities. By contrast, more challenging scenario, in which a black-box teacher only returns the top- class. Moreover, in real-world scenarios, these black-box models usually charge for each query. To achieve good per- formance, these methods require millions of queries, which consume a lot of computing resources and money in real-world scenarios.
study considers much
2.3 COMPARISON WITH RELATED WORKS
The most related work is 7SDB3 Wang 2021 which also studied data-free black-box distillation with hard labels. It proposes generate pseudo samples distinguished by the teacher's decision boundaries and then reconstruct the soft labels for distillation. More specifically, it calculates the minimal æn-ormm distance between the current sample and those of other classes (measured by the teacher model) and uses the zeroth-order optimization method to estimate the gradient of the teacher model, which requires large number of queries, making ZSBD3 not practical in real-world sce- narios. By contrast, we consider more challenging and practical setting where only a very small number of queries (to the teacher model) is allowed, i.e., query efficient. For example, ZSDB3 re- quires about 1000 queries to reconstruct the soft label (logits) of a single sample on MNIST dataset, while our method only requires one query, which hugely reduces the number queries by 1000x.
MAIN METHOD
3.1 NOTATIONS
We use G. S, and T to denote the generator, the student model, and the teacher model, respectively.
og and Ag denote the parameters of generator G and student model S, respectively. & and ý denote
3

---

Published as a conference paper at ICLR 2023
Random Label
Black-box Model
Random Noise
Update
Query API
HI
Top-1 label
Lee + Linfor
Logits
Update
Distillation Loss
Synthetic
Logits
Generator
Synthetic Data
Student (fixed)
Student
(a) Data Generation
(b) Model Distillation
Figure 1: Illustration of the training process of our proposed IDEAL. The left panel demonstrates the data generation stage. this stage, we train a generator, that can generate desired synthetic samples, with the student model. The right panel shows the model distillation stage, which trains student model that has similar predictions as the teacher model on the synthetic samples.
the synthetic sample (generated by G) and the corresponding prediction score. Subscript i denotes the i-th sample, e.g., 2t denotes the i-th synthetic sample. Superscript_ j denotes the j-th epoch, e.g. D denotes the set of synthetic samples generated in j-th epoch. C' is the number of classes and B the batch size. We use (f)x to denote the k-th clement of outputs g. i.e., the prediction score of the k-th class.
3.2 OVERVIEW
In data-free black-box KD, the teacher's model and data are not accessible, and we are only given the prediction of a sample by the teacher model. In particular, focus challenging setting where only a category index for each sample (i.e., hard-label) is given by the teacher model. Since each query to the teacher costs money, consider the scenario with limited queries, i.e., query-efficient. Our goal is to query-efficiently learn from black-box models to train good student without any real data.
practical and
To achieve this goal, we propose novel method called IDEAL, which consists of two stages: data generation stage and model distillation stage. In the first stage, instead of utilizing the teacher model (as in previous methods Truong et al. (202Tb); Kariyappa et al. (202T)). we propose to adopt the student model train the generator, which can solve the hard-label issue and largely reduce the number of queries to the teacher model. In the second stage, utilize the teacher model and synthetic samples to train the student model. The generator and student model are iteratively trained for E epochs. The training procedure is demonstrated in the Appendix (see Algorithm I and the illustration the training process of IDEAL is shown in Fig.
3.3 DATA GENERATION
In data-free setting, Therefore, in the first stage, we aim to train generator to generate the desired synthetic data (to train the student model). According to the finding in ZZang et al. (2022), we reinitialize the generator each epoch. The data generation procedure is illustrated in Fig.
unable access the original training data for training the student model.
The first step is generating the synthetic sample. Given random noise (sampled from standard Gaussian distribution) and a corresponding random one-hot label (sampled from a uniform dis- tribution), the generator aims generate desired synthetic sample á corresponding label y. Specifically, we feed into the generator G and compute the synthetic sample as follows:
a = (2; 0g),
(1)
where Bg is the parameter of G. The synthetic samples used to train g.
In the second step, we compute the prediction score of â. A straightforward way is to use the teacher model compute the prediction score. The prediction score is used update 0g. but the parameter

---

Published as a conference paper at ICLR 2023
of the teacher model is not accessible in black-box setting, thus unable to conduct backpropagation. Previous black-box box KD methods Truong et al. (2021b); Wang (202T); Kariyapaa et al. (2021) used gradient estimation methods to obtain approximate gradient. Nevertheless, they need estimate the gradient from the black-box teacher model, which requires a large number of queries (to the teacher model), which is not practical. Moreover, in the hand-label setting, the prediction score is not accessible. To this end, we propose to use the student model (instead of the teacher model) to compute the prediction score of &. The detail of the student model discussed in Section 3.4 Note that in this stage, we do not train the student model and keep the parameter of the student model fixed. By utilizing the student model, we can directly conduct backpropagation and compute the gradient of the model without querying the teacher model. In this way, we can avoid the hard-label problem and the large number of queries at the same time. The prediction score computed as follows:
ÿ = S(;; 0s),
where S and As are student model and model parameters.
The third step is optimizing the generator. We propose to train a generator that considers both confidence and balancing.
3.3.1 CONFIDENCE
First, need consider confidence, i.e., the synthetic sample is classified the specified class with high confidence. To achieve this goal, we minimize the difference between the prediction score y and the specified label y:
Le = CE(Ú y):
(3)
where CE(,,)) is the cross-entropy (CE) loss. Actually, in the training process, the generator G can quickly converge when using Lee. Since the generated data & is fitted for the student S, and intend to generate data according to the knowledge of the teacher's s model, we must avoid overfitting to S. Therefore, iterations may lead to poor data, while too many iterations may lead to overfitting. See the detailed experiments in the Section 4.1
need to control the number of iterations Eg in data generation. Too few
3.3.2 BALANCING
Second, need to consider balancing, i.e., the number of synthetic samples in each class should be balanced. Although we uniformly sample the specified label y, we observe that the prediction score ý not balanced, i.e., the prediction score is high on some classes but low the other classes. This leads to class imbalance the generated synthetic samples. Motivated by [Chnn et al.] (2019), employ the information entropy loss to measure the class balance of the synthetic samples. In
particular, given a batch of synthetic samples ( B and corresponding prediction scores (ú) )
where B is the batch size, first compute the average of the prediction scores follows:
Dovg =
1
Σ
D
(4)
B
1=1
Then, compute the information entropy loss follows:
Linfo = C Σ (Bavg)k k (og((Gogg))))
(5)
where (fang); is the k-th element of Dougs i.e., the average prediction score of the k-th class. When Linfo takes the minimum, each element in Dang would equal to t.. which implies that can generate synthetic samples of each class with an equal probability.
By combining the above losses, we can obtain the generator loss follows:
Lyen Loe + XLinfos
where is the scaling factor. By minimizing Eyens we train a generator that generates desired balanced synthetic samples.
5

---

Published as a conference paper at ICLR 2023
3.4 MODEL DISTILLATION
In the second stage, train the student model S with teacher model T and the synthetic samples. The training process is illustrated in Fig. [b). Our goal is to obtain a student model that has the same predictions teacher model T on the synthetic samples (generated by generator 9).
In particular, we first sample the random noise and generate synthetic sample with the generator. Second, feed ft into the black-box teacher model and obtain its label as follows:
VT T())
We treat yT as the ground-truth label of i. Since the teacher model only returns hard-label, yr is ground-truth one-hot label. Afterwards, we feed into the student model and obtain the prediction score follows:
ý ((;;0s))
Last, we optimize the student model by minimizing the CE loss as follows:
Lmdd CC yT)
(9)
By minimizing Lind, the student model can have similar predictions as the teacher model on the synthetic samples, which leads to a desired student model.
In each epoch, our proposed IDEAL only queries the teacher model once for each sample, while previous black-box methods (e.g., ZSDB3 Wang (2021) and DFME (FMonn al. (2021a) may need more than 100 queries. Moreover, IDEAL requires similar epochs (compared with previous black- box methods) to converge. As result, IDEAL requires much less query budget than previous methods, which saves a lot of money and becomes more practical in reality.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
4.1.1 DATASET AND MODEL ARCHITECTURE
Our experiments are conducted on real-world datasets: MNIST LeCun et al. (1998), Fashion- MNIST (FMNIST) Xiao et al.] (2017). CIFAR10 and CIFAR 100 Krizhevsky et al. (2009). SVHN Netzer et al. (201)). Tiny- ImageNet Le Yang (2015), and ImageNet subset Deng et al. 2009). The ImageNet subset generated by Li et al. (022)) which consists of 12 classes. We size the original image with size 224*224*3 to 64*64*3 for fast training. Note that student models cannot access any raw data during training. Only teacher models trained on these datasets. In this work, we study the effectiveness our method on several network architectures. including Lecun et al. (2016), VGG-16 Simonyan & Zisserman (2015). and ResNet-34 He et al. (2016). For each dataset, we train several different teacher models evaluate the effectiveness use the generator proposed in StyleGAN Karras et al. (2019) as the default generator.
MLP al. 1990), Krizzhesk et al. (2012). LeNet Lecun al. (1998), ResNet-18 He
our method. We
4.1.2 BASELINES
As discussed in the related work, we compare our approach with the following baselines: 1) SOTA data-free distillation methods that are originally designed for white-box scenarios (DAFL Chen al. (219)) ZSKT Micaelli Micalli & Storkey (2019), DeepIn Yin et al. (2020), CMI [ang et al. (001bb)) Here we adapt them to the black-box scenarios in which only hard labels are provided. 2) Besides, we also compare with the SOTA methods in model extraction attack (DFME Truong et al. (202 and transfer-based adversarial attack (DaST Zhou et al. (2020a)). In fact, these techniques are essential data-free distillation methods in black-box scenarios. 3) Furthermore, compare method with ZSDB3 Wang (20T), which also focuses on improving the performance of the black-box data-free distillation in label-only scenarios.

---

Published as a conference paper at ICLR 2023
Table 2: Accuracy (%) of student models trained with various teacher models MNIST, FMNIST, SVHN, CIFARIO, and ImageNet subset. Best results are in bold. Best results of the baselines are underlined. " "Improvement" denotes the improvements of IDEAL compared with the best bascline.
Dataset
Model
Teacher
DAFL
ZSKT
Decplo
CMI
DaST
ZSDB3
DRME
Ours
Improvement
MLP
98.25
16.97
13.84
16.68
13.89
15.62
30.13
56.32
88.41
32,097
MNIST
LeNet
99.27
18.92
22.96
22.71
35.98
62.86
96.32
33.46
AlexNet
99.35
20.43
27.97
29.54
28.87
23.86
37.33
66.45
96.51
30.06
MLP
84.54
14.23
16.86
12.24
10.30
12.93
24.52
$2.29
76.95
24.66
PMNIST
LeNet
90.23
16.89
18.52
16.89
14.44
22.72
32,46
56.76
83,92
27.16
AlexNet
92.66
21.78
20.22
22.38
21.24
25.21
34.47
63.59
86.14
22.551
AlexNet
89.82
16.89
13.96
16.71
17.63
24.47
33.96
58.92
84.42
25,50
SVHN
VGG-16
94.41
19.24
21.03
24.65
24.55
25.17
36.35
62.53
86,91
24.381
18
95.28
21.25
20.95
24.75
28.55
24.33
37.40
64.82
87.65
22.831
CIFARIO
AlexNet
13.75
12.56
14.54
13.98
14.54
29.38
35.73
29.881
ResNet-34
93.85
16.08
14.31
15.99
15.95
15.41
32.18
37.91
68.82
30.91
ImageNet subxet
AlexNet VGG-16
72.96
15.15
15.89
17.75
17.31
16.96
27.83
32.89
53.72
20.831
78.53
19.36
20.16
19.66
22.10
22.03
29.46
34.65
57.95
23,30
4.1.3 QUERY BUDGET AND TRAINING SETTINGS
Since we consider the limited query budget scenario, we adopt the same query budget Q for all methods. In particular, we set the query budget Q = 25K for MNIST, Q 100K for FMNIST and SVHN. Besides, the default query budget Q 250K for CIFARIO and ImageNet subset. For large datasets with a large number of classes (i.e., CIFAR 100 and Tmay-lmag,Net), we set the query budget Q = 2M. For our method, each sample only needs to query the teacher model once, so the total number of queries is Q B x E, where B is the batch size and E denotes the training epochs. To update the generator, we use the Adam Optimizer with learning rate 1lg = le 3. To train the student model, we use the SGD optimizer with momentum=0.9 and learning rate T/S Ie 2. We set the batch size B = 250 for MNIST, FMNIST, SVHN CIFARIO, and ImageNet subset, and B 1000 for CIFAR100 and Tiny- ImageNet datasets. By default, we set the number of iterations in data generation Fgg = 5 and the scaling factor X = 5. The number of epochs E is computed according the query budget. For evaluation, We experiments for times, and report the average top-1 test accuracy.
4.2
EXPERIMENTAL RESULTS
4.2.1
PERFORMANCE COMPARISON ON SMALL DATASET
First, show the results of different KD methods MNIST, FMNIST, SVHN CIFAR1O, and ImageNet subset using various teacher models in Table From the table, we observe that:
(1) Our proposed IDEAL outperforms all the baseline methods all datasets. For instance, method achieves 87.65% accuracy on SVHN dataset when the teacher model is ResNet-18, 18, whereas the best baseline method DFME achieves only 64.82% accuracy under the same query budget. In general, IDEAL improves the performance of the best baseline by at least 20% under the same settings.
(2) The black-box teacher models trained on MNIST are much easier for the student to learn. Even with very few queries, the student model of our proposed IDEAL achieves over 96% accuracy on MNIST. We argue that this is reasonable because this task is simple for neural networks to solve, and the underlying representations are easy learn. However, even for such simple task, other methods cannot derive a good student model with the same small query budget. For example, when learning from the black-box AlexNet trained on MNIST, the best baseline DFME only achieves 66.45% accuracy.
(3) DAFL and ZSKT have the worst performance on all datasets. For example, the accuracy of ZSKT is only 12.56% when the teacher model is AlexNet on CIFAR1O, which is close to random guess (10%). We conjecture this is because white-box KD methods are not suitable in black-box scenarios. These methods mainly depend on woxte-box information, such as model structure and probability or logits returned by the teacher model. Therefore, using these methods in blxck-box scenarios will significantly reduce their effectiveness.
7

---

Published as a conference paper at ICLR 2023
Table 3: Accuracy (%) of student models datasets with hundreds of classes. We use ResNet- 18
as the default student network, and all results are tested under the same query budget Q = 2M.
Dataset
Model
Teacher
DAFL
7SKT
Deepln
CMI
DaST
ZSDB3
DFMB
Ours
Improvement.
CIFAR100
AlexNet
66.24
9.65
10.19
14.53
12.30
14.22
14.38
16.48
23,85
7.37
ReaNet-34
89.45
12.76
13.54
18.35
15.93
17.92
17.51
20.01
36,96
16.95t
VGG-16
52.96
6,31
7.24
11.95
9.79
8.42
10.98
15.06
21.72
6.66"
Tiny ImageNet
ResNet-34
64.53
9.78
10.35
15.79
12.82
11.21
12.16
17.64
27.95
10.311
4.2.2 PERFORMANCE COMPARISON ON LARGE DATASETS
In addition to the performance small datasets, the performance of the black-box distillation method on large datasets deserves further investigation. Data-free knowledge distillation has his- torically performed poorly Zhou al. (2020a); Chen et al. (2199) for datasets with large number of classes (c.g. Tiny-ImageNet and CIFARI00), since is very difficult to generate synthetic data with particularly rich class diversity. Thus, also conduct experiments classes (at least 100 classes). Table demonstrates the results all methods on CIFAR 100 and Tiny-ImageNet. As shown in Table it is also difficult for all these methods produce a good student model in the black-box scenario. However, our proposed IDEAL consistently achieves the best performance on these large datasets. For example, IDEAL outperforms the best baseline DFME by 16.95% on CIFAR 100 with ResNet-34. When compared with other baseline methods, our model achieves significant performance improvement by a large margin of over 18%.
datasets with more
4.2.3 PERFORMANCE ON MICROSOFT AZURE
100
Following the settings in DaST Zhou et al. 2020a)), also conduct KD in real-world scenario. In particular, we adopt the API provided by Microsoft Azure] (trained MNIST dataset) as the teacher model and uti- lize LeNet Lecun et al. (1998) as the student model. As illustrated in Fig. 2 our method con- verges quickly and is very stable compared to other methods. Actually, our method achieves over 98% test accuracy after 10,000 queries, which implies that our proposed method is also effective and efficient for real-worke APIs.
DaST
Acouracy (%g
80
ZSDB3
DFME
Mm
60
Ours
40
20
20
40
60
100
Query 250)
Figure 2: Transferring the knowledge of the on- line model on Microsoft Azure to the student.
4.2.4 PERFORMANCE UNDER DIFFERENT QUERY BUDGET
In previous experiments, we consider training the student model with limited query budget. As described in previous studies Zhou et al. (2020a); Truong et al.] (202 [Wang] (2021), these methods require millions of queries to the black-box model. Therefore, we have in- creased the number of queries of other base- line methods to provide a more comprehensive comparison, but without increasing the number of queries in our method. More specifically, increase the number of queries required by other baseline methods (ZSDB3 and DFME) on
Query budget = 200K
Querr budget G = 10M
ZSDB3
Peoo9e
DFME
Z5DB5
16
90
DRME
8
12
20
20
Quary (H 10K)
Quary (x 600K)
Analyses of method and other com- parison methods (ZSDB3, DFME) with a small query budget (Q 200K) and large query bud- get (Q = 10M).
Figure 3:
CIFARI0 10 dataset from 200K to 10M. Fig. 3 illustrates the training curves of these methods with Q = 200K and Q = 10M, respectively. Note that 7SDB3, DFME can achieve the highest ac- curacy of 56.39% and 57.94% respectively (right panel in Fig. 3). when large number of queries are involved. By contrast, our approach achieves 63.77% with only 0.02x the query budget of both ZSDB3 and DFME. It validates the effectiveness of our method perform query-efficient KD.
8

---

Published as a conference paper at ICLR 2023
Table 4: Ablation studies by cutting of different modules.
Method
MNIST
SVHN
CIFARIO
ImageNet subset
Ours
96.32
86,91
68.82
57.95
w/o Linfor
95.76
83.21
62.68
54.31
w/o Tige
15.21
12.48
10.68
9.84
w/o generator re-initializing
95.26
80.53
55.32
51.25
4.2.5 VISUALIZATION OF SYNTHETIC DATA
In this subsection, present synthesised examples of ZSDB3, DFME, and our method evaluate the visual diversity. As can be seen in Fig. 4. images generated by ZSDB3 are all of very low quality, which cannot show any meaningful patterns. And the image samples generated by ZSDB3 and DFME both exhibit very similar patterns, which implies that the synthetic data has low sample diversity. By contrast, proposed approach can synthesize more meaningful and diverse data. We observe that the images generated by method have more different patterns, which indicates that our proposed IDEAL can synthesize more diverse data. It also proves that it is feasible and ef- fective for our model replace with S in generator training without gradient estimation.
0
8
2SDR3
DFME
Onrs
Figure 4: Visualization of data generated by dif- ferent methods on MNIST. Our approach can syn- thesize more diverse data, there is clear visual distinction between samples in different classes.
4.2.6 EFFECT INVESTIGATION OF DIFFERENT MODULES
In this section, we evaluate the contributions of different loss functions in Equation 6 used during data generation, and discuss the effect of re-initializing the generator. As shown in Table removing both the generator and information loss Lin can lead to sigmificant performance degradation. Moreover, our model suffers from an obvious degradation when the generator re-initializing strategy is abandoned, especially on SVHN, CIFAR10, and ImageNet-subsst. In fact, since the generator reinitialized in each epoch during training. our method does not depend on the generator from the previous round. In other words, we do not need to train the generator and the student model adversarially, and therefore we do not require large number of training iterations to guarantee convergence. Besides, we find significant degradation when we remove Lee, which demonstrates its effectiveness the data generation. The ablation experiments verify that all modules are essential in our method.
4.2.7 EFFECT INVESTIGATION OF Fgg
We also conduct ablation study to investigate the effect of Eg on the data generation stage. As show in Table 5 in Appendix, we modify the value of Eg and report the top-1 test accuracy. We can observe that too small or too large Fgg is hard to obtain the optimal solution. To better understand the impact Eg, we show the t-SNE visualization of synthetic data in Fig. 5 in Appendix. More detailed results can be referred to the Appendix A.O.I
5 CONCLUSION
In this paper, we propose query-efflcient Data-free lEarning from blAck-box model.s (IDEAL) in order to query-efficiently train a good student model from black-box teacher models under the data- free and hard-label setting. To the best of our knowledge, challenging to date. Extensive experiments on various real-world datasets show the effectiveness of proposed IDEAL. For instance, IDEAL improve the performance of the best baseline method DFME by 5.83% on CIFARIO dataset with only 0.02x x the query budget of DFME. We envision this work milestone for query-efficient and data-free learning from black-box models.
setting is the most practical and

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENT
This work funded by Sony Al.
REFERENCES
Kartikeya Bhardwaj, Naveen Suda, and Radu Marculescu. Dream distillation: A data-independent model compression framework. CoRR, abs/1905.07072, 2019. URL http://arxiv.orrg
abs/ 11905.90.000 07072
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3514-3522, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, 2009.
Gongfan Fang. Yifan Bao, Jie Song, Xinchao Wang. Donglin Xie, Chengchao Shen, and Mingli Song. Mosaicking to distill: Knowledge distillation from out-of-domain data. Advances in Neural Information Processing Systems, 34, 2021a.
Gongfan Fang, Jie Song, Xinchao Wang, Chengchao Shen, Xingen Wang, and Mingli Song. Con- trastive model inversion for data-free knowledge distillation. CoRR, abs/2105.08584 2021
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-18)) 2021.
Matan Haroush, Itay Hubara, Elad Hoffer, and Daniel Soudry. The knowledge within: Meth- ods for data-free model compression. In 2020 IEEE/CVF Conference on Computer Vision
and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 8491-8499.
Computer Vision Foundation IEEE, 2020. doi: 10. N0.109//CVRR-22002200000000 https::/cpenaccess..tt thecvf com/ haaam/contett_ttee CVPR_ 20tp/22000///////// The Khowleegge_Wittie Within Methods for Data-Free Model Compression.
URL
CVPR CVPR_2020_pp 20e02_0ape.hhtmm html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.00 |tthttss::/ /tt:::///oooooooo www.i.orgg/i.. 1109//1PR.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.02555 2015.
Yuenan Hou, Zheng Ma. Chunxiao Liu, Tak Hui, and Chen Change Loy. Inter- region affinity distillation for road marking segmentation.
2020 IEEE/CVF Con-
ference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA,
June 13-19, 2020, pp. 12483-12492 Computer Vision Foundation IEEE, 2020. doi:
10.1109/CVP4426002000 202.000002000
URL https: httpas////peacccsss.ees tww.wh.ecw..... com/
content_cvve CVPR Bateeeett_ccee_222//eeee 2020022tm/000ooo////
Inter- TeeeRRegooon__ae www.Affityiissiilllletttt CVPR 2020 paper.htm.
For Road Marking BaiiniinnSSggetttttaaa
SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu. De-rrd: A knowledge dis-
tillation framework for recommender system. In Proceedings of the 29th ACM International Conference on Information Knowledge Management, pp. 605-614, 2020.
Sanjay Kariyappa, Atul Prakash, and Moinuddin K. Qureshi. MAZE: data-free model stealing attack using zeroth-order gradient estimation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021. virtual. June 19-25, 2021, pp. 13814-13823 Computer
Vision Foundation / IEEE, 2021. URL htRt:tt:::2 http:///es.eeccc...eeeeee thecv com/ contonttnt// CVPR202//fxm/KKxx/aappp MAZE Data- Model Beloode_Steeelle Stealine Attack Us 1agnneeertth--rdrrrrraa
10

---

Published as a conference paper at ICLR 2023
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for gen-
erative adversarial networks. In IEEE Conference on Computer Vision and Pattern
Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 4401-4410. Computer Vision Foundation IEEF, 2019. doi: 10.1109/CVP.22019.000 http:://peeaccest thecccessss..eec..... com content, CVPR_ CWVPR_2199/16 htt_2209/htmmm/aaraa... Style Generator or wwterrrrehhhhiieee for Generative Adversariai
URL
Networks CVP_s____________ 2019 paper.htm.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Peter I.. Bartleut Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 1106-1114, 2012. URL https: http::sss//.rroceiiggg..ooo wew.eeeep...nns.nn.............. http:/eerrrppp...pppppeeeeeeeeee httpc//ppppee/////aaaaaaa//
Abstract .ttll
Wonbin Kweon, SeongKu Kang, and Hwanjo Yu. Bidirectional distillation for top-k recommender system. In Jure Leskovec, Marko Grobelnik, Marc Najork, Jie Tang, and Leila Zia (eds.), WWW '11: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pp. 3861-
3871. ACM IW3C2, 2021, doi: 00.1145/3442381.3449887 1145/3442 3449878
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7);3, 2015.
Y. Lecun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied to document recog- nition. Proceedings of the IEEE, 86(11) 36(11):228-2224 1998. doi: 10. 109/5/726791.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, pp.
1998.
Boren Li, Po-Yu Zhuang, Jian Gu, Mingyang Li, and Ping Tan. Interpretable foreground object search knowledge distillation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings. Part XXVIII, volume 12373 of Lecture Notes Computer 0100779978--00058644---88
Science, pp. 104. Springer. 2020. doi: 7/doi. org. 0t0p:/000000017777----0--00----0
URL https:
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learn- ing: Training clean models poisoned data. Advances in Neural Information Processing Sys- tems, 34, 2021.
Paul Micaelli and Amos J. Storkey, Zero-shot knowledge transfer via adversarial belief matching, In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alhé--Buc Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, pp. 9547-9557, 2019.
Jonghwan Mun, Kimin Lee, Jinwoo Shin, and Bohyung Han. Learning to specialize with knowledge distillation for visual question answering. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Biandn Bianchi, and Roman Gamett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8092-8102, 2018. URL
hew.eeeeeedinn.........ee...ee nwwrins.ii..... htc/paperr221//aatt. html
Ndapandula Nakashole and Raphael Flauger. Knowledge distillation for bilingual dictionary in- duction. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 2497-2506. Association for Computational Linguistics, 2017. doi: 10. 10.18653/w1/117-12664 URL | http::tppp/............pp //t.///ii.oo. aaeii.rg//11 /10. 1ww111/////11.//

---

Published as a conference paper at ICLR 2023
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Heterogeneous knowledge distillation using information flow modeling. In 2020 IEEE/CVF Conference on Computer Vision and
Paitern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 2336-2345.
Computer Vision Foundation IEEE, 2020. doi: 10. D0110///CVP4440000000000000
URL
thecvf..e...cc.... wwm/ content CVPR 2020 10120/htmm/Passali. Heterogeeneos Knowledge Distillation Using Inf Information. Flow
Modeling Sww_iinng__PRR___ 2a0__22200_ppp_00 Satao_ppperr.th.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arx.w.orr//bb////
6550
Dennis W Ruck, Steven K Rogers, and Matthew Kabrisky. Feature selection using a multilayer perceptron. Journal of Neural Network Computing, 2(2):40-48, 1990.
Jiayi Shen, Haotao Wang. Shupeng Gui, Jianchao Tan, Zhangyang Wang, and Ji Liu. UMEC: unified
model and embedding compression for efficient recommendation systems. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. bH_RSh
OpenReview.net, 2021. URLLhttpps//openeeeieeee wttp:viwwwwweee//outmte...
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL
abs/ / 1409. 1556
Jean- Man-BBatiste Truong, Pratyush Maini, Robert Walls, and Nicolas Papernot. Data-free model extraction. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 4771-4780, 2021a.
Jean-Baptiste Truong, Pratyush Maini, Robert Walls, and Nicolas Papernot. Data-free model extraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 4771-4780, 2021b.
Shuai Wang, Kun Zhang, Le Wu, Haiping Ma, Richang Hong, and Meng Wang. Privileged graph distillation for cold start recommendation. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '1: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 1187-1196. ACM, 2021a. doi: 10.1445330483534662999 URL https: 7/doi wrw..org//...o 1ee.111154444438 3462929
Wenxuan Wang, Bangjie Yin, Taiping Yao, Li Zhang, Yanwei Fu, Shouhong Ding, Jilin Li, Feiyue Huang, and Xiangyang Xue. Delving into data: Effectively substitute training for black-box attack. In IEEE Conference Computer Vision and Patter Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 4761 Computer Vision Foundation / IEEE, 2021b. URL https: : hptp//ccpccccs..epppp thecvf.ccc com wttp///et/ttttt///t////// http://w2221/mmmmp// h/tpm///mtm////gggg//// Delving into Iatt_Daata____aaa Effectivee Str.SStsttttterrraanaae for Borpaiiiinooo_Baaaaeoooa Attack KEtack_CVPR_C. 2021 paper.ppppp......... html
Zi Wang. Zero-shot knowledge distillation from a decision-based black-box model. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 10675-10685, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 7708.07777 2017.
Shoukai Xu, Haokun Li, Bohan Zhuang, Jing Liu, Jiezhang Cao, Chuangrun Liang, and Mingkui Tan. Generative low-bitwidth data free quantization. In Andrea Vedaldi, Horst Bischof, Thomas
12

---

Published as a conference paper at ICLR 2023
Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII, volume 12357 of Lecture Notes in Computer Science, pp. 1-17. Springer, 2020. doi: 10. https::/ddo .rrg 1007 --00888---000--1--------0---
1. URL
Jingwen Ye, Yixin Ji, Xinchao Wang, Xin Gao, and Mingli Song. Data-free knowledge amal- gamation via group-stack dual-gan.
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle. WA, USA, Jume 13-19, 2020, pp. 12513- 12522. Computer Vision Foundation IEEE, 2020. doi: 10.1109/CVVP422002222222 202000012022 URL
https: / http:/ccpppccccss..eeoos thecvf. com/ content_ctttttoeceeee CVPR 20l__222//100 /htmll /Ye_
Data FrraaFFeeeeee Knowledge Amalgamation, wiwavvvavavvaaevaeeee Group Stack Dual
GAN CVPR
2020 paper.html
Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8712-8721, 2020.
Jaemin Yoo, Minyong Cho. Taebum Kim, and Kang. Knowledge extraction with no observable data. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' F'Aché-BBce Emily Fox, and Roman Garnett (eds.), Advances Neural formation Processing Systems 32: Anmual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver. BC, Canada, pp. neurips. CC/ pappr//019/hast. .html
2701-2710, 2019. URL
Mengran Yu and Shiliang Sun. Fe-dast: Fast and effective data-free substitute training for black-box adversarial attacks. Comput. Secur, 113:102555, 2022. doi: 00.1016/.cose.2221.102222 URL
1025555
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao
Wu. Dense: Data-free one-shot federated learning. Advances Neural Information Processing Systems.
Jie Zhang, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Lei Zhang, and Chao Wu. Towards efficient data free black-box adversarial attack. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15115 15115-15125, June 2022.
Mingyi Zhou, Jing Wu, Yipeng Liu, Shuaicheng Liu, and Ce Zhu. Dast: Date-free substitute training for adversarial attacks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 231-240, 2020a.
Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. More grounded image captioning by distilling image-text matching model. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 4776- 4785. Computer Vision Foundation IEEE, 2020b. doi: 10.1 1009////RR220002000000000
URL
https:://peeaccesst wwwcrf..eeccccccccee com httpecccv..coo/oooeettteeee wttpoo/mccooeet_ceeeeeeeeee SWVVPP_2020/bmm//2aaaa More____________
Grounded Image http://aamaaaaaaaiiiangooooo www.by__sssllinnn.ee. Baciio_mmaeeeeee Text, Matttt__acching._.
13

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.0.1 EFFECT INVESTIGATION OF Eg.
In Fig. 5 clearly, the student can easily identify the synthetic data when Egg 50 (the training accuracy on synthetic data is 100%, while the test accuracy on CIFARIO is 58.69%). but when Eg = 10, the student cannot distinguish the data accurately (the training accuracy is 63.78%, while the test accuracy is 68.82 (8.8.%). We guess that, small value of Eg leads to poor quality of the generated data (a large loss) while large value of Eg leads to student model that overfits to the synthetic data. Thus, from the empirical experiments in Table 5 Fgg = for CIFARIO and ImageNet subset.
we set Eg 5 for MNIST,
Iterations (Fig)
Teacher
Student
3
10
30
50
MNIST
AlexNet
LeNet
65.36
96.51
95.24
88.75
80.15
CIFARIO
ResNet-34
ResNet-18
30.25
57.56
68.82
62.45
58.69
ImageNet subset
VGG-16
ResNet-18
26.38
52.59
$7.95
48.67
41.73
Table 5: The influence of different number of iterations Eg on data generation. We report the top-1 test accuracy (%).
20
Beraation
10 iterations
iterations
Loss generator
Figure 5: T-SNE visualization of synthetic data on CIFARIO and the corresponding training loss of the generator. When Fgg 10, the features well separated, indicating that the student can still learn from synthetic data.
A.0.2 EFFECT OF THE GENERATOR
For fair comparisons, we use the same generator StyleGan for methods in our experiments. We also introduce the effects of different sizes of generators as shown in Table where DCGAN, Style- GAN and Transformer-GAt have small, medium and large parameters. Different generative models have negligible effect the performance of method. Besides, our method still outperforms the best baseline when using generators with different sizes.
A.0.3 CLASS IMBALANCE IN SYNTHETIC DATA
To avoid class imbalance, we generate the same number of samples per class. As illustrated in Table even we use some SOTA re-weighting methods to assign different weights to our model, the accuracy drop caused by the class imbalance can not be entirely eliminated. Hence, it is effective to consider class- balanced generation for each class, i.e., the number of synthetic samples per class balanced.
Generator
DCGAN
StyleGAN
Transformer based GAN
DFME
31.23
34.65
33.61
Ours
55,81
57.95
57.62
Table 6: The effect the generator.
14

---

Published as a conference paper at ICLR 2023
Method
CIFARTO
SVHN
Ours+LDAM
62.14
82.24
Ours+CB-Focal
63.31
81.58
Ours(same number of samples in each class)
68.82
87.65
Table 7: The influence of class imbalance in synthetic data.
Method
Model training
SVHN
Method A
Train from scratch with the synthetic data
22.64
Method.B
Using out-of-domain data to distill (CIFARIO)
39.68
Ours
Using synthetic data to distill (our method)
87.65
Table 8: Domain gap between synthetic data and original data
A.0.4 DOMAIN GAP BETWEEN SYNTHETIC DATA AND ORIGINAL DATA
We find that there is significant difference between the data synthesized by our method and the test set. As shown in Table 8 we introduce more detailed experiments to investigate such distribution discrepancy. Specifically, (1) Method denotes performance when training from scratch with the synthetic data (i.e. no distillation) and directly evaluating the test set. (2) Method_B denotes the performance of using out-of-domain CIFARIO (i.e. no synthetic data) to perform knowledge distillation. (3) and Ours denotes the performance of using synthetic data to perform distillation.
Obviously, Ours outperforms Method_A by large margin over 65%. validates the significant distribution discrepancy between the synthetic data and the test set, and Ours effectively address such domain gap via knowledge distillation. Besides, Ours performs better than domain adaptation strategy Method B, which also verifies the effectiveness of our model for black-box.
A.0.5 DETAILED ALGORITHM
Algorithm Training process IDEAL
Input: Generator G with parameter 0g. student model S with parameter Os. teacher model T, number training rounds Eg for generator each epoch, number classes C, training epochs E, learning rate of generator 1/z, learning rate of student model rJs. scaling factor X, and batch size B.
for = E do
Stage 1: data generation for round 1,... Egg do
Sample a batch of noises and labels {zi, yi
Generate a batch of synthetic samples { | by Equation
Compute Lgen by Equation 6
Update fg by minimizing Ligen
end for
Stage 2: model distillation
Sample a batch of noises {
Generate a batch of synthetic samples { by Equation
Compute Lmd by Equation 9
Update Os by minimizing Cmd
end for return 0s
15


Reviews:
[{"review_id": "DdeOSIrMXO0", "reviewer": "Reviewer_XB73", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The paper proposes a zero-shot Knowledge Distillation (\"ZSKD\") approach (i.e., does not require any samples from teacher's training data).\n- Similar to DFME (Truong et al.) and MAZE (Karyiappa et al.), the approach consists of two phases per epoch: (a) training a generator to produce synthetic labels; and (b) training a student network using generator's synthetic images annotated by a black-box teacher model. Unlike previous approaches, the proposed approach is query-efficient since gradients for the generator no longer requires 0-th order gradient estimates. Rather the gradients are generator by backprop-ing through the partially trained student network.\n- Results are verified on a number of datasets (CIFAR10, TinyImages, ...) and compared with many zero-shot KD baselines.", "summary_of_the_review": "Overall, the paper proposes a straight-forward approach with strong results (significant improvements over ZSKD baselines). However, a big concern I have is that the rationale of the approach is not evident to me (why does it work, given large losses in ideal circumstances). A somewhat minor concern I have is unfairly using KD as a straw-man (e.g., KD requires white-box teacher).", "strength_and_weaknesses": "### Strengths\n\n**1. Insight: Identifying query-inefficient bottleneck**\n- Although the proposed approach follows a similar approach to baselines (i.e., iterative training of generator and student model), it rightly identifies and tackles a relevant pain-point: estimating the gradients to update the generator incurs a large query budget. It appears that the drastic improvement in query efficiency can be attributed to addressing this pain-point.\n\n**2. Results - significantly better than prior art**\n- I appreciate the extensive comparison of the proposed approach with 7+ baselines and on 7+ datasets. The results are furthermore promising: the proposed approach results in drastic improvements over baselines e.g., 37.91→68.82 in CIFAR10.\n\n### Concerns\n\n**1. (Major concern) Counter-intuitive formulation: Generator loss**\n- I found counter-intuitive how the generator is trained in stage 1. For simplicity, let's assume a perfect generator and student network. In this case, the $L_CE$ would appear to be extremely large (in expectation) -- since the GT target $y$ is randomly drawn and compared with a random generation ($\\hat{y} = S(G(z)), z \\sim N(0, I)$). (Side-note: wouldn't a conditional generator make more sense?)\n- On a more general note, while I understand the algorithm steps, I cannot understand the reasoning behind it. Why is the gradient signal though a noisy student network with mislabeled examples informative enough to perform ZSKD?\n\n**2. KD vs. Model Stealing approaches**\n- I get the impression that the paper tackles the *Model stealing/extraction* problem (black-box teacher, query budget, \netc.), but constantly and unfairly compares to *Knowledge Distillation*.\n- For instance, \"KD methods are based on several unrealistic assumptions ... access teacher's training data ... white-box teacher\". I don't think this is a correct claim, since KD approaches are predominantly tailored for model compression -- Hinton et al. 2015 claim that it can be used to \"... compress the knowledge in an ensemble into a single model...\". In which case, it's a perfectly valid assumption to use a white-box teacher and its corresponding training data.\n- In contrast, the proposed approach is more consistent with model stealing works (several references missing here in the paper; please fetch and discuss relevant citations from DFME) where indeed the teacher model is a black-box prediction API. Unfortunately, the results fall short here -- randomly using a pool of publicly-available images over generated images results in significantly better accuracy scores as well as sample efficiency (typically with 50K queries, see references in Kariyappa et al., CVPR '20).\n- Overall, my concern is that the paper considers KD works as a straw-man, as opposed to model extraction literature. However, I'm slightly overlooking this as the paper is \"data-free\" and does not rely on a publicly-available image data pool (at the expense of results degradation).\n\n**3. Some misc. concerns**\n- Table 1 \"with a limited number of queries\": please mention how many queries per method.\n- \"Contributions ... new problem ... training models with hard-labels\": This has been studied plenty of times before e.g., Tramèr et al. '16, Knockoff Nets '20)\n- APIs return only top-1 class: This is another unfair claim. Plenty of pay-per-query cloud APIs provide probabilities e.g., [Google Cloud](https://cloud.google.com/vision/docs/labels).\n- The paper many times mentions a \"distillation loss\", but rather refers to simple cross-entropy given that there is no temperature-scaling factor.\n- (Nitpick) Would be nice to have an additional \"upper bound\" column in Table 2, where the numbers reflect the teacher model trained on GT data with the relevant budget.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BE_T--a2poK", "reviewer": "Reviewer_Ruz3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on a very interesting issue: how to learn from black-box models using label-only data? For this purpose, they propose a novel method called IDEAL, which aims to query-efficiently learn from black-box model APIs in order to train a good student model without any real data. In detail, the proposed method trains the student model in two stages: data generation and model distillation. Their proposed IDEAL is query-efficient, as it does not require any query in the data generation stage and queries the teacher only once for each sample in the distillation stage. Results can well support that the proposed method consistently outperforms all the baselines.", "summary_of_the_review": "A strong paper in black-box KD. Technical details are clearly described. It seems like a pioneering contribution towards black-box KD. Hence, I would like to vote for acceptance.", "strength_and_weaknesses": "Strengths：\n\n- This paper focuses on learning from label-only black-box models, an interesting but less explored topic. From my understanding, the proposed method is by far the most practical one that considers data-free, label-only, black-box, and query-efficient learning. In Table 1, it is clear that query-efficiently training a good student model from black-box models with hard labels is very practical, but challenging. Generally, I think the investigated problem is sound and interesting. I think this can be an extremely strong paper in black-box KD.\n\n- The idea is novel and straightforward. The approach is technically sound. The experiment showcases solid performance improvement over baselines. Particularly in Table 2, the proposed  method has significantly better results than other baselines. In Fig. 3, this paper provides convincing and solid comparisons between baselines with small and large query budgets.\n\n-The paper also conducted very detailed and convincing ablation studies. The ablation studies are quite thorough in my view. It’s easy to understand why this method works.\n\nOverall, this paper is very interesting and to my knowledge novel. The presentation is clear and easy to follow. Technical details are clearly described. It seems like a pioneering contribution towards black-box KD. \n\nWeaknesses:\nI didn't see major weakness in this paper actually. I am just curious about below questions:\n\n1. How different generator sizes affect the results. Since in my opinion, synthetic data are affected by the quality of the generator. Maybe the authors can give some explanation on this and also in the future version some supplementary materials can be provided for more detailed analysis.  \n\n2. Additionally, it will be interesting to investigate why DFME and ZSDB3 perform much worse than IDEAL, what's the cause? \n\n3. Also, is there any domain gap between synthetic data and original data? What will happen if we train the model from scratch with synthetic data?\n\n4. Last, in the Balancing section: why it is necessary to generate the same number of samples in each class? Is there an imbalanced data distribution? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6XRqvVyY8Ee", "reviewer": "Reviewer_S4b4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of query-efficient learning from a black-box teacher model in a data-free way. This paper considers the additional constraint that only the hard labels (categorical predictions) of the teacher are available, rather than the soft-labels. The authors propose the IDEAL method for this problem which operates in two stages. First, the method uses the student model to train a generator that generates synthetic data. In the second stage, the synthetic data is used to perform hard-label distillation, i.e., train a student model so that its predictions match those of the teacher. The authors present empirical evaluations that demonstrate the query efficiency and effectiveness of the proposed approach.", "summary_of_the_review": "This paper addresses a challenging problem that is motivated by real-world applications. The authors propose an approach that performs exceptionally well in practice compared to state-of-the-art. I have some concerns regarding the sensitivity of the method to its hyperparameters and some clarifying questions that I raised in the sections above. Overall, I lean towards acceptance and would be willing to raise my score if my concerns are adequately addressed.", "strength_and_weaknesses": "## Strengths\n* The problem of query-efficient, data-free learning from a black-box teacher that outputs only hard labels is well-motivated by real-world examples (e.g., Google BigQuery). It is a challenging problem that is of high relevance to the ML community.\n* The proposed approach contains novel components compared to prior work and using the student model to generate a diverse set of synthetic data is interesting.\n* Empirical evaluations on various data sets are presented that support the improved effectiveness of the proposed work. Based on the results, IDEAL improves over the state-of-the-art by more than 20% on the evaluated scenarios, which is quite impressive. This trend holds for larger data sets containing a larger number of classes as well. The hyperparameters for the experiments are reported for reproducibility.\n* The authors present ablation studies that help justify the various components of the method (Table 4).\n* The paper is well-written with a clear exposition overall. For example, the accompanying visualization of the data generated by different methods (Fig. 4) is quite interesting and helps understand the benefit of the method.\n\n## Weaknesses\n* The proposed two-stage approach of generating synthetic data -> distilling knowledge using generated data appears in prior work (Wang, 2021). The novelty seems to lie in the synthetic data generation stage.\n* The synthetic data generation seems to be highly sensitive to the number of epochs $E_{\\mathcal G}$ that the generator is trained for (see Appendix A.0.1). It has to be not too high and not too low. This is quite puzzling and the explanation in Sec. A.0.1 is not very compelling. My understanding of the synthetic data generation stage is to generate a diverse set of examples with varying (student predicted) labels. Why would training the generator for more epochs lead to “overfitting of the student?” Shouldn’t it be more conducive in generating a diverse set of synthetic data points?\n* Standard deviations of the results in Sec. 4 averaged over 3 trials are not reported.\n* Why is a scaling factor of $\\lambda = 5$ used for the experiments? How was this choice made and how sensitive is the algorithm to this choice?\n* How does the method fare in computational complexity relative to the compared approaches? My understanding is that the two stage process is done on a per-epoch basis, so it is not clear how much of a computational burden this imposes. To add to this concern, the authors mention downsizing the original images for the ImageNet subset for “fast training.” \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6DAijo2qBBm", "reviewer": "Reviewer_Epxx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose the “IDEAL” algorithm that can improve the distillation process when there is no data to be used for the distillation process, and also when the teacher provides only the hard-labels (no soft predictions over the labels). IDEAL employs a generator to generate examples on-the-fly in this data-free scenario. The generator aims to balance the class distribution and fit to one of the classes while the distillation progresses. In this particular setting (no-data and hard-labels), authors claim IDEAL can reduce the number of examples to be annotated significantly and still can achieve better performance on the given task.", "summary_of_the_review": "As discussed above, the paper has some contributions to the community. However, the paper does seem to be limited in a very specific setting and lack the clarity/quality for the acceptance bar.\n\n---\n\nPost-rebuttal: After reading author's response as well as other reviewer's recommendation, I agree that the paper has some original contribution to the community (although it's limited). Hence, I updated the score accordingly.", "strength_and_weaknesses": "**Strength**\n* Empirically strong in the particular distillation setup (no data, teacher can only provide hard-labels).\n* Utilizing the generator in the data-limited distillation setup has some novelty, particularly in the joint training of the generator during the distillation process.\n\n**Weakness**\n* The impact is limited on the very specific distillation setup (data-free & hard-label setting). Moreover, the algorithm can only be applicable for the classification set-up, and does not seem to scale up when there are a large set of labels (thousands or millions).\n* Authors seem to be over-claiming their impact. Particularly, I disagree with the author's claim that most KD assumes “users can directly access teacher’s training data.” Widely, KD focuses on a large set of unlabeled data because a teacher can provide pseudo-labels for them.\n* Writings can be improved to focus on their core motivation, core idea, and core techniques. Please see the “clarity/quality section.” Because of this reason, many of the design choices in Section 3 feel a bit arbitrary.\n* One of the baseline authors chosen, ZSDB3KD does seem to show 96.54% in their paper unlike the very low number in the current paper. Why is the number significantly different?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster