Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation
Authors: Kien Do
Hung Le
Dung Nguyen
Dang Nguyen
Haripriya Harikumar
Truyen Tran
Santu Rana
Svetha Venkatesh
Venue: NeurIPS 2022 Accept (2022)
Keywords: adversarial
data-free knowledge distillation
forgetting

Abstract: Data-free Knowledge Distillation (DFKD) has attracted attention recently thanks to its appealing capability of transferring knowledge from a teacher network to a student network without using training data. The main idea is to use a generator to synthesize data for training the student. As the generator gets updated, the distribution of synthetic data will change. Such distribution shift could be large if the generator and the student are trained adversarially, causing the student to forget the knowledge it acquired at the previous steps. To alleviate this problem, we propose a simple yet effective method called Momentum Adversarial Distillation (MAD) which maintains an exponential moving average (EMA) copy of the generator and uses synthetic samples from both the generator and the EMA generator to train the student. Since the EMA generator can be considered as an ensemble of the generator's old versions and often undergoes a smaller change in updates compared to the generator, training on its synthetic samples can help the student recall the past knowledge and prevent the student from adapting too quickly to the new updates of the generator. Our experiments on six benchmark datasets including big datasets like ImageNet and Places365 demonstrate the superior performance of MAD over competing methods for handling the large distribution shift problem. Our method also compares favorably to existing DFKD methods and even achieves state-of-the-art results in some cases.

Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation
Kien Do, Hung Le, Dung Nguyen, Dang Nguyen, Haripriya Harikumar, Truyen Tran, Santu Rana, Svetha Venkatesh Applied Artificial Intelligence Institute (A212), Deakin University, Australia do. thai.le, dungeng d.nguyen, h.rarikumar, svetha.venkatest. venkatesh/ @ deakin. edu.an
trayen.tran, santu, rana, svetha.
Abstract
Data-free Knowledge Distillation (DFKD) has attracted attention recently thanks its appealing capability of transferring knowledge from teacher network to student network without using training data. The main idea is to use a generator synthesize data for training the student. As the generator gets updated, the distribution of synthetic data will change. Such distribution shift could be large if the generator and the student trained adversarially, causing the student forget the knowledge it acquired at previous steps. To alleviate this problem, we propose simple yet effective method called Momentum Adversarial Distillation (MAD) which maintains an exponential moving average (EMA) copy of the generator and uses synthetic samples from both the generator and the EMA generator train the student. Since the EMA generator can be considered as an ensemble of the generator's old versions and often undergoes a smaller change in updates compared to the generator, training on its synthetic samples can help the student recall the past knowledge and prevent the student from adapting too quickly new updates of the generator. Our experiments on six benchmark datasets including big datasets like ImageNet and Places365 demonstrate the superior performance MAD over competing methods for handling the large distribution shift problem. Our method also compares favorably to existing DFKD methods and even achieves state-of-the-an results in some cases.
Introduction
With the development of deep learning, more pretrained deep neural networks have been released to the public [5] [0] I8 38 43). However, their superior performances often come with big sizes, causing difficulties in deployment of these pretrained networks on resource-constrained devices. This leads to the demand for transferring knowledge from a cumbersome pretrained source network (called "teacher") to compact target network (called "student") with a minimal loss of performance. This task regarded as Knowledge Distillation (KD) [99]
The original idea of KD is to make use of class probabilities predicted by the teacher which encapsulate the hidden correlation among classes as training signals for the student. Note that such "dark" knowledge [19 is generally not available if the student is trained directly raw data with one-hot labels. Later, various KD methods have been proposed improve the quality of knowledge transfer. For example, AT [47] matches the spatial attention maps at intermediate layers of the student and the
teacher. SPD 42 encourages the similarity between the student's 's and teacher's feature correlation
matrices. PKT [36] transfers the conditional probability between two samples computed via kernel density estimation on the feature space. RKD (5] exploits relational knowledge for distillation. VID
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

maximizes variational lower bound of the mutual information (MI) between the student's 's and teacher's 's representations. CRD 41 uses contrastive learning as a proxy for maximizing MI.
The common drawback of these methods is the reliance on samples from the teacher training set. However, in practice, accessing the original training data is usually infeasible due many reasons such as data privacy (e.g,, healthcare data containing personal information) or data regarded as intellectual property of the vendors. Addressing this critical issue, Data-Free Knowledge Distillation (DFKD) methods have been introduced [7] 27
[0 A common DFKD approach is to use generator network synthesize training data and jointly train the generator and the student in an adversarial manner [33 31 46). Under this adversarial learning scheme, the student attempts to make predictions as close as possible to the teacher's 's on synthetic data generated by the generator, while the generator tries to create samples that maximize the mismatch between the student's 's and the teacher's predictions. This adversarial game enables a rapid exploration of synthetic distributions useful for knowledge transfer between the teacher and the student. At the same time, it could also lead to large shifts the synthetic distributions, causing the student to forget useful knowledge acquired at the previous steps and suffer from performance drops 3
34
In this paper, we propose a simple yet effective method called Momentum Adversarial Distillation (MAD) to mitigate the large distribution shift problem in adversarial DFKD. MAD maintains an exponential moving average (EMA) copy of the generator which is responsible for storing information about past updates of the generator. By using synthetic samples from the EMA generator as additional training data for the student besides those from the generator, MAD can ensure that the student can recall the old knowledge, hence, is less prone to forgetting. Moreover, to reduce the negative effect caused by spurious solutions of an unconditional generator when learning on large datasets such as ImageNet, we propose to use a class-conditional generator that takes the sum of a noise vector and a class embedding vector as input, and train this generator with a new objective that suppresses the presence of spurious solutions. This technique requires only a small change in the generator's s architecture but enables MAD (and possibly other adversarial DFKD methods) to learn surprisingly well on large datasets. Through extensive experiments on three small and three large image datasets, we demonstrate that our proposed method far better than related baselines [1] in dealing with the large distribution shift problem. In some cases, MAD even outperforms current state-of-the- oftthe-rrt methods g 14].
2 Adversarial Data-Free Knowledge Distillation
Let T be teacher network pretrained some dataset Diain and S be fresh student network. Let T(.) and s(.) denote outputs of the teacher and student networks before the softmax activation, respectively. In Data-Free Knowledge Distillation (DFKI)), T to S so that S performs as well as or even better than T on the original test set Diest but with a constraint that no training data for S is available. An intuitive way to deal with such constraint is learning an additional generator network G that can generate synthetic data for training S from a noise distribution p(2) usually chosen to be the standard Gaussian distribution (0, I). Adversarial Belief Matching (ABM) [31] proposed an adversarial learning framework between S and G via optimizing the following min-max objective:
want transfer knowledge from
min max Ee-p(e) [Gx))
$
min max E E2-r( [<x(())],
(2)
S
where (x)(x) denotes the knowledge distillation (KD) loss, i.c., the discrepancy between S(x) and T(x). ABM, (x) is the Kullback-Leeis Leibler (KI.) divergence between class probabilities of T and S computed on a::
(x)( DKL (Tp(x) = Σ Tp(x)]c] (log [p(x)[c] - log (c())
where Tp(#) = softmax(T(x)) and Sp(#) = (oftmax(Sxx)) denote the class probabilities of and computed on x, respectively; and C is the total number of classes.
The core idea behind the optimization in Eq. Dis to encourage to generate samples on which the outputs of are very different from those of T (or the KD loss is large). Typically, the generated

---

T
G
L LDD
S
E
T
y
G
L LD
S
Figure An illustration of our proposed Momentum Adversarial Distillation (MAD) consisting of a teacher (T), student (S), class-conditional generator (G), EMA generator (G), and class embeddings (E). Networks with dashed borders (T, G) are not optimized during training. z and = are random noises sampled from N I). Ey and ey gathered from E at index y and y/ respectively.
samples have not been observed by S during training (otherwise, the KD loss will be small) and the learning task of S is to match on these novel samples. It is expected that in the ideal case, continuous adversarial exploration of G, S can be exposed to diverse enough set of synthetic samples which allows S to match with an arbitrarily small prediction error Diest-
In practice, we usually implement Eq. I by alternately optimizing S and G in nig and ng steps, respectively. Thus, avoid confusion later, refer nt--tep update of update stage of and similarly, ne-Step update of G as an update stage of G.
3 Momentum Adversarial Distillation
3.1 Handling large distribution shifts with an additional EMA generator
Although theoretically sound, the optimization in Eq. Thas practical problem: If the update of varies too much, the distribution of synthetic samples generated by will change significantly over two consecutive update steps of S, which in turns causes $ to catastrophically forget what it has learned at the previous stages 3 to adapt to the new update of G. order to address this problem, we propose maintain exponential moving average (EMA) of the generator G, denoted by G during learning and use synthetic samples from both G and G to train S as follows:
min La AoEzup(s) [AxD (G(=)) A(E [<xD()(='))
where No. A1 2 are coefficients. The parameters o of the momentum generator õ at the update stage of S (after ng steps update of G) computed as:
0
0 (1- 0g
(5)
where a (0 < < is the momentum. If is close to 1, G will change very slightly compared to G. Therefore, by using synthetic samples from G as additional training data for besides those from we can alleviate the large exploratory distribution shift caused by the large update of G and achieve stable update We name our proposed method Momentum Adversarial Distillation (MAD). See Fig. for an illustration.

---

3.2 Enabling MAD to learn well on large datasets
During our experiments, we observed that MAD, with the unconditional generator described in Section 2 is not able to learn well on large datasets such as ImageNet. Our hypothesis is that in case of large datasets, the objective of Eq. iinduces very large number spurious solutions, which hampers the learning of G. To see this, first let us recall the objective of G which is maximizing the KI. divergence between Sp(z:) and Tp(æ) over some synthetic sample x: G(z). If we, for example, assume that there are classes in total and Tp( is fixed at [1, 0, 0). then we could train to generate x so that Sp(x) is either [0. 1, 0] or [0, 0, 1]. In this toy example, we see spurious solutions for 3 classes. If dataset has 1000 classes like ImageNet, there will be at least 999 spurious solutions. Besides class numbers, larger input sizes also increase the space and the number of (spurious) solutions. Unfortunately, the more spurious solutions, the more likely G could jump from one spurious solution to another in successive update steps, causing instability in training G.
To overcome this problem. we propose to condition G on a class label y as G + +ey) where ey a trainable embedding of y. We found that using the sum of = and ey as input to instead of concatenation allows our method to learn much better, possibly because the noise in updating ey is absorbed into the stochasticity of z via summation rather than concatenation. We illustrate this idea in Fig. [ and provide an empirical justification in Appdx. [3.] Denoted by E the list of trainable embedding vectors for all classes (E (e1. Ec)), we train both G and E together by minimizing the following loss:
min La.E e - [- >x<Kk((z) + AGCNLL v) + X (ooonnRog(ee) (6) G.E
where Cat(C) is the uniform categorical distribution over C classes; X2, As, da 2 0 are hyperparam- eters; and ENLL(,, y) and LNommeg (Cy) are the negative log-likelihood and the norm regularization losses respectively, defined follows:
LNL.L. (ar, v) (p()))))
LNommReg (ey) max (Ie,) - v Vd..o).
(8)
where de denotes the dimensionality of ey: and 7 2 1 is a scaling hyperparameter. LNormRee (ey) servers as a constraint that restricts the norm of ey to be smaller than y x vTg. An explanation of the formula x vdz is provided in Appdx. C Intuitively, via minimizing LNLI (xr y). G could maintain its focus on predicting y throughout entire update stage rather than jumping between different spurious solutions. Besides, since y is sampled uniformly, the synthetic data will be evenly distributed among classes.
In case the teacher T has BatchNorm 20 layers, we can make use of the running mean t and running variance We of each BatchNorm layer E of T guide the data synthesis of G by adding the BatchNorm moment matching (BNmm) loss [5) below to La.8 (weighted by As 2 0):
CaNmm e
Σ llue - - + ||ws- - Well2:
where uu, We are the empirical mean and variance of the features at the BatchNorm layer é w.r.t. the batch of synthetic samples from G. During our experiment, we observed that using CBNmm improves knowledge distillation of the student
4 Related Work
In Data-Free Knowledge Distillation (DFKD), it is critical to synthesize data that are useful for transferring knowledge from T to S. Existing DFKD methods differ mainly in their objectives to guide the data synthesis. These methods generally fall into either the adversarial camp or the non-adversarial camp. Non-adversarial DFKD methods 77] 27 29
46) make use of certain heuristics to search for synthetic data that resembles the original training data Dr. For example, ZSKD (34 and DAFL 7 consider prediction probabilities of and the confidence of T as heuristics. In DAFL, synthetic data is generated via a generator G rather than being optimized directly like in 7SKD. KegNet [46] extends DAFL by making G conditioned on class labels. Adversarial DFKD
methods [8]
[4
[7 37
45 50 leverage adversarial learning to explore the data space more efficiently.

---

Most methods of this type are derived from the ABM [3]] discussed in Section 2 with additional objectives improve the quality and/or diversity of synthetic data. For example, RDSKD [7] uses
[14] uses an inverse contrastive loss to improve diversity; and DeepInversion [45] uses Batch Norm moment matching (BNmm) and DeepDream's inception losses
diversity-seeking loss [30]:
CMI
(total variation, 1.2)
32
to generate visually interpretable images.
29 45 that have been shown learn well on the full-size ImageNet dataset. Its success has inspired subsequent works on large-scale continual learning 39 and data-free object detection [6]. However, training DeepInversion on ImageNet is very teme-consuming because this method optimizes synthetic images directly and each complete
DeepInversion 45 one of a few DFKD methods 12
optimization requires very large number of iterations (about 20,000). LS- LS-GDDD (29) learns to generate synthetic data instead but requires one generator for each class (that is, 1,000 generators for 1,000 classes) to avoid the "mode collapse" problem and achieve good performance. FastDFKI
12 leverages meta-learning to speed up the training process significantly. On the contrary, by applying our proposed technique Section 3.2 our MAD only needs generator (compared to 1,000 of LS-GDFD and 6,000 training iterations (compared to x of DeepInversion) to achieve reasonably good performance on ImageNet (Section 5.2.2). In addition, our technique is orthogonal to and can combine with the meta-learning idea in 022 for further improvement.
Besides the DFKD methods discussed above, there a line of works that consider less extreme scenario in which unlabeled transfer sets are given for training. These transfer sets can be very different from the original dataset in terms of distribution and semantics and can be freely collected from open resources. Nayak et al. [3 analyzed various kinds of transfer sets ranging from random noise images to natural images and found that the target-class balance property of the transfer set and the similarity between the transfer set and the original training set play important roles in improving knowledge distillation results. Fang et al. [I] argued that although the transfer images can be semantically different from the original traning images, they still share common local visual patterns. Therefore, the authors proposed an interesting method called MosaicKD which combines different local patches extracted from the transfer images to craft synthetic mosaic images that capture the semantics of the original data while enjoy realistic local structures.
Adversarial training methods like GANs has been shown to suffer from the catastrophic forgetting problem of the discriminator 226 40 These works consider GANs training as continual learning problem and leverage existing methods to mitigate the issue. For example, Liang et al. (26) use either EWC 23 or SI [49 to enforce similarity between the discriminator's 's weights at the current step and at the last checkpoint, Hoang et al. [40] suggest some other tricks such as using a momentum optimizer (e.g., Adam (21) or applying gradient penalty [16] to the discriminator. In this paper, we consider the catastrophic forgetting problem of the student as a consequence of the large distribution shift problem caused by the generator. Therefore, we focus on regularizing the generator rather than the student. Besides, from this perspective, the catastrophic forgetting problem in GANs is indeed less severe than that in adversarial DFKD. is because in GANs, the discriminator is trained on real data with fixed distribution, which can somewhat reduce the effect of the distribution shift of fake (synthetic) data while in adversarial DFKD, real data is not available. Recent works that also address the student forgetting problem in DFKD like ours include DFKD-Mem B and
2). DFKD-Mem [ stores the past synthetic samples in a memory bank and uses these samples as additional training data for the student. Our method, on the other hand, uses an EMA generator to generate old synthetic samples on-the-fly. which is more memory efficient and adapts better to the student update. PRE-DFKD [2] models past synthetic data via VAE 2 and treats the decoder of this VAE as replay generator. However, training a VAE on a continuous stream of synthetic samples could be unstable and could lead to another catastrophic forgetting problem on its own. In addition, VAE is not effective for generating images with large size (e.g., ImageNet images) due to the "posterior collapse" problem (28).
5 Experiments
5.1 Experimental Setup
Datasets We consider the image classification task and evaluate our proposed method on 3 small image datasets (CIFARIO 24]. C0O (4), TinyImageNet (25)), and 3 large image datasets
(ImageNet [9], Places365 511. Food101
4). Details are provided in Appdx. A.I
5

---

Datasct
Arch.
Tea." a
Stu.
DAFL.
7SKT"
ADI
DEAD
DDAD*
DFQ
CMP
MAD
95.70
95.20
92.22
93.32
93.26
93.30
94.81
94,61
94.84
94.90
CIFARIO
94.87
93.95
81.55
89.66
89.72
92.01
92.52
92.64
78.05
77.10
74.47
67.74
61.32
69.43
75.04
77.01
77.04
77.31
CIFARI00
75.83
73.56
40.00
28.44
61.34
59.01
68.75
64.05
TinyIN
66.44
64.87
63.73
64.01
62.32
Table I: Classification accuracy (in %) of the student trained by different DFKD methods on 3 small image datasets. The teacher/student architecture settings are ResNee34/ResNet18 (V) and
WRN40- 1AN40-2/WRNI6-2 (+). and à denote results taken from [14] and [50], respectively. Tea. and Stu.
denote the teacher and student trained from scratch on Dinin- The best and second best results are highlighted in bold and italic, respectively.
Network architectures For the small datasets, we follow g 14 and consider ResNer33//eeNer11 [18] and WRN40-2/WRN16-2 [48] for the teacher/student. We observed that knowledge distillation with the WRN architectures more challenging than with the ResNet architectures, possibly because WRN40-2/WRN 16-2 have much fewer parameters than ResNet34/ResNetl
For the large datasets, we use the AlexNet architecture for both the teacher and student for fast training. Since AlexNet does not have any BatchNorm layer, we exclude the BNmm loss (Eq. 9 from the total loss of the generator in our experiments on the large datasets. Architectures of the generator w.r.t. different datasets are given in Appdx. A.4
Training settings of teacher For ImageNet, we make use of pretrained networks provided by PyTorch. For other datasets, we train the teacher from scratch. Detailed training settings of the teacher are given in Appdx. A.2
If not otherwise specified, we set the momentum a in Eq. 5 to 0.95 and the length of the noise vector to 256. We train the student S using SGD and Adam for the small and large datasets, respectively. We train the generator G using Adam for both the small and large datasets. To reduce the difficulty of training G for the large datasets, we pretrain G for some steps before the main KD training by setting the coefficient of LXD to O and only optimizing the remaining losses in Eq. 6 The generator G is classcconiitinnal (as described in Section 3.2) for the large datasets, and unconditional for the small datasets. We empirically found that this leads to better results. For further details about the training settings of MAD, please refer Appdx. A.3
Training settings of MAD
5.2 Results
5.2.1
Comparison with existing DFKD methods
we compare MAD with existing DFKD methods 7 datasets. The results of baselines are taken from [14 50). We checked the results of the teacher trained by us and found that they are quite similar to the results of the teacher reported in 14 and in Ta in Appdx. B. This means our results of MAD are comparable with those of the basclines. We see that MAD outperforms all the DFKD methods on CIFARIO and CIFAR 100 in case the teacher and student are ResNets 18 (2)) In case the teacher and student are WideResNets [48] (0), our method still achieves significantly better results than most of the baselines such as DAFL 7 and ZSKT [34] and only performs worse than CMI [4. We hypothesize it is mainly because of the differences in model design and training settings of CMI and MAD. For example, CMI has an additional contrastive- loss- based objective that encourages the generator's 's diversity while MAD does not. CMI also performs more updates of the student and generator per training step than our method"
In Table D
45 50 on the small
However, our ultimate goal for this section is not to show that MAD can achieve state-of-thh-art results all cases (which often requires intensive hyper-parameter tuning) but to verify the soundness of our implementation of MAD. In order to see clearly the advantage of MAD, we need to compare MAD with its related baselines under the same settings. This will be presented in Section 5.2.2
We could not find the supplementary material containing the training settings of CMI. However, from the offficial code provided by the authors] we saw that they set n. = 200 and nss = 2000 for NideResNet teacher/student on CIFARIO while we set R2 = 3 and nig = 60 (see Appdx.[A.3) In general, setting larger no and ni often leads better results (Section 5.3 but will increase the training time.

---

Dataset
CIFARIO
CIFARIO0
TinyIN
ImageNet
Places365
Food101
Arch.
Teacher
94.65
75.65
66.47
56.52
50,80
65.15
ABM
92.38
62.39
59.75
41.23
41.84
60.37
DFKD-Mem
92.09
61.25
58.66
43.30
42.38
61.25
MAD
92.64
64.05
62.32
45,48
43.67
61.74
Table 2: Classification accuracy (in %) of MAD and its related baselines on 3 small and 3 large image
datasets. The teacher/student architecture settings are ResNet34/ResNet1 88 (V), WRN40-2/WRN162
(D), and Alex (.). The teacher's results are from our own runs (Appdx. B.T. The best results are highlighted in bold.
MAD
10
DHiD-Van
12
15
Stp
(a) CIFARI00
(b) TmnyImageNet
(c) ImageNet
(d) Places365
Figure 2: Test accuracy curves of S trained via MAD, ABM, and DFKD-Mem on some datasets.
5.2.2 Comparison with related baselines
We consider related baselines of MAD which are ABM [31 and DFKD--Me 3 the student S with only synthetic samples from G. DFKD-Mem, on the other hand, stores past synthetic samples
ABM learns
memory bank, and uses samples from this memory bank (dubbed "memory samples") as well as those generated by G as training data for S. We trained ABM and DFKD-Mem using exactly the same settings for training MAD. For DFKD-Mem, we set the memory size to 8,192. For other memory sizes, the results remain relatively similar as shown in Appdx. B.2
From Table and Fig. it is clear that MAD significantly outperforms both ABM and DFKD- -Me on all datasets. In addition, the performance gaps between our method and the two baselines tend to be larger for larger datasets. For example, MAD achieves about 1.5/2.8% 2.5/3.6%, and 4.2/2.2% higher accuracy than ABM/DFKD-Ment on CIFAR 100, TinylmageNet, and ImageNet, respectively. These empirical results suggest the importance of the EMA generator in mitigating the large distribution shift caused by G.
In this experiment, found that DFKD-Mem often performs worse than ABM CIFAR100 and TinylmageNet. We found the decay of the student learning rate is the main reason for this. As shown in Fig. 3a. the distillation loss memory samples surges when the (student) learning rate is decayed and cannot recover if the new learning rate too small (Irs = le-4), which in contrast the distillation loss on synthetic samples from G or G (Figs. 3b 3c This implies potential issue of storing old samples in a memory bank instead of using the EMA generator as memory samples could be completely out-of-date if the student suddenly change its state (e.g.. via learning rate decay). However, even when the learning rate does not change (c.g., from step 0 to step 100 on CIFAR 100 or on ImageNeuPPaace365). DFKD-Mem still performs worse than our method.
5.2.3 Comparing the changes in update of and G
In order to see whether G actually has smaller changes in update than G or not, we perform the following experiment: Let G, and G, be the versions of the generator and the EMA generator respectively at step f, and S be the version of the student at step t - T (0 < < t). We then measure two different average Jensen-Shanmee of S,-, and T on two separate sets of synthetic samples from Ge and G. We hypothesize that if G has a smaller change in the distribution of synthetic samples than G, will memorize the samples from G more and will match T better on those samples, which leads to smaller average JS divergence. This hypothesis clearly reflected on results in Fig. which verifies the reasonability of using G in alleviating the large distribution shift problem caused by G.
(JS) divergences between the prediction probabilities
7

---

0.6-
512
1024
2048
1221
0.9
-... M0.mm cmo
4098
0.5
KL Diverrgence
0.9
KI Divargence
0.4
0.7
0.7
0.1
0.5
0.5
step (x1000)
step (x1000)
step (x1000)
(a) LXD memory samples
LKD on samples from
(c) LKD on samples from
Figure 3: Distillation loss curves on memory samples (a) and samples generated by (b) and G (c). The numbers in the legends denote DFKD-Men with the corresponding memory sizes. "no_ema" and "ema" denote ABM and MAD, respectively.
1.19
320
6:275
0.
2.0
0.3
6.99
R/34
201.1T7 LET
312
210
240 4ET_ETTA
0.00
20.001
03001
(a) CIFARI00
(b) TinyImageNet
(c) ImageNet
(d) Places365
Figure 4: Average Pensen-Shannon divergences between the prediction probabilities of St-r and computed on synthetic samples from G, (labeled as "gen") and G, (labeled as "gen_ema") for different datasets. For CIFARIOS and TinylmageNet, we E [500, 5500] with step size of 500 and 50. For ImageNet and Places365, we set E [100, 1000] with step size of 100 and = 10. Note that the sudden drops step 5000 in (a), (b) correspond to the decay of the learning rate by 0.1.
5.2.4 Visualization of synthetic samples
In Fig. visualize the synthetic data generated by and G. Although the generated images visually realistic, they are visually diverse, suggesting no mode collapse has occurred during training MAD. Besides, samples generated by G are different from those generated by G which indicates that the EMA generator could act as a complement for the generator in our model.
5.3 Sensitivity Analysis
Below investigate some choices of hyperparameters that could affect the performance MAD. Unless stated otherwise, the dataset we use is CIFAR100.
Generator
EMA Generator
Generator
EMA Generator
(a) CIFAR100
(b) ImageNet
Figure Synthetic data generated by G and in case the original dataset CIFAR100 (a) and ImageNet (b).

---

63
Ao
1.0
0.3
0.1
0.0
1.0
64.05
63.26
62.43
1.08
0.3
63.65
A1
0.1
62.80
10.
50
50
120
160
0.2
0.4
0.6
0.8
0.95
momennkum
0.0
62.59
Figure 6: Test accuracy of our method w.r.t different values of student update steps ns (left), of the momentum a (middle), and of the coefficients (Ao, A1) in Eq. (right). The dataset is CIFAR100.
Different values of the coefficients in Ls We can control the relative importance of samples from over those from by changing the values of the two coefficients Ag and X in Ls (Eq. 4 As shown in Fig. (right), the best result is obtained when Åg d = 1 which is our default setting for MAD. Decreasing either Ag or X will lead to worse performance. In the worst case when Xg
and A = 1, the model learns for some epochs and then suddenly stops learning. The main reason is that often updates much slower than G and S so we need adversarial samples from to keep $ learning. Otherwise, S will overfit the samples from and learn nothing.
Different values of the momentum In Fig. 6 (middle), show the classification accuracy of the student of MAD with a in {0.2, 0.4, 0.6, 0.3, 0.95, 0.99, 0.999, 1.0). We observe that the performance of method increases when becomes larger as G is more different from However, a is too large (e.g., 0.999), the performance drops since the update of very small, resulting in almost no change in the distribution of synthetic samples. In about 50% prediction accuracy on test data.
1.0, MAD only achieves
Different update steps of S From Fig. 6 (left), we see that increasing the number of update steps for usually leads to better performances MAD since learns match T better, In exchange, the training time will increase.
Conclusion
We have presented Momentum Adversarial Distillation (MAD), simple yet effective method to deal with the large distribution shift problem in adversarial Data-Free Knowledge Distillation (DFKD). MAD maintains exponential moving average (EMA) copy of the generator which, by design, encapsulates information about past updates the generator and updated at a slower pace than the generator. By training the student samples from both the generator and the EMA generator, MAD can prevent the student from adapting too much to the generator at the current step and forgetting old knowledge learned from the generator at the previous steps. We have also described new type of conditional generator along with a new loss for training it which enable our model to learn well on large datasets. Our experiments on various datasets demonstrated the superior performance of our method over competing baselines that either use only generator or use memory bank in place of an EMA generator.
We note that our idea of using an EMA generator to mitigate large distribution shifts is general and can be generalized other machine learning problems besides DFKD. For example, the technique can be adapted for general continual learning (in which large distribution shifts can happen gradually or suddenly with clear task boundaries), and source-data-free domain adaptation. Our method is also well suited for DFKD with other data types such as video, text, or graph.
Limitations
In our current implementation of MAD, we perform additional forward pass through ? for every training step S. This increases the total training time of by about 40% compared to ABM. However, this technical problem can be somewhat addressed by first storing synthetic samples from and buffer before each training stage of and then training with samples from the buffer only. This will be left for future work.

---

Negative Social Impacts The DFKD problem may have negative social impacts related to data privacy as generated data could somehow reveal the original training data. However, as shown in Fig. 5 our proposed method does not attempt to improve the visual interpretability of synthetic data but addresses the large distribution shift problem. This target seems be harmless to the society.
Acknowledgement
This research was partially funded by the Australian Govermment through the Australian Research Council (ARC). Prof. Venkatesh the recipient of an ARC Australian Laureate Fellowship (FL170100006).
References
[1] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D) Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9163-9177 2019.
[2] Kuluhan Binici, Shivam Aggarwal, Nam Trung Pham, Karianto Leman, and Tulika Mitra. Robust and resource-effcient data-free knowledge distillation by generative pseudo replay. arXiv preprint arXiv:2201.03011 2022.
[3] Kuluhan Binici, Nam Trung Pham, Tulika Mitra, and Karianto Leman. Preventing catastrophic forgetting and distribution mismatch in knowledge distillation via synthetic data. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 663-671, 2022.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 21 213-229. Springer, 2020.
[6] Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and Jose Alvarez. Data-free knowledge distillation for object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3289-3298, 2021.
[7] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang. Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. Proceedings of the IEEF/VE International Conference on Computer Vision pages 2514-3522, 2019.
[8] Yoojin Choi, Jihwan Choi, Mostafa El- El-Khamy, and Jungwon Lee. Data-free network quantiza- tion with adversarial knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 710-711, 2020
[9] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
[11] Gongfan Fang. Yifan Bao, Jie Song, Xinchao Wang. Donglin Xie, Chengchao Shen, and Mingli Song. Mosaicking to distill: Knowledge distillation from out-of-domain data. Advances in Neural Information Processing Systems, 34:1 11932, 2021.
[12] Gongfan Fang, Kanya Mo, Xinchao Wang, Jie Song, Shitao Bei, Haofei Zhang, and Mingli Song. Up to 100x faster data-free knowledge distillation. arXiv preprint 1rXiv:2/12.06255. 2021.
[13] Gongfan Fang, Song, Chengchao Shen, Xinchao Wang, Da Chen, and Mingli Song. Data-free adversarial distillation. arXiv preprint arXiv: 2019.
10

---

[14] Gongfan Fang, Jie Song, Xinchao Wang. Chengchao Shen, Xingen Wang, and Mingli Song. Con- trastive model inversion for data-free knowledge distillation. arXiv preprint arXiv:2105.08584, 2021.
[15] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 026777 2017.
[16] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
[17] Pengchao Han, Jihong Park, Shiqiang Wang, and Yejun Liu. Robustness and diversity seeking data-free knowledge distillation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2740-2744. IEEE, 2021.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503 2015.
[20] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, In International conference on machine learning, pages 448-456. PMLR, 2015.
[21] Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: 1416.9880 2014.
[22] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[23] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barwikk, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3322 2017.
[24] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[25] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. Technical report, 2015.
[26] Kevin J Liang, Chunyuan Li, Guoyin Wang, and Lawrence Carin. Generative adversarial network training is continual learning problem. arXiv preprint arXiv:/811.11083 2018.
[27] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv: 2017.
[28] James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Understanding posterior collapse in generative latent variable models. 2019.
[29] Liangehen Luo, Mark Sandler, Zi Lin, Andrey Zhmoginov, and Andrew Howard. Large-scale generative data-free distillation. arXiv preprint arXiv:2012.05577, 2020.
[30] Qi Mao, Hsin- Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 1429-1437, 2019.
[31] Paul Micaclli and Amos J Storkey. Zero-shot knowledge transfer via adversarial belief matching. Advances in Neural Information Processing Systems, 32, 2019.
[32] Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. 2015.

---

[33] Gaurav Kumar Nayak, Konda Reddy Mopuri, and Anirban Chakraborty. Effectiveness of arbitrary transfer sets for free knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1430-1438, 2021.
[34] Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation deep networks. In International Conference on Machine Learning, pages
PMLR, 2019.
[35] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
[36] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference Computer Vision (ECCV), pages 268-284, 2018.
[37] Xiaoyang Qu, Jianzong Wang, and Jing Xiao. Enhancing data-free adversarial distillation with activation regularization and virtual interpolation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3340-3344. IEEE, 2021.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748 8763. PMLR, 2021.
[39] James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9374-9384, 2021.
[40] Hoang Thanh-Tume Tung and Truyen Tran. Catastrophic forgetting and mode collapse in gans, In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-10. IEEE, 2020.
[41] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2019.
[42] Frederick Tung and Greg Mori. the IEEE/CVF International Conference on Computer Vision, pages 1365-1374, 2019.
knowledge distillation. In Proceedings of
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[44] Zi Wang. Data-free knowledge distillation with soft targeted transfer set synthesis. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10245-10253, 2021.
[45] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8715-8724, 2020.
[46] Jaemin Yoo, Minyong Cho, Tacbum Kim, and U Kang. Knowledge extraction with no observable data. Advances in Neural Information Processing Systems, 32, 2019.
[47] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.033928. 2016.
[48] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
[49] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987-3995. PMLR, 2017.
12

---

[50] Haoran Zhao. Xin Sun, Junyu Dong, Milos Manic, Huiyu Zhou, and Hui Yu. Dual discriminator adversarial distillation for data-free model compression. International Journal of Machine Learning and Cybernetics, pages 1-18, 2021.
[51] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
13


Reviews:
[{"review_id": "wI7W-lwtlP9", "reviewer": "Reviewer_tq3D", "summary": "This paper shows that momentum adversarial distillation can help data-free knowledge distillation by solving the distribution shift problem. Extensive experiments are conducted on six datasets to demonstrate the effectiveness of the proposed method, including: CIFAR10, CIFAR100, Tiny-ImageNet, ImageNet, Places365 and Food101. ", "questions": "-\tClarify the key contribution and novelty of the proposed method since the momentum updating method is not a new technique and has been widely studied.\n\n-\tProvide more explanations on how momentum update handles the distribution shift issue in the proposed framework.\n\n-\tThe performance in the paper is not competitive and the comparison is not sufficient in the experiments.\n", "limitations": "This paper has discussed the limitations and negative social impacts.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-\tThe proposed method is simple and seems effective, also easy to follow for other research.\n-\tThe paper is generally well-written with sufficient experiments on six datasets.\n\nWeaknesses:\n\n-\tThe momentum updating method itself is not novel and has been used in many problems, such as semi-supervised learning [1], self-supervised learning [2], etc. I’m a little bit concerned about the novelty.\n\n\t[1] Tarvainen, Antti, and Harri Valpola. \"Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.\" Advances in neural information processing systems 30 (2017).\n\n\t[2] He, Kaiming, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. \"Momentum contrast for unsupervised visual representation learning.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738. 2020.\n\n-\tIt seems the performance of the proposed method is not competitive if compared to other existing DFKD methods, as shown in Table 1. The comparison with other DFKD methods in Table 2 is also not sufficient.\n\n-\tThere can be more analyses and visualizations to show how momentum update helps alleviate the distribution shift issue, not just the synthetic samples.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "e4qc2f4aDRG", "reviewer": "Reviewer_Lniz", "summary": "In data-free knowledge distillation, a generator is used to synthesize data for training the student. This paper found that the distribution of synthetic data will change when the generator gets updated.  Thus, this paper proposes a new method called  Momentum Adversarial Distillation for data-free knowledge distillation. Experiments are done on six benchmark datasets to show the effectiveness of the proposed method. \n", "questions": "Using EMA to stabilize the model training is a common technique in existing works. This paper applies it to the generator of images for data-free knowledge distillation. Thus the novelty of the proposed method is limited. Another concern is from Figure5, the EMA generator fails to improve the quality of generated images, which makes me concerned about the cause of the improvement of student performance. How about adding EMA to the student?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The presentation of this work is clear and well-written.\n2. The idea of the proposed method is straightforward.\n3. I appreciate the sufficient experiments on different datasets and study on CIFAR100. They deliver the strong message that the proposed method is efficient.\n\nWeakness\n1. In figure5,  it is hard to conclude that the generated images by EMA Generator are better than the Generator. Could you use a metric to show how the difference for example the FID score?\n\n2. In numerical experiments,  the results seems better than baselines. Results are mainly given the teacher and student are similar networks, how about the performance diverse architecture (resnet as teacher, alexnet as student)?\n\nAfter rebuttal:\nThe reviewers have addressed my concerns. I insist my previous score after considering other reviewers' comments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Lkb8GLW0_9-", "reviewer": "Reviewer_qSj6", "summary": "This work focuses on Data-free Knowledge Distillation (DFKD). The main idea of existing DFKD is to use a generator to synthesize data for training the student. The authors propose Momentum Adversarial Distillation (MAD) to avoid forgetting the knowledge it acquired at the previous steps.", "questions": "Please see the weakness.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\nMany existing data-free knowledge distillation methods adopt an adversarial training paradigm, where the generator is adversarial trained for distillation, making the data distribution different from the earlier. This work focuses on the important yet understudied distribution shift problem in DFKD and propose a reasonable method to solve the problem.\n\nThe paper is overall well written and easy to follow. The experiments are well designed and the results well demonstrate the superiority of the proposed method.\n\nCons:\n\nThe experiments of this work have already addressed most my concerns about the paper. The only one remaining concern is that the difference or superiority of the proposed method from some highly related works should be clarified. For example, MosaicKD [1] also studies the domain shift problem in knowledge distillation, the authors should make the differences from the work clearer or make some comparisons if possible.\n\n[1] Fang G, Bao Y, Song J, et al. Mosaicking to distill: Knowledge distillation from out-of-domain data[J]. Advances in Neural Information Processing Systems, 2021, 34: 11920-11932.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6A1-fOKbeb-", "reviewer": "Reviewer_5Fok", "summary": "The paper tackles the problem of data free knowledge distillation, wherein access to a (teacher) model trained on some data is assumed, however the training data is not available. DFKD tries to train a student model using just the teacher model. The paper focuses on one method for DFKD which trains a generator network to generate synthetic examples for distilling the knowledge of the teacher. The main contribution of the paper is to use the generator along with an exponential moving average of the generator’s weights for producing samples. The paper claims that this improves training stability, since the data distribution that the student network sees does not change abruptly. This helps limit catastrophic forgetting of the student network on data from generators at previous iterations. The paper has a thorough empirical evaluation of the proposed method on multiple image benchmarks.", "questions": "1. Can the authors provide more evidence around how EMA helps with the catastrophic forgetting of the student? An ablation comparing the training and test accuracy curves with and without EMA could help\n\n2. Can this trick of using an additional generator be plugged into other DFKD methods which rely on synthetic data? If so does it always help, or are other methods of making training more stable better?\n\n3. How would the performance change if the train and test set have some distribution shift within themselves? In that case, would EMA help?\n\n4. Can EMA also be applied on the student to stabilize the generator's training?\n\n5. While EMA might mitigate catastrophic forgetting to some extent, it is also possible that there is a large shift within the EMA generator as well, leading to some amount of catastrophic forgetting. While this effect can be controlled by the alpha parameter of decay, an alternative could be to maintain an EMA of the EMA generator and so on. Do the authors have any insights around this?\n", "limitations": "Some of the limitations are listed in the weaknesses above. More analysis of the generators and their impact on the student would be appreciated. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths - \n1. The paper proposes a simple trick to prevent catastrophic forgetting which does better than memory store techniques with lower overhead. The compute overhead is also lower than other DFKD methods.\n2. The paper does extensive empirical evaluation of their method, along with a sensitivity analysis to various components of the approach.\n3. The related works section is very well written.\n\nWeaknesses - \n1. The empirical evaluation is over a single run, so it is hard to ascertain the significance of the results. Adding error bars would definitely help here.\n2. The claim of the student undergoing catastrophic forgetting is not sufficiently explored. While the experiments show that having a higher momentum (and weight for the ema generator’s images) is better to an extent, a class wise breakdown of where such models do better, and how this improvement is related to catastrophic forgetting could help.\n3. The method is not interpretable. This is a general shortcoming of such adversarial methods, and should not be taken as a very specific shortcoming of this work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept