A Unified Knowledge Distillation Framework for Deep Directed Graphical Models
Authors: Yizhuo Chen
Kaizhao Liang
Zhe Zeng
Yifei Yang
Shuochao Yao
Huajie Shao
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Directed Graphical Models
Knowledge Distillation
Reparameterization trick
Model compression

Abstract:  Knowledge distillation (KD) is a technique that transfers the knowledge from a large teacher network to a small student network. It has been widely applied to many different tasks, such as model compression and federated learning. However, the existing KD methods fail to generalize to general \textit{deep directed graphical models (DGMs)} with arbitrary layers of random variables. We refer by \textit{deep} DGMs to DGMs whose conditional distributions are parameterized by deep neural networks. In this work, we propose a novel unified knowledge distillation framework for deep DGMs on various applications. Specifically, we leverage the reparameterization trick to hide the intermediate latent variables, resulting in a compact DGM. Then we develop a surrogate distillation loss to reduce error accumulation through multiple layers of random variables. Moreover, we present the connections between our method and some existing knowledge distillation approaches. The proposed framework is evaluated on three applications: deep generative models compression, discriminative deep DGMs compression, and VAE continual learning. The results show that our distillation method outperforms the baselines in data-free compression of deep generative models, including variational autoencoder (VAE), variational recurrent neural networks (VRNN), and Helmholtz Machine (HM). Moreover, our method achieves good performance for discriminative deep DGMs compression. Finally, we also demonstrate that it significantly improves the continual learning performance of VAE.

Under review as a conference paper at ICLR 2022
A UNIFIED KNOWLEDGE DISTILLATION FRAMEWORK FOR DEEP DIRECTED GRAPHICAL MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge distillation (KD) is technique that transfers the knowledge from a large teacher network to small student network. has been widely applied many different tasks, such as model compression and federated learning. However, the existing KD methods fail to generalize to general deep directed graphical models (DGMs) with arbitrary layers of random variables. We refer by deep DGMs to DGMs whose conditional distributions are parameterized by deep neural net- works. In this work, propose novel unified knowledge distillation framework for deep DGMs on various applications. Specifically, we leverage the reparam- eterization trick to hide the intermediate latent variables, resulting in a compact DGM. Then we develop a surrogate distillation loss to reduce error accumulation through multiple layers of random variables. Moreover, we present the connections between our method and some existing knowledge distillation approaches. The proposed framework is evaluated on three applications: deep generative models compression, discriminative deep DGMs compression, and VAE continual learning. The results show that our distillation method outperforms the baselines data- free compression of deep generative models, including variational autoencoder (VAB), variational recurrent neural networks (VRNN), and Helmholtz Machine (HM). Moreover, our method achieves good performance for discriminative deep DGMs compression. Finally, we also demonstrate that significantly improves the continual learning performance of VAE.
INTRODUCTION
Knowledge distillation (KD) aims at transferring the knowledge of a large teacher model to a small student model, which tries to mimic the behavior of the teacher model to attain a competitive or superior performance Hinton et al. (2015); Gou et al. (2021). The goal of this work is to develop a unified knowledge distillation (KD) framework for deep directed graphical models (DGMs). Applications of the proposed framework include: (i) data-free deep generative models compression, (ii) discriminative deep DGMs compression, and (iii) continual learning with VAEs.
Deep directed graphical models (DGMs) refer DGMs whose conditional distributions are param- eterized by deep neural networks (DNNs), which in contrast to the regular DGMs with tabular conditional probability. One good example is variational autoencoders (VAEs), whose posterior probability of latent variables is parameterized by DNNs. A general deep DGM may have complex structures, consisting of arbitrary number of input variables, target variables, and latent variables. Deep DGMs have been widely used in various applications, such as image generation Vahdat & Kautz (2020), text generation Bowman al. (2015), and video prediction Wu al. (2021).
This work is motivated by the growing popularity of recent wver-parameterized deep DGMs with millions of parameters to improve their accuracy in various tasks. However, the large models are very computationally expensive. As result, it is not practical to deploy them on resource-constrained edge devices, such as mobile phones and IoT systems Li et al. (2020). One possible solution to this problem is KD, which enables smaller student model approximate the performance of large teacher. Recently, KD has been widely applied to many different tasks, such as model compression Hinton et al. (2015), continual learning Li & Hoiem (2017); Zhai et al. (2019), and federated learning Lin et al. (2020b); Li & Wang (2019). To our knowledge, the existing KD methods, however, are only applicable to some specific DGMs, including generative adversarial networks (GANs) Aguinaldo

---

Under review as a conference paper at ICLR 2022
accumulated error (KL divergence)
Cx
x
x
local-disillatioon
2.5
our-distilation
Ex11
E1
2.0
Y2
1.0
222
17
67
12
y2
Y/2
Y/2
0.0
9 13 17
(n)
(b)
(0)
number of layers of latent variables
Figure 1: Illustration of DGM in four different Figure 2: Toy example of accumulated error forms. Diamonds are deterministic variables and
(KL divergence) between the teacher and stu- circles are random variables. (a) Original form; (b) dent for local distillation and our method. Ex-
Auxiliary form; (c) Our coni-auxiliary form; (d) perimental settings are presented in the last para- Compact remi-auxiliary form.
graph in Section 3.2.
et al. (2019); Li al. (2020), auto-regressive models in natural language processing (NLP) Lin et al. (2020a), and VQ-VAE Roy et al. (2018). They fail to generalize to the general deep DGMs, especially to those with multiple latent variables or complex dependence structures, as illustrated in Fig. 1.
Generalizing knowledge distillation to deep DGMs poses two major challenges. First, distillation by marginalizing all latent variables is generally intractable. Secondly, distilling each layer locally and independently may suffer from error accumulation, shown Fig. 2. We can observe that the accumulated error (i.e., KI. divergence) between the teacher and student grows linearly for local distillation. To address these challenges, we propose novel unified knowledge distillation framework for deep DGMs. Specifically, we first adopt the reparameterization trick Kingma (2013); Kingma al. (2014) to convert DGM into compact semi-auxiliary form. By semi-auxiliary form, we mean the latent variables, 2, in both the student and teacher models are converted to deterministic variables with auxiliary variables, while the input variables and target variables remain unchanged, as shown in Fig. (c). Note that different from the classical reparameterization for model training that requires continuous latent variables, ours can be applied to both continuous and discrete variables. Then surrogate distillation loss is derived as a new objective of KD. To mitigate gradient vanishing. we further incorporate a latent distillation loss that penalizes the dissimilarity of latent variables between the teacher and student into our objective. Moreover, we illustrate the connections between our approach and some existing KD methods for specific DGMs. We demonstrate that our method is proper generalization of these existing methods.
We apply our distillation method to three different applications: deep generative models compres- sion Rezende al. (2014); Chung et al. (2015), VAE continual learning Rao al. (2019), and discriminative deep DGMs compression Alemi et al. (2016). For deep generative models compres- sion, the student model distilled by our method in data-free manner outperforms that trains from scratch and the other baselines. In addition to generative modeling, our method is able to compress discriminative deep DGMs with high accuracy. Finally, we also illustrate that can better mitigate the catastrophic forgetting issue than the generative replay approach in continual learning.
In summary, our contributions include: a new unified KD framework proposed for general deep DGMs based on reparameterization trick, 2) we derive novel distillation loss that combines the latent distillation loss and surrogate distillation loss to improve the performance of KD, 3) evaluation results on multiple benchmark datasets show that our approach can not only achieve high accuracy for deep DGMs compression but also improve the continual learning performance of VAEs.
2 PRELIMINARIES
Directed Graphical Models (DGMs) DGMs such as Bayesian Networks (Darwiche, 2009) are an expressive class of probabilistic graphical models (Koller & Friedman, 2009), in which the joint distribution is factorized into the product of many conditional distributions according to a directed acyclic graph (DAG) that captures variable conditional dependencies. In this work, primarily
2

---

Under review as a conference paper at ICLR 2022
study knowledge distillation for deep DGMs, especially for those with complicated dependency structures.
For deep DGMs, we are interested in modeling the conditional distribution pe po(y, xxx) for targer variables y and latent variables = given input variables x, paramcterized by 0. Specifically, when there is input variables, i.e., 0, we actually model the joint distribution pe (y. z). Let Pa(+)
denote the parent random variables of a certain variable which is defined by the DAG of a DGM. Then the conditional distribution pe (y, z (x) has its factorized form as follows,
po x x|2)= = II (U, Pa(u,). x) II po po(z;|Pa(z,),
(1)
where y, denotes the jth target variable in y. z, denotes the ith latent variable in 2. Without loss of generality, assume for two variables y, and y,, if y, is an ancestor of y,, then it holds that i j-
Knowledge Distillation (KD) KD aims to transfer the knowledge of a large teacher model to a smaller student model. One commonly used vanilla distillation method is encourage the student to mimic the output of the teacher model Hinton et al. (2015). Given an empirical distribution of the training data Pdata (ar), its distillation loss is an expected dissimilarity measure between the output of the student and that of the teacher. A general form of distillation loss is given by
Exd = Epans (ax) (d(p)(ylx), po x))],
where p& (u(y))r)) and (y(()z)) denote the output conditional distribution of the teacher and student models, respectively. o and denote their corresponding parameters. d(.,-) is a dissimilarity measure between two probability distributions. Kullback- Hellbeck-Leibler (KL) divergence (Hinton et al., 2015; Chen et al., 2017), for example, is one of some typical choices.
The above Eq. (2) and its extended version have been applied to different DGMs, such as vanilla neural networks Chen et al. (2017), GANs Aguinaldo et al. (2019), and fully-visible auto-regressive models (e.g., Transformer) Kim & Rush (2016); Lin al. (2020a). Take fully-visible auto-regressive models as an example. When we set d(.,)) to KL divergence, Eq. (2) can be factorized as
Lkd = E (ppss( (xr)
Epnoivesles [KL(po ( V)) (pllylye x)1)
(3)
A tractable estimation to Led above can be obtained by Monte Carlo method. It can also be viewed as distilling each conditional distribution in a local and independent manner, as shown in Fig. 3 (d).
However, to our best knowledge, the existing KD approaches are only designed for some specific DGMs. They fail to be applied to a general DGM with multiple latent variables or complicated dependence structures. Hence, the question is, how can we distill the knowledge from the teacher to the student given a general DGM structure?
Intuitively, there two naive methods: (i) marginalized distillation: it marginalizes all latent variables to get p(z/Iz) = I p(y, x z|z)dx, However, the integration p'y x(x) dz is generally intractable and thus the loss as in Eq. (2) is also intractable. (ii) local distillation: it treats latent variables z equally as target variables and then distills each conditional distributions for both z and y locally and independently, as shown in Fig. 3(d). The local distillation may suffer from error accumulation through multiple layers if each conditional distribution in the student slightly deviates from the teacher. Fig. 2 shows toy example that the accumulated error of local distillation, i.e., the KL-divergence between teacher and student, increases linearly as the number of layers of latent variables rises. For detailed discussions, please refer to Appendix A.
Reparameterization Trick Reparameterization trick Kingma (2013): Kingma et al. (2014), also called the auxiliary form of a DGM, originally proposed to backpropagate through a random node that is not differentiable during training. Its basic idea is introduced as follows. Given conditional distribution of random variable 2, in a DGM, p(z, Pa(z;), x). we convert it to a deterministic variable by adding auxiliary variable e; to its dependence, as shown in Figure (c). Here e; is root node of the DGM with an independent marginal distribution of p(c,). By choosing appropriate p(e,) and deterministic transformation g(-) Kingma (2013), we can have 2, g (PPazz)), æ, e,), where z; is determined by its parent variables Pa(z;), input variables æ. and the corresponding auxiliary variable e, E; serves the source of stochasticity of z;.
3

---

Under review as a conference paper at ICLR 2022
teacher
student 0
taachor
studant 9
teacher
student
Teather Ø
Sludkent
teacher d
student 8
X
displ
clsoN
7,1
7
disti
desie
Y1
V1
Y/1
Y1
Y/1
F2
disaN
dball
Zz
Z2
Y/
y2
y/
Y2
Y2
Y2
(c)
(c) Our method
Figure 3: Illustration of different distillation methods. Each pair of conditional distributions marked with same color represents an independent distillation component. (a) Distillation on vanilla neural network. (b) GAN distillation. (c) Distillation on 2-layer fully-visible auto rugr-reegessive DGM. (d) Local distillation on the original DGM. (e) Our distillation method with semi- auxiliary form of DGM.
In this work, note that we do not primarily use reparamcterization trick for model training. Rather, leverage it to convert the latent variables in DGMs to deterministic variables so that we can effectively distill knowledge from a compact form of DGM. Note that different from the classical reparameterization for model training that requires continuous latent variables, ours can be applied to a wider range of variables z, including both continuous and discrete variables. Besides, the transformation function g(.) Kingma & Welling (2013) in our framework be either differentiable or non non-differenntable. Hence, our method can be applied to much more DGMs than the classical one. Below, we will elaborate this general idea in more detail.
MODELING
In this section, we first introduce the semi-auxiliary form of DGMs using reparametcrization trick. Then we propose a new surrogate loss function and latent distillation loss for our KD method.
3.1 SEMI-AUXILIARY FORM
As discussed in Section 2, the two naive methods, marginalized distillation and local distillation, do work well due to intractable distillation loss or error accumulation. order address these issues, we propose novel idea that converts DGM to its semi-auxiliary form based reparameterization trick. Specifically, we convert all the latent variables, 2, both the teacher and student to deterministic variables with auxiliary variables, while keeping target variables and input variables unchanged. This is because our ultimate goal is to encourage student to mimic the output (target variable) of the teacher based input variables. Hence, yielding a compact semi-au semi-auxilian auxiliary form that only consists of target variables, input variables and auxiliary variables. In this way, each target variable has tractable and direct dependence input variables or prior target variables, as shown in Fig. (c).
omit the deterministic (latent) variables in DGM,
Fig. 1 illustrates toy example of converting DGM to its semi-auxiliary form. Fig. 1 (a) is the original form of a DGM, and its corresponding auxiliary form is shown Fig. (b). In this paper, we only assign auxiliary variables to latent variables z, as shown in Fig. (c). We can observe from it that latent variables 2 and 22 are deterministic when their ancestors are given. We thus can omit the deterministic variables, leading compact eemi-auxiliary form, as shown in Fig. (d).
3.2 SURROGATE DISTILLATION LOSs
After obtaining the semi-ause semi-auxiliary form of DGM in Fig. (d), our distillation loss becomes the dissimilarity between Pa(yles e, x) and pa (y) We call it surrogate distillation loss. Our goal is to minimize the surrogate distillation loss w.r.t. student's parameters below.
Led Ee (e)paaaa) [d(polyle, x) po (ylye, x))]
(4)
4

---

Under review as a conference paper at ICLR 2022
where denotes set of auxiliary variables The expectation of dissimilarity is taken over both empirical data distribution Pdota (z) and auxiliary variable distribution Po(E). The expectation can be estimated using Monte Carlo method. Note that (c( is generally chosen to be simple and fixed distribution with no parameters, such as unit Gaussian or standard uniform distribution. Thus, implies that teacher's (e(e) is equivalent student's po(e), there is no need to distill ps (c) to (e). An illustration is given in Fig. 3(e).
Proposition 3.1. The surrogate distillation loss as defined in Eq. (4) is an upper bound of the distillation loss as defined in Eq. (2) when the dissimilarity measure is chosen to be KL divergence.
We provide the detailed proof in Appendix B.
Next, we discuss the advantages of our method over the two naive methods mentioned above. Firstly, the proposed method bypasses the intractable computation in marginalized distillation. While marginalized distillation measures (()))r) which is intractable in general, we instead measure p(yle, le, a:) which is easily tractable by function composition with no need of integral sum operation. Here p(y/e, æ) is tractable because p(yle, x) = II; ply, Egi Ve: Z).
Secondly, our method can make the DGMs shallower than local distillation, mitigating error ac- cumulation through multiple layers of latent variables. As illustrated in Fig. 3(d) and 3(e), we can see our method only constraints the target variables y in the student while local distillation constrains both latent variables z and target variables y. As result, local distillation may suffer from error accumulation issue. Fig. 2 shows toy example of comparing the accumulated error, i.e., KI. divergence between the teacher and student, for our method and local distillation. In this illustrative experiment, we let student mimic the output distribution of the teacher with L-layers latent variables for L E 1, .,,.... 20. Each layer of the teacher follows the Gaussian distribution p(Z++1/z,) = (z;)),, 0.014), where p(z;) is 2! for z; 2 0 and - (- for z; < O. p(z1) is uniform distribution U 1, 1]. The student is parameterized by neural networks with proper residual structure He et al. (2016). Then density ratio estimation Nguyen et al. (2010); Sugiyama al. (2012) is used to measure the KL divergence between the teacher and student. We can observe from Fig. 2 that the accumulated error (KI. divergence) grows linearly w.r.t the number of layers for local distillation because each layer of the student deviates from the teacher to an extent. In contrast, the accumulated error (KL divergence) of our method increases slowly. Furthermore, we show that our method can still mitigate error accumulation issue for a DGM whose latent variables are discrete in Appendix I.
3.3 LATENT DISTILLATION LOSS
The surrogate distillation loss in Eq.(4) can sufficiently achieve satisfactory performance for knowledge distillation. Nevertheless, there are still two limitations of the proposed method for some special DGMs. Firstly, it fails to back- pack-propagate the gradients when there exist discrete latent variables. Secondly, it might suffer from gradient vanishing when the network structure of latent variables is very deep and complex.
To deal with these issues, we propose to penalize the dissimilarity of latent variables z in the teacher and student model. The resulting latent distillation loss for latent variables 2, is given by
E = Epple)paana ot po x)]]
(5)
= Epp (d(ps(z m), po (2;
where egi is set of all the ancestral auxiliary variables of z; z; is deterministic when 6g; and i are given. In order to better penalize the dissimilarity of latent variables 2,, one good choice is to (only) convert the current z, back to its original form by removing e, from its dependence. Then have the following latent distillation loss
C.,s = Epsle) (d(po(z x), (6 (2. x))]
(6)
The proposed latent distillation loss can benefit our optimization process. When latent variables are continuous, the latent distillation loss may provide shallower and supplementary supervisory signals to hasten the convergence. When latent variables are discrete, can deal with the back proppaapaaa cutting off problem.
5

---

Under review as a conference paper at ICLR 2022
3.4 FINAL TARGET-FREE LOSS
Combining the above surrogate distillation loss with latent distillation loss, our final distillation loss given by
Lour = Lad + A Σ Lz.i:
(7)
where X a hyper-parameter that controls the importance of latent distillation loss. Similar to Hinton
et al. (2015), our loss in Eq. (7) is a target-free distillation loss, which means target data is not
required for calculating it. Eq. (7) can also be applied to DGMs with no input variables (i.e. æ = 0).
In this case, Eq. (7) can be computed in completely data-free manner. We summarize our KD method for a general DGM in Algorithm in Appendix C.
3.5 CONNECTIONS TO OTHER DISTILLATION METHODS
We present the connections between our method and some existing knowledge distillation methods. Vanilla KD and Sequence-Level KD. When there is no latent variable in a DGM, our distillation method in Eq. (4) is naturally reduced to Eq. (2) by removing the dependence on auxiliary variables e, which is typical vanilla KD Hinton et al. (2015). Also, when the DGM is a fully visible auto- regressive model, by removing dependence on and letting d(..-) -) be KL divergence, Eq. (4) can be reduced to Eq. (3), which is the Monte Carlo approximation of intractable sequence-level distillation loss Kim & Rush (2016). Please note that further simplification has been made Kim & Rush (2016) to enhance the practicability.
Feature Based KD Feature based knowledge distillation uses the intermediary representations of a teacher network to supervise student network Romero et al. (2014). When the teacher and student share the same size of latent features, the feature distillation loss can be written as
f ry (a) fo(x))
(8)
where fx()) and [a) are intermediary deterministic features of the teacher and student, respec- tively. -y(-:-) is distance between two vectorized feature maps. A vanilla neural network with multiple intermediate features can be viewed as a DGM with multiple deterministic latent variables. Deterministic variables can be viewed as following the degenerate distributions. In general, for p 2 1,
p- Wassersteii distance (inf ([[(a1 ) between two degenerate distributions located at a and a2 is equivalent to r(a1,a) Thus, by viewing the intermediate features as latent variables following degenerate distributions, and choosing d(.,-) in our latent distillation loss Lz,i as Wasserstein distance
(inf >)) our latent distillation loss is reduced to feature distillation loss in Eq. (8).
GAN Distillation. GAN distillation in Li et al. (2020); Aguinaldo et al. (2019) also incorporates the idea of feature distillation into their model, which is given by
Eyan = To (Go (z). Gz(z)) + (r((f(().
The first term above is the output distillation loss of the generator and the second one is the inter- mediary feature distillation loss. Similar to feature distillation loss, by viewing the intermediate features and generator output as latent variables following degenerate distributions, and choosing d(.,.) in Lad and Zz,; as p- Wasserstein distance (inf (yr yo ) and (inf Elr, (ZØ, zn)²)) respectively, our final distillation loss in Eq. (7) can be reduced to GAN distillation loss as well.
3.6 APPLICATIONS
We evaluate the performance of our method in three applications below.
Data-Free Deep Generative Models Compression. We first apply our method to compress deep generative models in a data-free manner. We conduct experiments on three representative models: VAE, VRNN, and Helmholtz Machine (HM) Chung et al. (2015); Saul et al. (1996). Recent studies show that VABS generate better with over-paramettiization Vahdat & Kautz (2020). However, it is challenging to deploy them to edge devices with limited computing resources. In this work, we apply apply method to compress a large 5-layer hierarchical VAE model Sønderby et al. (2016) to smaller models, as illustrated in Fig. 6(a) in Appendix D. Then it is applied to compress a  sequence generative model with many latent variables, VRNN. Finally, adopt our method to

---

Under review as a conference paper at ICLR 2022
compress the HM model with discrete variables. The classical HM only has one target variable and each layer is parametrized by a linear transformation. In order to demonstrate the applicability of our method, we extend it to 5-layer deep HM with two targets, shown in Fig. 6(d).
VAF Continual Learning. In addition to model compression, we evaluate the performance of our method VAE continual learning. Our goal is model a new distribution while retaining the ability of modeling a learned old distribution without access to the old dataset. Prior work on VAE continual learning Ye & Bors (2020); Ramapuram et al. (2020) mainly resort to generative replay strategy, in which set of samples generated from VAE model learned on old datasets P/ (y) are reused when learning new dataset. In fact, our experiments will show that generative replay is inferior to our distillation method for VAE continual learning because of the blurry nature of VAE generation.
Discriminative Deep DGMs Compression. Finally, we apply the proposed method to compress a 2-layer discriminative DGM in Alemi et al. (2016), as shown in Fig. 6(c). Incorporating our distillation loss into the original training loss, we have the following loss function.
Lyoint (1 frrinn + aLLur-
(10)
4 EXPERIMENTS
In this section, we evaluate the performance of our method on three applications mentioned in Sec- tion 3.6. We carry out extensive experiments on seven benchmark datasets: Old Faithful Geyser Härdle et al. (1991), IAM online handwriting Liwicki & Bunke (2005), SVHN Netzer et al. (201 1), CI- FARIO Krizhevsky et al. (2009), CelebA Liu et al. (2015), and Fashion-MNIS' Sohn et al. (2015). The detailed data splitting for training and testing is described in Appendix E. In addition, we present the detailed model configurations and hyperparameter settings in Appendix F. Source code will be publicly available upon publication.
4.1 EVALUATION ON DATA-FREE DEEP GENERATIVE MODELS COMPRESSION
We first apply our distillation method to data-free compression of deep generative models, including VAE, RVNN, and HM. We compare the performance of the student model using our method and two baselines: training from scratch and local distillation Achille et al. (2018). Frechet inception distance (FID) Heusel et al. (2017) is adopted to evaluate their generative performance.
We compress a large 5-layer hierarchical VAE Sønderby et al. (2016) smaller model. Experiments are conducted on three benchmark datasets: SVHN, CIFARIO, and CelcbA. Fig. 4 illustrates the comparison of FID for different methods. We can see that method consistently outperforms the baselines for different student model sizes. Importantly, our data-free distillation method outperforms student trained from scratch. This is because directly optimizing a capacity-limiter student VAE may not learn a decent model that can generate high-fidelity samples. Conversely, our method can help student VAE learn better performance from the teacher.
Data-Free Hicrarchical VAE Compression.
To demonstrate the similarity of the teacher and student, we further compare the FID between samples generated from the student and teacher for different methods, as shown in Fig. 7 in Appendix G. We observe that our distillation method helps the student better approximate the teacher's 's performance. We also demonstrate that our method not very sensitive to the hyper- ppprmmprraettrr Please refer to Appendix G.3 for more details.
Data-free VRNN Compression. We then adopt our distillation method compress a deep sequence generative model, VRNN. Fig 5 shows the generated strokes by our method and the baselines. It can be observed that our method can generate more readable and clearer strokes than the baselines. The student VRNN distilled by our method in Fig 5 (c) has comparable generative performance to that of the teacher in Fig 5 (b). Hence, we can conclude that the proposed method is able to achieve good performance for data-free VRNN compression.
Data-Free HM Compression. In addition, our method is applied to compress Helmholtz Machine (HM) with discrete latent variables. The experiment results, as shown in Fig. 9 in Appendix H. show that it can still achieve good performance as the teacher for HM compression. For more details, please refer to the experimental results in Appendix H.
7

---

Under review as a conference paper at ICLR 2022
338
our deniiiilatoon Iccal-ctollaiion
out-dsillaton
100
cur-ditilaato
loca--dee lafion
trah-dcrrtthh
toocter
180
100
0008 (1600)
10M 1st0 student size
5.3m (164)
(3:80
BIM 11806
150 (Maa)
0 0.8MM190 PSC
000 1-900
(502
00M 1150
soudene model
Student size
(a) CelebA
(b) CIFARIO
(c) SVHN
Figure 4: Comparison of FID for different methods in data-free VAE compression on three datasets: CelebA, CIFARIO, and SVHN. We compute the FID between the generated samples by student and ground truth averaged over 3 random seeds. Teacher model size is fixed to 5.4M parameters.
froms Meephet! In Yhowe Tal oet San did becuee
he's
del
Swntent Inv-
Wid
Swate of
Fis paint
Rollnkk juacce Aerooniee
oigiictoon.." We wgy be
Bren colorrall
Phulo Proteeoooogggggoooo Pow.geaeeee
(a) Ground truth
(b) Teacher
(c) Our method
(d) Local distillation
(e) From scratch
Figure 5: Strokes generated by VRNNs. Different strokes are represented with different colors. (a) Ground truth dataset. (b) Teacher VRNN (12M parameters) trained from scratch. (c) Student VRNN
(2.4M parameters, 5.0 times smaller) distilled by our method. (d) Student VRNN distilled using local distillation. (e) Student VRNN trained from scratch.
4.2 EVALUATION ON VAE CONTINUAL LEARNING
We also evaluate the performance of the proposed method on VAE continual learning using CelebA dataset. For each experiment, one of forty ground-trute -rrth attributes is selected. Based this attribute, we divide the whole dataset into two parts for continual learning. Specifically, we first learn to model one part of images, then we learn model the other part while trying retain the ability of modeling the first part. We repeat this experiment for four different attributes. The single input variable x in this model is binary variable used to indicate which part the generated images belong to.
We compare our method with local distillation and other two generative replay approaches: CURL Rao al. (2019)) and LGM Ramapuram et al. (2020). Table illustrates the comparison of FID for different methods. We can observe from Columns Old that the increased FID of our method is much smaller than that of baselines, which means our method suffers from the least catastrophic forgetting. Besides, we can see from Columns New that our method achieves comparable FID to LGM and CURL on new distribution, because these methods put only necessary constraints on the model to prevent catastrophic forgetting. However, local distillation does not work well on new distributions because of its redundant constraints, leaving little free space for learning new distributions.
Table 1: FID of different methods after learning new distributions on Celeb dataset. Columns Old show FID between the generated images and real images on old distribution. A(+B) denotes that FID increased by B to after learning new distributions. Columns New illustrate FID between generated images and real images on new distributions. FID and (+B): lower is better.
Eyeglasses
Eyeglasses New
Female
Male
No Beard
Beard
Not Smiling
Smiling
Methods/FID
Old
Old
New
Old
New
Old
New
Our methed
79,0(+12)
79.8
85.1(+2.8
69.3
80.0(+0.4)
79.8
90.1(44.7)
78.1
Local distillation
178.7(+100.9)
122.2
90.0(+7.7)
80.9
96.7(+17.1)
109.9
91.7(+2.3)
85.7
CURL
99.9(422.1)
84.9
112.4(+30.1)
65.9
105.15+25.5)
78.1
108.2(+18.8)
76.8
LGM
99.2(+21.4)
86.5
114.6(+32.3)
67.9
104.9(425.3)
82.0
108.6(+19.2)
79.2
4.3 EVALUATION ON DISCRIMINATIVE DGMS COMPRESSION
Finally, we apply the proposed method to compress discriminative 2-layer DGM with the same MLP parameterization as Alemi et al. (2016). The training loss in Alemi et al. (2016) is denoted by LyIR- We compare the performance of different models using method (our-VIBB) in Eq. (10) and
8

---

Under review as a conference paper at ICLR 2022
the other five baselines: (i) combining Lvis with local distillation loss (Local-VIB) (ii) combining C CVI with Monte Carlo based marginalized distillation loss (MC-VIB), (iii) model trained with VIB only (VIB), (iv) model trained with vamilla cross entropy loss only (Vanilla), and (v) model trained using vanilla cross entropy loss with dropout (Dropout). The experimental results are illustrated in Table 2. We can see that our method beats the baselines in terms of classification accuracy and negative log likelihood (NLL) on Fashion MNIST and SVHN datasets. Besides, our method has lower ECE than all the baselines except for Dropout method.
Table 2: Classification accuracy of different methods averaged over 5 random seeds.
Fashion MNIST
SVHN
Methods
*Parameters
Accuracy
NLL
ECE
Accuracy
NLL
ECE
Teacher
2.60M
90.40
0.4657
0,0592
83,97
0.6589
0.0484
Vanilla
0.40M (6.5x)
89.69=0.12
0.651=0.005
0.077=0.001
81.07:1.668
0.719::0.036
0.0296:10.009
Drepout
0.40M (6.5x)
89.69
0.653=10.009
0.078=0.001
76.46:2.95
0.831=0.119
0.0229110005
VIB
0.40M (6.5x)
89, 09.00-00.24
0.480=0.0122
0.064-0.003
83.01+00.36
0.645=00.007
0.0244:40.000
MC-VIB
0.40M (6.5x)
90.35:=0.08
0.459=0.002
0.065=0.001
83.77=0.32
0.689=0.023
0.0494:10.0017
Local-VIB
0.40M (6.5x)
90.38+00006
0.467=+0.001
0.059=0.0001
83.94:+0.01
0.656+00.001
0.047++00003
Our
0.40M (6.5x)
90.4
0.429:00.006
0.0576:10.001
84.04:00.09
0.6307:10.008
0.0346:10.001
RELATED WORK
In the past few years, there is line of work on knowledge distillation (KD) for different DGMs. Most of existing approaches focus on vanilla DNNs or multi-target DNNs Hinton et al. (2015); Chen et al. (2017); Müller et al. (2019); Romero al. (2014); Zagonuyko & Komodakis (2016). For instance, Chen et al. Chen et al. (2017) proposed a weighted cross entropy loss for multi-class object detection models using KD. Zagoruyko al. Müller al. (2019) developed an attention based mechanism to transfer knowledge from the teacher CNN to the student for image recognition tasks. These models can be viewed as DGMs with only one stochastic layer, consisting of one input variable and one or multiple conditionally independent target variables.
Some researchers also study KD for fully visible auto-regressive networks Kim & Rush (2016); Jiao et al. (2019): Oord et al. (2018); Huang et al. (2018). Among them, sequence-level knowledge distil- lation (SeqKD) Kim Rush (2016) is promising strategy that supervises student with the teacher's sequence distribution over the space of all possible sequences. For example, some researchers Lin et al. (2020a) adopted imitation- knowledge distillation (ImitKD) method to compress autore- gressive models for language generation tasks. Tan et al. (2019) proposed multilingual distillation framework for multilingual Neural Machine Translation (NMT). These - models are specific DGMs with multiple stochastic layers, but with no latent variables.
In addition, recent studies apply KD to compress deep generative models with one latent variable. To reduce the number of parameters used in GANs, researchers Aguinaldo et al. (2019); Li al. (2020) devised new knowledge distillation methods for compressing GANs. Besides, Lee et al. (2020) distilled the learned representation from VAE models GAN for high-fidelity synthesis. Roy et al. (2018) leveraged VQ-VAE with KD to develop a con-autoregressive machine translation model. However, these works did not apply reparameterization trick to deal with the latent variables.
In summary, the above existing KD methods are mainly focused on specific DGMs, but fail to generalize to the general deep DGMs, especially to those with multiple layers of random variables or complex dependence structures. Different from prior work, we developed a novel unified KD framework for general deep DGM using reparameterizztion trick.
CONCLUSION
This paper proposed a new unified KD framework for deep directed graphical models (DGMs). Specifically, we first adopted reparametrization trick to convert latent variables into deterministic variables with auxiliary variables, resulting in a compact semi-auxiliary form of DGM. Then a novel objective that combines the surrogate distillation loss and latent distillation loss was proposed to improve the performance of KD. We further illustrated that our framework is a proper generalization of some existing KD methods. We evaluated the performance of our method on different tasks. The results showed that it can better compress VAE models in a data-free manner than the baselines. It can also achieve high accuracy for discriminative DGMs compression and DGMs with discrete variables.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
To replicate our experimental results in Section 4, we have uploaded the source code on GitHub (https: / /gi gut..b.. com/pape hwtm//ppppsoocccccccccco - git). In our ex- periments, we adopt seven benchmark datasets: Old Faithful Geyser, IAM online handwriting, SVHN, CIFARIO, CelebA, MNIST, and MNNIS.. The detailed data splitting for training and testing is described in Appendix E. In addition, we present the detailed model configurations and hyperparameter settings Appendix F. For Proposition 3.1, we provide the detailed proof in Appendix B.
REFERENCES
Alessandro Achille, Tom Fccles, Loic Matthey, Christopher P Burgess, Nick Watters, Alexander Lerehner, and Irima Higgins. Life-long disentangled representation learning with cross-domain latent homologies. arXiv preprint arXiv:1806 06508, 2018.
Angeline Aguinaldo, Ping-Yes Yeh Chiang, Alex Gain, Ameya Patil, Kolten Pearson, and Soheil Feizi. Compressing gans using knowledge distillation. arXiv preprint VrXiv:1902.00// 2019.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 2016.
Jörg Bornschein and Yoshua Bengio. Reweighted wake-sleep. arXiv preprint vrxiv.1406.270 2014.
Jorg Bornschein, Samira Shabanian, Asja Fischer, and Yoshua Bengio. Bidirectional helmholtz machines. In International Conference Machine Learning, pp. 2511-2519. PMLR, 2016.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowiez, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint 15 2015.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. recurrent latent variable model for sequential data. Advances in neural information processing systems, 28:2980-2988, 2015.
Adnan Darwiche. Modeling and reasoning with Bayesian networks. Cambridge university press, 2009.
Marco Fraccaro, Søren Kaae Sønderby, Urrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. arXiv preprint 1rXiv:1605.07555 07551, 2016.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):189-18118) 2011.
Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. Deep autoregressive networks. In International Conference on Machine Learning, pp. 1242-1250. PMLR, 2014.
Wolfgang Karl Härdle et al. Smoothing techniques: with implementation in S. Springer Science & Business Media, 1991.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint /003.02500.25.3 2015.
10

---

Under review as a conference paper at ICLR 2022
Mingkun Huang, Yongbin You, Zhehuai Chen, Yanmin Qian, and Kai Yu. Knowledge distillation for sequence model. In Interspeech, pp. 3703-3707, 2018.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: arXiv 10351, 2019.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. 1eXiv:1606.0770 2016.
arXiv preprint
Diederik P Kingma. Fast gradient- based inference with continuous latent variable models in auxiliary form. arXiv preprint arXiv: 100.00333 2013.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes. arXiv preprint
arXiv:1312.61 14, 2013.
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Rahul G Krishnan, Uri Shalit, and David Sontag. Deep kalman filters. arXiv:1511.05121, 2015.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Wonkwang Lee, Donggyun Kim, Scunghoon Hong, and Honglak Lee. High-fidelity synthesis with disentangled representation. In European Conference on Computer Vision, pp. 157-174 Springer, 2020.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.033 2019.
Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Yan Zhu, and Song Han. Gan compression: Efficient architectures for interactive conditional gans, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5284-5294, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947 2017.
Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. arXiv preprint arXiv:2009.00 072553, 2020a.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. arXiv preprint arXiv:2006.0720 2020b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Mareus Liwicki and Horst Bunke. lam-ondb-an on-line english sentence database acquired from handwritten text on a whiteboard. In Eighth International Conference on Document Analysis and Recognition (ICDAR'0S '05), pp. 956-961. IEEE, 2005.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv:1906.0260 2019.
Radford M Neal. Connectionist learning of belief networks. Artificial intelligence, 56(1):71-113, 1992.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 201 1.
11

---

Under review as a conference paper at ICLR 2022
XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-5861, 2010.
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference on machine learning, pp. 3918-3926. PMLR, 2018.
Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong generative modeling. Neurocomputing. 404:381-400, 2020.
Dushyant Rao, Francesco Visin, Andrei A Rusu, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Continual unsupervised representation learning. arXiv preprint arXiv:1900.1448: 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.65550 2014.
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments vector quantized autoencoders. arXiv preprint arXiv: 1063, 2018.
Lawrence K Saul, Tommi Jaakkola, and Michael I Jordan. Mean field theory for sigmoid belief networks. Journal of artificial intelligence research, 4:61-76, 1996.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28:3483-3491, 2015.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalge, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29:3738-377 -746, 2016.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the bregman divergence: unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 1405):0009-100 2012.
Xu Tan, Yi Ren, Di Hc, Tao Qin. Zhou Zhao, and Tie- Yan Liu. Multilingual neural machine translation with knowledge distillation. arXiv preprint 1rXiv:1902.10461 2019.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03888 2020.
Eszter Vértes and Maneesh Sahani. Flexible and accurate inference and learning for deep generative models. arXiv preprint 1805. 2018.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy hierarchi- cal variational autoencoders for large-scale video prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 2318 2228, 2021.
Fei Ye and Adrian G Bors. Learning latent representations across multiple data domains using lifelong vaegan. In European Conference on Computer Vision, pp. 777-795. Springer, 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: (612 2016.
Mengyao Zhai, Lei Chen. Frederick Tung, Jiawei He, Megha Nawhal, and Greg Mori. Lifelong gan: Continual learning for conditional image generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2759-2768, 2019.
12

---

Under review as a conference paper at ICLR 2022
A TWo NAIVE METHODS
We present two naive methods for distilling knowledge from a teacher to a student for general DGM as follows.
A.1 MARGINALIZED DISTILLATION
To minimize the desired distillation loss in Eq. (2), one candidate solution is to marginalize over all latent variables, which, however, is generally intractable in DGMs with latent variables. This because in general there is no analytical solution for the likelihood pyy p(y, z (z))dz, much less for (po (y (x), pa palyll (y)x))). Thus, we try to use Monte Carlo method to estimate the conditional distribution pyy (z) as below
p(y| Σ ply) (*) x))
(11)
where (=(")) ) is set of samples drawn from p(z|ar). However, there are two main issues of this method. First, when y consists of more than random variables (e.g., in deep generative modeling) or it a continuous variable (e.g, auto-regressive models), p(y (z) is often not a simple distribution. As result, Eq. (2) generally does not have closed-form solution. Besides, it is not often scalable to the scenario where z contains many latent variables or each latent variable is a high-dimensional vector, since the variance of Monte Carlo estimation may be increased.
The two limitations discussed above suggest that marginalized distillation is only applicable to single- label classification tasks, where latent variables in the deep DGMs must be very low dimensional.
A.2 LOCAL DISTILLATION
Another naive method is called local distillation, which applies distillation in layer-wise manner in deep DGMs, as shown in Fig. 3(d). local distribution, each of student's local component is encouraged to mimic its corresponding teacher's local component in independent manner. With this method, we can factorize the joint distribution of a deep DGM into a product a series of conditional distributions,
shown in Eq. (I), so that the distillation loss of each conditional distribution can be calculated separately. Formally, the distillation loss for target variable y, is given by
Ly. Epacto (m) ( (Paty, )az) [d (po (Pa(y,) Po Pa(y,,),
(12)
Similarly, the distillation loss for latent variable z; is
Cz2 = Epana (Pa(p))a)()))e))) [d (p.(2,/Pa(z)) x), Po (a((Pa(zz) x))]
(13)
For the above equations, all the expectations be estimated using Monte Carlo method, and Monte Carlo samples can be drawn using ancestral sampling from the teacher model.
Finally, we up all the local distillation losses and then jointly minimize the overall distillation loss as follows.
Lid Enud Ened
(14)
Let d (1.)) be KI. divergence, then Eq. (14) can be simplified as
Lid (x) [KL(p (y, 2| po(y)z(x))
(15)
The above equation is the KL divergence between the joint probability on all the latent variables and target variables of the teacher and that of the student model. It can easily proved by factorizing p(y, x/x).
For the above Eq (15), if there are no latent variables in a DGM, and if we choose d (---) to be KL divergence, the local distillation loss can be simply reduced to the distillation loss in Eq. (2) or can be expanded to Eq. (3). However, when there exist latent variables, local distillation will add extra constraints latent variables, which is redundant and dampens the distillation performance in practice. This is because we are only interested in minimizing the dissimilarity between target variables rather than latent variables.
13

---

Under review as a conference paper at ICLR 2022
Next, provide insight of why local distillation does not work well in deep DGMs. Take the DGM in Fig. 3(d) as an illustrative example. After distillation, each conditional distribution in the student model may still slightly deviate from the teacher model because of the inferior capacity of the student model or the randomness optimization process. We call this deviation from student teacher imitation error. Imitation errors will accumulate through multiple layers. For instance, in the imitation process of variable Y/. its local distillation loss in Eq. (12) is Ly,1 = (x)p [d (p)(3)) 2 1),, Pe (y []|=1))]] Even if this local distillation loss for y/1 is very well optimized, when the student's 's latent distribution pa (z)) deviates far from the teacher's distribution, Pa(=1 Pa(=1(<), there will be no guarantee that pe (3, (<) is good approximation for p (x) because p (x) Ep(zz (m) .1). pe [=1) 1 is only well constrained in the domain of p 21 (z).. As result, imitation errors may accumulate through layers of latent variables in a deep DGM.
We also conduct toy experiment show the error accumulation issue in local distillation. In our experiment, we let student mimic the output distribution of the teacher with L-layers latent variables for L E ,,....,2 20. N(p(z;), 0.01 I), where (u(;;) is 2!11 for 2; 2 0 and - (-z;)=_. for 2; < 0. p(z1) is uniform distribution U1-1, 1]. The student parameterized by neural networks with proper residual structure He et al. (2016). Then density ratio estimation Nguyen et al. (2010); Sugiyama et al. (2012) is used to measure the KL divergence between the teacher and student. We can observe from Fig. 2 that the accumulated error (KL. divergence) grows linearly w.r.t the number of layers for local distillation because each layer of the student deviates from the teacher to an extent. In contrast, the accumulated error (KL divergence) of our method increases slowly.
20. Each layer of the teacher follows the Gaussian distribution p(z;+ | =
B ABOUT UPPER BOUND OF VANILLA DISTILLATION Loss
We present the detailed proof of Proposition 3.1 below.
Proof. Since the distributions of the auxiliary variables for both student and teacher are the same, i.e, (c) (e), the surrogate distillation loss as in Eq. (4) can be rewritten as follows.
log Po(yle, a)) x))
Ldd (e) (m)) (ule,
log Po(yle, pl)))
Notice that Polyle, ())) = poly, = pa(ely, ,, x x)Px (yl=). Similarly, also holds that Po(yle e, (po(e)) po(Ely, xp (z)(z)) Thus, we can further rewrite the surrogate distillation loss Led as below.
log (o(E(y,x) + log (o(()x)) po po(cly,x)) KL (po(E)y, po(ely, x))] + Epancs (2) [KL (lylp) po(y x))]
Ltd Epint (n (vlar)
(3)(2z)
E (w)p)
Since the first term is also non-negative, further holds that
Lad 2 E (x))
[KL (loggooyyl (y()) || po(y)[=)) Led
which finishes our proof.
C ALGORITHM SUMMARY
We summarize our knowledge distillation method for a general DGM Algorithm 1. Please note that we v; denote non-root random variable in DGMs, which is either latent variable or
14

---

Under review as a conference paper at ICLR 2022
target variable yy- Ve,i and va denote sampling point of node v; in the teacher DGM and the student, respectively. It can be observed from the distillation process that we only need to traverse the graph once to compute both the surrogate distillation loss and the latent distillation losses. Line 9 is our latent distillation loss. In Line 10-12, we descrihe the latent variables of the teacher and student model that shared indirectly via sharing the auxiliary variables Line 14 computes the surrogate distillation loss, Line 15-16 describe that the target variables of teacher and student model are shared directly. Line 19 presents the back propagation and gradient descent.
Algorithm 1 Our Distillation Method
Require: teacher DGM Gg. student DGM Go. learning rate 7), hyperparameter X, empirical distribu- tion of input dataset Pdata (x) (only required when x o 0)
Ensure: distilled student DGM Go
1: while 0 not converged do
2: sample mini-batched x from Pdata (x) // (only required when x: + 0)
3: set Toour = 0
4: while 9 not fully traversed do
5: grab variable where Pa(v;) have all been sampled
6:
set P (v; Pa(
7:
set p = po (v. )a(
8:
if v; is latent variable then
9:
(d(pa.i:Po.s) sample E; from (e.)
Cour Lour
10:
sample va,i from Pá.ı according to E; // set vg =.
12:
sample Vo, from Pa,; according to E; // set Vo = -o((a(v)), (.,))
else (v; target variable}
14:
Cour +- Lour + d(Po.i.Po.))
15:
sample V4.i from Pø.i 16: Vg.i U,b
17:
end if
18:
end while
19: 0 <- 1l 00
&Cour
20:
end while
D ILLUSTRATION OF GRAPHICAL MODELS FOR DIFFERENT APPLICATIONS
Fig. 6 shows different forms of deep DGMs used in our experiments.
7
7.2
22
Z3
Z3
y1
Z
Z
Z--
Vr-1
Z
7
V1
7
y
y2
(d)
Figure 6: DGMs used in our experiments. (a) 5-layer DGM without input variables. (b) 6-layer DGM with one binary input variable indicating which domain the sample comes from. (c) DGM with as many latent variables and target variables the sequence length and input variable. (d) 5-layer DGM with two target variables and no input variable. (e) 2-layer DGM with one input variable
15

---

Under review as a conference paper at ICLR 2022
E DATASETS
To evaluate the performance of the proposed distillation method, we conduct experiments on the following benchmark datasets.
Old Faithful Geyser Härdle et al. (1991): this is 2-dimensional Geyser data. We use all the 272 samples for training.
IAM Online Handwriting Liwicki & Bunke (2005): it has 13040 samples and each sample is a sequence of (z, y) coordinate together with binary indicator of pen up/down. use all the examples for training.
SVHN Netzer et al. (2011): This dataset about house numbers in street views with categorical label of 10 classes. We use 73257 images for training and 26032 images for testing respectively.
CIFARIO Krizhevsky et al. (2009): This dataset consists of 60000 natural world images, with categorical label of 10 classes. The data is split into 50000 and 10000 images for training and testing.
CelebA Liu et al. (2015): It contains 182, 732 RGB 128 x 128 x 3 images of celebrity faces. We use 162770 images for training and 19962 images for testing.
Mashion-MNIST Sohn et al. (2015): This dataset about Grey-scale dressing images. We use 60000 images for training and 10000 images for testing in our experiment.
F MODEL CONFIGURATIONS AND HYPERPARAMETER SETTINGS
summarize the detailed model configurations and hyperparameter settings for the proposed method in different tasks below.
For all the tasks, we choose d(.; :) the surrogate distillation loss Lad to be KI. divergence. When latent variables are discrete, d(-,) in the latent distillation loss L2 is chosen to be KL divergence and À is set to 1. When latent variables are continuous, we choose d(.,-) in C., to squared 2- Wassersttii distance because it has analytical solution two Gaussian distributions. In addition, we normalize Zz.i w.r.t. the size of vector z;. We empirically found that normalized latent distillation loss works slightly better than unnormalized one. In all the following experiments, À is chosen after simple grid search.
The main training configurations for different tasks summarized in the Table 3 below.
Table 3: Configurations summary
Configuration
VAE Compression
VRNN Compression
HM Compression
Discriminative DGM Compression
Teacher size
5.4M
12M
354
2.6M
Student size
0.01M-1.4M
2.4M
102
0.40M
Optimizer
Adam
Adam
Adam
Adam
Learning rate
le-4
le-4
le-2
le-3
Weight decay
le-4
le-4
Batch size
128-256
32
272
256
F.1 DATA-FREE VAE COMPRESSION
In our experiments, we distill a VAE with the same DGM and parameterization as the Ladder VAE Sønderby et al. (2016). The DGM structure of its generative model (decoder) shown in Fig. 6(a). Slightly different from Sønderby et al. (2016), we use Convolutional Neural Network (CNN) to parameterize both the bottom-up and top-down network in VAE, instead of simple MLP. addition, we adopt spatial latent variables (latent variables are spatially arranged, as the feature map in CNN) in the neural networks. Each CNN is composed of consecutive residual blocks He et al. (2016). We fix the size of the teacher model to 5.4M parameters, with almost symmetric structure the bottom-up network and top-down network. To demonstrate the robustness our method, we vary the student model size with different compression times.
16

---

Under review as a conference paper at ICLR 2022
In the experiments, set the depth of latent variables to 32 for both the teacher and student VAE. The hidden dimension residual blocks varies from 4 to 64 in the student model, and is set to 128 for the teacher VAE. Mini-batch sizes are set to 256 for SVHN and CIFARIO and 128 for CclebA. When VAE is trained from scratch, we adopt linear 3 annealing, which is called warm up in Sønderby al. (2016) for first half of the iterations stabilize model learning. For CelebA dataset, we resize the images to the shape 64 x 64 x 3. Model training is stopped after 57k, 46k, 63k iterations for SVHN, CIFARIO, and CelebA, respectively.
F.2 VAE CONTINUAL LEARNING
In our experiment, we still use the same 5-layer VAE as the above VAE compression experiment. Following Rao et al. (2019), we add an additional discrete input variable to the graph for the sake of indicating where the data comes from. As result, our VAE model is turned to a 6-layer Conditional VAE, as illustrated Fig. 6(d).
For VAE continual learning, we have the following distillation loss.
i Lecontinaal (Poas Pro (vi) [- log (b)]
(B pall)) (ogggoo))) + E E |- log pa (y))
(16)
= 2 (KL(pota()) |l po(y)]] + HLLKLLaa KL
(ey)) +
where è is a constant independent of 0. (y) and Pnew (1) are the empirical distribution of old and new dataset, respectively. For the above loss, since Pola is not accessible, we the old learned VAE's distribution (y) as a replacement. Accordingly, we replace (L(Pota(y) || po()) with KL (t) pe po(y)). This results in our desired distillation loss, transferring knowledge from teacher Po (y) to student pe (y), which, however, is intractable as discussed earlier. To tackle this issue, we use our distillation loss Cour instead. As result, the total loss for continual learning is rewritten as
1 (Lour + KLlpaew (y) po()))
Leontivenal
(17)
In this experiment, follow the model configurations the above VAE compression, except that we resize images to 32 x 32 x 3 and do not use running statistics for batch normalization layers during evaluation.
F.3 DATA-FREE VRNN COMPRESSION
In this experiment, we build and distill VRNN with the same DGM and parameterization in Fraccaro al. (2016), as illustrated in Fig 6(c). VRNN consists of as many latent variables and target variable as its sequence length. We fix the number of hidden layer in all densely connected networks to 1, and fix the RNN to be 1-layer LSTM as in Fraccaro et al. (2016). The dimension of latent variables is set to 16. For the teacher model, the hidden dimension of LSTM is set to 1200. Feature size target variables, latent variables, and hidden dimension of all densely connect networks are set to 512. For the student model, the hidden dimension of L.STM is set to 600. Feature dimension of target/latent variables, and hidden dimension of all densely connect networks are set to 128.
F.4 DATA-FREE HM COMPRESSION
Helmholtz Machine (HM) Neal (1992); Saul al. (1996) is classical generative modeling technol- ogyGregor et al. (2014); Bornschein & Bengio (2014); Vértes & Sahani (2018): ; Bornschein et al. (2016) that consists of: 1) Sigmoid Belief Network (SBN) as generative graphical model; 2) another SBN as amortized inference network; 3) Wake-Sleep learning strategy. The classical HM has only one target variable and each layer is parameterized by a linear transformation. In our experiment, in order to demonstrate the applicability of our distillation method, we extend it to 5-layer DGM with two target variables, as shown in Fig. 6(d). The inference network is also modified to match the true posterior dependency structure. Following the similar idea in Krishnan et al. (2015) and Bornschein & Bengio (2014), each conditional distribution both generative and inference network parameterized by 2-layer MI.P instead of linear transformation, which turns our HM to deep HM.
17

---

Under review as a conference paper at ICLR 2022
In experiment, we fix latent variables two-dimensional vectors with binary elements. The hidden dimension of MLP is set to and for teacher and student model respectively.
F.3 DISCRIMINATIVE DEEP DGM COMPRESSION
In this experiment, we distill Discriminative Deep DGM with the same 3-layer MLP parameteriza- tion as Alemi et al. (2016). We fix the hyperparameter 3 in original training loss to be 1e-3. Latent variable set be 256 dimensional vector. The hidden size of MLP teacher and the student model, respectively. Models in our experiment are trained for 46k iterations for MNIST, Bashion-MNISS dataset, and are trained for 57k iterations for SVHN dataset. Images in SVHN dataset are first turned to grayscale.
1024 and 256 for the
G EXPERIMENTS ON VAE COMPRESSION
G.1 SIMILARITY BETWEEN TEACHER AND STUDENT
Fig. compares the FID between samples generated from the student and teacher for different methods. We can observe that method beats the baselines: local distillation and trained from scratch.
240
out-dill leton
local cooll-tllliee atio
tral-scratth
disilation lncaldiell beion
our distiatoon Baloon
0/200/1000)
0.100 24e]
11000100
14M
D.N (380
10M (5ec)
0.0MM (9600
(186s)
5.10 (154
(a) CelebA
(b) CIFARIO
(c) SVHN
Figure 7: FID between samples generated by the student and teacher averaged over 3 random seeds.
G.2 QUALITATIVE RESULTS ON CELEBA
We also conduct experiments to compare the generated samples by different knowledge distillation (KD) methods on the CelebA dataset, as illustrated in Fig. 8. It can be observed from Fig. 8(c) that images generated by our method are high-fidelity, as good as ground truth and those generated by the teacher, which means method limited-size VAE model is incapable of learning such a complicated distribution from scratch on its own. Hence, the images generated by the student VAE trained from scratch low quality, as illustrated in Fig. 8(d).
imitate teacher's performance very well. However,
G.3 HYPERPARAMETER STABILITY IN THE LOSS
We further demonstrate the hyper- ppramepar stability of our method. Fig. shows the relationship between \ and FID) score. It can be observed that as À varies from to 0.0 to 1e3, the FID score of our distillation method does not vary too much. means that our method stable and robust to the hyper-parameter A.
H EVALUATION ON DATA-FREE HELMHOLTZ MACHINE COMPRESSION
We apply the proposed approach to compress 5-layer HM trained with Wake-Sleep algorithm to smaller HM in a data-free manner. We still compare it with the baselines: training from scratch and local distillation. When model is trained from scratch instead of distillation, we add an inference model (encoder) with symmetric structure and almost the same number of parameters as its generative model. The experimental results shown in Fig. 10.
18

---

Under review as a conference paper at ICLR 2022
(a) Ground truth
(b) Teacher
(c) Our method
(d) Local distillation
(c) From scratch
Figure 8: Images generated by VAE trained on CelebA dataset. (a) Ground truth dataset. (b) Teacher VAE (5.4M parameters) trained from scratch. (c) Student VAE (1 (1.4M parameters, 3.9 times smaller) distilled by our method. (d) Student VAE distilled using local distillation. (c) Student VAE trained from scratch.
110
90
80
our-distilaaton
70
local-distilation train-scratch
50
1e3
1e1
16-1
1e-3
1e-5
0.0
lambda
Figure 9: Effect of À in Eq. (7) on the performance of our method,
We can observe that our distillation method still outperforms the student model trained from scratch, just VAE and VRNN compression. In addition, it shows that optimizing capacity- limited HM may not learn a powerful model that can generate high-fidelity samples. In contrast, our method enables small student HM to attain good performance larger teacher model.
19

---

Under review as a conference paper at ICLR 2022
(a)
(h)
(c)
(d)
(e)
Figure 10: Generation distribution of HMs. We performed and plot kernel density estimation using 272 generated samples for each HM. (a) Ground truth. (b) Teacher HM (354 parameters) trained from scratch. (c) Student HM (102 parameters, 3.5 times smaller) distilled using our method. (d)
Student HM distilled using local distillation. (e) Student HM trained from scratch.
In our experimental results, find that there is tiny difference between our method and local distillation, partly because this 2-dimensional modeling task is too easy for highly flexible HMs to learn Both our method and local distillation can converge after few iterations. Hence, they have comparable performance to each other.
I SCALABILITY ON LARGE NUMBER OF DISCRETE VARIABLES
Our proposed method can also be scalable to a DGM with large number of discrete latent variables. We take a DGM with L (e.g, 20) layers of discrete latent variables as an example to demonstrate the good performance of our approach, as shown Fig. 11. In our experiment, we let student mimic the output distribution of the teacher with L-layers latent variables for L E l,..., 20. Each layer of the teacher follows binomial distribution p( 1 z.) (11000, p(z))) where p(z,) = =(Tom p(z1) is uniform distribution U10 1]. The student is parameterized by neural networks with residual structure He et al. (2016). Then density ratio estimation Nguyen et al. (2010); Sugiyama al. (2012) is used to measure the KI. divergence between the teacher and the student. We observe from Fig. 11 that even all the latent variables are discrete, our method still has the capability of mitigating the imitation error occurred in the shallow layers and achieves a very good imitation performance. For the local distillation method, however, its imitation errors are accumulated as the number of layers increases.
E 1.2; 1000
accumulated error (KL divergence)
0.4
local-distillation our-distillation
0.3
0.2
0.1
0.0
4
8
12
16
20
number of layers of latent variables
Figure 11: A toy example of accumulated error (KL divergence) between the teacher and student for local distillation and our method in a DMG with discrete random variables.
20

---

Under review as a conference paper at ICLR 2022
J DISCUSSION AND FUTURE WORK
We further discuss the limitation of the proposed framework. Our framework is only applied to DGMs where latent variables can be reparameterized. However, we would like point out that the definition of reparameterization trick in our framework is slightly different from the classical one for VAE training Kingma & Welling (2013). Our reparameterization is more general than that for propagating gradients for model training. Specifically, reparameterization in our work is used for compacting the DGMs cemi-auxiliary forms rather than propagating gradients. Different from the classical reparameterization for model training that requires continuous latent variables, ours can be applied to wider range of variables z, including both continuous and discrete variables. addition, the transformation function g(-) in our framework can be either differentiable non-differentiable. Hence, our method can be applied to much more DGMs than the classical one. Table 4 compares the different requirements for method and the classical reparameterization trick.
Table 4: Different requirements for our method and the classical reparamcterization trick. Traditional reparameterization differentiable continuous
Requirements
Our framework differentiable / niffiffferettiable
Transformation function g()) Latent variable
continuous/ / discrete
Another limitation of our framework is that it needs to consist of the same number of random variables (these variables can be arranged in different structures). For future work, we plan to extend the proposed method to deal with more difficult cases where the teacher and students have different number of random variables
21


Reviews:
[{"review_id": "VDzfVvMeJgG", "reviewer": "Reviewer_XcHW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Writing:\n* The paper is relatively readable. The main idea came across clear, but it can enjoy thorough editing, definitely in the experiment section.\n\n# Method:\n* The paper's main claim is that the local method can result in the accumulation of errors. However, the second term in Eq 7 somewhat does the same. In fact, one can view the proposed method as a combination of the marginalization and local methods together. \n\n* I am not convinced by the idea of the semi-auxiliary graph that yields the loss function in Eq 3,4. The example in Fig 1 is too simple. For example, when we factor out the z's, the resulting graph on the observed variables is no longer DAG. Consider,  $y_1$ <-- z --> $y_2$ if $z$ is factored out then y_1 and y_2 are connected with undirected edge. How does Eq 3 handle such a case? \n\n* The advantage of the method is mostly for DGMs with continuous latent variables. For discrete latent variables, the local distillation is used. I'm not sure how well this scales to models with the large number of discrete latent variables.\n\n* One limitation is that the teacher and students should have the same architecture. Can it be extended to cases when the architectures are not necessarily the same?\n\n\n# Experiment:\n* The results in the experiment section are not convincing. For example, the difference between the performances of different methods is within the same standard error, especially for the local method and the proposed method. \n\n* The choice of some of the tasks is questionable for the knowledge of DGM distillation. For example, why is continual learning a good task to show that the knowledge distillation method working? The argument made in the paper is that the Delta of forgetting the previous task is small, which is acceptable for continual learning; however, it is not clear why continual learning is a good task to show knowledge distillation. \n\n* In figure 5: it is not clear why the authors claim that the proposed method produces better results. For example, why (c) is better than (e)? This experiment is very qualitative and subjective.\n\n* None of the experiments (except HM) really motivate the knowledge distillation for DGM. VAE is a very simple DGM, and there is no real structure in the graphical model. The authors could have simulated data from a hierarchical graphical model and used a complex teacher to learn that model, then applied their method to show their approach can recover the true model from the teacher. For example, see [1] for examples such simulation -- a simple mixture model.\n\n* I strongly recommend the authors look into the metrics introduced in [2]. Several metrics and experiments presented in that paper can be adopted or adapted for the DGM knowledge distillation.   \n\n# References\n\n[1] https://arxiv.org/pdf/1603.06277.pdf\n[2] https://arxiv.org/pdf/2106.05945.pdf", "summary_of_the_paper": "The authors propose a method for knowledge distillation specifically for  Deep Directed Graphical Models. The authors compare their method with marginalization methods, which integrates the latent variables out, and the factorized (local) method, which distills knowledge between teacher and student on each factor. They validate their approach on continual learning, model compression, and discriminative learning. \n\n", "summary_of_the_review": "* The paper is relatively clear.\n\n* The experiments are not well-chosen and the results are not convincing. \n\n* The method section does not explore the idea of knowledge distillation for DGM deeply. There are questions about the generalizability and scalability of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Gv0cEnOxsMP", "reviewer": "Reviewer_sRSN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the proposed distillation framework is novel, and the entire paper is well-organized. However, I have the following concerns/questions,\n\n1. The title claims for a unified KD framework for deep DGMs. However, the proposed distillation loss only applies to DGMs where the latent variable z has a reparameterization form. I believe the author should highlight this limitation of their framework.\n\n\n2. In equation (3), the expected KL divergence is computed over $p_{\\phi}(y_{<j}|x)$.  I am wondering if the following three ways will improve the performance of local distillation? Though they are no longer equivalent to equation (2), my intuition is that, for the original objective (equation (3)), the conditional KL divergence is computed with respect to the teacher's distribution $p_{\\phi}(y_{<j}|x)$, however, at inference time,  the conditional distribution of the student model is computed based on its own distribution $p_{\\theta}(y_{ <j}|x)$. This makes the training objective does not consistent with the test objective.\n\n\n- $\n\\mathcal{L}\\_{kd} = \\mathbb{E}\\_{p_{data}(x)}\\left[\\mathbb{E}\\_{p_{\\theta}(y\\_{<j}|x)} [KL(p_{\\phi}(y_j|y_{<j}, x)|| p_{\\theta}(y_j|y_{<j}, x) )] \\right]\n$\n\n\n- $\\mathcal{L}\\_{kd} = \\mathbb{E}\\_{p_{data}(x)} \\left[\n\\mathbb{E}\\_{p_{\\theta}(y_{<j}|x)} [KL( p_{\\theta}(y_j|y_{<j}, x) || p_{\\phi}(y_j|y_{<j}, x)) ] \n \\right]$  \n\n- $\\mathcal{L}\\_{kd} = \\mathbb{E}\\_{p_{data}(x)} \\left[\\sum_j (1-\\lambda)\\cdot \\mathbb{E}\\_{p_{\\phi}(y_{<j}|x)} [KL(p_{\\phi}(y_j|y_{<j}, x)|| p_{\\theta}(y_j|y_{<j}, x) )]   + \n\\lambda\\cdot\\mathbb{E}\\_{p_{\\theta}(y_{<j}|x)} [KL(p_{\\phi}(y_j|y_{<j}, x)|| p_{\\theta}(y_j|y_{<j}, x) )] \n \\right]$  (for some $\\lambda \\in [0, 1]$)\n\n3. Can you explain more on why do you remove $ \\epsilon_i$ in equation (6)? I don't understand why such a change will better penalize the dissimilarity of latent variables $z_i$? \n\n4. Can you provide some qualitative comparisons between the samples from the distilled DGMs on the CelebA dataset? ", "summary_of_the_paper": "This paper proposes a new knowledge distillation framework for directed graphical models based on the reparameterization trick. The new distillation framework overcomes the intractable marginals in marginalized distillation, and error accumulation in local distillation. Empirically, the new distillation framework surpasses the baselines on deep generative model compression, VAE continual learning, and discriminative DGMs compression.  ", "summary_of_the_review": "Overall, I like the solution proposed by this paper to address the issues in marginalized distillation and local distillation. However, I believe the authors did not fully investigate the failure reasons for local distillation. The argument in the current paper is kind of vague and not well-supported. I believe a more rigorous analysis and carefully designed experiments are needed to illustrate the argument (e.g., as I mentioned above). I will increase my rating if the authors can provide more convincing results.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FtndfHh3Nk", "reviewer": "Reviewer_twdH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The paper proposes a knowledge distillation framework for deep DGMs on various applications.\n\nWeaknesses: After reading the paper several times I still don't see the significant novelty of this paper. (A) The paper converts each hidden random variable to a deterministic variable via the reparameterization trick. This is a well-known technique. The VAE, for example,  uses this technique during training. Although the paper argues that \"we do not primarily use reparameterization trick for model training. Rather, we leverage it to convert the latent variables z in DGMs to deterministic variables so that we can effectively distill knowledge from a compact form of DGM\", but isn't this very straightforward? I don't see any big difference between using the reparameterization trick during training and KD. The authors should provide a discussion on this. (B) I don't see a difference between equations (4) and (2) when applying to VAE because, during VAE training, the sampling over the auxiliary random variables \\epsilon is implicitly included even though we just apply equation (2). (3) Equations (5) and (6) look very intuitive and straightforward. I am more interested in knowing what theoretical guarantee we can have when using these losses. (C) For experimental evaluation, could you compare your model with more state-of-the-art KD baselines? (eg., Figures 4 and 5 and table 1).  ", "summary_of_the_paper": "This paper proposes to use the reparameterization trick to convert the latent variables in DGMs to deterministic variables in the context of KD. It then proposes a surrogate distillation loss and latent distillation loss and evaluates the performance of the proposed method in three applications. Experimental results confirm the effectiveness of the proposed model.", "summary_of_the_review": "I am mainly concerned about the novelty and clarity of this paper. At the current stage, I don't recommend the paper for acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1AjWjKZ2-S5", "reviewer": "Reviewer_1kU6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1. The authors do a good job of giving basic preliminaries and of ruling out the naïve marginalized distillation and local distillation approaches \n2. After the compact DGM reductions for both the teacher and student networks, each target variable has direct dependence on the input x and prior y_i’s. This is a neat approach. \n3. Proposition 3.1 looks correct to me, when KL divergence is chosen.\n\nConcerns & questions:\n1. Just verifying: The reduction to a compact semi-auxiliary form is a novel contribution of this work, correct? (Fig. 1c, 1d, 3e)\n2. How do we get the correct choice of deterministic transformation g(.) for the reparameterization trick of the original teacher network? p_\\phi() is a neural network, so I am curious how to get g(.) without loss of accuracy. (Algo.1, lines 11-12). Please give a detailed example.\n3. The chain error accumulation is reduced as the number of layers reduces in the semi-auxiliary form, right (pg5, para2) ? Or is there any other rationale to it?\n4. I wonder how the performance is with only using L_{sd} in eq(7), \\lambda=0 setting?\n5. I am a bit confused about the VAE compression expts. Each layer is considered a latent variable `z’, I presume. In this case, what is the student network chosen (I might have missed it)? Also, curious to know, how well the proposed L_{sd} loss works with \\lambda=0 setting. Kindly clarify this expt.\n", "summary_of_the_paper": "The authors propose an unified Knowledge Distillation technique for general deep directed graphical models. They use the reparameterization trick on the intermediate latent variables of the original DGM network and the student network. This converts the networks to a compact semi-auxiliary form. Then they use a surrogate distillation loss (combined with latent loss) to reduce the error accumulations over the chain of random variables. They discuss the similarity of their technique with others and demonstrate its performance for 3 applications. ", "summary_of_the_review": "The work is good and the paper is well written. I feel the contribution of the work is not that novel. My confidence in evaluation will increase once the authors address the concerns raised above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework for distilling deep directed graphical models where the teacher and student models have the same number of latent variables z. The key idea is to reparameterize both models in terms of standardized random variables epsilon with fixed distributions and train the student to match the conditional distributions of the observed variables/targets given the values of the standardized RVs epsilon. The approach aims to avoid error compounding that affects the local distillation approach, where the student is trained to match conditional distributions of the teacher model (without the above reparameterization). To deal with discrete latent variables and vanishing gradients the authors augment the target matching loss with the latent distillation loss that matches the local distribution for each z_i given the standardized variables epsilon it depends on.

Positives
-The paper tackles an important problem.
-The idea of using reparameterization for distillation in this way makes a lot of sense for continuous latent variables and could be impactful.
-The experiments provide some evidence in support of the idea.

Negatives
-There are considerable issues with the clarity of writing: for example, it is really not clear how (and why) the method is supposed to work for discrete latent variables. The explanation provided by the authors in their response to the reviewers was helpful but still not clear enough.
-The fact that the teacher and student models need to have the same number of latent variables (and perhaps even the same structure) is a big limitation of the method given the claim of its generality, and thus needs to be clearly acknowledged and discussed. For example, the method cannot be used to train a student model with fewer latent variables than the teacher, which seems like a very common use case.
-The experimental evaluation is extensive but insufficient, in large part due to the evaluation metrics. Given that VAEs are trained by maximizing the ELBO (and distilled by minimizing a sum of KLs), it makes sense to also evaluate them based on the ELBO rather than solely on the FID, is done in the paper. The VRNN experiment would be much more informative if it included a quantitative evaluation (e.g. based on ELBO).

In summary, the paper has considerable potential but needs to be substantially improved before being published.
Decision: Reject