Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new knowledge distillation framework for directed graphical models based on the reparameterization trick. The new distillation framework overcomes the intractable marginals in marginalized distillation, and error accumulation in local distillation. Empirically, the new distillation framework surpasses the baselines on deep generative model compression, VAE continual learning, and discriminative DGMs compression.  
main_review: I think the proposed distillation framework is novel, and the entire paper is well-organized. However, I have the following concerns/questions,

1. The title claims for a unified KD framework for deep DGMs. However, the proposed distillation loss only applies to DGMs where the latent variable z has a reparameterization form. I believe the author should highlight this limitation of their framework.


2. In equation (3), the expected KL divergence is computed over $p_{\phi}(y_{<j}|x)$.  I am wondering if the following three ways will improve the performance of local distillation? Though they are no longer equivalent to equation (2), my intuition is that, for the original objective (equation (3)), the conditional KL divergence is computed with respect to the teacher's distribution $p_{\phi}(y_{<j}|x)$, however, at inference time,  the conditional distribution of the student model is computed based on its own distribution $p_{\theta}(y_{ <j}|x)$. This makes the training objective does not consistent with the test objective.


- $
\mathcal{L}\_{kd} = \mathbb{E}\_{p_{data}(x)}\left[\mathbb{E}\_{p_{\theta}(y\_{<j}|x)} [KL(p_{\phi}(y_j|y_{<j}, x)|| p_{\theta}(y_j|y_{<j}, x) )] \right]
$


- $\mathcal{L}\_{kd} = \mathbb{E}\_{p_{data}(x)} \left[
\mathbb{E}\_{p_{\theta}(y_{<j}|x)} [KL( p_{\theta}(y_j|y_{<j}, x) || p_{\phi}(y_j|y_{<j}, x)) ] 
 \right]$  

- $\mathcal{L}\_{kd} = \mathbb{E}\_{p_{data}(x)} \left[\sum_j (1-\lambda)\cdot \mathbb{E}\_{p_{\phi}(y_{<j}|x)} [KL(p_{\phi}(y_j|y_{<j}, x)|| p_{\theta}(y_j|y_{<j}, x) )]   + 
\lambda\cdot\mathbb{E}\_{p_{\theta}(y_{<j}|x)} [KL(p_{\phi}(y_j|y_{<j}, x)|| p_{\theta}(y_j|y_{<j}, x) )] 
 \right]$  (for some $\lambda \in [0, 1]$)

3. Can you explain more on why do you remove $ \epsilon_i$ in equation (6)? I don't understand why such a change will better penalize the dissimilarity of latent variables $z_i$? 

4. Can you provide some qualitative comparisons between the samples from the distilled DGMs on the CelebA dataset? 
summary_of_the_review: Overall, I like the solution proposed by this paper to address the issues in marginalized distillation and local distillation. However, I believe the authors did not fully investigate the failure reasons for local distillation. The argument in the current paper is kind of vague and not well-supported. I believe a more rigorous analysis and carefully designed experiments are needed to illustrate the argument (e.g., as I mentioned above). I will increase my rating if the authors can provide more convincing results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for knowledge distillation specifically for  Deep Directed Graphical Models. The authors compare their method with marginalization methods, which integrates the latent variables out, and the factorized (local) method, which distills knowledge between teacher and student on each factor. They validate their approach on continual learning, model compression, and discriminative learning. 


main_review: # Writing:
* The paper is relatively readable. The main idea came across clear, but it can enjoy thorough editing, definitely in the experiment section.

# Method:
* The paper's main claim is that the local method can result in the accumulation of errors. However, the second term in Eq 7 somewhat does the same. In fact, one can view the proposed method as a combination of the marginalization and local methods together. 

* I am not convinced by the idea of the semi-auxiliary graph that yields the loss function in Eq 3,4. The example in Fig 1 is too simple. For example, when we factor out the z's, the resulting graph on the observed variables is no longer DAG. Consider,  $y_1$ <-- z --> $y_2$ if $z$ is factored out then y_1 and y_2 are connected with undirected edge. How does Eq 3 handle such a case? 

* The advantage of the method is mostly for DGMs with continuous latent variables. For discrete latent variables, the local distillation is used. I'm not sure how well this scales to models with the large number of discrete latent variables.

* One limitation is that the teacher and students should have the same architecture. Can it be extended to cases when the architectures are not necessarily the same?


# Experiment:
* The results in the experiment section are not convincing. For example, the difference between the performances of different methods is within the same standard error, especially for the local method and the proposed method. 

* The choice of some of the tasks is questionable for the knowledge of DGM distillation. For example, why is continual learning a good task to show that the knowledge distillation method working? The argument made in the paper is that the Delta of forgetting the previous task is small, which is acceptable for continual learning; however, it is not clear why continual learning is a good task to show knowledge distillation. 

* In figure 5: it is not clear why the authors claim that the proposed method produces better results. For example, why (c) is better than (e)? This experiment is very qualitative and subjective.

* None of the experiments (except HM) really motivate the knowledge distillation for DGM. VAE is a very simple DGM, and there is no real structure in the graphical model. The authors could have simulated data from a hierarchical graphical model and used a complex teacher to learn that model, then applied their method to show their approach can recover the true model from the teacher. For example, see [1] for examples such simulation -- a simple mixture model.

* I strongly recommend the authors look into the metrics introduced in [2]. Several metrics and experiments presented in that paper can be adopted or adapted for the DGM knowledge distillation.   

# References

[1] https://arxiv.org/pdf/1603.06277.pdf
[2] https://arxiv.org/pdf/2106.05945.pdf
summary_of_the_review: * The paper is relatively clear.

* The experiments are not well-chosen and the results are not convincing. 

* The method section does not explore the idea of knowledge distillation for DGM deeply. There are questions about the generalizability and scalability of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use the reparameterization trick to convert the latent variables in DGMs to deterministic variables in the context of KD. It then proposes a surrogate distillation loss and latent distillation loss and evaluates the performance of the proposed method in three applications. Experimental results confirm the effectiveness of the proposed model.
main_review: Strengths: The paper proposes a knowledge distillation framework for deep DGMs on various applications.

Weaknesses: After reading the paper several times I still don't see the significant novelty of this paper. (A) The paper converts each hidden random variable to a deterministic variable via the reparameterization trick. This is a well-known technique. The VAE, for example,  uses this technique during training. Although the paper argues that "we do not primarily use reparameterization trick for model training. Rather, we leverage it to convert the latent variables z in DGMs to deterministic variables so that we can effectively distill knowledge from a compact form of DGM", but isn't this very straightforward? I don't see any big difference between using the reparameterization trick during training and KD. The authors should provide a discussion on this. (B) I don't see a difference between equations (4) and (2) when applying to VAE because, during VAE training, the sampling over the auxiliary random variables \epsilon is implicitly included even though we just apply equation (2). (3) Equations (5) and (6) look very intuitive and straightforward. I am more interested in knowing what theoretical guarantee we can have when using these losses. (C) For experimental evaluation, could you compare your model with more state-of-the-art KD baselines? (eg., Figures 4 and 5 and table 1).  
summary_of_the_review: I am mainly concerned about the novelty and clarity of this paper. At the current stage, I don't recommend the paper for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an unified Knowledge Distillation technique for general deep directed graphical models. They use the reparameterization trick on the intermediate latent variables of the original DGM network and the student network. This converts the networks to a compact semi-auxiliary form. Then they use a surrogate distillation loss (combined with latent loss) to reduce the error accumulations over the chain of random variables. They discuss the similarity of their technique with others and demonstrate its performance for 3 applications. 
main_review: Pros: 
1. The authors do a good job of giving basic preliminaries and of ruling out the naïve marginalized distillation and local distillation approaches 
2. After the compact DGM reductions for both the teacher and student networks, each target variable has direct dependence on the input x and prior y_i’s. This is a neat approach. 
3. Proposition 3.1 looks correct to me, when KL divergence is chosen.

Concerns & questions:
1. Just verifying: The reduction to a compact semi-auxiliary form is a novel contribution of this work, correct? (Fig. 1c, 1d, 3e)
2. How do we get the correct choice of deterministic transformation g(.) for the reparameterization trick of the original teacher network? p_\phi() is a neural network, so I am curious how to get g(.) without loss of accuracy. (Algo.1, lines 11-12). Please give a detailed example.
3. The chain error accumulation is reduced as the number of layers reduces in the semi-auxiliary form, right (pg5, para2) ? Or is there any other rationale to it?
4. I wonder how the performance is with only using L_{sd} in eq(7), \lambda=0 setting?
5. I am a bit confused about the VAE compression expts. Each layer is considered a latent variable `z’, I presume. In this case, what is the student network chosen (I might have missed it)? Also, curious to know, how well the proposed L_{sd} loss works with \lambda=0 setting. Kindly clarify this expt.

summary_of_the_review: The work is good and the paper is well written. I feel the contribution of the work is not that novel. My confidence in evaluation will increase once the authors address the concerns raised above.
review_title: null

Meta-review: The paper proposes a framework for distilling deep directed graphical models where the teacher and student models have the same number of latent variables z. The key idea is to reparameterize both models in terms of standardized random variables epsilon with fixed distributions and train the student to match the conditional distributions of the observed variables/targets given the values of the standardized RVs epsilon. The approach aims to avoid error compounding that affects the local distillation approach, where the student is trained to match conditional distributions of the teacher model (without the above reparameterization). To deal with discrete latent variables and vanishing gradients the authors augment the target matching loss with the latent distillation loss that matches the local distribution for each z_i given the standardized variables epsilon it depends on.

Positives
-The paper tackles an important problem.
-The idea of using reparameterization for distillation in this way makes a lot of sense for continuous latent variables and could be impactful.
-The experiments provide some evidence in support of the idea.

Negatives
-There are considerable issues with the clarity of writing: for example, it is really not clear how (and why) the method is supposed to work for discrete latent variables. The explanation provided by the authors in their response to the reviewers was helpful but still not clear enough.
-The fact that the teacher and student models need to have the same number of latent variables (and perhaps even the same structure) is a big limitation of the method given the claim of its generality, and thus needs to be clearly acknowledged and discussed. For example, the method cannot be used to train a student model with fewer latent variables than the teacher, which seems like a very common use case.
-The experimental evaluation is extensive but insufficient, in large part due to the evaluation metrics. Given that VAEs are trained by maximizing the ELBO (and distilled by minimizing a sum of KLs), it makes sense to also evaluate them based on the ELBO rather than solely on the FID, is done in the paper. The VRNN experiment would be much more informative if it included a quantitative evaluation (e.g. based on ELBO).

In summary, the paper has considerable potential but needs to be substantially improved before being published.
Decision: Reject