Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors focus on alleviating the catastrophic forgetting problem in continual learning.  The authors propose a discriminative variational autoencoder (DiVA) to solve this problem under the generative replay framework. DiVA modifies the objective function of VAE by introducing an additional term that maximizes the mutual information between the latent variables and the class labels.  

The authors do not thoroughly explain the motivation of this paper. The authors do not explicitly define continual learning, incremental learning, and catastrophic forgetting problem. It is also not clear to me why these problems are important. 

The idea that introduces labels in VAE is not novel. For example, Narayanaswamy et al. [1] also propose to utilize labels to VAE. I do not understand why making use of labels is important for solving the catastrophic forgetting problem and how the labels are useful in the generative replay process. It is also not clear to me how domain translation is relevant to continual learning. 

In terms of modeling, since the input into the prior network has finite possible discrete values, we do not need a fully connected network to generate $\hat{\mu}_c$ and $\hat{\sigma}_c$. Instead, we can directly optimize $\hat{\mu}_c$ and $\hat{\sigma}_c$ for each $c$ as parameters.

The paper provides some good experimental results. But the problem settings are not clear to me. I do not understand how the model is trained to solve multiple tasks. Do the same model is trained for multiple tasks? Is each of the tasks trained sequentially or simultaneously? It is also not clear to me why CIFAR datasets involve two domains and how these domains are relevant in each of the tasks.

In summary, since DiVA gives a good experimental performance, the proposed method might be promising. However, it looks to me that the authors need to better explain the motivation of DiVA, the differences of DiVA from existing supervised VAE, and the experimental settings, before the acceptance of this paper.

References
[1]Narayanaswamy, Siddharth, T. Brooks Paige, Jan-Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. "Learning disentangled representations with semi-supervised deep generative models." In Advances in Neural Information Processing Systems, pp. 5925-5935. 2017.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: -- This paper seeks to combine several ideas together to propose an approach for image classification based continual learning tasks. In this effort, the paper combines previously published approaches from generative modeling with VAEs, mutual information regularization and domain adaptation. 

I am a making a recommendation for reject for this paper with the main reason being that I believe the primary derivations for their method appear flawed. 

--In the main section describing the approach (Section 4), the authors start with a claim that Equation 1 and 2 are equal; I don’t believe 1 and 2 are equal.

--In Section 4.1, it appears that they are instead making a claim about Equation 2 being a bound for equation 1; but even this derivation appears to have a problem. The following is the concern:

--In the second line of Equation 5, the KL term appears to be measuring a distance between distributions on two different variables; z|c and c|z. If one were to interpret the second one as the unnormalized distribution on z defined via the likelihood for c given z; even this has an issue because then the expression for KL where we plug the unnormalized density in place of the normalized need not be positive which is something they need to derive their bound.

--Another issue is that the regularization lambda should apply to both the terms in the bound but in Equation (7) only appears selectively for one of the two terms. 

It is also not clear how the loss function proposed differs from that of the CDVAE, etc.  If the novelty is in applying to continual learning and new datasets, it is not clear that this is sufficient.

Additional feedback for authors (not part of the main decision reasoning):

- What is dt in Algorithm 1 description?

Figure 1:
-typo “implmented”
-What’s the 3d plot supposed to represent?
Doesn't the classification loss have a dependency on the input condition?

--What does a "heavy classifier" imply concretely? 

--“Redundant weights” seems like not a very strong constraint especially for a small cardinality label space (like 10, in the case of this paper).

--The notation for the proposed parameters theta, theta’, phi, phi’ are not consistent with the notation in the intro section, where phi was used for the encoder and theta for the decoder. In later sections they use theta and theta’ for encoder/decoder resp.

-- “When the encoder and decoder networks are sufficiently complex, it is enough to implement each the prior and classification network as one fully-connected layer” → what do the authors mean “when … networks are sufficiently complex” or do they actually mean when the “when the problem is simple enough”?

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper devises a pipeline that aims to address catastrophic forgetting in continual learning (CL) by the well-known generative replay (GR) technique. The key ingredient of the pipeline is a modern variational auto-encoder (VAE) that is trained with class labels with respect to a mutual information maximization criterion.

The paper does not follow a smooth story line, where an open research question is presented and a solution to this problem is developed in steps. The flowchart in Fig 1 is rather a system design consisting of many components, the functionality of which is not clearly described and existence of which is not justified. This complex flowchart does not even describe the complete task. It is in the end plugged into a continual learning algorithm which also performs domain transformation. All of these pieces are very well-known methods (e.g. VAEs, conditional VAEs, CL, catastrophic forgetting, domain transformation) in the literature and this paper puts them together in a straightforward way. Hence, I kindly do not think the outcome is truly a research result. It is more system engineering than science. 

The next submission of the paper could choose one or few of these pieces as target research problems and develop a thoroughly analyzed novel technical solution for them. If this solution can be proven to improve a valuable metric (e.g. accuracy, interpretability, theoretical understanding, or computational efficiency) of a setup, it is then worthwhile being published.

Minor: The abstract could be improved by providing more clear pointers to the presented novelty.
review_title: Official Blind Review #2

Meta-review: The paper presents a method for continual learning with a variant of VAE. The proposed approach is reasonable but technical contribution is quite incremental. The experimental results are limited to comparisons among methods with generative replay, and experimental results on more complex datasets (e.g., CIFAR 100, CUB, ImageNet) are missing. Overall, the contribution of the work in the current form seems insufficient for acceptance at ICLR.
Decision: Reject