Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a scalable approach to train generative classifiers using information maximizing representation learning, with the motivation that generative classifiers could be more robust to adversarial attacks than discriminative classifiers. An off-the-shelf mutual information maximizer (MINE, DIM) is used to learn low-dimensional representations of images. Then, class-conditioned generative models of the representations are learned avoiding full generative modeling of the images. An additional loss is used to train the generative classifier which maximizes likelihood margins. Finally, percentile-based thresholds of the class log-probabilities is proposed to be used to reject classification for out-of-manifold inputs.

The paper cites existing literature which indicates that generative classifiers might be more robust to adversarial attacks, and uses recently-proposed representation learning techniques to scale up learning generative generative classifiers. The motivation of the proposed technique is clear, and the problem itself is relevant. Similar prior work use generative modeling at the pixel-level. Generative modeling of representations is novel, afaik.

The technique is evaluated on out-of-distribution sample detection (FashionMnist->MNIST and Cifar10->SVHN) and adversarial attacks. FGSM, PGD, CW-L2 attack, deepfool.

The experiments section is not very clearly written. Some of the evaluation itself is nonstandard in which only the first example of digit 0 of MNIST is used. The paper needs to have a clearer explanation and interpretation of the results. It’s not clear what the logits in table 4 and table 6 are, and what is being shown by the comparison.

Summary: The authors propose a new technique for training generative classifiers with the aim to improve robustness to adversarial attacks and confidence on out-of-distribution samples. The method is well-motivated and explained, but the experiment section is not very clearly written and I’m not confident whether the technique represents an advancement in the state-of-the-art or not.

Misc comments:

“By adopting a generative approach p(x, y) = p(y)p(x|y), we assume that the data follows the manifold assumption: the (high-dimensional) data lies on low-dimensional manifolds corresponding to their class labels.”

This sentence seems to be making a stronger claim than is needed, which is possibly incorrect. Assuming a generative approach doesn’t require assuming the manifold assumption. 

Nit: “possible MI low-bounds”
Nit: “state-of-the-arts”
Typo: “The original image is the fist sample of class 0”

review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This paper studies classification problems via a reject option. A reject option could be useful in prediction problems to handle Out-of-distribution examples. The classification procedure studied in this paper builds on three components 1. An auto-encoder that obtains a latent low-dimensional representation of the data point 2.  A generative model that models the class-conditional probability model and 3.  a margin based loss function that learns a classifier that provides a large probability mass to the class-conditional distribution corresponding to the correct class.  The final decision procedure is to reject an input if the best class conditional probability is small and to use the class corresponding to the best class conditional probability otherwise. 

On the whole I like the paper and think that the problem tackles an important problem. I have a few comments
1. I would like to see what is the log-likelihood assigned by the proposed procedure on OOD samples and would like to see a comparison of the log-likelihood assigned by other procedures.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposed a supervised method for robust classification. By utilizing mutual information constrain on the encoder, supervised probabilistic constraint on the class conditional probability, and introducing a margin to the maximum likelihood, the proposed method can manage a high classification accuracy and detect the out of distribution data. 

The motivation is good, the writing is OK. The structure of the paper needs to be refined.  The experiments are not very strong. Several concerns are listed:

1.	It will be better if the author can show a structure map for Encoder, classification, class condition embedding, and mutual information evaluation networks, to clarify their relationships.
2.	In Table 2, what is the accuracy for the rejection (outlier detection)?
3.	The choose of the threshold for rejection is tricky. It will be better if the author can provide some rules for choosing the threshold which can be generalized to a new dataset.
4.	The overall loss functions have three components, what is the contribution of different components to the final performance experimentally?
5.	The architecture for encoder is large. Is it proper for Mnist which is a relatively simple dataset?
6.	The comparison talked in the paragraph “Is Fully Generative Model Necessary for Generative Classification”, are these accuracies obtained from a comparable network size? It only makes sense if they are obtained using a comparable parameter size.

review_title: Official Blind Review #1

Meta-review: This paper combines a well-known, recently proposed unsupervised representation learning technique technique with a class-conditional negative log likelihood and a squared hinge loss on the class-wise conditional likelihoods, and proposes to use the resulting conditional density model for generative classification. The empirical work appears to validate the claim that their method leads to good out of distribution detection, and better performance using a rejection option. The adversarial defense results are less clear. Reporting raw logits is a strange choice, and difficult to interpret; the table is also difficult to read, and this method of reporting makes it difficult to compare against existing methods.

The reviewers generally remarked on presentation issues. R1 asked about the contribution of various loss terms, a matter I feel is underexplored in this work, and the authors mainly replied with a qualitative description of loss behaviour in the joint system, which I don't believe was the question. R1 also asked about the choice of thresholds and the issues of fairness of comparison regarding model capacity, neither of which seemed adequately addressed. R3 remarked on the clarity being lacking, and also that "Generative modeling of representations is novel, afaik." (It is not; see, for example, the VQ-VAE line of work where PixelCNN priors are fit on top of representations, and layer-wise pre-training works of the mid 2000s, where generative models were frequently fit on greedily trained feature representations, sometimes in conjunction with a joint generative model of class labels).  R2's review was very brief, and with a self-reported low confidence, but their concerns were addressed in a subsequent update.

There are three weaknesses which are my grounds for recommending rejection. First, this paper does a poor job of situating itself in the wider body of literature on classification with rejection, which dates to at least the 1970s (see Bartlett & Wengkamp, 2006 and the references therein). Second, the empirical work makes little comparison to other methods in the literature; baselines on clean data are self-generated, and the paper compares to no other adversarial defense proposals. In a minor drawback, ImageNet results are also missing; given that one of the purported advantages of the method is scalability, a large scale benchmark would have strengthened this claim. Third, no ablation study is undertaken that might give us insight into the role of each term of the loss. Given that this is a straightforward combination of well-understood techniques, a fully empirical paper ought to deliver more insight into the combination than this manuscript has.
Decision: Reject