Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The manuscript proposes a generative approach to detect which samples are within vs. out of the sample space of the training distribution. This distribution is used to adjust the classifier so it makes confident predictions within sample, and less confident predictions out of sample, where presumably it is prone to mistakes. Evaluation on several datasets suggests that accounting for the within-sample distribution in this way can often actually improve evaluation performance, and can help the model detect outliers.

The manuscript is reasonably well written overall, though some of the writing could be improved e.g. a clearer description of the cost function in section 2. However, equation 4 and algorithm 1 were very helpful in clarifying the cost function. The manuscript also does a good job giving pointers to related prior work. The problem of interest is timely and important, and the provided solution seems reasonable and is well evaluated.

Looking at the cost function and the intuition, the difference in figure 1 seems to be primarily due to the relative number of samples used during optimization -- and not to anything inherent about the distribution as is claimed. In particular, if a proportional number of samples is generated for the 50x50 case, I would expect the plots to be similar. I suggest the authors modify the claim of figure 1 accordingly.

Along those lines, it would be interesting if instead of the uniform distribution, a model that explicitly models within vs. out of sample might perform better? Though this is partially canceled out by the other terms in the optimization.

Finally, the authors claim that the PT is approximately equal to entropy. The cited reference (Zhao et. al. 2017) does not justify the claim. I suggest the authors remove this claim or correctly justify it.

Questions:
 - Could the authors comment on cases where such a strong within-sample assumption may adversely affect performance?
 - Could the authors comment on how the modifications affect prediction score calibration?
 - Could the authors comment on whether they think the proposed approach may be more resilient to adversarial attacks?

Minor issues:
 - Figure 1 is unclear using dots. Perhaps the authors can try plotting a smoothed decision boundary to clarify the idea?
review_title: interesting idea for robust classification

final_score: rating: 6: Marginally above acceptance threshold
review_content: I have read authors' reply.  In response to authors' comprehensive reply and feedback. I upgrade my score to 6.

-----------------------------

This paper presents a novel approach to calibrate classifiers for out of distribution samples. In additional to the original cross entropy loss, the “confidence loss”  was proposed to guarantee the out of distribution points have low confidence in the classifier. As out of distribution samples are hard to obtain, authors also propose to use GAN generating “boundary” samples as out of distribution samples. 

The problem setting is new and objective (1) is interesting and reasonable. However, I am not very convinced that objective (3) will generate boundary samples. Suppose that theta is set appropriately so that p_theta (y|x) gives a uniform distribution over labels for out of distribution samples. Because of the construction of U(y), which uniformly assign labels to generated out of distribution samples, the conditional probability p_g (y|x) should always be uniform so p_g (y|x) divided by p_theta (y|x) is almost always 1. The KL divergence in (a) of (3) should always be approximately 0 no matter what samples are generated. 

I also have a few other concerns: 
1. There seems to be a related work: 
[1] Perello-Nieto et al., Background Check: A general technique to build more reliable and versatile classifiers, ICDM 2016, 
Where authors constructed a classifier, which output K+1 labels and the K+1-th label is the “background noise” label for this classification problem. Is the method in [1] applicable to this paper’s setting?  Moreover, [1] did not seem to generate any out of distribution samples. 

2. I am not so sure that how the actual out of distribution detection was done (did I miss something here?). Authors repeatedly mentioned “maximum prediction values”, but it was not defined throughout the paper. 
Algorithm 1. is called “minimization for detection and generating out of distribution (samples)”, but this is only gradient descent, right? I do not see a detection procedure. Given the title also contains “detecting”, I feel authors should write explicitly how the detection is done in the main body. 

review_title: Interesting idea, but not yet convinced

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new method of detecting in vs. out of distribution samples. Most existing approaches for this deal with detecting out of distributions at *test time* by augmenting input data and or temperature scaling the softmax and applying a simple classification rule based on the output. This paper proposes a different approach (with could be combined with these methods) based on a new training procedure. 

The authors propose to train a generator network in combination with the classifier and an adversarial discriminator. The generator is trained to produce images that (1) fools a standard GAN discriminator and (2) has high entropy (as enforced with the pull-away term from the EBGAN). Classifier is trained to not only maximize classification accuracy on the real training data but also to output a uniform distribution for the generated samples. 

The model is evaluated on CIFAR-10 and SVNH, where several out of distribution datasets are used in each case. Performance gains are clear with respect to the baseline methods.

This paper is clearly written, proposes a simple model and seems to outperform current methods. One thing missing is a discussion of how this approach is related to semi-supervised learning approaches using GANS where a generative model produces extra data points for the classifier/discriminator. 

 I have some clarifying questions below:
- Figure 4 is unclear: does "Confidence loss with original GAN" refer to the method where the classifier is pretrained and then "Joint confidence loss" is with joint training? What does "Confidence loss (KL on SVHN/CIFAR-10)" refer to?

- Why does the join training improve the ability of the model to generalize to out-of-distribution datasets not seen during training?

- Why is the pull away term necessary and how does the model perform without it? Most GAN models are able to stably train without such explicit terms such as the pull away or batch discrimination. Is the proposed model unstable without the pull-away term? 

- How does this compare with a method whereby instead of pushing the fake sample's softmax distribution to be uniform, the model is simply a trained to classify them as an additional "out of distribution" class? This exact approach has been used to do semi supervised learning with GANS [1][2]. More generally, could the authors comment on how this approach is related to these semi-supervised approaches? 

- Did you try combining the classifier and discriminator into one model as in [1][2]?

[1] Semi-Supervised Learning with Generative Adversarial Networks (https://arxiv.org/abs/1606.01583)
[2] Good Semi-supervised Learning that Requires a Bad GAN (https://arxiv.org/abs/1705.09783)
review_title: simple, effective method, some discussion/understanding missing

Meta-review: Meta score: 6

The paper approaches the problem of identifying out-of-distribution data by modifying the objective function to include a generative term.  Experiments on a number of image datasets.

Pros:
 - clearly expressed idea, well-supported by experimentation
 - good experimental results
 - well-written

Cons:
 - slightly limited novelty
 - could be improved by linking to work on semi-supervised learning approaches using GANs

The authors note that ICLR submission 267 (https://openreview.net/forum?id=H1VGkIxRZ) covers similar ground to theirs.
Decision: Accept (Poster)