Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: In this draft, the models are only evaluated (as is standard) by accuracy. But often model probabilities are used as input to decision theory.

It seems one consequence of the entropy regularization is that it could un-calibrate probabilities. 

Consider a problem with two classes. The first class has base rate 20% and the second has base rate 80%. 

In the extreme case, if you crank up the entropy regularization you might get a model which always gives both classes 50% probability. But this overestimates the frequency of the first label and underestimates the frequency of the second.

In short, this problem arises in part because this form of the confidence penalty pushes label distributions towards uniform. Another formulation might push label distributions towards the base rate distribution instead. Have you considered such an alternative formulation?

*UPDATE* Thanks for the answer. This makes sense.

Question 2:
It might seem that achieving arbitrarily high entropy while preserving prediction accuracy should be trivial to accomplish. Any softmax softening maintains the order of the predictions exactly while increasing the entropy.

So while the experimental results appear strong and are clearly the interesting component of this paper, I suggest there's a big question mark over why precisely this method works. This regularizer doesn't limit the model complexity.

One explanation given in the paper suggests that the confident predictions are a symptom of overfitting, therefore by addressing overconfident predictions we might mitigate overfitting.

This argument appears doubly dubious.

First, there's a logical error: curing a symptom doesn't cure a disease (although it might be a valid source of hypothesis generation).
Second It's not clear that having more confident predictions is necessarily bad, or precisely at what point it becomes bad.

An alternative hypothesis is that you might get sharp errors (from cross entropy) when probabilities are very confident. Entropy regularization might effectively be providing some gradient clipping. 

This could lead to faster convergence and a growing body of work (see Moritz Hardt's "Train faster, generalize better" suggests that faster convergence leads to better generalization).

I'm interested to hear your thoughts and a deeper explanation / discussion of what precisely is working - especially in light of the triviality of satisfying the entropy maximization goal while generating predictions with identical accuracy. 





review_title: The effect of entropy regularization on calibration

final_score: rating: null
review_content: Have the authors compared their results with other smoothing techniques? E.g.:
  - the virtual adversarial training
  - Other smoothing technique is to force the network to estimate uniform output, which is then interpolated with the true CE objective:
    L = -log(p_c) - lambda * Sum_i(t_i * log(p_i))  where t_i = 1/N, N: num of classes, p_c is the estimated posterior of the correct output.

MNIST result: Comparison with similar model size or compared to label smoothing: is the 0.08% or 0.06% improvement really significant?
  confidence penalty: 1.0 is maybe not optimal. Have the authors tried larger value?

CIFAR: confidence penalty, 0.1 might be not optimal, have you tried \beta<0.1?  

review_title: questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper experimentally investigates a slightly modified version of label smoothing technique for neural network training, and reports results on various tasks. Such smoothing idea is not new, but was not investigated previously in wide range of machine learning tasks.

Comments:
The paper should report the state-of-the-art results for speech recognition tasks (TIMIT, WSJ), even if models are not directly comparable.
The error back-propagation of label smoothing through softmax is straightforward and efficient. Is there an efficient solution for BP of the entropy smoothing through softmax?
Although the classification accuracy could remain the same, the model will not estimate the true posterior distribution with this kind of smoothing.
This might be an issue in complex machine learning problems where the decision is made on higher level and based on the posterior estimations, e.g. language models in speech recognition.
More motivation is necessary for the proposed smoothing.

review_title: Experimental investigations on a slightly modified version of label smoothing.

final_score: rating: null
review_content: Your current related work section claims: "Penalizing the [low] entropy of a network's output distribution has not been explored in supervised learning."  Could you please clarify the sense in which this is true?

Surely you've heard of the maximum entropy principle (Jaynes 1957), which breaks ties among distributions that satisfy the data constraints by choosing the one with maximum entropy?  Or the whole line of work in the 1990's on deterministic annealing and information bottleneck methods?  These explicitly add an entropy regularizer similar to yours.  And it's not as if these papers were limited to unsupervised methods (Rao et al. 1990; Tishby et al., 1999) or to single-layer probabilistic classifiers (Berger et al. 1996).  For example, consider Miller et al. (1995) https://scl.ece.ucsb.edu/sites/scl.ece.ucsb.edu/files/publications/b95_4_0.pdf : "... probability distributions chosen to maximize entropy subject to a constraint on the expected classification error.  ... Our method is applicable to a variety of classifier structures, including ... multilayer perceptron-based classifiers."

Conversely, in some settings it makes sense to take beta < 0 and penalize HIGH entropy, on the grounds that the true classification is unambiguous and thus the true classifier will be confident.  This yields the entropy regularization technique of Grandvalet and Bengio (2005), which, in their semi-supervised setting, can be seen as a probabilistic analogue of transductive SVMs. 

These ideas are well known and have been applied in many settings and with many variants.  Thus, I hope you can spend a day identifying a wider swath of relevant previous work (including work before 2014, and any work that may have conducted experiments similar to yours) and situating your own contribution more precisely with respect to it.  

---------------------

I'm not sure I understand your result tables.  Usually they have two sections.  Is the idea that the numbers above the line are copied from the literature, while the numbers below the line are from your own implementations?  You seem to boldface the lowest test number in each section, except in Table 4 - why is Table 4 different?  And are the differences among systems statistically significant?  

In some sections you compare label smoothing or confidence penalty to dropout, while in other sections you combine them with dropout and compare to dropout alone.  Why the change?

How sensitive were your results to the choice of hyperparameter?  (You say something about this in section 4.4.2, although you don't seem to report your actual hyperparameters in section 4.4.2.)  Also, are you sure you did enough of a grid search in sections 4.1.1 and 4.1.2, given that you often picked one of the extreme values?

The results are missing in the last line of Table 6.  Do you have these results now?

---------------------

I'm interested in AnonReviewer5's question 2, but AnonReviewer5 seems to be assuming that the main objective only cares about the ranking of the classes y, e.g., that it scores argmax_y p(y | x).  Is your answer that it's ok because your main objective term is log p(y|x)?  (\sum_y p(y|x) loss(y,y*) would also work, as in the many papers on deterministic annealing in supervised settings.)

---------------------

NOTE: I may edit this post to add additional questions here.  Thought I'd be able to post them separately, but apparently not.
review_title: isn't this a well-known idea with a long history?

final_score: rating: 5: Marginally below acceptance threshold
review_content: Specifically, this paper suggests regularizing the estimator of a probability distribution to prefer high-entropy distributions.  This avoids overfitting.

I generally like this idea.  Regularizing the behavior of the model often makes more sense than regularizing its parameters.  After all, the behavior is interpretable, whereas the parameters are uninterpretable and work together in mysterious ways to produce the behavior.  So one might be able to choose a more sensible prior over the behavior.  In other words, prefer parameters not because they are individually close to 0 but because they jointly lead to a distribution that is plausible or low-risk a priori.

Pro: I believe that the idea is natural and sound (that is, I do not share the doubts of AnonReviewer5).

Pro: It's possible that this hasn't been well-explored yet in neural networks (not sure).

Pro: The experimental results look good.  So maybe everyone should use this kind of regularizer. 

Con: It is a kind of pollution of the scientific literature to introduce this idea to the community as if it were unconnected to (almost) anything else in machine learning.  There are many, many papers that include a scaled entropy term in the optimization objective!  It's not just for reinforcement learning.  Please see the long list of connections in my pre-review questions / comments.  

Con: Experimental results should always be accompanied by significance tests and error analysis.  Is your trained model actually doing better on the distribution of test data, or was your test set too small to tell?  Are the improvements robust across many different training sets?  What errors does your model fix, and what errors does it introduce?  

Summary recommendation: Revise and resubmit.  ICLR has lots of submissions.  I would prefer to reward authors who not only tried something, but who properly contextualized it and carefully evaluated it.  Otherwise, there's a race to the bottom where everyone wants to be the first to try something, so that readers are confronted with a confusing sea of slapdash papers with unclear relationships.
review_title: perfectly sensible idea, with apparently good results, but lacks scholarship 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a simple idea. They penalize confident predictions by using the entropy of the predictive distribution as a regularizer. The authors consider two variations on this idea. In one, they penalize the divergence from the uniform distribution. In the other variation, they penalize distance from the base rates. They term this variation "unigram" but I find the name odd as I've never seen multi-class labels described as unigrams before. What would a bigram be? 

The idea is simple,  and while it's been used in the context of reinforcement learning, it hasn't been popularized as a regularizer for improving generalization in supervised learning. 

The justifications for the idea still lacks analysis. And the author responses comparing it to L2 regularization have some holes. A simple number line example with polynomial regression makes clear how L2 regularization could prevent a model from badly overfitting to accommodate every data point. In contrast, it seems trivial to fit every data point and satisfy arbitrarily high entropy. Of course, the un-regularized optimization is to maximize log likelihood, not simply to maximize accuracy.  And perhaps something interesting may be happening at the interplay between the log likelihood objective and the regularization objective. But the paper doesn't indicate precisely what.

I could imagine the following scenario: when the network outputs probabilities near 0, it can get high loss (if the label is 1). The entropy regularization could be stabilizing the gradient, preventing sharp loss on outlier examples. The regularization then might owe mainly to faster convergence. Could the authors analyze the effect empirically, on the distribution of the gradient norms? 

The strength of this paper is its empirical rigor. The authors take their idea and put it through its paces on a host of popular and classic benchmarks spanning CNNs and RNNs. It appears that on some datasets, especially language modeling, the confidence penalty outperforms label smoothing. 

At present, I rate this paper as a borderline contribution but I'm open to revising my review pending further modifications. 

Typo:
In related work: "Penalizing entropy" - you mean penalizing low entropy

review_title: Simple idea, analysis lacking, comprehensive experiments

Meta-review: The reviewers agreed that the idea proposed in this paper is sensible and possibly very useful, and that the experiments are thorough with good results. However, they share strong doubts regarding the novelty of the proposed approach. Hopefully the discussion will help the authors refine this work. With a more thorough discussion of related ideas, such as entropy maximization, within the machine learning literature and a careful placement of this idea within that context, this could be a strong submission to a future conference.
Decision: Reject