Title: SoftTarget Regularization: An Effective Technique to Reduce Over-Fitting in Neural Networks
Authors: Armen Aghajanyan
Venue: ICLR 2017
Keywords: Deep learning
Optimization
Computer vision

Abstract: Deep neural networks are learning models with a very high capacity and therefore prone to over-fitting. Many regularization techniques such as Dropout, DropConnect, and weight decay all attempt to solve the problem of over-fitting by reducing the capacity of their respective models (Srivastava et al., 2014), (Wan et al., 2013), (Krogh & Hertz, 1992). In this paper we introduce a new form of regularization that guides the learning problem in a way that reduces over-fitting without sacrificing the capacity of the model. The mistakes that models make in early stages of training carry information about the learning problem. By adjusting the labels of the current epoch of training through a weighted average of the real labels, and an exponential average of the past soft-targets we achieved a regularization scheme as powerful as Dropout without necessarily reducing the capacity of the model, and simplified the complexity of the learning problem. SoftTarget regularization proved to be an effective tool in various neural network architectures.


Review discussion:
- The method as proposed requires memory that grows linearly both with the number of training samples and the output dimensionality (i.e. number of target classes). How would you propose to scale this to large datasets?
- A commonly used regularization scheme is early stopping, whereby you monitor a validation objective of interest, persist the parameters (e.g. to disk) whenever a sufficiently good new configuration is found, and terminate training when the validation objective either stagnates or gets worse. What advantage does this method offer over the conceptually simpler (and computationally less demanding) early stopping formulation?
- Your benchmarks involve exclusively classification. Does this method meaningfully impact test set *accuracy*? The value of the loss function on test data is less informative close to a minimum of the empirical loss as the model doubling down on its confidence in a few wrong answers will cause its test loss to rise but not its test misclassification error. In general, a lack of test error reporting makes this work incomparable to the literature.
Thank you for reading the paper.
1. The memory growth is rather marginal as the size we have to store in memory is equal to (# of samples kept in memory, # of classes).
When training large networks on a large amount of data (such as ImageNet) data is usually fed in, in the form of a generator, and therefore is soft on memory.
With respect to the amount memory images takes (# of samples kept in memory, # of channels, row size,column size), this amount of memory is marginal. Optimization routines that store past gradients take up much more memory than our approach.
2. This is a very good question. Early stopping is an effective technique to stop overfitting but tends to produce suboptimal results. This is why methods such as dropout/drop connect have been really popular. Our experiments also show that the results obtained with soft-target regularization have been better than standard vanilla networks (check out the minimum loss on all the tasks).
3. The reason the paper only shows loss results and not accuracy is because the loss is what we are directly optimizing over, even though it acts as a surrogate for accuracy. This was a conscious decision not to include accuracy in the paper.
Would you suggest I include the accuracy graphs in the paper?
Thank you again for reading my paper.
Armen.
1. As I understand it you accumulate all synthetic targets for the entire training set and then engage in an epoch of training. Is this not the case? (If it isn't the case and you are generating de novo targets on-the-fly as you process each minibatch, then are you backpropagating through the targets in the weighting term?).
In the ImageNet case, this amounts to about 5 GB of storage in float32 that needs to be frequently updated (and therefore is most practically stored in memory), whereas training data is usually streamed read-only from disk in such a setting.
2. I agree that early stopping may not be sufficient on its own (for this choice of architecture and hyperparameters) but early stopping is usually combined with other regularization methods where necessary.
3. Average test loss can get bigger without a corresponding increase in misclassification rate if the model simply becomes more confident in a few of its already incorrect predictions.
Thus I'd consider it important to compare test error, for practical purposes.
Relatedly, the paper would be stronger if efficacy were demonstrated on architectures that aren't arbitrarily introduced in this paper but have been drawn from the literature and were chosen by previous authors for their favorable performance in the control setting. With that baseline established, a gain in accuracy with the same architecture (or in test loss without a noticeable decrease in accuracy), or a gain in these metrics with an architecture that is somehow preferable (less computation, less parameters, etc.) would solidify the empirical claim.
1. In this case, you would only keep in memory the predicted classes the subset of images from ImageNet that are contained in memory at that time.
2. I agree with you on this point. SoftTarget with stopping would be more optimal as seen from the minimal loss.
3. I see what you mean. I will update the paper to include accuracy rates as well. The majority of architectures picked were from standard papers (such as the ResNet), others were standard convolution -> pooling networks that are usually applied as baseline architectures. I will do some edits and will update the paper.
Thank you once again for the comments.
Hello,
I have reported accuracy terms for all my experiments, as well as a few comments on choice of hyper-parameters.
Overall SoftTarget regularized networks outperformed other networks in terms of accuracy as well.
Thank you.
Armen.
- You definitely need to report misclassification error results on test data for obvious reasons related to losses and final test misclassification error. Currently comparisons are not conclusive.
-  Can you explain better the reason for using the particular updates in (3) and (4) better? Why don't you do for example totally corrective update, e.g. take convex combination of all \cal{F}'s (or some portion) up to current iteration in (3)?
Therefore \beta and \gamma should be tuned reasonably well to see whether (3) and (4) is really helping or not and the range for cross validation should be reported.
- The reason to set n_t n_b is not satisfactory.  It is crucial to cross-validate such parameters. Isn't  n_t = {1,2} unreasonably small number that can cause unstable results? why all n_b and n_t are equal?Are there results on other n_b and n_t's that were tried?
- It is stated that colabel similarities disappear when network starts to overfit.
However distillation ( Hinton et.al. ,2015 ) captures colabel similarities after training a model and using distillation. This method seems an iterative extension of distillation without using a bigger teacher model. Does proposed method gives better results then a two step version of distillation ?
- How do you tune \lambda for weight decay?
- From paper: "We considered a frozen set of hyper-parameters for the SoftTarget regularization to show that SoftTarget regularization can still work without a having to conduct a large grid search". This argument is not valid in ML, maybe if you did a reasonable search, you would get worse results (since you should not look test error until you finish the cross-validation).   Why a common hyper parameter tuning procedure is not used e.g. random search (Bergstra and Bengio, JMLR 2012) or Bayesian optimization (Snoek et al ,NIPS 2012) ?  Setting the hyper parameters to some numbers without searching a range or set can dramatically ruin fair comparison.
Thank you for comments.
I am editing the paper right now. I will include hyper-parameter optimization for soft-target, and also explanations for the choice of parameters.
To refer to the distillation paper, distillation was described as a way to transfer knowledge from large models to smaller models utilizing soft-targets. Soft-Target regularization can be thought of doing weighted distillation where the donor model is the state of the previous training, and the weighting target are the hard-targets.
Thank you for reading.
Hello,
I wanted to comment on the new revisions made to the paper. I ran a small brute force grid search on SoftTarget hyperparameters as well. I also described the use of small numbers in n_t and n_b. I also added a mention in the "similarities to other methods" of the interpretation of SoftTarget as a stateful implementation of distillation.
I also wanted to quickly comment on your question on tuning the lambda parameter. The lambda parameter can be seen as the general weighting for any reguralizer.
Therefore I did not include a mention on an algorithm on how to tune it since it depends on problem set up. Lambda was found through a small grid-search.
Thank you for your comments.
It seems the only baseline that author has compared against is dropout. Weight decay can be very effective regularizer for the negative log likelihood of a model. Author should at least do some comparison with the weight decay.
Also, ultimately we care about the classification accuracy of a model. A terrible negative log likelihood on the test set does not always translate to overfitting in terms of the classification performance.
Thank you for reading.
I'll run weight-decay experiments as soon as I can.
The next edits for this paper will include hyper-parameter optimization and accuracy reports.
Hello,
I have added accuracy reports in the paper. SoftTarget models performs better than other models in terms of accuracy as well. The paper also compares Batch Normalization as a regularization scheme.
Thank you.
Armen A.
This manuscript tries to tackle neural network regularization by blending the target distribution with predictions of the model itself. In this sense it is similar in spirit to scheduled sampling (Bengio et al) and SEARN (Daume et al) DAgger (Ross et al) which consider a "roll-in" mixture of the target and model distributions during training. It was clarified in the pre-review questions that these targets are generated on-line rather than from a lagged distribution, which I think makes the algorithm pseudocode somewhat misleading if I understand it correctly.
This is an incremental improvement on the idea of label softening/smoothing that has recently been revived, and so the novelty is not that high. The author points out that co-label similarity is better preserved by this method but it doesn't follow that this is causal re: regularization; a natural baseline would be a fixed, soft label distribution, as well as one where the softening/temperature of the label distribution is gradually reduced (as one would expect for this method to do as the model gets closer and closer to reproducing the target distribution).
It's an interesting and somewhat appealing idea but the case is not clearly made that this is all that useful. The dropout baselines for MNIST seem quite far from results already in the literature (Srivastava et al 2014 achieves 1.06% with a 3x1024 MLP with dropout and a simple max norm constraint; the dropout baselines here fail to break 1.3% which is rather high by contemporary standards on the permutation-invariant task), and results for CIFAR10 are quite far from the current state of the art, making it difficult to judge the contribution in light of other innovations. The largest benchmark considered is SVHN where the reported accuracies are quite bad indeed; SOTA for single net performance has been less than half the reported error rates for 3-4 years now. It's unclear what conclusions can be drawn about how this would help (or even hurt) in a better-tuned setting.
I have remaining reservations about data hygiene, namely reporting minimum test loss/maximum test accuracy rather than an unbiased method for model selection (minimum validation set error, for example). Relatedly, the regularization potential of early stopping on a validation set is not considered. See, e.g. the protocol in Goodfellow et al (2013).
Inspired by the analysis on the effect of the co-label similarity (Hinton et al., 2015), this paper proposes a soft-target regularization that iteratively trains the network using weighted average of the exponential moving average of past labels and hard labels as target argument of loss. They claim that this prevents the disappearing of co-label similarity after early training and  yields a competitive regularization to dropout without sacrificing network capacity.
In order to make a fair comparison to dropout,  the dropout should be tuned carefully. Showing that it performs better than dropout regularization for some particular values of dropout (Table 2) does not demonstrate a convincing advantage. It is possible that dropout performs better after a reasonable tuning with cross-validation.
The baseline architectures used in the experiments do not belong the recent state of art methods thus yielding significantly lower accuracy. It seems also that experiment setup does not involve any data augmentation, the results can also change with augmentation. It is not clear why number of epochs are set to a small number like 100 without putting some convergence tests..
Therefore the significance of the method is not convincingly demonstrated in empirical study.
Co-label similarities could be calculated using softmax results at final layer rather than using predicted labels.  The advantage over dropout is not clear in Figure 4, the dropout is set to 0.2 without any cross-validation.
Regularizing by enforcing the training steps to keep co-label similarities is interesting idea but not very novel and the results are not significant.
Pros :
- provides an investigation of regularization on co-label similarity during training
Cons:
-The empirical results do not support the intuitive claims regarding proposed procedure
Iterative version can be unstable in practice
Thank you for your comments.
We tested dropout extensively in multiple experiments, by electing dropout values ranging from 0.2-0.8.
"Regularizing by enforcing the training steps to keep co-label similarities is interesting idea but not very novel": I don't believe I have seen any paper that approach regularization from this perspective, papers that build up to our approach are mentioned and talked about extensively. Could you please reference a paper that does this?
I would kindly ask you to reconsider. SoftTarget outperformed any other regularization in virtually all of our experiments.
Showing results from 0.2 to 0.8 with increments of 0.2 is not satisfactory as changes smaller than 0.2 can effect results when the parameters are tuned with random sampling or spearmint.
Regularization effect of softlabels is discussed in Section 3 and Section 6 in Hinton et.al. (2015). Using this for the network itself is an incremantal approach.
My decision did not change after rebuttal since the author could not provide convincing arguments/improvements to concerns above.
The paper introduced a regularization scheme through soft-target that are produced by mixing between the true hard label and the current model prediction. Very similar method was proposed in Section 6 from (Hinton et al. 2016, Distilling the Knowledge in a Neural Network).
Pros:
+ Comprehensive analysis on the co-label similarity.
Cons:
- Weak baselines. I am not sure the authors have found the best hyper-parameters in their experiments. I just trained a 5 layer fully connected MNIST model with 512 hidden units without any regularizer and achieved 0.986 acc. using Adam and He initialization, where the paper reported 0.981 for such architecture.
- The authors failed to bring the novel idea. It is very similar to (Hinton et al. 2016). This is probably not enough for ICLR.
Thank you for your comment.
Hyper-parameter optimization was done with uniform initialization on all of the various architectures tested. The purpose of all the tests was to show relative comparisons between varying architectures and the fact that SoftTarget regularization performed better in virtually all the cases builds a strong case for the effectiveness of SoftTarget.
I believe the paper you are referring to is Hinton's distillation paper, which I reference and have a subsection dedicated too. Hinton et al, describe a method to distil knowledge from bigger networks to smaller networks but don't create a framework for regularizing a single network through soft-targets. This paper creates that framework.
I would ask kindly to reconsider.
Thank you.
A particular naive but seemly related baseline is to compare with a model regularized by a linear combination of hard labels and uniform soft targets( 1/#classes ) instead of the previous model predictions. The uniform soft target has the maximum co-label similarity. If this naive baseline also work well as an effective regularizer, the authors need to rethink about the proposed methods in this paper.
Alternatively, the authors can directly penalize the negative co-label similarity in the loss function.
Overall, the baseline is weak and there is very little insight into why the proposed method provide generalization benefits. I stay by my original review.

Meta-review:
role: PC
data: comment: The reviewers unanimously recommend rejection.

Decision: Reject