Softmax Gradient Tampering: Decoupling the Backward Pass for Improved Fitting

Abstract: We introduce Softmax Gradient Tampering, a technique for modifying the gradients in the backward pass of neural networks in order to enhance their accuracy. Our approach transforms the predicted probability values using a power-based probability transformation and then recomputes the gradients in the backward pass. This modification results in a smoother gradient profile, which we demonstrate empirically and theoretically. We do a grid search for the transform parameters on residual networks. We demonstrate that modifying the softmax gradients in ConvNets may result in increased training accuracy, thus increasing the fit across the training data and maximally utilizing the learning capacity of neural networks. We get better test metrics and lower generalization gaps when combined with regularization techniques such as label smoothing. Softmax gradient tampering improves ResNet-50's test accuracy by $0.52\%$ over the baseline on the ImageNet dataset. Our approach is very generic and may be used across a wide range of different network architectures and datasets.

Reviews and author responses:

 Given that there's no rebuttal from the authors, and my fellow reviewers are also not very positive, I am keeping my score and voting for a reject for this conference. 

For the authors I found this paper: https://arxiv.org/abs/2010.07344
Which describes things similar to this paper, but in a very comprehensive way with a more in-depth look into the practice of changing the temperature of the softmax.

In this paper, the authors propose a technique called Softmax Gradient Tampering, which transforms the predicted output class probabilities to improve training performance of neural networks. The authors show that the proposed technique results in a smoother output probability distribution for lower values of the hyperparameter $\alpha$. On standard benchmarks on the ImageNet and CIFAR-10 datasets, the authors show that the proposed method results in better training accuracy performance as well as improved generalization performance. Strengths
-

- Overall I think this is a fairly well-written and easy-to-follow paper, although there are a couple of typos/grammatical errors which need to be fixed.
- The proposed method does introduce a hyperparameter $\alpha$, but tuning this hyperparameter seems to result in reliable, albeit small, performance improvements on the training set across the 3 benchmark tasks considered.

Weaknesses
-

- One of the claims in the paper is that the proposed softmax gradient tampering method smoothens the gradients. This indicates that the authors are implying that the proposed method improves the Lipshitzness of the gradients. However, this claim is never made concrete, and it is not clear to me whether Theorem 1 implies this. It would be great if the authors can clarify this claim with a bit more detail, either through a theoretical result or through an experiment which shows smoothened gradients.
- The paper will be improved with a much more thorough experimental section. While the baselines in the benchmarks considered have been tuned well, it would have been good to also present results across multiple independent seeds. It would also have been good to check how well the $\alpha$ values of around 0.25 or 0.3 transfers to another task, particularly a transformer. 

 While the proposed method looks promising in terms of results, the paper would benefit a lot from a more thorough experimental section, and from clarifying the claim of how softmax gradient tampering smoothens the gradients.

The paper introduces gradient tampering, a method whereby the gradient of of the cross-entropy loss function at the end is overwritten by using a smoothed p' output instead of the original output.  The reasoning for why this method would or should work is unclear to me. There is a lot of grandstanding in the paper like "the networks are just training better and arriving to better optimas at convergence" and "that improves the learning capacity of neural networks" that just don't follow directly from any logical argument in the paper.

Also the paper claims in several spots that "We theoretically analyze how softmax gradient tampering works from the perspective of
gradient smoothness." But section 5 only shows that (1) indeed smoothes out p, but that is already fairly obvious from the definition itself. It really isn't necessary to devote 1.5 pages of math to this, this could easily go in the appendix as it seems trivial. The question remains on why gradient tampering is a smart idea. I could not glean this from the paper. 

To me, it's also not clear why all gradients in the neural network would be 'smoothed' by having p redefined as p'. First of all, what does smoothing mean. I can see that a probability distribution is smoother as it tends more towards low entropy, but what does this exactly mean for the gradients themselves. This is too hand-wavy. Furthermore, having a flat probability distribution for the output, does not necessarily mean the gradients in the rest of the network show less peaks in their behavior. If anything, once my model is converged properly to p = q, then there is only a 0 gradient everywhere. That's as smooth as could be. 

In the end, the method is really doing something similar to adding a temperature parameter to the softmax function. This is not much of a reach in terms of the idea. See e.g. https://www.sciencedirect.com/science/article/abs/pii/S1568494618302758?via%3Dihub , https://arxiv.org/pdf/1612.02695.pdf , or in deep metric learning:  https://arxiv.org/abs/1808.04699 , https://arxiv.org/pdf/1811.12649.pdf,  or for other methods that change the outputs that are e.g. too confident: https://arxiv.org/pdf/1701.06548.pdf . It would be good if the authors did a better search on similar methods and explorations of the temperature parameter. 

The experimental results in this paper are far from sufficient. There is no comparison to other types of methods that are cited, such as label smoothing and other things you could do to your network for improvement. I understand your method does something slightly different, but it's similar enough to warrant a comparison. The alpha values are picked on the test set, which is not good practice. Results on the ResNets show only marginal improvements, and would benefit significantly from confidence bounds and reran experiments. From Fig. 2 it also looks like the network exhibits quite some noisy behavior when changing this alpha parameter. I'm not convinced the results show that training with a smaller alpha works better. 

Some smaller comments:

Lemma 1 - It would be better to define the threshold before or inside of the lemma for clarity. 

I don't think it's OK to give a non-anonimous link to your code. This goes against the anonimization idea of double blind reviewing. 

Editorial Notes:

page 2 | I assume an exp is missing in your softmax definition?
page 5 | Why does Softmax Gradient Tampering Works -> Work
page 5 | why softmax gradient tampering work -> works
page 6 | accuriacies -> accuracies The paper seems to amount to changing the temperature parameter of the softmax function during training. this has been investigated and done before, although not a very common practice. The paper does not show sufficiently why this would help during training, and resorts to unproven comments about smoothing of gradients. The theoretical section is insufficient to show why the method works, and the experimental section is not elaborate enough to prove beyond a doubt that the method works well. The paper also misses references to other works that have tuned the temperature parameter of the neural network.
All-in-all, the quality of this work is not high enough to warrant an accept. 

This paper introduces a simple technique that manipulates the predicted values by the exponentiation at the backward step. The authors search best performing hyper-parameters to carry out the experiments to show the accuracy improvements over the baselines. Pros)
1. The idea looks simple and interesting.
2. The accuracy improvement by ResNet18 without BN is quite intriguing.

Cons)
1. The paper has a clear message, but the novelty is limited, and the any intuitions behind why the method works are not clearly presented.
2. Experiments are insufficient to support the claim; provided training setups are very limited to support whether this works really well on a broad scale; some improvements are negligible such as in the case of ResNet-18. 
3. The experimental results are not convincing; all the accuracy gains should be validated by the multiple experiments with error bars, thus the authors should repeat the experiments against the randomness.
3. Some figures such as Figure 1 and Figure 3b are not referred to in the manuscript.
4. The manuscript should be refined: many typos (IN contains 50 images per class for the validation and the test sets both); need to refine all the citations for clarity. 
5. The works related to involving gradient noise are not cited such as the papers below. The authors should compare with such methods in the manuscript. 
[1] Wu, Jingfeng, et al. "On the noisy gradient descent that generalizes as sgd", ICML 2020;
[2] Neelakantan, Arvind, et al. "Adding gradient noise improves learning for very deep networks." arXiv preprint arXiv:1511.06807 (2015).

Comments)
1. Searching for alpha depicted in FIgure 2 and Section 3.1 is not clearly stated. What is the rule of thumb for picking alphas in the searches? In Figure 2b, alpha=0.2 has better test accuracy compared with others but was not picked; due to the large train accuracy? Why did the authors experiment only with a single experimental setup to determine the parameter? Overall, It is hardly agreeable for the entire practice of searching the parameter, and because of this, the following results that could have come out better became worse.

2. In the training setup on the ImageNet training, why did the authors use the 10-epochs cooling phase at the end of the training? Does the high weight decay (5e-4, the widely-used standard for ResNets is 1e-4) affect the results?

3. Please clarify why the authors use the half or mixed-precision training in the paper?

4. The following "arxiv" works do not need to be cited, but I recommend the authors to do more studies with such softmax manipulation methods below comparing with the proposed method:
[1] Oland, Anders, et al. "Be careful what you backpropagate: A case for linear output activations & gradient boosting." arXiv preprint arXiv:1707.04199 (2017).
[2] Agarwala, Atish, et al. "Temperature check: theory and practice for training models with softmax-cross-entropy losses." arXiv preprint arXiv:2010.07344 (2020).

5. Can this method work well with stronger data augmentations such as dropout, stochastic depth, rand/autoaug, or mixup? Please see the main review.

This paper proposes the softmax gradient tampering to modify the gradients in the backward pass to enhance the accuracy. The predicted probability value is transformed using a power-based probability transformation, and the gradient profile is more smooth. The experimental results show the slight accuracy increase. Cons:
1. The writing needs to be improved a lot to enhance the readability. The motivation is very vague, what is the problem of standard BP? Meanwhile, the benefits of the presented methods are not well explained. Why the softmax gradient tampering can better approximate the true gradient?

2. The experiments are not convincing. For example, the hyperparamters $\alpha$ in Table 1 are very important to the final performance. However, only two results with $\alpha=1$ and $\alpha=0.25$ are reported, remaining a large blank to evaluate the effectiveness of softmax gradient tampering.

3. She authors argue the generalization gap between the training and test dataset can be controlled by softmax gradient tampering which influences the gradient accuracy. However, no clear conclusions are drawn from the empirical study to show their correlation or rules.

4. There are a lot of blanks in Page 6 and 7.

5. The discussion does not convey any useful information.

6. The connection between the proof in Page 6-7 and the motivation is weak.
  Please see the main review.

Meta-review: This paper introduces Softmax Gradient Tampering, a technique for modifying the gradient of the softmax loss to make the loss more smooth. On standard benchmarks the authors demonstrate improved training and test accuracy.

The reviewers are unanimous in their recommendation to not accept the paper. They identify the following problems:
* a lack of theoretical understanding and rigor, and a lack of support for the claims that are made
* insufficient experimental results to convince the reviewers of the merit of the proposed technique in the absence of theoretical understanding

The authors did not provide a rebuttal, and I see no special reasons to question the assessment made by the reviewers. I therefore recommend to not accept this paper.

Decision: Reject