Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##  Summary of the paper
This paper proposed the new time scaling analysis for the cross-entropy loss with temperature in the deep neural networks. The authors introduced the effective time scale, and then provided the linear and non-linear time scale based on the output of logits with temperature and stepsizes by using the NTK theory.

## Strong and weak points of the paper
### Strong points
- Provided the novel time scaling analysis for the cross-entropy loss with many empirical validations.
### Weak points 
- The choice of optimal $\beta$ still remains unclear and left to the future work.  I think clarifying the tuning strategy of $\beta$ is the most important from practical view point.

## Rating
- Clarity: The authors should clarify which part corresponds to the previous work proposed or this work proposed
- Correctness: I did not check all the proof in detail.
- Novelty: Since I am unfamiliar with this field, I thought that introducing the linear and non-linear time scale in Sec 2.3 and 2.4 seems intereting tools in experimetns.

## Comments and Questions
- Q) In Sec 2.2 to 2.4, what types of models and datasets are used in the numerical experiments ? Although I might overlooked, it should be explained explicitly.

- Q) As far as I understood, from the ovservation of Sec 2.2 to 2.4, setting large $\beta$ makes the times scales $\tau_z$ and $\tau_{nl}$ smaller, which means that  the early linear learning timescale and nonlinear timescale smaller. So, setting large $\beta$ enhance escaping from the linear dynamics at the beggining, but the time period of the nonlinear dynamics become shorter, which results in the worse final performace. Is my understanding correct ?

- Q) What is the definition of optimal $\eta^*$ in Sec 3.1?

- Following is just a comment. In Sec 3.3, the authors found that using small $\beta\in [10^{-2},1)$ might improve the final performance in a variety of networks.  On the other hand, when we consider Bayesian deep learning, using cold posterior that uses the temperatrue below $1$ results in better final perfomance compared to using $\beta=1$ (For example see, Wenzel, Florian, et al. "How good is the bayes posterior in deep neural networks really?." arXiv preprint arXiv:2002.02405 (2020). ). So, I just thought that there might be some connection between this small temperature phonomena.

review_title: Introducing the novel time scaling analysis for the cross entropy loss with temperature

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies how temperature scaling affects training dynamics of neural networks (with softmax layer and cross-entropy loss). The theoretical analysis shows that neural networks trained with smaller inverse temperatures (beta) exit the linear regime faster, which implies better performance. Experiments on image classification and sentiment analysis confirm that tuning temperature improves neural network generalization, even for state-of-the-art models.

Strengths:
* The topic is novel and interesting.
* Empirical results confirm the importance of tuning temperature.

Weaknesses:
* Theoretical contribution seems unclear.
* Experiments are limited to two tasks.

I am leaning towards rejecting the paper. My biggest concern is about the theoretical contribution. The main conclusion of the theory seems to be that smaller beta is better, because it helps neural networks exit linear regime faster. But this doesn’t explain the experiment results: optimal beta varies a lot across models, and it is sometimes quite large (beta >= 1). The paper empirically shows that small beta causes more instability as an explanation, but there is no theoretical explanation. Therefore, it is unclear how to use the current theory.

Ideally, I hope the theory can be extended to explain why small beta causes instability (the conclusion section mentions this as future work), and/or how neural network architecture affects optimal beta, but these extensions do not seem obvious.

The experiments can also be expanded. Currently, there are only two tasks in the experiments. While the results are impressive, I would be more convinced if there are more tasks/models. For example, I wonder what the optimal beta is for state-of-the-art BERT-based models.  What about structured prediction tasks? Does the size of training set affects optimal beta?

On the positive side, I think the topic is novel and interesting, and the current empirical results are solid. If the theory can be extended to explain the tradeoff between small/large beta and the role of architecture, I would recommend this paper. Alternatively, the paper can also be improved by expanding experiments.

Other suggestion and question:
- Section 2 (theory) may be easier to read if there is a short summary of main claims/results.
- For IMDb experiment, is there a reason for choosing GRU instead of more recent BERT-based models?
- Some of the figure fonts are too small.

Feedbacks after author response:
I am maintaining my rating after reading the author response. It is a close decision. I like the topic, but I think the draft still has room for improvement to become a great paper. The updated draft is much clearer and answers some of my questions. Most importantly, the updated theory section explains why small beta can be bad: it slows training. While I appreciate the clarification, I think this argument still doesn't fully align with the experiment results. As the CIFAR-10 experiment points out, using small beta can still lead to good performance, so the main problem for small beta seems to be instability (rather than slow training), which the current theory couldn't explain. To improve the theory, I hope the paper can provide more insights into the instability caused by small beta. For example, I wonder if this is somehow connected to the slow training argument; perhaps the failed runs indeed suffer from slow training. 
review_title: Topic is interesting, but theoretical contribution is unclear

final_score: rating: 6: Marginally above acceptance threshold
review_content: The soft-max layer of the neural network typically does not scale the outputs of the incoming layers, i.e., the scale is set to one. This paper analyses the how the dynamics of the training is effected by a non-unit scale $\beta$. The mathematics involves elementary calculus, and it is clear and easy to follow. The analysis primarily concerns the initial/early phases of the training, or what the authors refer to as short times --- note to authors: I think there should be a better phrasing than using "short times" --- where $\tau$ is small. The analysis is divided into two cases: linear dynamics where the Hessian/second-order-term is negligible , and the non-linear dynamics where it is not. The experiments are illustrating but inconclusive.

Additional comments and suggestions:
1. Sections 2.2 to 2.3 can be restructured to more bring out that both the linear and non-linear dynamics concern the early phases of the training --- unless of course I misread these sections.
2. Sections 3.1 and 3.2 does not use batch -normalisation. Although some readers can reasonably guess the reason, it is worthwhile to explicitly say why this is the case.
3. I don't think section 3.3 contributes to the central message of the paper. Hence, I think it can be move to the appendix, and some from the appendix can be moved to the main paper.
4. It is not mentioned how the first plot of Figure 2 and the two plots in Figure 3 are obtained.

The paper deserves an *accept* because it is fundamentally correct and it is one of the "secrets-of-the-trade", and I am glad that it is written. It is *not a stronger accept* because I feel the "right units for comparison" (see section 4) could already be common knowledge among those who has looked hard at soft-max.




review_title: This paper analyses theoretically and empirically how the temperature/scaling of the soft-max output layer affects the dynamics of the training of the neural network.

final_score: rating: 3: Clear rejection
review_content: This paper investigates how the inverse temperature parameter $\beta$ in the softmax-cross-entropy loss impacts the learning and the generalization. In the theory part, this paper introduces the concepts of early learning timescale and nonlinear timescale, and shows how the learning dynamics depend on the parameter $\beta$. The empirical investigations are carried out with wide Resnets on CIFAR10, Resnet50 on ImageNet, and GRUs on IMDB. The results suggest that the optimal $\beta$ is architecture sensitive. 

The theory developed in this paper is rudimentary. I don't see how the theory in Section 2 is crucial for the understanding of the impact of $\beta$. The results basically say proper normalization on the timescale is needed for a fair comparison among different $\beta$. There are neither formal statements nor proper discussions on the significance of the results. 

From the experiment results, the performance is relatively insensitive to the parameter $\beta$ with batch normalization. I don't see a clear motivation for dropping batch normalization or tuning $\beta$ with batch normalization. It is apparent that tuning for the best $\beta$ is always no worse than setting $\beta=1$. But the impact of $\beta$ seems to be insignificant, and the so-called optimal $\beta$ varies wildly across different experiments. The performance with different $\beta$ is not presented in Sections 3.2.2 and 3.2.3. 

Finally, the manuscript is poorly written and needs to be largely reworked to be considered for publication. None of the key concepts identified in the paper are formally defined. For example, it is unclear what is the mathematical meaning of $\ll$ and $\sim$ in Section 2.3, and what is the precise meaning of the phrase "no longer be neglected" in Section 2.4. Abbreviations like NTK are not introduced. Curves in the plots like Figure 6(c) dashed orange are not labeled.
review_title: Review

Meta-review: Three reviewers are mildly positive, while one is negative. The substantive comments of the reviewers are consistent with each other; it is merely their evaluations that differ. 

One contribution of the paper is that it shows how using temperature tuning can yield similar accuracy to using batch normalization; this is useful because batch normalization is not always possible. The revised paper shows improvements, and we appreciate the engagement of the authors with the reviewer comments. However, there are remaining weaknesses such as a weak argument based on the empirical.results.

This paper can be improved based on the comments made by the reviewers. We encourage the authors to resubmit to a future venue.
Decision: Reject