Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use temperature scaling during the training process, thus improving accuracy and reducing the need for learning rate schedules. In the theoretical part of the paper, the gradients have been calculated analytically, comparing the gradients with and without the proposed modifications. In the experimental part, the proposed methods have been applied on several tasks and neural architectures, showing that the methods result in improvements in classification accuracy.
strength_and_weaknesses: Strengths:

* The paper is well-written and clear.
* The idea of using calibration during training to improve performance can be considered novel and the methods are well justified and simple.
* The empirical advances are nicely complemented with a theoretical study and discussion about what might be the reasons for the achieved improvements.


Weaknesses:

* My main concern is that in CIFAR10 classification the authors have only considered an architecture that results in accuracies much below 80%, whereas modern architectures (plus data augmentation) go well beyond 90%. It would be important to experiment with at least one better architecture also. In CIFAR100, a better architecture has been used, but also for this dataset there is room for improvement so that the baseline method would be at least 70% accurate.

* In the introduction, the paper promises to measure ECE (expected calibration error) but never gets to doing it, not even in the appendix. On one hand, this promise could simply be dropped, but actually it would be an important thing to do, because calibration is an important part of the motivation and intuition behind the proposed method. For example, it would be good to have plots about how ECE (evaluated on test data) is evolving during training alongside with accuracy improvements.

* Figure 1 (left) starts from the cooling factor value of about $0.5$. In order to support the discussion better, lower values of the cooling factor could have also been considered in that plot.

* It would have been good to see more figures about gradient norm evolution in the appendix, perhaps even for all situations considered in Table 1. This would help to compare whether the evolution achieved with the proposed methods is different or similar to the evolution observed with different learning rate schedules.

Minor weaknesses:

* Some parentheses are missing in the proof of Lemma A.1., e.g. the first formula should be $x'_1=\beta\rho(\beta^{-1}(W'_1 x_0+b'_1))$.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and clear, with good quality and original ideas. The experiments seem reproducible.

summary_of_the_review: A strong paper with one key problem that the considered architectures are quite far from the state-of-the-art (CIFAR-10 accuracies below 80% as opposed to more than 90%). While in principle, it is good to do experiments with architectures requiring less resources, in this particular case it would be important to know whether the observed accuracy improvements are specific to the considered architectures or whether they also carry over to the state-of-the-art architectures. Also, more information could hav
e been provided in the appendix, in particular about gradient norm evolution as well as calibration evolution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed the temperature-based cooling algorithm to achieve better performance in optimization. The authors show better gradient scaling and reducing the need for scheduling. The starting point of the cooling algorithm is that miscalibration can worsen network learning. Specifically, the authors consider last-layer cooling, distributed cooling, etc. The gradient analysis is interesting, and experimental results provide the effects of the cooling algorithm compared to naïve approaches.   
strength_and_weaknesses: Pros:
This paper proposed an algorithm to address the optimization problem in deep networks. The algorithm is simple to be applied in practice, and the simulation results can provide the basic effect of the cooling algorithm.

Cons:
Calibration has its area. Therefore, the authors must show the precise relationship between calibration and cooling in optimization in detail. Also, this paper is more related to optimization topics. Experiments are too limited because only scheduling is examined. Maybe Batch-Normalization Dropout and other regularized methods should be compared. 


clarity,_quality,_novelty_and_reproducibility: Clarity: 
The cooling algorithm’s effect is clearly explained in an aspect of the gradient. In addition, I have a question that vector scaling can be helpful to address the heating in learning, and the precise relationship to calibration is vague to me. In conclusion, the authors stated that there are indications that well-calibrated networks are more stable in training and less reliant on the ‘right’ learning rate schedule. However, there is no clear evidence in the experiments that the cooling algorithm is more calibrated (although it can be expected, it should be clarified). If you can report this, I have a question, can the degree of improvement in calibration cause more stability in learning? 

Quality:
Well-written but has some missing in the formulation, such as $\beta^{-}, \beta^{i}$ in Definition 3.2. The formulation is not well-organized and to be understood easily. The motivation is simple in the optimization aspect. However, there are rooms to explore and research, such as vector scaling and intensive experiments.  

Novelty & Reproducibility:
The paper provides a somewhat novel algorithm, and it is expected that reproducibility can be possible.      

summary_of_the_review: I’ve not convinced of the relationship between calibration and learning stability in a transparent manner. The paper is 
well-written. However, it can have many issues to be solved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to incorporate temperature scaling, a popular post-training confidence calibration technique, into training to improve performance and training stability.  Temperature scaling in confidence calibration changes the confidence of a network but does not affect the ordering of predictions. The paper hypothesizes that temperature scaling can regulate gradients if it is used during training.  Specifically, three variants: last layer cooling, distributed cooling and periodic cooling are proposed. For example, last layer cooling uses a temperature parameter in the last linear layer which is calibrated on a held-out set after every training epoch. 
strength_and_weaknesses: Pros: 

1, the analysis of the effects of different cooling strategies on gradients is insightful. 

Cons:

**1, Lack of experimental support.** While the method is well developed and studied analytically, the experiments do not fully support the strong claims of the paper. Specifically, classification experiments are only conducted on CIFAR10 and CIFAR100	,¬ both which are considered small datasets by today’s standard. This is especially true when the main claim of the paper is improved performance and stability. Moreover, the accuracy reported for CIFAR10 using VGG seems too low compared to the performance using the same architecture by other open-sourced codes, e.g., [1]. 

**2, Contribution of distributed cooling and periodic cooling not strong.** A large component of the paper is on distributed cooling and periodic cooling. However, according to Table 1, neither of them is noticeably better than simple last layer cooling, if not worse. 

[1] https://github.com/chengyangfu/pytorch-vgg-cifar10

clarity,_quality,_novelty_and_reproducibility: The paper is relatively clear, and the method is novel. The reviewer has doubts on the reported results and no code is provided. 
summary_of_the_review: The paper aims to show an overlooked benefit of temperature scaling during training. While the analytical analysis is detailed, it lacks enough experiments to support the claims, making the results less convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to use temperature scaling during neural network training as a means to improve model performance and reducing the need for learning rate schedules.
The novelty of the methods was found to be moderate.
Related methods and different settings for neural network optimization are not sufficiently evaluated. The authors have added additional experiments on  settings for neural network optimization. Yet their additional experiments did not show consistent improvements over the state of the art. 
A key problem pointed out by a reviewer is that the considered architectures are quite far from the state-of-the-art on important benchmark data sets. While the authors provided additional experiments on these benchmarks, these still fell short of reaching state of the art. 
Minor issues with the mathematical notation where pointed out.
justification_for_why_not_higher_score: After the rebuttal period two out of three reviewers vote for rejection. The third reviewer who voted for a borderline accept also points out that the model  architectures are quite far from the state-of-the-art for important image benchmarks.
Decision: Reject