How to Keep Cool While Training
Authors: Martin Trimmel
Mihai Zanfir
Richard Hartley
Cristian Sminchisescu
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural network
calibration
network calibration
cooling
temperature scaling
classification

Abstract: Modern classification neural networks are notoriously prone to being overly confident in their predictions. With multiple calibration methods having been proposed so far, there has been noteworthy progress in reducing this overconfidence. However, to the best of our knowledge, prior methods have exclusively focused on the factors that affect calibration, leaving open the reverse question of how (mis)calibration impacts network training. Aiming for a better understanding of this interplay, we propose a temperature-based Cooling method for calibrating classification neural networks during training. Cooling has a substantial effect on the gradients and reduces the need for a learning rate schedule. We investigate different variants of Cooling, with the simplest one, last layer Cooling, being also the best-performant one, improving network performance on a range of datasets, network architectures, and hyperparameter settings.

Under review as a conference paper at ICLR 2023
How TO KEEP COOL WHILE TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Modern neural networks used for classification are notoriously prone to overly confident predictions. With multiple calibration methods proposed so far, there has been noteworthy progress in addressing overconfidence issues. However, to the best of our knowledge, prior methods have exclusively focused on those factors that affect calibration, leaving open the question of how (mis)calibration circles back to negatively impact network training. Aiming to better understand such de- pendencies, we propose a temperature- based Cooling method to calibrate classifi- cation neural networks during training. Cooling results in better gradient scaling and reduces the need for learning rate schedule. We investigate different variants of Cooling, with the simplest, last layer Cooling, being also the best-performing one, improving network performance for a range of datasets, network architec- tures, and hyperparameter settings.
INTRODUCTION
Training neural networks can be challenging task, with optimal performance depending on the right setting of hyperparameters. For this reason, finding suitable network configuration can often take multiple costly training runs with varying parameters of the learning rate schedule, the optimizer and the batch size. Apart from standard learning rate schedules like piecewise constant schedules and exponential decay schedules, there has been activate research in developing better schedules: Among the most prominent of these are learning rate warmup (Goyal et al., 2017 He et al. 2016a)
and cosine decay (Loshchilov & Hutter 2017) schedules.
Complementary to these challenges, (Guo et al. 2017) found that modern convolutional classifica- tion networks are often poorly calibrated, leading to overly confident predictions. They investigated multiple methods to improve calibration, with a simple temperature scaling method performing best: the network's output logits are multiplied by a temperature parameter, optimised on a valida- tion dataset after training. Importantly, this leaves the maximal value and therefore the predicted class label unchanged since all the logits multiplied by the same temperature value.
Since then, multiple papers (Kull et al., 2019 Kumar et al. 2019 2018 Müller et al., 2019 Gupta al. 202I) proposed methods aiming even better calibrated networks. More recently, (De- sai & Durrett, 2020, Minderer et al., 202T) investigated the calibration of state-of-the-art non- convolutional Transformer networks (Vaswani et al., 2017 Dosovitskiy et al., 2021) and MLP- Mixers (Tolstikhin et al., 2021). They concluded that such architectures may have benefits, with further work needed to fully understand the factors contributing to calibration.
Despite initially leaving the accuracy unchanged, we have noticed that temperature scaling can have an intriguing effect as training continues: scaling the output logits results in a change in the cross- entropy loss, which in turn leads to scaled gradient updates and subsequently new parameter values. During training, this can lead to a significant increase in accuracy. To the best of our knowledge, temperature scaling has until now only been applied post hoc after completing network training. However, our investigation shows that networks become gradually overconfident during training (they overheat), which seems to have detrimental effect learning. This has motivated us to modify the original temperature scaling and propose Cooling method to calibrate neural networks during training.

---

Under review as a conference paper at ICLR 2023
Our Contributions
A Cooling method for calibrating classification neural networks during training. We pro- pose two basic variants called last layer Cooling and distributed Cooling, and one hybrid variant called periodically redistributed Cooling.
A mathematical analysis of the effect of Cooling on the network gradients, with a compar- ison of different Cooling variants.
An empirical investigation of the effects of Cooling on range of metrics, including net- work weights, gradients, output logits and the ECE (expected calibration error) calibration measure.
A broad set of experiments for different tasks (image classification and semantic segmen- tation). datasets and network architectures. We also include an extensive ablation study, in- volving different activation functions, optimizers, and hyperparameters such as the learning rate schedule, the Cooling factor and the use of weight decay and data augmentation. Our experiments indicate an interplay between the learning rate and calibration during training. Importantly, if well-calibrated, networks can train well without the use of learning rate schedule.
2 BACKGROUND AND NOTATION
Let fo : R R denote the function of a classification neural network with parameters 8, mapping a d-dimensional input (in our case an image) an s-dimensional logits vector z = fx().. During training, each input comes with a class-probabiligy or label vector y, denoting probabilities of x belonging each of classes. This usually (but not necessarily) one-hot vector corresponding to so-called ground-truth class label, i*
In its simplest variant, we suppose the network consists L affine (dense or convolutional) layers, each followed by non-linear activation function. For the 7t layer (1 S i S L) this gives an expression of the form x; = =(p,) = p(W,,,( + b,) with weight matrices W, bias vectors b, non -ineaiieesss p, pre-activation values P, and layer inputs and outputs x;-1 and output x,, respectively. (More generally, our method can be applied to any neural network, involving arbitrary functions and layers like e.g. attention, batch normalization and skip connections.) The output logits are then passed through the softmax function a which results in a vector ý = (z) of class probabilities. The classification network is trained to minimize the categorical cross-entropy loss function
L(z) H(y, =
Σ y log (1))
(2.1)
We say that network is well-calibrated if the output values ÿ can be interpreted as true probabilities. Intuitively, if network makes 100 predictions with 90% confidence, we would expect that 90% are correctly classified. (Guo al., (017)) observed that convolutional neural networks tend to display over-confidence in their results, in that Th- gives over-estimate of the probability that X; is the correct label. Thus the networks are badly calibrated, which we metaphorically express by saying that the networks become overheated.
(Guo et al.) 2017) found that simply multiplying the pre-softmax logits z; by a factor T does an excellent task of improving the network's 's calibration. Thus, the task of correcting the calibration of the network is to find a constant 7 to correct its output, so that it becomes
== oTz) = =T(
(2.2)
The optimal T is found by minimizing the log- ligllikel cost function on a small calibration set, held back from the training data. This operation is carried out when the network is fully trained. Usually, one finds that the optimum value is T < 1. This process is known as temperature scaling.
We refer Guo al. 2017) for detailed introduction network calibration.
2

---

Under review as a conference paper at ICLR 2023
3 METHOD: COOLING WHILE TRAINING
3.1 FUNDAMENTALS
When the network overheats, the predicted values Ø become too close to 0 or 1. This can cause problems with gradients becoming large. We hypothesise therefore that keeping the network the correct temperature during training can lead to improved convergence. Our proposed operation is to periodically correct the network, by multiplying the logits by the optimal temperature correcting constant. There are multiple ways to implement Cooling, the most basic being last layer Cooling:
Definition 3.1. A network performs last layer Cooling if before the softmax function there is a final scaling layer, multiplying the network output logits z by a constant scalar T >> This value not modified during the batch updates of the gradients, but is corrected using a held-out validation set at the end of the Cooling period.
Cooling factor. We investigate the effect of taking the optimal temperature parameter some power which we call the Cooling factor. Let us assume that < 1 (which is mostly the case). Then for H > multiplying by 7* results in smaller logits, a scenario which we call overcooling. Conversely, K 1 produces larger logits, resulting in an undercooling of the network. We note that as K + 0, we approach standard network training without Cooling. As we show in the experiments, using a suitable value for can have positive effects on the training stability and performance.
Cooling period. We call the periodic time interval after which perform temperature scaling the Cooling period. Typically, we let the Cooling period be equal to one epoch of training. We trained the VGG-style network described in $4.2 on CIFAR1O. 10, using Nvidia GeForee GTX Titan GPU and a 16-core Intel Xeon CPU E5- 2640 v3. The actual training takes approximately 56 seconds per epoch, whereas Cooling takes approximately 1.05 seconds. Hence, takes approximately 1.9% of the time of one training epoch. We experimented on CIFARIO with Cooling period of one batch, but found additional performance benefits. Thus, it seems that Cooling once per epoch is sufficient.
of Cooling
3.2 DISTRIBUTED COOLING
Instead of waiting to perform scaling after the final layer, it is possible to redistribute the temperature correction across the network, by scaling layers other than the last.
Suppose the optimal temperature is R When we redistribute the temperature across the network, we would like the temperature correction to gradually take effect. More precisely, we want to ensure that (1) each layer multiplies its input by an additional factor of B r1/L, so that the output of the th layer is now x = 3'x,. Moreover, (2) the inputs to the non-linearities have to be the same as before scaling because we would otherwise change the network output non-linear manner. Finally, (3) the output logits should be multiplied by 3 =T.
Definition 3.2. Let the notation be in Let 71/4, A network performs distributed Cooling if after each Cooling period,
1. the weight matrix W, is multiplied by 3, resulting in a new matrix W == .WW:
2. the bias vector b, is multiplied by 3 resulting in new vector by B'b
3. the activation p is changed p3. defined by (a)(2)) 3 p(8-1x).
Lemma 3.3. a network performs distributed Cooling, then, compared no Cooling,
1. the output of the tt layer is X =3'xx, for LisL-1
2. the input each of the non-linearities is left unchanged;
3. the output logits are scaled by a factor of T: z' = TZ.
Proof. We give the proof in in Appendix A
3

---

Under review as a conference paper at ICLR 2023
Algorithm 3.1 Periodically Redistributed Cooling (simplified)
Inputs:
Neural network fe with linear layers (W;, b;) Final Cooling layer 8,
Validation data set Aval Cooling factor K
Maximal temperature Tmax reset temperature Tre
Output: Calibrated Network for
After each Cooling period:
1: 7 + temperature minimising cross entropy on X
2: T + T
over- or undercooling
3: if Tmax then
4: excess f- T T/T
5: T Tre
6: kernel scale + excess
7: bias scale 1
8: for i= to L do
9: bias scale bias scale kernel scale
10:
W; < Wi/kernel scale
b, - b bias scale
12:
Update 87 using new 7
may be observed that the ReLU function, then Pgt = P. so the activation layer is not changed. (More generally, this holds for piecewise-lines linear ReLU variants such as CReLU, Leaky ReLU and PRELU.) Note also that 7 is usually less than 1, indicating overheating of the network, and hence multiplications by 3 and 3 result in decrease in the values of the parameters W. and b,
When performing last layer Cooling, we have observed that the network can correct for this by overheating even more. This means that the required temperature correction becomes smaller and smaller, towards zero. At the same time, this means that the parameters of the layers (for instance W and b in affine layers) become larger and larger, eventually causing numerical overflow. A possible solution to this problem to keep track of the overheating parameter т, and when it becomes too small, redistribute the temperature correction over all layers:
Definition 3.4. A network performs periodically redistributed Cooling if for some Tmax,
0.
it performs last layer Cooling as long as 7 < Tmax (i.e. the optimal temperature is less than specified maximal temperature);
it redistributes the excess temperature T/Tre across the layers in Definition 3.2 if
Tmax-
The values Tmax and Tre are manually specified. In our experiments, we used Tmax periodically redistributed Cooling, but other values can also be considered.
100 for
Summary. Algorithm 3.1 gives the pseudocode of our proposed Cooling method. The algorithm
shows periodically redistributed Cooling, since is the most general case. We can recover last layer Cooling and distributed Cooling as special cases with (Tmax, Tr)) = (oo, oo) and (Fmax; Tre) = (0, 1), respectively. The algorithm displayed above is simplified for the sake of clarity: it does not take heed of layers like attention and batch normalization and excludes skip connections. These layers are straightforward to address in general implementation of the Cooling method.
3.3 EFFECTS ON GRADIENT VALUES
We now perform analysis of the effect Cooling the network gradients.
4

---

Under review as a conference paper at ICLR 2023
3.3.1 LAST LAYER COOLING
Proposition 3.5 (Gradients of last layer Cooling), Let the notation be as in & IfC ((a()z)) where a is the softmax function and L is the cross-entropy loss, the derivative with respect to any network parameter w E 0 given by
ac
θTZ dw - (0(Tz) - Ow
(((((()))))))) y, Ow
Oz
(3.1)
expressed as the inner-product of two vectors.
Proof. When 7 1, Equation 3.1 is known result of simple computation. The general case is an application of the chain rule.
The difference = a()) may be termed the residual, namely the difference between the ground-truth label probabilities y and the label probabilities (2) computed by the network. The derivative 8C/0w is then the inner product of this residual with the vector 0z/Ow.
Interpretation.
We can use this formula to analyze the effect of temperature-scaling by T. Suppose is no heating correction. As is known, in this case probabilities tend to be overestimated, so that a (z) approaches y. All values of (z) other than the ground truth become very small, meaning that values of 0z;/0o /ōw are multiplied by small values, and so are ultimately ignored, with harmful effects on convergence.
1, there
Setting T < results in more evenly distributed (less peaked) vector (TZZ), meaning that all values of 2, and 0z,/Ow have an effect on the gradient.
3.3.2 DISTRIBUTED COOLING
Now, we consider what happens to gradients when distributed Cooling is applied. We simplify the analysis by thinking of scaling occurring in two steps. First global temperature scaling is applied by modifying the final layer so that its output is multiplied by т. Subsequently. distributed scaling is applied to all layers resulting in the output of the i-th layer being multiplied by 8, in way that the network output is unchanged. The effect of the final-layer scaling on gradients was addressed earlier. Now we concentrate on the effect of distributed scaling on gradients in the network.
Consider network with N layers, labelled 0 to N - 1, let x, be the input to the i-th layer (which is also the output of the 1-th layer), and XN the output of the last layer. Let another network have layer inputs denoted by x/.
Definition 3.6. We will say that two networks are scale-equivalent if for inputs x = x6 there are constants 3: with 9o IN 1 such that x B.x.. Evidently, for the same input xo X/, the outputs X = x'y are the same, since 3x =
will be observed, however that the gradients of the parameters of these networks will be different. Let the first (unprimed) network be represented by ri+1 Pa (W (:( ca,, + b.) where pa is activation function, possibly different for each i. Then, given numbers 3 with 3o = By = 1, an equivalent network is given by 2'41 = A W/ + by) where
W 1, W. and = B and P Pa.11
(3.2)
and Pg is modified activation function given by A (x) = 3x(B-1x). (It should be noted that if is a ReLU activation, then P) Then, x = 3, for all i, as required.
Thus, with T. representing the transformation x; + (WW, xx + b,) = x (and similarly T') we compare the two networks:
T Tw-20... To(xo) and a T TV- o...... (x(x))
where represents the final softmax layer.
is evident that these two networks carry out the same operation. However, it will be shown that optimized using gradient-descent based method, the update of their parameters will be different,
5

---

Interpretation. The effect of distributed scaling is to individually change the relative effect of gradients over the network. In particular, 3; = /'/ for O,..., - and 3y = 1, distributing the scale evenly across the network, with < 1, then the effect is to modify the gradients and relative Under review as a conference paper at ICLR 2023
and the trajectory of the parameters in the path towards the optimum during training will be quite different.
Let W iijk be one of the entries of W, and bi be one of the parameters of b, Similarly, let W Likk and b be the corresponding parameter of the primed (distributively scaled) network. The following will be shown:
f'g (xx) for scale equivalent networks fe and for Functions a and L are softmax and loss functions. Let W Wijh be the (j, k)-th entry of parameter matrix W; and b the j-th entry of parameter vector b;. Then
Theorem 3.7. Let C = L(z) where
fa
ac
B,
OC
OC
OC
and
(3.3)
OW Ljk
B+11
OWisk iijk
Bi+1 Ōb.s
Proof. Define Pi+1 = W, + b, and x = p: (P;+1)) and similarly primed quantities, We see that x = 3,x; and p = 3pp for all Let fi+1 be the mapping defined by
C (P o Tx-1 o - .... Ti+1 o P:(Pi+1)
namely, the part of the network "downstream" from Pi+1 (including the activation function pi the i-th layer, and the softmax and loss functions). Function f/41 is similarly defined for the primed network.
We apply the chain rule:
aC
OC
0p
dW
i,jk
0p.41
ÖW,,gk Ligh
A similar formula holds for the primed case.
Now, since C = fs+1(P++1) f441 (P'+1)
S44 (Bi+1Pit1)
we see
dc
OC
B
(3.4)
0p'+1
Öpit1
Next, we compare 0p.+1/ /0 ii and /OW' Ligk"
Let Eşs be the matrix with an entry in position (j.b) k) and 0 elsewhere. Then 0p:+1/8W,, E,ax, On the other hand,
0 fjjk E,ax/- = E5 = B: 0p.+1/8W.. 1,jk
Putting this together with equation equation 3.4 we see that @C/0W i.jk = (8./8L1) OC/0W.. as required.
In the where bigj is entry of b; that 0p /06 0p:-1/0b;,j
ac/061 (1/8x11) 0C/Ob...
Relative gradients. Since small change to a small parameter more important to the same change to large parameter, it is perhaps more important to determine the ratio (aC/00)/0 /0) which determines by what ratio a parameter is changed during gradient update. This gives:
8C/oW
Ligh
8
ac/ow i.tk
Wiig iijk
W
B
W ijj
0C/OV/
1
0c/abi.
B
bi.j
gradients across the network. This may have the effect of mitigating the effect of gradient vanishing.
Distributed scaling be used control the magnitudes of the output x; each level.

---

Under review as a conference paper at ICLR 2023
Impac: Cooling factor
Impact ing factor
70
27.2
0.5
Figure 1: Plots showing the test accuracy of networks trained with last layer Cooling and various Cooling factors. Left: VGG network trained CIFARIO. Right: ResNet50 network trained CIFAR The non-Cooling baselines are shown as dashed. All Cooling factors K S. 1 outperform (All means and
the baseline, the best one by 4.6%. Training diverges for Cooling factors standard deviations are computed over three runs.)
4 EXPERIMENTS
4.1 GENERAL SETUP
We explore the use of Cooling two image classification datasets and semantic segmentation dataset. We use 99% of the CIFAR "training sets" (corresponding to 50,000 images in total) for training and 1% validation either the SGD optimizer with a momentum of 0.9 or the Adam optimizer (Kingma & Ba 2015) with 0,1, 3 0.9, 32 0.999. All networks trained using the TensorFlow (Abadi al. 2015) framework.
optimise the temperature 7 on. We train networks using
4.2 IMAGE CLASSIFICATION CIFARIO
Setup. We train small VGG-style network (Simonyan Zisserman 2014 on the CIFARIO dataset (Krizhevsky 2009). The network consists of a sequence of 6 convolutional layers of filter size 3 x 3 with 32, 32, 64, 64, 128 and 128 channels, respectively, followed by two dense layers with 128 and 10 output nodes, respectively. In total, the network has approximately 620,000 trainable parameters. The hidden layers either use the ReLU (Fukushima) 1980 Nair & Hinton 2010) or the CReLU (Shang et al.) 2016) activation. When use learning rate warmup, we linearly increase the learning rate from O to 0.01 over 2 epochs. When we do not use warmup, we directly start with a learning rate of 0.01. In our learning rate schedule ablations,
experiment with (1) schedule, (2) a piecewise linear schedule which drops by a factor of 0. after 30 and 40 epochs, (3) schedule with linear decay from the initial rate to
(4) schedule with exponential decay with a total drop by either factor of 0.01 ("slow") or 0.001 ("fast") and (5) cosine decay schedule (Loshchilov & Hutter 2017). We train the network for epochs. We use batch size of 64.
Results. As shown in Table Cooling can have a significantly positive impact on network perfor- mance. We see stark difference between various LRR schedules. Whereas smooth schedules (where the learning rate changes after each batch) hardly benefit from Cooling, there a noticeable benefit for piccewise constant schedules and a drastic improvement when schedule is employed. Starting from the lowest performance at 74.7% test accuracy, last layer Cooling increases the test accuracy by 4.6%.
Figure [ (left) shows an ablation of last layer Cooling, involving various Cooling factors k. Last layer Cooling shows little sensitivity to the Cooling factor, long S 1.0. Values greater than 1.0 lead to divergence. On the other hand, all values x S 1.0 produce networks outperforming the non-Cooling baseline.

---

Under review as a conference paper at ICLR 2023
LR Schedule
No Cooling
Distributed
Periodic
Last Layer
Cosine
78.6
18.
18.9
18.9
Exp Decay Fast
77.0
77.1 10.1
77.3
76.810.6
Exp Decay Slow
77.7
77.3+0.
77.640.1
77.5+0.3
Linear
78.7=0.6
78.4
78.8+0.5
78.7+0.1
None
74.7+1.1
77. 77.640.2
79.2+0.4
79.2.0.2
Piecewise Const
78.0+0.2
79.1
79.1
79.2.t..0
Table I: Test accuracy of VGG network trained on CIFARIO with various learning rate schedules and Cooling modes. The different Cooling modes perform at least on par with baselines on all LR schedules. Cooling considerably outperforms the baseline when schedule is used. Significant gains are also achieved when a piecewise constant schedule is used. (All means and standard devia- tions computed over three runs.)
shows that Cooling works well for both ReLU and CReLU activation functions. In particular, CReLU diverged in all our experiments without learning rate schedules, but converged when Cooling was used. As for ReLU, we note that last layer and periodically redistributed Cooling outperform pure distributed Cooling.
Activation
No Cooling
Distributed
Periodic
Last Layer
CReLU ReLU
divergence 74.7
79.7
84.2+0.1
84.4+0.2
77.6
79.2
79.2
Table 2: Test accuracies of VGG network trained without a learning rate schedule on CIFARIO with different activation functions and Cooling modes. Last layer scaling and periodically redis- tributed scaling perform best for both activation functions. Training diverges for CReLU without Cooling and without a LR schedule. (All means and standard deviations are computed over three runs.)
The effect of the Cooling factor on the inverse of the network temperatures is shown in Figure 2 In the left plot, where smaller Cooling factor and last layer Cooling used, the inverse temperature grows much more slowly and only exceeds 100 after 19 epochs. In the right plot, where periodically redistributed Cooling with Tíe 9 epochs.
Tmax 100 is used, the first temperature reset already happens after
We present further experimental results on CIFARIO in Appendix
4.3 IMAGE CLASSIFICATION: CIFARI00
Setup. We train ResNet50 network (He et al. 2016b) the CIFAR100 dataset (Krizhevsky 2009]. The network consists of a sequence of 50 convolutional layers of varying filter sizes and channel numbers, followed by global average pooling layer. The network uses skip connections and has approximately 23.5 million trainable parameters in total. The hidden layers either use the ReLU (Fukushima 1980; Nair & Hinton| 2010) or the CReLU (Shang et al. (0166) activation. In our learning rate schedule ablations, we try out (1) no schedule, (2) piccewise linear schedule with drops by a factor of 0.1 after 80, 120, 160 and 180 epochs. We train the network for 200 epochs. We use batch size of 64.
Results. Similar to the CIFARIO experiments, we present an ablation on the effect of different Cooling factors last layer Cooling. In Figure [ (right) we notice the same pattern: Cooling factors x that do not exceed 1.0 yield neural network models that outperform the baseline. On the other hand, observe once more that x > leads to the divergence of network training.
4.4 SEMANTIC SEGMENTATION: ADE20K DATASET
Setup. We train a small U-Net Ronneberger et al. (2015) architecture on the challenging ADE20K dataset Zhou al. (2019). which includes 150 semantic categories. This dataset contains 20,000
8

---

Under review as a conference paper at ICLR 2023
Cooling Factor: 0.75
Cooling Factor: 1.0
230
120
Tomperature
150
80
00
50
20
5
10
15
20
25
0
15
20
25
Epochs
Epochs
Figure 2: Comparison of the increase of network temperatures for different Cooling factors. Both images show the training of VGG network on CIFARIO with ReLU activations. Left: last layer Cooling, CF: 0.75. Right: periodically redistributed Cooling, CF: 1.0. We note that larger Cooling factor causes a much steeper increase in the temperature. For last layer Cooling, the temperature keeps on growing, whereas for periodically redistributed Cooling, the temperature is redistributed whenever it exceeds a value of 100.
images for training and 2,000 images for validation, on which we report results. We leave aside 320 images from the training set when performing Cooling. We work with images of size 256 256 and our U-Net architecture has = million trainable parameters.
Results. We compare our proposed last layer Cooling against the baseline, with temperature scaling. For the former we obtain 22.1 mloU and 71.9% accuracy, while for the latter we obtain 21.0 mloU and 70.9% accuracy. This shows that our proposed cooling method also beneficial on denser, pixel-wise classification tasks. Further investigation on larger architectures and across multiple design choices could further reveal the full potential of Cooling.
DISCUSSION AND CONCLUSION
Our proposed Cooling method adaptively calibrate classification neural networks produces sig- nificant benefits in terms of network performance and training stability. Theoretical and empirical findings point to significant benefits resulting from differently scaled gradients during network train- ing. As result of experiments on different tasks, datasets and network architectures, as well as an ablation study on different hyperparameter settings we find that that Cooling gives significant per- formance benefit over relevant basclines. In particular, notice that Cooling greatly reduces the need for a learning rate schedule.
Even though all versions of Cooling re-scale the network the same mathematical function, they all produce differently parameterised networks. This reparameterisation has a strong impact on gradients, resulting in different network functions training progresses. This raises the question: What are the general conditions on the parametrisation of network to achieve optimal training?
Another highlight of work is the connection between calibration and the learning rate. There indications that well-calibrated networks are more stable in training and less reliant on the 'right' learning rate schedule. Since training stability is critical in number of classification tasks (e.g. when training the discriminator of a GAN), a deeper investigation into the relation between calibra- tion and training stability could be promising direction for future research.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https:// available from tensorflow.org.
tensorferrerro www.eorrrooo.ooooooooo.o org/ Software
Shrey Desai and Greg Durrett. Calibration of Pre-trained Transformers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image worth 16x16 words: Transformers for image recognition at scale, 2021. URL http::://pperevvieeeeeee/// http://wwwweeetroommddddddd-aaae
cNTTT
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36:19) 193-2222 1980.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training im- agenet hour. CoRR, abs/1 706.02677, 2017. URL 02677
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 06-11 Aug 2017. URL guol 7a html
ml wtt.ssss/.....//////m
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. In International Confer- ence on Learning Representations, 2021. htttttt://pperreeeee @@@eDDENNN2W)
net. net/forum?id
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 630-645, Cham, 2016b. Springer International Publishing. ISBN 978-3-319- 46493-0.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2015. URL http://arxiv.co wwtp:/o////aae/ee..o..o 6980
Alex Krizhevsky. Learning multiple layers features from tiny images. Master's thesis, University of Toronto, 2009.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-But E. Fox, and R. Gareut (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL BBao0eee2207900ff377844446630000
pdf
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In H. Wallach, H. Larochelle, A. Beygelzimer, E. d'Alché-Bue, E. Fox, and R. Garnett
10

---

Under review as a conference paper at ICLR 2023
(eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL 3B003388225503333303144888888228
pdff
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research. pp. 2814. PMLR, 10-15 Jul 2018. URL https::/proceeding
#ress/v80/ea kumaa
html
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, ternational Conference Learning Representations, 2017. URL https://openreview
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 15682-15694, Curran As- neur sociates, Inc., 2021. URL httpppt:///p/ccccp/eeiiiatc neur ps. cc httpppaprr/2221/55552 8420635940402456 0ttp:/445661aafffa222222202 44220333440222555555e6aa22222222
Rafael Müller, Simon Korbblith, and Geoffrey E Hinton. When does label smoothing help? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: http:///poceed/deeeeeeeeoe neur pptpccc/ppper///1//////p./p 1174866b0fd9443999 77446666000093111111166766622833 Paps-- pdf
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines, 2010.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer. 2015.
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improv- ing convolutional neural networks via concatenated rectified linear units. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16. pp. 2217-2225. JMLR.org. 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprini arXiv: 1409. 1556, 2014.
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. CoRR, abs/2105.01601, 2021.
abs 21..2.00111011 601
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Attention all you need,
Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.
URL https://procediggs htwp:rrrceeeeinn.....i..oo....s. cc wttpereips..ppppeeeeeeae cc/ppaper2017/Elle 3efeee44Ssreee1tabbbaaaaaaaaaaa3 pdf
2017.
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-324 2019.
11


Reviews:
[{"review_id": "z1PcsWDc9y", "reviewer": "Reviewer_DpjZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use temperature scaling during the training process, thus improving accuracy and reducing the need for learning rate schedules. In the theoretical part of the paper, the gradients have been calculated analytically, comparing the gradients with and without the proposed modifications. In the experimental part, the proposed methods have been applied on several tasks and neural architectures, showing that the methods result in improvements in classification accuracy.", "summary_of_the_review": "A strong paper with one key problem that the considered architectures are quite far from the state-of-the-art (CIFAR-10 accuracies below 80% as opposed to more than 90%). While in principle, it is good to do experiments with architectures requiring less resources, in this particular case it would be important to know whether the observed accuracy improvements are specific to the considered architectures or whether they also carry over to the state-of-the-art architectures. Also, more information could hav\ne been provided in the appendix, in particular about gradient norm evolution as well as calibration evolution.\n", "strength_and_weaknesses": "Strengths:\n\n* The paper is well-written and clear.\n* The idea of using calibration during training to improve performance can be considered novel and the methods are well justified and simple.\n* The empirical advances are nicely complemented with a theoretical study and discussion about what might be the reasons for the achieved improvements.\n\n\nWeaknesses:\n\n* My main concern is that in CIFAR10 classification the authors have only considered an architecture that results in accuracies much below 80%, whereas modern architectures (plus data augmentation) go well beyond 90%. It would be important to experiment with at least one better architecture also. In CIFAR100, a better architecture has been used, but also for this dataset there is room for improvement so that the baseline method would be at least 70% accurate.\n\n* In the introduction, the paper promises to measure ECE (expected calibration error) but never gets to doing it, not even in the appendix. On one hand, this promise could simply be dropped, but actually it would be an important thing to do, because calibration is an important part of the motivation and intuition behind the proposed method. For example, it would be good to have plots about how ECE (evaluated on test data) is evolving during training alongside with accuracy improvements.\n\n* Figure 1 (left) starts from the cooling factor value of about $0.5$. In order to support the discussion better, lower values of the cooling factor could have also been considered in that plot.\n\n* It would have been good to see more figures about gradient norm evolution in the appendix, perhaps even for all situations considered in Table 1. This would help to compare whether the evolution achieved with the proposed methods is different or similar to the evolution observed with different learning rate schedules.\n\nMinor weaknesses:\n\n* Some parentheses are missing in the proof of Lemma A.1., e.g. the first formula should be $x'_1=\\beta\\rho(\\beta^{-1}(W'_1 x_0+b'_1))$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "SEC3JbRNw4", "reviewer": "Reviewer_kQKT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to incorporate temperature scaling, a popular post-training confidence calibration technique, into training to improve performance and training stability.  Temperature scaling in confidence calibration changes the confidence of a network but does not affect the ordering of predictions. The paper hypothesizes that temperature scaling can regulate gradients if it is used during training.  Specifically, three variants: last layer cooling, distributed cooling and periodic cooling are proposed. For example, last layer cooling uses a temperature parameter in the last linear layer which is calibrated on a held-out set after every training epoch. ", "summary_of_the_review": "The paper aims to show an overlooked benefit of temperature scaling during training. While the analytical analysis is detailed, it lacks enough experiments to support the claims, making the results less convincing. ", "strength_and_weaknesses": "Pros: \n\n1, the analysis of the effects of different cooling strategies on gradients is insightful. \n\nCons:\n\n**1, Lack of experimental support.** While the method is well developed and studied analytically, the experiments do not fully support the strong claims of the paper. Specifically, classification experiments are only conducted on CIFAR10 and CIFAR100\t,¬ both which are considered small datasets by today’s standard. This is especially true when the main claim of the paper is improved performance and stability. Moreover, the accuracy reported for CIFAR10 using VGG seems too low compared to the performance using the same architecture by other open-sourced codes, e.g., [1]. \n\n**2, Contribution of distributed cooling and periodic cooling not strong.** A large component of the paper is on distributed cooling and periodic cooling. However, according to Table 1, neither of them is noticeably better than simple last layer cooling, if not worse. \n\n[1] https://github.com/chengyangfu/pytorch-vgg-cifar10\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "4fbzxzOarZ", "reviewer": "Reviewer_ZSwR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed the temperature-based cooling algorithm to achieve better performance in optimization. The authors show better gradient scaling and reducing the need for scheduling. The starting point of the cooling algorithm is that miscalibration can worsen network learning. Specifically, the authors consider last-layer cooling, distributed cooling, etc. The gradient analysis is interesting, and experimental results provide the effects of the cooling algorithm compared to naïve approaches.   ", "summary_of_the_review": "I’ve not convinced of the relationship between calibration and learning stability in a transparent manner. The paper is \nwell-written. However, it can have many issues to be solved. \n", "strength_and_weaknesses": "Pros:\nThis paper proposed an algorithm to address the optimization problem in deep networks. The algorithm is simple to be applied in practice, and the simulation results can provide the basic effect of the cooling algorithm.\n\nCons:\nCalibration has its area. Therefore, the authors must show the precise relationship between calibration and cooling in optimization in detail. Also, this paper is more related to optimization topics. Experiments are too limited because only scheduling is examined. Maybe Batch-Normalization Dropout and other regularized methods should be compared. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject