Neural Clamping: Joint Input Perturbation and Temperature Scaling for Neural Network Calibration
Authors: Yung-Chen Tang
Pin-Yu Chen
Tsung-Yi Ho
Venue: Accepted by TMLR ()

Abstract: Neural network calibration is an essential task in deep learning to ensure consistency between the confidence of model prediction and the true correctness likelihood. In this paper, we propose a new post-processing calibration method called $\textbf{Neural Clamping}$, which employs a simple joint input-output transformation on a pre-trained classifier via a learnable universal input perturbation and an output temperature scaling parameter. Moreover, we provide theoretical explanations on why Neural Clamping is provably better than temperature scaling. Evaluated on BloodMNIST, CIFAR-100, and ImageNet image recognition datasets and a variety of deep neural network models, our empirical results show that Neural Clamping significantly outperforms state-of-the-art post-processing calibration methods. The code is available at github.com/yungchentang/NCToolkit, and the demo is available at huggingface.co/spaces/TrustSafeAI/NCTV.

Published in Transactions on Machine Learning Research (07/2024)
Neural Clamping: Joint Input Perturbation and Tempera- ture Scaling for Neural Network Calibration
Yung-Cher Chen Tang The Chinese University of Hong Kong National Tsing Hua University
yctang@cse.cuht, edu.hk
Pin- Yu Chen IBM Research
pin-gn..ceeniibm.com
Tsung- Yi Ho
The Chinese University of Hong Kong
Reviewed OpenReview: https:// opeeevve//enrrevieeuu net/ forum? id= gSFToMaLcg
Abstract
Neural network calibration is an essential task deep learning to ensure consistency be- tween the confidence of model prediction and the true correctness likelihood. In this paper, propose new post-processing calibration method called Neural Clamping, which em- ploys simple joint input-output transformation on pre-trained classifier via learnable universal input perturbation and an output lemperal ure scaling parameter. Moreover, we provide theoretical explanations on why Neural Clamping is provably better than temper- ature scaling. Evaluated on BloodMNIST, CIFAR-100, and ImageNet image recognition datasets and & variely of deep neural network models, our empirical results show that Neu- ral Clamping significantly outperforms state-of taeate-of-thh--rt post-processing calibration methods. and the demo available at
The code is available at /ittutb..om//yuuuuddennnn///AAAA httfffacccccoccccsseeeeeess Imggingface.co /pacees/ http///cee/attSSaaaa///////// TrustSafeAI/NCTV
1 Introduction
Deep neural notworks have been widcly deployed in real-world machinc learning empowered applications such as computer vision, natural language processing, and robotics. However, without further calibration, model prediction confidence usually deviates from the true correctness likelihood (Guo et al., 2017). The issue of poor calibration in neural networks is further amplified in high-stakes or safety-critical decision making scenarios requiring accurate uncertainty quantification and estimation, such as discase diagnosis (Jiang et al.
2012
Esteva et al. 2017) and traffic sign recognition systems in autonomous vehicles (Shalaei et al. 2018
Therefore, calibration plays an important role in trustworthy machine learning (Guo et al. 2017 K Kumar
et al. 2019 Minderer et al. 2021)
Recent studies on neural netw calibration can be mainly divided into two categories: in-processing and post-processing. In-proccssing involves training or fino-tuning neural networks mitigate their calibration errors, such as Müller et. al. (2019) Liang al. (2200) Tian et al. (2021); Qin et al. (2021); Tao et al. (2023) Post-processing involves post- hoc intervention
pre-trained neural network model without changing the given model parameters, such as adjusting the data representations of the penultimate layer (i.c., the logits) to calibrate the final softmax layer's output of prediction probability estimates. As in-processing calibration tends to be time-consuning and computationally expensive, in this paper, we opt to focus on post- wost-processing calibration.
Current post-processing calibration methods are predominately shed on processing or remapping the output
logits of neural networks, e.g., Guo et. al. (2077 Kull et. al. (019h: Gupta et al. (0000) Tian et. al. (021

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Xiong al. 2023 However, aim to provide new perspective and show that joint input-output model calibration can further improve neural network calibration. The rationale is that active adjustment of data inputs will affect their representations in every subsequent layer, rather than passive modification of the output logits.
In this paper, we propose a new post-processing calibration framework for neural networks. We name this framework Neural Clamping because its methodology is based on learning a simple joint input-output transformation for calibrating pre-trained (frozen) neural network classifier. Figure 1 illustrates the entire procedure of Neural Clamping. We consider K-way neural network classifier fa(-) R k with fixed model parameters 0. The classifier outputs the logits for K classes and uses softmax on the logits to obtain the final confidence on class predictions (i.e., probability scores). To realize joint input-oulput calibration, Neural Clamping adds a trainable universal perturbation 8 to every data imput and a trainable temperature scaling parameter at the output logits. The parameters 5 and T are jointly learned by minimizing the focal loss Lin et al. (2017) with a weight-decay regularization term trained on calibration set (i.c. validation set) {1i; yi Va == for calibration. The focal loss assigns non-uniform importance
{a:) during training and includes the standard cross entropy loss special case. Finally, in the evaluation (testing) phase, Neural Clamping appends the optimized calibration parameters 5* and T" to the input and output of the fixed classifier fo(-), respectively.
Neural Clamping
Training
Updiate and gradient descent
Pg
Focal
Apr
xx
Training Data Calbratiin (0/
Learnable Universal Perturbation
Fixed X-way Image Classifier fol:)
Logils foln,
Leamable Temperaturo
Probabilly Scores in Caloration
Softmax & Scaling
+
Testing
Learned Porturbation
Fixad K Image Glassifier fx()
Logits f-0 8*)
Learned Temperalure
Probabillyy Scores
8*
Testing
Figure 1: Overview of Neural Clamping: joint input-output post-processing calibration framework.
Our main contributions are summarized as follows:
We propose Neural Clamping as novel joint input-outpun post- post-processing calibration framework for neural networks. Neural Clamping learns a universal input perturbation and a temperature scaling parameter at the model output for calibration. It includes temperature scaling as a special case, which a strong baseline for post-processing calibration.
We develop theoretical results to prove that Neural Clamping is better than temperature scaling in terms of constrained entropy maximization for uncertainty quantification. In addition, we use first-order approximation to optimize the data-driven initialization term for the input perturbation, improving the stability of Neural Clamping in our ablation study. Furthermore, we leverage this theoretical result to design computationally efficient algorithm for Neural Clamping.
2

---

Published in Transactions on Machine Learning Research (07/2024)
Evaluated different deep neural network classifiers (including ResNet (e et al. 2016 Vision Transformers (Dosovitskiy et al., 2020 and MT.P-Mixer (Tolstikhin et al., 2021 trained on Blood- MNIST Yang et al. 2023,, CIFAR-100 (Krizhevsky et al. 2009,, and ImageNet- Deng et al. 20090 datascts and three calibration metrics, Neural Clamping outperforms state-of-the-art post- processing calibration methods. For instance, when calibrating the ResNet-110 model on CIFAR- 100, the expected calibration error improved by 34% when compared the best baseline.
2 Background and Related Work
In this section, we begin by introducing the problem formulation for calibration and describing the notations used in this paper. Furthermore, we define different metrics used to measure calibration error and conclude this section with an overview of the post-processing calibration methods.
2.1 Probabilistic Characterization of Neural Network Calibration
Assune pair of data sample and label (æ,y) is drawn from a joint distribution D C XxY, where E X is data sample, and E Y = {1....,A K} is the ground-trute truthtuthhhhhhhh class label. Let fo - R denote K-way neural network classifier parametrized by 8, where z = (x) = [A,,.... 2x] is the model's output logits of a given data input æ. Following the convention neural network implemennatiioss, the prediction probability score of obtained by applying the softmar function a on 2, denoted as ((2) E [0, 1] The k-th component of (z) is defined as dz)x = exp(z*)/ Σ exp(zæ), which satisfies Σ az)) = and a()); 2 0. Suppose the model predicts a most likely class ÿ = arg max; (z)) with confidence p = o (z))g. Formally, the model
So is called calibrated if
F(y = Slp = =
(1)
where P denotes probability and p demotes the true likelihood. Equation 1 only considers the prediction confidence of the most likely (top-1) class. We can extend to consider the prediction confidence of every class. Let the class- wise prediction confidence be P. = o(2), for i (1...., K), the network is called classuisseccalirrttedd if
P(y ipp; p.) = pi, VieY
(2)
where Pt is the true likelihood for class
2.2 Calibration Metrics
Expected Calibration Error (ECE). Calibration error aims to compute the difference between confidence and accuracy as
(( [IP flp -0- - -
(3)
Unfortunately, this quantily cannot be exactly computed from equation 3 the underlying data distribution ID is unknown. The most popular metric to measure calibration is the Expected Calibration Error (ECE) Guo et al.) 2017 Naeini et al., 2015). ECE approximates the calibration error by partitioning predictions into m intervals (bins) (m The calibration error is calculated by first taking the difference between the confidence and accuracy in each bin and then computing the weighted average across all bins, i.e.,
M
B
ECE =
Σ 1=1
Jacc( B,) conf( (B:]
(4)
where |B:| is the number of samples in bin B;, is the total number of data, and acc(B:) and conf( is the accuracy and confidence in B, respectively.
Adaptive Expected Calibration Error (AECE) MMukhoti et al., 2020). Since most data for a trained model fall into the highest confidence bins, these bins mostly determine the value of the ECE. Instead of pre-defined intervals for bin partitioning, in AECE, adaptive interval ranges ensure each bin has the same number of samples. AECE is defined
3

---

Published in Transactions on Machine Learning Research (07 (024)
Bil Σ
AECE
acc conf(B:)
(5)
i=1
subject
|B
[B,]
Vi,j
where |B:] is the number of samples in bin B;, is the total number of data, and acc(B:) and conf( R;)) is the accuracy and confidence in Bi, respectively.
Static Calibration Error (SCE) (Nxoo et al., 2019). ECE does not take into account the calibration error for all classes. It only calculates the calibration error of the top-1 class prodiction. SCE extends FCE and considers multi-class predictions based on Equation
M
B n
SCE = K k=
|acc(i, k) - conf(i,
(6)
where is the number of classes, |B]] the number of samples in bin i of class k, n the total number of data, and acc(i, k) and conf(i, k) the accuracy and confidence B,, respectively.
2.3 Post-Processing Calibration Methods
Temperature Scaling (Guo et al., 2017). Temperature scaling is the simplest variant of Platt scaling PPatt al. 1999,, which a method of converting classificr's output into probability distribution over all classes. Specifically, all classes have the same scalar parameter (i.e., temperature) T > in the softmax output such that q o2/TT), where q [o, 1]K denotes the calibrated probability scores. It is worth noting that by definition temperature scaling only changes the confidence but not the class prediction. Moreover, the entropy of & increases with T when T 2 1. The temperature T is optimized via the Negative Log Likelihood (NLL) over a calibration training set {ar)]== where NL. is defined as - -E log( and L. is the prediction on the correct class y for the i-th sample.
Vector Scaling and Matrix Scaling (Guo et al., 2017). They are two extensions of Plat scaling Pllat et al. 1999). Let be the output logits for input Vector Scaling and Matrix Scaling adopt linear transformations
such that I o(W + b), where W E kx* and b E g for both settings. Vector scaling is variation of matrix scaling when W is restricted to be diagonal matrix. The parameters W and b are optimized based on NLL.
MS-ODIR and Dir-ODIR (Kull et al., 2019). The authors in Kull et al. (019 proposed Dirichlet calibration and ODIR term (Off-Diagonal and Intercept Regularization). The difference between matrix scaling and Dirichlet calibration is that the former affects logits while the latter modifies pseudo-logits through 4 o In(o(z)) + b), where In(.) is component-wise natural logarithm function. The results in Guo et al. 2017 indicated poor matrix scaling performance, because calibration methods with large number of parameters will over-fit to a small calibration set. Therefore, Kull et al. 2019 proposed new regularization method called ODIR to address the overfitting problem, i.e. ODIR K(K-IJ Ejts Wyy +
bis where Wj, and by are elements of W and b, respectively. MS-ODIR applies Matrix Scaling Guo
ct al. 2017 with ODIR, while Dir-ODIR uses Dirichlet calibration (Kull et al. 2019).
Spline-Fitting GGupta et al.) 2020 and Density-Ratio Calibration (Xiong et al. 2023 The
Spline-fitting approximates the empirical cumulative distribution with splines to re-calibrate network outputs for calibrated probabilitics. However, this method is limited to calibration on the top-k class prediction but cannot be extended to all-class predictions. The Density-Ratio Calibration approach focuses on estimating continuous density functions, which aligns well with calibration methods that produce contimuous confidence scores, such as scaling-basod calibration techniques. However, this method is limited in that can only calibrate the predicted probabilities, and cannot calibrate the full probability distribution output by the model. Therefore, these methods beyond our studied problem of all-class processng calibration.
4

---

Published in Transactions on Machine Learning Research (07 (07/2024)
3
Neural Clamping
Based on the proposed framework of Neural Clamping as illustrated in Figure 1 in this section we provide detailed descriptions on the joint input-output calibration methodology, the training objective function, the influence hyperparameter selection, and the theoretical justification on the improvement over temperature scaling and the date-driven initialization.
3.1 Joint Input-Output Calibration
To realize joint input-output calibration, Neural Clamping appends a learnable universal perturbation ò the model input and learnable temperature scaling paramcter T for all classes at the model output. In contrast to the convention of output calibration, Neural Clamping introduces the notion of imput calibration by applying simple trainable transformations on the data inputs prior to feeding them to the model. In our implementation, the input calibration is simply a universal additive perturbation d. Therefore, Neural Clamping includes temperature scaling as a special case when setting = O.
Modern neural networks often suffer from an overconfident issue, resulting a lack of well-calibration and low output entropy (Guo al. 2017 Mukhoti et al. 2020). Calibrating neural networks, a common approach to address the overconfident issue, typically results in increased entropy a byproduct. In the seminal work neural network calibration by Guo et al. (2017), it is noted that adjusting the temperature parameter to improve calibration aligns with the objective of maximizing the entropy of the output probability distribution under additional constraints.
Building upon this notion, we extend the problem formulation presented in Guo et al. 2017 which utilizes
entropy to study output calibration. We evaluate this problem on calibration set { (=,, V4 J
J with an input
perturbation 5. The objective to find the best-calibrated outpul q² that maximizes the entropy of q" while satisfying the specified calibration constraints:
n
Maximize
GERK
Σ ((2)) log(g(z,)) 1=1
subject to ((2)) 2 0 Vi fl..... n) and k e {l..... KK
(7)
Σ 1 [T())) =1 E 2 e
Dim 23 ((z))
where T denotes voctor transpose, 2; = folx; + 5) the logit of x; + 1 is an all-one vector, e(v)) an one-hot vector corresponding to the class label yi, and log(.) an element-wise log operator. The first two constraints guarantee that q probability distribution, whereas the third constraint restricts the range of possible distributions, which stipulates that the average true class logit equals to the average weighted logit.
To motivate the utility of joint inpul-oulput calibration, the following lemma formally states that the pro- posed form of joint input perturbation and temperature scaling in Neural Clamping is the unique solution q" to the above constrained entropy maximization problem.
Lemma 3.1 (optimality of joint input-output calibration). For any input perturbation 5, let fo(.) 116" ...... 5 [{)] be fized K K-way newral network classifier and let be the oulput logils of perturbed data
input + 8. Then the proposed form of joint input-output calibration in Neural Clamping is the unique
expls!" " (x+5)/T]
solution qº (x)) Σ " vk (1...., ...,K to the constrained entropy marimization problem in equalion
Proof. The proof is given in Appendix
3.2 Training Objective Function in Neural Clamping
Neural Clamping uses the focal loss Lin al. 2017) and a weight-decay regularization term as the overall objective function for calibration. It has been shown that the focal loss is upper bound of the regularized
5

---

Published in Transactions on Machine Learning Research (07 (07/2024)
ResNet-50 on BloodMNIST
0.000
0.13
0.337
ResNet-110 CIFAR-100
1.05
-1.00
0:33
WideRessNet440-10 CIFAR-100
435
4.60
5.00
Loss Entropy
ECE
0..90
0.933
Loss
1.00
0.373
000
0.44
Entropy
0..90
0.75
-0.95
3.00
5.00
0.903
10.42
Ennropy
0.923
0.35
05
0.302
0003
Loss
2.00
5.00
0.49
ECE
2.000
0.370
0.025
0.70
0.822
0.93
Loss
335
-330
0.802
Entropy
0.05
100
12
06
2 0.728
02
04
0.
02
04
x2 55
Gamma
Gamma
Gamma
Figure 2: Neural Clamping on ResNet-50/Reshe 110 and Wide- RelieRReeNet-0-1100 with different Y values and the resulting expected calibration error (ECE), training loss, and entropy on BloodMNIST and CTFAR-100. When y = 0, focal loss reduces to cross entropy loss. The experiment sctup is the same as Section !
KL-divergence (Charoenphakdee et al. 2021 Mukhoti et al. 2020 Therefore, minimizing the focal loss aims to reduce the KI, divergence between the groundtruth distribution and the predicted distribution while increasing the entropy of the predicted distribution. Focal loss is an adjusted cross entropy loss with a modulating factor (1-ft.y) ) and y 0, where fi..; is the prediction probability given by neural network the correct class yı for the i-th sample. When y 0, focal loss reduces to the standard cross entropy loss. Formally, it is defined as
(f((s)))) = -11-11111111 - -pr.))" log(ps.)))
(8)
Given a calibration training set (ær, Mi the optimal calibration parameters å and T in Neural Clamping are obtained by solving
8", T = arg min + 2
(9)
Like other post-processing calibration methods, Neural Clamping only appends a perturbation at the model input and a temperature scaling parameter at the model output. It does not require any alternations on the given neural network for calibration.
3.3 How to Choose a Proper 7 Value in Focal Loss for Neural Clamping?
The focal loss has a hyperparameter 7 governing the assigned importance on each data sample in the aggregated loss. To understand its influence on calibration, in Figure 2 we performed a grid search of y value between 0 and 1 with an interval of 0.05 to calibrate 2016) and DenseNet-12. (Huang et al. 2017 models trained on CIFAR-100 (Krizhevsky et al. 2009) dataset. While the entropy continues to increase as the gamma value increases, ECE attains its minimum some intermediate value and is better than the ECE of using cross entropy loss (i.e., ? = 0). This observation verifies the importance of using focal loss for calibration. In our implementation, we select the best value that minimizes ECE of the calibration dataset from a candidate pool of y values with separate runs.
Resnet-40-11 (Zagoruyko & Komodakis
3.4 Theoretical Justification on the Advantage of Neural Clamping
Here we use the entropy after calibration as quantifiable metric to prove that Neural Clamping can further increase this quantity over temperature scaling. Note that temperature scaling is a special case of Neural Clamping when there is no input calibration (i.e., setting 5 = 0). For ease of understanding, we define gi as the gradient of the output entropy H (((((())/T) with respect to the input data æ; [a "...,..,,""" ((m) E [a, $]. where [a, [] RM x RT means the bounded range of all feasible data inputs (e.g., every image pixel value is within [0, 255)). We further define é E Rm and μ E I as the lower bound and the upper bound over all calibration data {a on each input dimension. That is, their j-th entry defined é, minee x!
and My max;e(1....)) xt respectively.
6

---

Published in Transactions on Machine Learning Research (07/2024)
With the use of first-order approximation, the following theorem shows that given the same temperature value 7, Neural Clamping increases the entropy of temperature scaling by 8 9, demonstrating the advantage of involving input calibration. Furthermore, based on our derivation and the data-driven bounds & and μ4, we can obtain a closed-form first-order optimal solution 8 for maximizing the entropy increment 8 g. We call 8 the dala-driven inilialization for the input perturbation 6. We will perform an ablation study to compare the performance and stability of data-driven versus random initialization in Section 4.3 In the following theorem, the notation sign, and denote element-wise absolute value, sign operation (i.e., =1), and product (i.e., Hadamard product), respectively.
Theorem 3.2. (provable entropy increment and data-driven initialization) Let [a, B] be the feasible range of data inputs and g = 2 9: =gg()) g(.). be the sum of local input gradients. Define n E I element-wiss such that πı = 1 aj g 0, n3 = - if gاوو 0, and fl = otherwise, for every j mm.....m.. Approaching by first-order approzimation and given the same temperature value T, Newral Clamping increases the endropy of lemperature scaling by 8 g. Parthermore, the optimal value å for mazimizing 8" g is ô sign(g) @ 17.
Proof. The proof is given in Appendix
Comparison with various calibration methods on BloodMNIST with ResNet-50. The reported re- sults are mean and standard deviation over runs, The see//seoon-bbss method highlighted by olue/green color. On ECE/AECE, the relative improvement of Neural Clamping the best baseline is 31% and 28%, respectively.
Table 1:
ResNet-50
Method
Accuracy (%)
Entropy 1
ECE (%) 4
AECE (%)
SCE (x10
Uncalibrated
85.79
0,2256
5.77
5.76
1.7003
Temperature Scaling
85.79 =0
0.3726 +0
1.77 =0
1.66
1.1067
TS by Grid Search
85.79 =0
0.3684 +0
2.13 =0
1.68 +0
1.1011 #0
Vector Scaling
85.79 =0.05
0.3653 +0.0023
1.97 00.11
1.94 +0.06
0.9264 +0.0571
Matrix Scaling
85.79 =0.38
0.2984 +0.0161
1.96 0.66
4.86 +0.71
1,4665 10.1314
MS-ODIR
85.79 =0.04
0.3726
#0.0001
1.94 =0.01
1.70 #0.03
0.9099 #0.0101
Dir-Oll OD-I
85.79 =0.02
0.3748
+0.0002
1.55 =0.04
1.71 +0.09
0,8366
+0.0034
Neural Clamping (CE)
85.79 =0.02
0.3820
+0.0005
1.54 =0.02
1.57 +0.05
1.1100 #0.0103
Neural Clamping (FL)
85.82 =0.03
0.4204
+0.0004
1.05 00.03
1.19 +0.06
1.0797
+0.0042
4 Performance Evaluation
In this section, we conducted extensive experiments evaluate the performance of our proposed Neural Clamping calibration method using the calibration metrics introduced in Section 2.2 We compared our method to several baseline and state-of-the-art calibration methods. All experiments are evaluated on three popular image recognition datasets (BloodMNIST, CIFAR100, ImageNet-1K) and six trained deep neural network models (e.g. ResNet, Vision Transformer (VIT), and MLP--ixer). An ablation study on Neural Clamping is presented at the end of this section.
4.1
Evaluation and Implementation Details
Experiment setup. We used ResNet-50 (He et al, 2016 BloodMNIST Yang et al. 2023 Res.Ne-- 110
He ct al.) 2016 and Widdo-RssNet-40-. (Zagoruyko & Komodakis 2016) models on CIFAR-100 Krizhevsky (He et al., 2016), VIT-S/16 (Dosovisskiy et al. 2020 and MLP-Mixer B/16 (Tossiikhin et al. 2021 models on ImageNet- 1K (Dng et al. 2009). Blood MNIST, a recognized medical machine learning benchmark, features 11 1,99/1,,12//,,
et al.
2009);
ResNet-101
samples for training /aliidaaioon/vvaaaattioo /evaluatioon. For CIFAR-100 and ImageNet, lacking default validation data, we divided CIFAR-100's training into 45,000 training images and 5,000 calibration images. For ImageNet, 25,000 test images were reserved for calibration,

---

Published in Transactions on Machine Learning Research (07/2024)
Comparison with various calibration methods on CIFAR-100 with different models. The reported re- sults are mean and standard deviation over runs, The best best/sccond- /scooddoddddae best method highlightod by color. On ECE, the relative improvement of Neural Clamping to the best baseline is 34/5 % on ResNet- 110/Wide ResNet-40-10, respectively.
Table 2:
ResNet- 110
Method
Accuracy (%)
Entropy
ECE (%)
AECE (%) 4
SCE (x10 2)
Uncalibrated
71.15
0.4742
10.74
10.71
0.2763
Temperature Scaling
74.15 +0
0.8991 +0
1.71 #0
1.63 #0
0.1711 #0
TS by Grid Search
74.15 =0
0.9239 +0
1.35 =0
1.38 +:
0.1717 +0
Vector Scaling
73.81 -0.55
0.8698 +0.0008
2.29 +0.07
2.15 +0.15
0.1949 +0.0046
Matrix Scaling
62.03 0031
0.1552 +0.0026
31.85 +0.29
31.85 10.29
0.6842 +0.0057
MS- MS-ODDR
74.07 #0.03
0.9035 +0.0001
1.79 +0.04
1.75 +0.03
0.1797 +0,0006
Dir- Dir-ODIR
74.10 =0.04
0.9160 +0.0002
1.36 +0.05
1.31 +0.03
0.1780 +0.0014
Neural Clamping (CE)
74.17 #0.07
0.8928 +0.0061
1.67 +0.16
1.63 +0.19
0.1709
#0.0020
Neural Clamping FFL)
74.16
10.09
0.9707 10.0049
0.89 10.06
1.01 10.11
0.1754 10.0015
Wide-ResNet- 10-10
Method
Accuracy (%)
Entropy t
ECE (%)
AECE (%) 4
SCE (x10-3)
Uncalibrated
79.51
0.4210
7.63
7.63
0.2188
Temperature Scaling
79.51 +0
0.7420 +0
2.30 +0
2.17 +0
0.1627 #0
TS by Grid Search
79.51 10
0.8359 10
1.75 1:
1.54 1:0
0.1659 10
Vector Scaling
79.08 +0.09
0.7079 #0.0012
2.52 +0.07
2.35 +0.05
0.1818 +0.0032
Matrix Scaling
68.48 10.16
0.1371 +0.0023
26.13 +0.15
26.12 +0.15
0.5657 #0.0024
MS-ODIR.
79.15 10.03
0.7529 10.0002
1.90 +0.07
1.95 10.03
0.1705 10.0008
Dir-ODIR
79.51 10.01
0.7707 10.0001
1.81 10.03
1.98 10.01
0.1625 10.0004
Neural Clamping (CE)
79.53
10.01
0.7461 10.0030
2.27 10.03
2,20
10.03
0.1624
10.0004
Neural Clamping (FL)
79.53 +0.04
0.8626 +0.0033
1.67 +0.14
1.66
+0.12
0.1683 +0.0014
and the remaining 25,000 were for evaluation. Uniform calibration dataset and test set were shared across all methods. Our experiments ran on an Nvidia Tesla V100 with 32GB RAM and Intel Xeon Gold CPU.
Comparative methods. We compared our method to all the post-processing calibration methods in- troduced in Scction 2.3 including Temperature Scaling (Guo et al. 2017). Vector Scaling, Matrix Scaling, Matrix scaling with ODIR (MS-ODIR) (Kull et al. 2019), and Dirichlet Calibration (Dir-ODIR) (Kull et al., 2019). For temperature scaling, we considered two implementations: (a) learning the temperature by mini- mizing NLL loss via gradient docent on the calibration dataset, and (b) taking a grid search on temperature over to 5 with a resolution of 0.001 and then reporting the lowest ECE and its corresponding tempera- ture, for which we call TS (Grid Searched). For MS-ODIR and Dir-ODIR, we trained their regularization coefficient with 7 values from 10-2 10 and chose the best result on the calibration dataset (See Section 4.1 in Kull al., 2019)). ). All methods were trained with 1000 epochs with full-batch gradient descent with learning rate 0.001. In addition to Neural Clamping with the focal loss (FL.), we also compared Neural Clamping with the cross entropy (CE) loss.
Neural Clamping implementation. The hyperparameters and y in equation are determined by the best parameter minimizing the FCE on the calibration dataset. The choice of ? was already discussed in See 3.3. The default value of y is set to 1 bocause we find it be stable across models and datascts. Regarding the choice of X, its purpose is to aid in regularization. Our default approach is to set this term to be 1/10 of the initial loss. The input calibration parameter 8 and the output calibration parameter T are optimized using the stochastic gradient descent (SGD) optimizer with learning rate 0.001, batch size 512, and 100 epochs. For initialization, uses random initialization and T is set to 1. The detailed algorithmic procedure of Neural Clamping is presented in Appendix A
Evaluation metrics. We reported evaluation measures on the test sets: Accuracy, Entropy, ECE, AECE, and SCE. All three calibration metries are defined in Section 2.2 and 15 bins were used. In all experiments, we report the average value and standard deviation over 5 independent runs.
8

---

Published in Transactions on Machine Learning Research (07/2024)
Comparison with various calibration methods on ImageNet with different models. The reported re- sults are mean and standard deviation over runs. The best best/sccet het//sscond-bess best method highlightod by blue/green color. On ECE, the relative improvement Neural Clamping to the best baseline is 11/6/13 % on ResNet- 101/ViT-S16/MML--Mictt B16, respectively.
Table 3:
ResNet-
Method
Accuracy (%)
Entropy
ECE (%)
AECE (%)
SCE (x10 3)
Uncalibrated
75.73
0.6608
5.88
5.88
0.3180
Temperature Scaling
75.73 =0
0.9376 +0
1.88 +0
1.91 #0
0.3117 #0
TS by Grid Search
75.73 =
0.9244 +0
2.02 =0
1.97 ::
0.3108 +0
Vector Scaling
75.67 00007
1.0463 +0.0017
2.04 10012
1.92 +0007
0.3192 +0.0009
Matrix Scaling
51.97 00.30
0.0593
+0.0008
45.61 +0.28
45.60 10.28
0.9037 +0.0052
MS- MS-ODIR
70.71 #0.10
0.9904
+0.0016
3.29 +0.06
3.28 +0.06
0.3448 #0.0011
Dir- DDr-ODRR
70.72 +.00
0.9841
+0.0007
3.47 +0.05
3.47 +0.05
0.3480 +0.0013
Neural Clamping (CE)
75.73 #0.01
+0.0240 0.9429
1.89 +0.13
1.88 #0.11
0.3114 #0.0007
Neural Clamping FFL)
75.73 10.01
1.0103 10.0245
1.68 10.04
1.71 10.03
0.3128 10.0001
VIT-S/16
Method
Accuracy (%)
Entropy
ECE (%)
AECE (%) 4
SCE (x10-3)
Uncalibrated
79.90
0.7161
1.28
1.30
0.2808
Temperature Scaling
79.90 =0
0.7314
1.08 +0
1.09
0.2817
TS by Grid Search
79.90 =0
0.7791 40
0.82 +:
0.80 1:0
0.2852 10
Vector Scaling
80.02 +0.03
0.9410 +0.0014
2.62 +0.02
2.69 +0.03
0.2983 +0.0015
Matrix Scaling
53.99 0.229
0.0646 +0.0010
43.36 +0.30
43.36 +0.29
0.8811 #0.0054
MS-ODIR
75.94 10.09
0.9810 10.0018
0.87 10.10
0.92 10.10
0.3163 10.0023
Di--ODIR
75.93 1..0
0.9788 10.0007
0.93 10.06
0.86 10.09
0.3149 10.0018
Neural Clamping (CE)
79.98
10.01
0.7898
10.0028
0.81 10.03
0.77 10.04
0.2801
10.0005
Neural Clamping FFL)
79.97
+0.01
0.7934 +0.0038
0.77 +0.01
0.72 +0.03
0.2804 +0.0004
MLP-Mixer B/16
Method
Accuracy (%)
Entropy
ECE (%)
AECE (%) +
SCE (x10-9)
Uncalibrated
73.94
0.6812
11.55
11.55
0.3589
Temperature Scaling
73.94 +
1.2735 +0
4.94 +0
4.98 +0
0.3188 t0
TS Grid Search
73.94 +0
1.6243 +0
2.60 +0
2.60
0.3258 +0
Vector Scaling
73.24 10.06
1.1474 10.0089
6.91 10.17
6.88 +0.20
0.3321 10.0027
Matrix Scaling
40.96 10.31
0.1137 10.0010
54.50 10.28
54.50 10.28
1.0979 10.0041
MS-ODIR
73.16 +0.02
+0.0016 1.8049
4.65 +0.08
4.73 +0.05
0.3477 +0.0018
Dir-ODIE
73.13 10.05
10.0013 1.8083
4.68 10.09
4.76 +0.09
0.3480 10.0018
Neural Clamping (CF)
74.14 +0.01
1.7952
+0.0302
2.43 +0.16
2.51 +0.18
0.3054
+0.0020
Neural Clamping (FL)
74.12 10.00
1.7673 10.0269
2.27 10.13
2.34 10.14
0.3029 10.0018
Table 4: Ablation study with ResNet-110 on CIFAR-100. The best result. is highlighted by blue color.
Method
Accuracy (%)
Entropy
ECE (%)
AECE (%)
SCE (x10-2).
Uncalibrated
74.15
0.4742
10.74
10.71
0.2763
Temperature Scaling
74.15
0.8991
0.1711
Temperature Scaling (FI.)
74.15
1.0542
2.01
2.02
0.1812
Iuput Calibration w/ 8*
74.16
10.09
0.4775 +0.03
10.62 00.1
10.61 10.10
0.2776 10.0019
Output Calibration w/ 7
74.15 +0
0.9648
+0.46
1.18 +0.05
1.35 +0.01
0.1731 +0.0003
Neural Clamping
74.16 10.09
0.9707 10.49
0.89 10.06
1.01 10.11
0.1754 10.0015
4.2 BloodMNIST, CIFAR-100, and ImageNet Results
BloodMNIST. BloodMNIST an 8-class microscopic peripheral blood cell image recognition task. The calibration results with ResNet-50 are shown in Table - Compared to the best existing method, Neural Clamping shows additional 31% 288 reduction in ECE/AECE.

---

Published in Transactions on Machine Learning Research (07/2024)
The experimental results CIFAR-100 are presented in Table which is divided into two sections corresponding to different models: ResNet-110 and Wide-ResNee-400110 Our method consistently achieves the lowest ECE and either the lowest or second lowest AECE and SCE when compared to other existing methods. Notably, in the ResNet-110 experiment, Neural Clamping reduced ECE and AECE by 34% and 23%, respectively, compared to the best existing method. important to highlight that our method not only reduces calibration error but also improves accuracy, which sets apart from existing approaches.
CIFAR-100.
ImageNet-11. Table g presents the experimental results on ImageNet, where the table is divided into three sections containing ResNet-101, VIT-S/16, and MLP-Mixer B/6. Neural Clamping consistently outperforms the compared methods by achieving the lowest ECE, AECE, and either the lowest or second-lowest SCE in all cases, similar to the CIFAR-100 experiments. Particularly, in the ResNet-101 experiment, Neural Clamping reduces ECE and AECE by 11% compared to the best existing method. Moreover, our method concurrently improves accuracy while reducing the calibration error for all three models, demonstrating its effectiveness in calibration for various model architectures.
Additionally, We also provide experimental results of different bins number in Appendix D which clearly demonstrates the same conclusion of the outstanding calibration performance of Neural Clamping over the baselines. To further compare how our method differs from the baselines, we also visualize the ECE results via plotting the reliability diagrams in Appendix
4.3 Additional Analysis of Neural Clamping
Data-driven random initialization for input perturbation 8. There are two initialization methods for the imput calibration d in Neural Clamping: data-driven initialization as derived from Theorem 3.2 and random initialization. In scrutinizing these two initialization methods, we found that the data-driven initialization managed to consistently deliver stable calibration results. shows that across all metrics, both initialization methods have similar mean values across 5 runs, while the data-driven initialization has a smaller variation and standard deviation. As a result of the experiment, can be concluded that data- driven value can not only offer a more reliable solution but also slightly improved outcomes. We also used this data-driven initialization to devisc computationally efficient Neural Clamping variant. Under similar runtime constraints, This implies that theoretically derived input perturbations can attain performance comparable to that of training results. Please see Appendix for details.
method achieves superior calibration performance compared to temperature scaling,
23113
0011
000377
mean=0.00mm DO89
6.29:
30105
0002
trean=000088 stal=0 100==0.0001
000079
38122
meen=0 9723
meen=0.0017
0963
mean= 0.9707
std=0.0018
mean-D.0101 d001
mean-0.comm 0.0096
mean=0.0017
std=6.6e-00
0084
$0d=0.0049
1000
std=1.5e-05
30005
Random
Random Inibolization
Intialization
Initialization
Initialization
Intialllzation
Intialization
Initialization
Indillssatinn
(a) Entropy
(b) ECE
(c) AECE
(d) SCE
Figure 3: Comparison of random (blue) and data-driven (green) initializations for input calibration 8 in
Neural Clamping. The reported results are (a) Entropy, (b) ECE, (c) AECE, and (d) SCE of ResNet-
110 on CIFAR-100 over 5 runs. This boxplot graphically demonstrates the spread groups of numerical data through their quartiles. The data-driven initialization shows better stability (smaller variation) than random initialization.
Ablation study with input calibration 8* and output calibration T After calibration, Neural Clamping learns 8* for input calibration and T* for output calibration. In Table g we performed an ablation study to examine the effects input calibration and output calibration separately with their jointly trained parameters 8* and T and "Temperature Scaling (FI.)' approach, where T optimized with focal loss.
10

---

Published in Transactions on Machine Learning Research (07/2024)
For imput calibration, inferred testing data with only the learned input perturbation 8*; for output calibration, we tested the result with only the learned temperature scaling parameter 7". One noteworthy finding from this exercise is that while output calibration alone already trims the ECE and AECE materially, a further 25% reduction in ECE and AECE can be achieved when is paired with input calibration (i.e. Neural Clamping)- Input calibration alone less effective because does not directly modify the prediction output.
In addition, the Temperature Scaling (FL) actually performed worse than both the original Temperature Scaling method and 'Output Calibration w/ T** across the various calibration metrics. This finding suggests that the improvements achieved by our method Neural Clamping (joint input-output calibration) is not simply due to the use of temperature scaling with focal loss. This ablation study corroborates the necessity and advantage of joint input-output calibration and the additional benefits only gained from joint calibration framework.
5 Conclusion
In this paper, we present a new post-processing calibration method called Neural Clamping, which offers novel insights into joint input-output calibration and significantly improves calibration performance. We also develop theoretical analysis to justify the advantage of Neural Clamping. Our empirical results several datasets and models show that Neural Clamping outperforms state-of-the-art post-processing calibration methods. We believe our method delivers practical tool that can contribute to neural network based technology and applications requiring accurate calibration.
Impact Statements
We see ethical or immediate negative societal consequence of our work, and it holds the potential for positive social impacts. By improving the accuracy of machine learning models' prediction probabilities, our research can benefit various domains.
Acknowledgments
would like to express my gratitude to Yu-Chich Cheng for his invaluable assistance in proofreading and editing.
References
Nontawat. Charoenphakdee, Jayakorn Vongkulbhisal, Nuttapong Chairatanakul, and Masashi Sugiyama. On focal loss for class-posterion probability estimation: A thcoretical perspective. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Poltern Recognition, pp. 5202 5211, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-2442222 255-255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXic:2010. 10. 11929, 2020.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologgistlevell classification of skin cancer with deep neural networks. nature, 542(7639): 115-118, 2017.
Chuan Guo, Gcoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1338 PMLR, 2017.
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. Xixio preprint or Xiv:2006. 12800, 2020.
11

---

Published in Transactions on Machine Learning Research (07/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pp- 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional nctworks. In Proccedings of the IEEE conference on computer vision and pattern recognition, pp.
4700-4708, 2017.
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. Calibrating predictive model estimates support personalized medicine. Journal of the American Medicol Informatics Association, 19(2):263 274, 2012.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Meelis Kull, Miquel Perello Nielo, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Be- yond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.
Ananya Kumar, Perey S Liang, and Tengyu Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019.
Gongbo Liang, Yu Zhang, Xiaoqin Wang, and Nathan Jacobs. Improved trainable calibration method for neural networks on medical imaging classification. arXiv preprint vrXiv:2009.04000 2020.
Tsung-Y? Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp. 2980 2988, 2017.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohna Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Infor- mation Processing Systems, 34, 2021.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 53:15528-152999 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. Tn Twenty-Ninth AAAT Conference on Artificial Intelligence, 2015.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration deep learning. CVPR Workshops, volume 2, 2019.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Yao Qin, Xuezhi Wang, Alex Beutel, and Fd Chi. Improving calibration through the relationship with adversarial robustness. Advances in Neural Information Processing Systems, 34:14358 14369, 2021.
Sina Shafaci, Stefan Kugele, Mohd Hafecz Osman, and Alois Knoll. Uncertainty in machine learning: A
safety perspective on autonomous driving. In International Conference on Computer Safety, Reliability, and Security, pp. 458-464. Springer, 2018.
Linwei Tao, Minjing Dong, and Chang Xu. Dual focal loss for calibration. In International Conference on Machine Learning, pp. 33833-33849. PMLR, 2023.
Junjiao Tian, Dylan Yung, Yen- Chang Hsu, and Zsolt Kira. A geometric perspective towards neural cali- bration via sensitivity decomposition. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 34:24261 24272, 2021.
Miao Xiong, Ailin Deng, Pang Wei W Koh, Jiaying Wu, Shen Li, Jianqing Xu, and Bryan Hooi. Proximity- informed calibration for deep neural networks. Advances in Newral Information Processing Systems, 36: 68511 (8138,, 2023.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Kc, Hanspoter Pfister, and Bingbing Ni. Medmnist. v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientifie Data, 10(1):41, 2023.
Sergey Zagoruyko and Nikos Komodakis, Wide residual networks. arXiv preprint arXie:1605.0m 00te-1600007146,, 2016.
13


Reviews:
[{"review_id": "x6BeebXakQ", "reviewer": "Reviewer_pRL2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The empirical evaluation is comprehensive, it studies three datasets and four architectures, and compares with about 7 baselines. The effectiveness of the proposed method as a good calibration method seems to be pronounced.\n2. The paper is well-motivated, it focuses on post-processing, which improves calibration without changing the model parameters, similar to prompt engineering in LLM. \n3. The paper is overall well-written, with preliminary definitions carefully stated. \n\n### Weakness\n1. The novelty of the proposed method is a concern. The improvement in calibration through input perturbation has been studied by [Qin et al](https://arxiv.org/abs/2006.16375) and temperature scaling is a traditional calibration approach. The technical novelty of the paper lies in making both components trainable at the post-training stage without changing the model parameters.\n2. The results in Table 4 suggest that input calibration may not be as necessary as output calibration, which raises questions about the authors' claim that \"This ablation study corroborates the necessity and advantage of joint input-output calibration.\"\nThe ablation study of Section 4.3, Table 4 shows that the ECE is reduced by 9.54 with output calibration (temperature scaling), but is reduced only by 0.1 - 0.3 with input calibration (perturbation).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new calibration method that combines input perturbation with temperature scaling. The prior method only considers using temperature scaling to modify output logits, while this work proposes using a learnable universal input perturbation to improve the calibration without changing the model parameters. It has theoretical result about optimal perturbation value and the improved entropy (calibration-related metric). It has empirical results in improving calibration metrics such as ECE in ResNets, Transformers and CIFAR, ImageNet datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to ask for some clarification.\n\n1. If I understand correctly, the $\\widetilde{\\boldsymbol{\\delta}}$ is derived from Theorem 3.2 and its closed form is $\\operatorname{sign}(\\boldsymbol{g}) \\odot \\boldsymbol{\\eta}$, the $\\widetilde{\\boldsymbol{\\delta}}$ is used as \"data-driven initialization\", $\\boldsymbol{\\delta}$ is then trained. Could the authors provide additional details, such as an algorithm diagram or text description, to explain how the \"data-driven initialization\" is implemented?\n2. What is the rationale for using the closed-form optimal value $\\widetilde{\\boldsymbol{\\delta}}$ that maximizes entropy as an initialization? If the theoretical result of Theorem 3.2 is correct, then one might suppose that the derived $\\widetilde{\\boldsymbol{\\delta}}$ can be directly used at test time (since it maximizes entropy). How is that compared to $\\boldsymbol{\\delta}$ after training, or randomly selected value in evaluations of Figure 3?\n3. Some of the theoretical results are confusing. Was Lemma 3.1 and its proof a replicate of Appendix S.2 of [Guo et al](https://arxiv.org/pdf/1706.04599) with $\\mathbf{x}$ changed to $\\mathbf{x} + \\boldsymbol{\\delta}$. I am confused about what are the new insights/differences between the presented one and the one in prior work.", "broader_impact_concerns": "The border impact has been discussed in the impact section, and no concerns regarding that.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mgNljecNW9", "reviewer": "Reviewer_7Ruf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+ interesting idea to modify both input and output, likewise to use the focal loss to tune  \n\\+ method appears to perform well across benchmarks  \n\\- experiments appear to be done on single pretrained models? would require multiple seeds *on models* to be meaningful  \n\\- having the same architectures across datasets would also make analysis better\n\\- also, statistical significance test should be done since the methods are sometimes quite close  \n\\- I have the suspicion that the benefit mainly derives from the focal loss and not from the input perturbation => should be done in an ablation  \n\\- slight test leakage required for the accuracy to improve (effectively, we are learning a slighlty adjusted initial centering operation)\n\\- nitpick: english is weird in some place s(\"Uncalibration\" instead of \"Uncalibrated\" in tables,\"methods predominently shed\" \"confidence made by model prediction\")", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents neural clamping, a generalization of temperature scaling scaling which adds a fixed, trainable, additive perturbation to the input of the model at hand, with both the perturbation and temperate fitted with focal loss instead of basic CE.\n\nThe method is evaluated on various datasets and model architectures and appears to improve ECE, AECE and SCE, sometimes also slightly improving accuracy. A theoretical analysis in terms of entropy maximisation and a data-derived init for the perturbation are also presented", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Critical\n\n1. run an ablation where you fit the temperature only with the focal loss - currently either temperature alone is fitted with CE or temperature+input are fitted with Focal loss,  but then only one is used\n2. pick at least one dataset (I'd propose cifar100) and train each architecture studied on it (multiple seeds for training), then perform the above ablation and your other tests and then perform a statistical significance test (e.g. welch test) or another rigorous assessment across independently trained models (I suggest also using multiple independent calibration datasets on each model)\n\n\nwithout these, I can't call them method rigorously supported\n## Nice to have\n1. above on all datasets => but I understand computational expense might be limiting\n2. if you can keep snapshots across training, you can try the calibration on various stages of undertraining to assess the importance of model capacity and fittedness", "broader_impact_concerns": "No need for broader impact assessment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o13AuEQybT", "reviewer": "Reviewer_GNEb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe introduction of noise perturbation is a good try. Very surprise  to see it works together with temperature scaling on calibration task.\n\n---\n\n**Weaknesses**\n\n1. The motivation is not clear. Why adding a noise together with the temperature scaling will benefit calibration? Better to adding more details or toy examples to illustrate the motivation in the introduction part.\n2. What is the calibration set mentioned in introduction? Is it a validation set? not clear in introduction\n3. Does it have to be focal loss to train the noise and temperature parameter? not clear in introduction\n4. In related works, did not mention the recent works in in-processing calibration methods such as [1]\n5. Lack of analysis and comparison with recent post-processing methods such as [2]\n6. Lack of constrain of perturbation, should be mentioned earlier than eq 7. Is it like the adversarial samples?\n7. It is not clear how you choose gamma in 3.3. Do you split another set for choosing gamma? Or using the same \"calibration set\"\n8. The introduction of noise perturbation is a good try. Very surprise  to see it works together with temperature scaling on calibration task. Can the author give more analysis on the ablation study why noise along does not work well.\n\n\n\n\n\n\n[1] Dual Focal Loss for Calibration\n[2] Proximity-Informed Calibration for Deep Neural Networks", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Neural Clamping, a novel post-processing calibration method for deep neural networks. This method enhances model calibration by jointly applying a learnable universal input perturbation and an output temperature scaling parameter to a pre-trained classifier. Theoretical proofs are provided to demonstrate the superiority of Neural Clamping over traditional temperature scaling. Empirical evaluations on datasets like BloodMNIST, CIFAR-100, and ImageNet reveal that Neural Clamping significantly improves calibration performance across various deep neural network architectures, achieving up to 34% better expected calibration error compared to state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. add more recent related works such as [1] [2]\n2. Compare to recent proposed post-processing works\n3. Answer the question in above section\n\n\n\n[1] Dual Focal Loss for Calibration\n[2] Proximity-Informed Calibration for Deep Neural Networks", "broader_impact_concerns": "The paper introduce a new framework of calibration, which is motivative.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there were some mixed impressions concerning the novelty of the method and the evaluation methods, these concerns were clarified during the discussion phase and changes were made in the manuscript. After the rebuttal, all reviewers recommend the acceptance of the paper at TMLR upon the following minor revisions:

- Reviewer *7Ruf* noted that "*the t-statistic and p value are possibly slightly misleading on the current stage, as they assume that the 25 samples are IID which one could argue is not likely (as they are derived from the same models). I think all that is required to fix this is to tweak the language in the camera ready and/or make an argument for why the authors chose this particular setup*"

- Reviewer *pRL2* noted that: "*based on the rebuttal, it is encouraging to know that the theoretically derived input perturbation can achieve performance similar to that of training results. I suggest highlighting this in the paper. Lastly, I still don't think that Lemma 3.1 and its proof made a significant contribution.*"

Therefore, I encourage the authors to implement these changes and I am happy to recommend the paper for publication at TMLR.
Decision: Accept (poster)