Calibration Attacks: A Comprehensive Study of Adversarial Attacks on Model Confidence
Authors: Stephen Obadinma
Xiaodan Zhu
Hongyu Guo
Venue: Accepted by TMLR ()

Abstract: In this work, we highlight and perform a comprehensive study on calibration attacks, a form of adversarial attacks that aim to trap victim models to be heavily miscalibrated without altering their predicted labels, hence endangering the trustworthiness of the models and follow-up decision making based on their confidence. We propose four typical forms of calibration attacks: underconfidence, overconfidence, maximum miscalibration, and random confidence attacks, conducted in both the black-box and white-box setups. We demonstrate that the attacks are highly effective on both convolutional and attention-based models: with a small number of queries, they seriously skew confidence without changing the predictive performance. Given the potential danger, we further investigate the effectiveness of a wide range of adversarial defence and recalibration methods, including our proposed defences specifically designed for calibration attacks to mitigate the harm. From the ECE and KS scores, we observe that there are still significant limitations in handling calibration attacks. To the best of our knowledge, this is the first dedicated study that provides a comprehensive investigation on calibration-focused attacks. We hope this study helps attract more attention to these types of attacks and hence hamper their potential serious damages. To this end, this work also provides detailed analyses to understand the characteristics of the attacks.

Published in Transactions on Machine Learning Research (11/2024)
Calibration Attacks: A Comprehensive Study of Adversarial Attacks on Model Confidence
Stephen Obadinma
16s00 @queensu.
Deportment of Electrical and Computer Engineering & Ingenaaty Labs Research Institute, Queen's University
Xiaodan Zhu
viaodan si www.wuueeuuuuuu.rr.rr
Deportment Electrical and Computer Engineering & Ingenaaity Labs Research Institute, Queen's University
Hongyu Guo
hongyu.gue @uottowo.cr
Digital Technologies Research Centre, National Research Council Canada
Reviewed on OpenReview: https:// openrevvee.. net/forum TXzz9xudpo
Abstract
In this work, we highlight and perform a comprehensive study on calibration attacks, a form of adversarial attacks that aim to trap victim models to be heavily miscalibrated without altering their predicted labels, hence endangering the trustworthiness follow-up decision making based on their confidence. We propose four typical forms of calibration attacks: underconfiience, overconfidence, marimum miscalibration, and random confidence attacks, conducted in both black-box and whito-box sctups. We demonstrate that the allacks are highly effective on both convolutional and allention-based models: with small number of queries, they seriously skew confidence. without changing the predictive performance. Given the potential danger, we further investigate the effectiveness of a wide range of adversarial defence and recalibration methods, including our proposed defences specifically designed for calibration attacks to mitigate the harm. From the ECE, and KS scores, we observe that there are still significant limitations in handling calibration attacks. To the best. of our knowledge, this the first dedicated study that provides comprehensive investigation
the models and
celbbraatio-foocusee attacks. We hope this study helps attract more attention to these types of attacks and hence hamper their potential scrious damagos. To this end, this work also provides detailed analyses to understand the characteristics of the attacks.
1 Introduction
While recent machine learning models have significantly improved the state-of-the-arl performance on a wide range of tasks (Bengio et al. 2021 Vaswani et al. 2017 LeCun et al. 2015 Krizhevsky et al. 2012)
these models are often vulnerable and easily deceived by perturbed inputs (Ren et al. 2020). Adversarial
attacks (Ren et al. 2020) have been shown to be crucial tool to reveal the suscoptibility of victim models
Tbiiooy et al. 2019 Zimmermann al. 2022 Xiao al. 2023). In the classic setup, adversarial examples are generated by introducing imperceptible modifications to original datapoint cause misclassification, where the focus is on trapping victim models to make incorrect predictions.
In this paper, we highlight and provide a comprehensive study on different type of threat, which we call calibration attacks. These attacks focus on manipulating the victim models' confidence scores without modifying their predicted labels, hence endangering any follow-up decision-making that based on the victim
Quur code available http://ghhaaaccmm//eeoddoCCoaaoa

---

Published in Transactions on Machine Learning Research (11/2024)
Underconfidence
Duerconfidence
Maximum Miscalibration
Random Confidence
07
0.7
0.7
07
0.4
0.
0.1
0.3
2 10111222445
101112131115
1 6 1011122234455
5433
323333355
Figure 1: Reliability diagrams of a ResNet-50 classifier (fine-tuned and tested on Caltech-101) before and after the four forms of calibration attacks. Red bars show the average accuracy on the test data binned by confidence scores (15 bins) and the blue bars are the average confidence of samples in each bin. The x-axis represents the bins and y-axis is the accuracy (for red bars) or confidence (for blue bars). The yellow line represents perfect calibration. To have the minimum possible ECE the red bars and blue bars have to completely overlap in each bin (shown in maroon), where no overlap represents miscalibration. Despite the accuracy being unchanged, the miscalibration is severe after the attacks.
models' confidence, We propose to conduct four forms of calibration attacks: underconfiiencee, overconfidence, mazimum miscalibration, and random confidence attacks, which can seriously skew confidence and cause heavy miscalibration, as demonstrated the reliability diagrams Figure [ As we will show and discuss in our study, calibration attacks are insidious and hard to detect. The intrinsic harm is that on the surface the models appear to still make correct decisions, but the level of miscalibration could make the models' decisions malicious for downstream tasks.
Our specific studics consist of four forms of attacks, spanning black-box and white-box setups. We attack typical convolutional and altention-based models, and investigate both attack and defence methods.
In summary, our main contributions are as follows.
To the best of knowledge, this is the first dedicated study that provides comprehensive invostigation on confidence-focuser calibration attacks.
We propose four typical forms of calibration attacks and demonstrate their effectiveness and danger to victim models from different perspectives. Detailed insights are provided to understand the characteristics of the attacks and the vulnerability of victim models.
We further investigate the effectiveness of wide range of adversarial defence and recalibration methods, including the harm. We hope our work helps attract more attention to these attacks and hence hamper their potential scrious damages in applications.
proposed defences specifically designed for calibration attacks to mitigate
2 Related Work
Calibration of Machine Learning Models. Among the two typical types of calibration methods, post- calibration methods are applied directly to the predictions of fully trained models at test time, which include classical approaches such temperature scaling (Guo et al.) 2017), Platt scaling (Platt] 1999). isotonic regression (Zadrozny & Elkan 2002). and histogram binning Zadrrzzuy & Elkan 2001). Training- based ap-
proaches, however, often add bias to help calibrate model during training (Zhang et al. 2018 Thulasidasan
ct al. 2019 Kumar et al. 2018 Tomani & Buettner 2021). ). We investigate a diverse range of calibration
methods against calibration attacks to reveal the limitations that require them to be overhauled to deal with attacks, examining the vulnerability of models similar to those on the convolutional architectures (Guo
et al. 2017 Minderer et al. 2021 and Transformer- based frameworks Dosovitskiy et al. 2021 (Refer to Appendix for more detailed summary of related work.)
Adversarial Attacks and Training. Adversarial attacks are divided into black-box and white-box ap- proaches. The former (Carlini & Wagner 2017) assume less information about victim models. Many
2

---

Published in Transactions on Machine Learning Research (11/2024)
methods are based on gradient estimation through querying the models and finding the finite differences (Bhagoji al. 2018). White-box attacks often have access to the full details of victim model such as its architecture and gradients [Goodfellow et al. 2015). Adversarial training and defence approaches have been introduced to improve the robustness of victim models (Chen et al. 2022 Qin et al., 2021 Patel ct al. 2021 Dhillon et al. 2018). Related to these are works on generating certilied robustiness guarantees Kumar ct al. 22020), where certified radii are generated for the predicted confidence of a smoothed classifier. The research in Emde et al. (2023) further extends this by certifying calibration through generating the worst- case bounds of calibration error, and discuss the importance studying attacks targeting calibration. [Stutz et al. (220)) develop modified adversarial training objective which focuses on biasing ad adversarial samples to be low-confidence predictions that can be easily filtered. Although this causes adversarial examples to be low confidence, it does not specifically larget making low or high confidence adversarial examples through perturbations as a form of attack. In this work, we propose two defence models against calibration attacks.
Attacking Uncertainty Estimates. Galil & El- El-ianiv (2021) first identified attacks on credible uncer- tainty estimates as an issue. Their models harm the potential of using uncertainty estimation on model's predictions by pushing correct datapoints closer to the decision boundary and incorrect ones further away using confidence without causing incorrect predictions. [Zeng et al. (2023) introduce a data poisoning atlack designed to alter the training process of models such that a high confidence region forms around an out-of- domain datapoint, harming the adversary-resistant uncertainty estimates on set of targeted out-of-domain datapoints while leaving the original labels. Nevertheless, none of the prior works provide a comprchensive study on the calibration attacks, nor do they systematically investigate how well victim models would remain calibrated under such attacks, including under range of different defence methods. Our work proposes and examines the effects of different attacks, the difficulty of detecting them, and the effectiveness of defence and calibration methods. We provide the study on both convolutional and more recent allention-based models, under both black and white-box approaches. Detailed analyses and insights are additionally discussed.
3 Calibration Attacks
Given the input X = (xi,..., X of N datapoints and their ground-truth labels y (un..... UNN ), where Yi E (L...., K} with K being the number of classes, classifier F makes prediction for an instance x, E X through a mapping F :x + (0.. where D is the predicted label which is often obtained by taking argmaz on the output distribution p. over the K classes: O, = argmaxf (0:y). When needed, P. is written as p(x:) and p(x;), for the input x; and its perturbation 8, respectively. Similarly, Åi can be rewritten as j(x.) or (Xx), which will be constrained to be same in calibration attacks.
3.1 Objective of Calibration Attacks
Calibration attacks aim to generate adversarial examples to optimize a predefined misenlibration function M k) for an adversarial example &; and the predicted class k. As will be detailed below, we propose four forms of calibration attacks where M(2,,k) k) takes different implementations. Following conventional
notation, an adversarial example x; is created by adding noise 8 to an input x,: x; ==xx x; + 8, bounded by in a 4m ball:
- - <e, m E oo),
(1)
where e controls the amount of allowed perturbation and m corresponds different norms that may be used. In general, our attacks are based on the most popular view of class-wise calibration (Guo et al. 2017] Kull et al. 2019). For datapoint (x,, y.) in the dataset D {(xx, vn) Wy well calibrated model aims to achieve:
F(y k (xx,)) qk) gh
(2)
where qk is the confidence of the predicted class i for x,. Any mismatch between the left and right hand sides of the equation creates undesirable miscalibration.
3

---

Published in Transactions on Machine Learning Research (11/2024)
3.2 Four Forms of Calibration Attacks
Building on the above notations and framework, we propose four approaches cover the most typical variants of calibration atlacks.
Underconfidencee and Overconfidence Attacks (UCA and OCA). These two types of attacks aim to solve the constrained optimization problem involving the miscalibration function M(x, k), making a victim model either underconfident or overconfident.
= MUCa(X; == (x(()) max Ps()),, DEk
(3)
Moca k)) (Xx).
(4)
s.t. (xx) = i(x)).
(5)
Calibration attacks focus on attacking the confidence victim models without altering the predicted labels, which is satisfied by the constraint in Eq. 3
By minimizing the loss Muca or MOCA, the attack models maximize calibration errors in these two setups respectively, with the corresponding adversarial examples X
max(P(y: = Pa (x,) 9b) - -.
(6)
Algorithm ] depicts an overview of calibration attacks, including UCA and DCA, but also MMA and RCA that we will introduce below. Unlike conventional attacks that modify the predicted labels and focus mainly on correctly classified examples, calibration attacks modify confidence and focus both the originally correctly classified and misclassified instances. As detailed later in Section E we will implement calibration attacks in popular black-box (Andriushchenko et al., 2020) and white-box frameworks (Madry et al. 2018).
Maximum Miscalibration Attacks (MMA). We propose MMA with the aim of exploring and understanding more serious scenarios of miscalibration. The main principle of MMA is to perturb a set inputs such that (i) all incorrectly classified datapoints (a sot where the model has zero accuracy) have 100% confidence, and (#) all correctly classified datapoints (a set where the model is 100% accurate) have the minimum possible confidence. MMA uses combination of OCA and UCA to accomplish this, shown in Algorithm [ Proposition 3.1 below states the property of MMA in terms of the oracle (upper-bound) ECE score that can be achieved in theory, with the proof provided in Appendix C
Proposition 3.1. Assume q the accuracy of a K-way classifier F on the datoset D {(xn, yn)) The Mazimum Miscalibration Atlack (MMA) marimizes the expected calibration error (FCF). The upper bound of ECE that con be achieved by MMA is K.
Random Confidence Attacks (BCA). We propose to perform RCA to decouple the victim model's 's confidence scores and predictive labels, in which the confidence scores produced by the attack model are randomized. RCA is performed by choosing a random target confidence score for each input, and then, depending on the original confidence score, running the corresponding underconfidence the target confidence score. Although RCA is theoretically less effective than MMA, it is less predictable, because unlike the other three types of attacks, RCA does not produce a predetermined direction (Lee. under- or overconfiencce) of attacks for a given input. The produced confidence scores are often less extreme and more reasonable-looking but largely meaningless due to randomization. Note that in addition to MMA and RCA, one may design other approaches to combine UCA with OCA, but MMA and RCA represent the two most typical compositions that create very harmful patterns of miscalibration.
overconfidence attacks to produce
3.3 Defence Against Calibration Attacks
As we will show the experiments, calibration attacks are very effective. In addition to studying wide range of existing defence methods, we introduce new methods specifically for calibration attacks: Calibration Attack A Adversarial Training (CAAT) and Compression Sealing (CS). The proposed CAAT is variation of PGD-based
4

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm A Brief Overview of Our Calibration Attack Framework
1: Input: A classifier F, imput X;, true label Vi. error bound max mumber of attack itcrations I, attack type T.
2: Output: Adversarial example X;
3: Di 4 F(x:); < argmax (Pt)) T UCA
11 k is the predicted label for the original (unattacced)) input x.-
OCA then
T T
clse if T MMA and ys + k then
// MMA combines UCA and OCA.
T' -CCA
8: else if T = MMA and = k then
9: T = UCA
10: else T = RCA then
11 RCA also considers both UCA and OCA. 1/ Get a random number the range [1/ /K, 1.0].
11: 9 4- randomm(1 K 1.0);
12: if g > (x:) then
13: T' OCA
14: else if y < (x, then
15: T' UCA
16: end
17: end
18: &; < x;i lold + (x; k)
19: for 1 to I do
20: = FendPerturb (Ri,t) c) // Find perturbation based on allack algoritlem (e.g., Square Allack) and
bound.
21: Xnew t- 82; +d;
22: Pi - F(((nen));; knom t- argmax
23:
hnew + -MT k)
24: if (Inew lold and knew = k) then
25:
X; -xnewile lold t Inew
26: end
27: if (lold < 0.01) (T RCA and (x( g) then
28: Break the for loop
29: end
30: end for
adversarial training that utilizes our white-box calibration attacks to generate adversarial training examples for each minibatch during training. Hence, both under- and overconfident examples with the model's original predicted label preserved are exclusively used train the model.
Our proposed CS defence is post-process scaling technique, based the assumption that effective classifier often has high level of accuracy and confidence, calibration attacks typically cause the most harm by lowering the confidence scores. CS hence aims to scale low confidence scores to high values to mitigate the damage of miscalibration. Specifically, the range of confidence is split into Bas equally sized bins. Datapoints in each bin bm E (1,..., B are mapped to new bin that has higher confidences more compressed range. For a datapoint x; with the output-layer logits (ri,..,, and the probability Pu (xx) for the predicated class k, a temperature T' is found to obtain a newly predicted probability Pnew explrr/T) so that Pnem = minconf(b(,) + {=-miucon((bm) range(K(,). where minconf(.) is the range(bb..)
arg max, Σ, eap(r;)/7))
minimum confidence level of a bin, and rango(-) represents the range of the confidence (i.c., the maximum level of confidence of that bin minus the minimum). is the bin that the original b is mapped using the corresponding T. We will demonstrate that even with defence, calibration attacks still highly effective.
5

---

Published in Transactions on Machine Learning Research (11/2024)
3.4 Discussion on the Importance of Remaining Well Calibrated Under the Attacks
In addition to the discussion of the technical details of calibration attacks for the victim models and de- fences, it is important to reemphasize why maintaining good calibration under such attacks is critical. Well-calibrated models are a crucial component for trustworthy complex systems. In terms of specific real- world applications, skewed confidence scores or uncertainties have been shown to have dramatic impacts on downstream tasks and human arrision-maainn. and previous studies such as those of Penso et al. (2024): Jiang et al. 2012 Kompa al. [021] have shown the effect of miscalibration
down-strcam author- ily mechanisms in the context of areas like medical imaging. Specific use-cases can also be found in child
maltreatment hotline screening models De Arteaga et al. (2020), legal domain Bernsohn et al. (2024), and
anomalous behaviour detectors Shashanka et al. 2016) where heavily miscalibrated outputs present a failure point to methods.
Moreover, prior works (Zhang al. 2020 Rechkemmer & Yin 2022) have examined the role of skewed confidence affecting the level of trust humans have in a system, which influences applications where humans and AI decision-making occurs symbiotically. These works demonstrate that when models produce high confidence scores, users are more prone to rely on the AT for decision-making Dhuliawala et al. (2023) conduct user studies that show that confidently incorrect predictions lead to over reliance and heavily undermine user trust in Auman-AI collaboration settings, requiring only a few incorrect instances with inaccurate confidence estimates to damage user trust and performance. Please refer to Appendix B for detailed discussion on downstream impacts.
4 Experiments
4.1 Experimental Setup
Implementation Details. As discussed earlier, calibration attacks can be built on different adversarial attack frameworks. In our implementation, we use Square Attack (SA) (Andriushchenko et al., 2020), which is one of the most popular black-box approaches and is highly effective. SA achieves frate-of-the-art (SOTA) performance in terms of query efficiency and success rate, even outperforming some white-box methods, Our white-box calibration attacks are based on the popular Projected Gradient Descent (PGD) framework (Madry
et al. 2018 The implementation details such hyperparameters are discussed in Appendix D
Models. We include both convolutional (ResNet (He et al. 2016 and non-convolutional attention-based modols (Vision Transformer (VIT) (Dosovitskiy et al. 2021 in our study. Dotails can be found in Appen-
dices and
Datasets.
We performed a comprehensive study on CIFAR-100 (Krizhevsky Hinton 2009) and Caltech- 101 Fei-FFii et al.) 2004). We also included the German Traffic Sign Recognition Benchmark (GT-
SRB) Houben et al. 2013 for its direct implication for safety.
Two calibration metrics are used in our experiments: the standard Expected Calibration Error (ECE) [Pakdaman Naeini et al.) 2015)) and the recent Kolmogorox-Smirnov Calibration Error (KS error) Gupta et al.) 2021).. In addition, we evaluate attacks' efficiency using the average and median number of queries for the attack to complete (Andriushchenko al., 2020). Average confidence of predictions also
Metrics.
leveraged judge the degree that the confidence scores are affected. (See Appendix D for details.)
Same in previous works (Guo et al. 2017 Kumar et al. 2018 Minderer al. 2021 focus on intrinsic
evaluation metrics in this rescarch, which isolate our analysis from downstream tasks to abstain from task- specific conclusions, particularly considering the downstream tasks here are very diverse. In addition, the harm from miscalibration in downstreem applications has been well studied in prior work, as detailed in Appendix
Detailed Attack Settings. Descriptions of other attack settings (c.g., Loc. and iterations) are in Ap- pendix Dand E
6

---

Published in Transactions on Machine Learning Research (11/2024)
4.2 Overall Performance
Table depicts the overall performance of black-box attacks under the lo norm. Ad- ditional results of l2 attacks
Table 1: Results of underconfidence, overconfidence, maximum miscalibration, and random confidence attack. Accuracy of vic- tim models are included.
included in Table 9 in Appendix H.10 The experiments show that the attacks are highly effective. UCA, MMA and RCA can bring ECE and KS to very high values. OCA can successfully raise aver- age confidences (the last column in Table [ to extremely high levels (in many cases, close to 100%), but given the original high accu- racy of the vietim models, increasing confidence levels will not have a drastic effect on cali- bration error. (OCA will be more harmful on less accurate models.) For the MMA, the theo- retical highest levels of miscalibration are not. reached due to the limited number of itera- tions that we run the attacks. Regarding dif- ferent architectures, the attention-based ViT models are seen to be more miscalibrated com- pared to the convolution-basel ResNet mod- els. The white- thxte-box results are included in Ap- pendix H.T While ResNet and VIT are the most representative frameworks for convolution and attention-based models respectively, in Ap- pendix H.S we include additional experiments on the Swin Transformer (Liu et al., 2021 which is famous extension of VIT. The results show similar trends as those presented in Ta- ble [ Again, Figure [ in the introduction sec- tion demonstrates the attack effects using cali- bration diagrams, visually showing the severity of miscalibration.
Res
Avg #q
Med. #q
ECE
KS
Avg. Conf.
CIFAR-100 (Acruracy: 0.881
20.002)
Pre-atk
.0522.000
.035 006
.965..00
UCA
71.3+a.4
42.7=1.
.540..o0
.179 001
A6552.05
OCA
16.0.0.
1.0g0.0
.1244.002
.124= 002
.996
MMA
72.9
41.5=2.
606g..om
.97
5022.00
RCA
68.9
42.7
.588
461g
514g 003
Caltech- 101
(Acruracy:
0.906g0.0m4)
Pre-atk
0.05..com
.031 004
.966..001
UCA
333.8
259.T2174
.661..om
.322 005
605 006
OCA
75.7
1.0g00
.028
.028 004
.992 000
MMA
182.6gn.
286.5
3977
.379
.1 618g
RCA
178.5
289.3g8
J444
342g
638
GTSRE
[Aocuracy:
0.972gn.00)
Pre-atk
.0192.oom
.008 002
.980g.002
UCA
197.5
103.0gr3
.366..07
.30 013
.51 011
OCA
12.1a1a
1.0g00
1099..00
.029 0os
.998
MMA
142.1a6.
102.2.3.6
.419
.022
.597
RCA
139.441.5
104.7.3.
.399 oon
336
.509 2.007
VIT
Avg #q
Med. #9
ECE
KS
Avg. Conf.
CIFAR-100 (Accuracy: 0.935g0.002)
Pre-atk
.064
.054g oos
.822 004
UCA
118.5g24
62.0ga3
5572..07
.558_
.404_.00
OCA
524.7
510.5-1145
.043
.013
.974
MMA
104.8
62.7
.116
.564g
.431
RCA
106.4ga.
70.3
24z
.505 003
.471 c. 000
Caltech-101
(Acewray: 0.961 20.024)
Pre-atk
.137..00
.136 oso
82552.05
UCA
325.5
273.7+28.7
4466.00a
.26
5366
OCA
62.144009
1.0.00
.081
.079
.881 881
MMA
150.7
269.
789.7
.415..00
414g 0s4
.51 551
RCA
129.0+17a
315.0+174
.3644.016
.364g 016
.508_.00 .400
GTSRR
(Aocurary:
(0.07))0000
Pre-atk
.040g.005
.026 017
.922 022
UCA
169.8
88.326.7
4559
.452 019
.488 u26
OCA
94.9
3.7=46
.029
030
.976..011
The general trend of the la calibration attacks (Appendix H. is similar to that of the Loo attacks, but the latter are found to be more effective in generating more significant miscalibration. Honce,
MMA 137.1g4.
88.326.7
.519
480g
.509
RCA
129.5
97.2299
.454
.432 016
.538 g 019
in the remainder of the paper, we focus on the lox attacks unless otherwise specified.
Overall, calibration attacks are generating severe miscalibration, which, compared to the pre-attack values, can increase ECE and KS by over 10 times in many cascs. Calibration attacks are very effective without changing the prediction accuracy, which could raise serious concerns for any down-stream applications relying on confidence.
4.3 Detection Difficulty Analysis
This section shows calibration attacks are also difficult to detect. To investigate this, run the popu- lar adversarial attack detection methods on the attacks against ResNet-50: Local Intrinsic Dinensionality
(LID) (Ma et al. 2018), Mahalanobis Distance (MD) (Lee el al. 2018), and SpectralDefense (Harder el al., 2021). The details behind the settings for each detection method can be found in Appendix G Table 2 depicts the main results of the effectiveness of the detectors in terms of Area Under the Curve (AUC) and Detection Accuracy under different types of calibration attacks and using both the whito- box and black-box approaches. We can see that there are consistent decreases in detection performances, particularly in SA,
7

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Attack detection results comparing original version of SA and PGD attacks with their calibration attack
counterparts.
SA
-A-DGA
SA-OCA
SA-RNA
PGD
PGTT-TCA
PGTT-DCA AUC
PCT
AUC
Ace
AUC
Aoe
AUC
Acc
AUC
Ase
AUC
Acc
AUC
Aoc
Ave
AUC
Aue
CIFAR-100
LID
90.1
82.9
54.2
54.3
65.9
61.6
54.5
54.0
93.77
81.7
64.5
67.1
88.7
84.3
63.3
66.0
90.8
98.9
90.2
80.5
78.1
74.8
89.4
70.9
08.3
98.5
83.1
96.7
93.8
81.9
73.3
Speet.
100
98.0
70.5
65.5
52.3
50.5
71.6
65.5
100
100
74.9
67.0
94.2
90.0
64.8
62.5
Calrech-101
LID
70.2
64.4
62.5
61.1
65.3
62.3
58.6
58.5
81.8
78.1
53.6
54.8
89.1
81.9
53.9
66.1
MDD
88.9
81.3
81.9
74.7
73.9
70.1
81.8
74.5
91.6
81.6
60.1
57.9
90.6
81.7
62.6
60.3
Speet.
98.0
94.5
69.1
53.0
51.8
50.0
56.9
53.0
95.4
88.5
67.8
66.5
95.77
90.5
66.2
02.0
CTSRB
LID
68.1
72.6
69.8
72.4
06.7
87.4
86.3
MD
95.5
89.6
83.7
77.8
74.4
74.9
85.6
73.0
100
90.8
94.6
93.6
97.9
97.7
92.99
92.6
Speet.
50.4
65-
OCA
MNA
Caltech101
CTSRB
VIT
Epslion
Epsion
Epsiion
Azback Itoradio
Figure 2: The inffuence perturbation noise levelse (the left three sub-figures) and attack iterations (the right sub-figure). Sub-figure1 (the left most) presents the comparison between the ECE scores of the different calibration attacks at differrnt values using ResNet-50 models trained on CIFAR-100. Sub-figure-2: ECE vs. e using maximum miscalibration attacks ViT models trained CTFAR-100. Sub-figure-: E.F. vs. using maximum miscalibration attacks on the ResNet-50 models trained on Caltech-101 and GTSRB. Sub-figure44: Effect of the numbers of attack iterations the ability of the attack algorithm. The first three sub-figures created at the 1000 attack iteration.
where the decreases are often more than 20%. The existing detection methods are shown to be less reliable for calibration attacks.
4.4 Insights on Key Aspects of Attacks
We analyze key aspects of calibration attacks: attack directions, noise bounds, and attack iterations under the -ased attacks. More details can be found in Appendix including attack effectiveness on data with varying imbalance ratios.
Comparison of Efficiency of Underconfidence vs. Overconfidence Attacks. The two directions of calibration attacks, underconfidence or overconfidence, basic building block for constructing the four forms of calibration attacks. We perform further studies understand which direction is most efficient. To the end, we identify all data points in the test set that are around certain predefined base confidence levels. In this study, we choose two base confidences: 80% and 90%. All the test cases that have confidences within 1% around these two base confidences are included in this experiment. We attack these examples by making the victim models produce either 10% increase or decrease in confidence. In Table 3 we can see a consistent pattern - for both base confidence levels, it takes notably fewer queries to create undercon- fidence than overconfidence adversarial examples. The former attack is also more effective in affecting the average confidences. This property could be further utilized to design calibration attacks (e.g., under the circumstances where computing resources or attack latency concerned.)
Perturbation Noise Levels. We study the effect of perturbation noise levels and how low the value could be in order to construct notable harm. As shown in the left three sub-figures in Figure 2 under different setups, the attacks are highly successful even at a low level of noise (e.g., e = 0.05 or even lower). We can also see the rise in ECE sharp when increases, and it plateaus quickly.
Iterations. The numbers of iterations can help measure the efficiency and cost of attacks (in terms of both time and computing expenditure). The last sub-figure in Figure shows ECE vs. iteration numbers when
8

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 3: The GradCAM visualizations shows the image regions most responsible for the decisions of ResNet-50 before (top row) and after (bottom row) attacks. The left three images are under UCA and the right three the OCA.
applying MMA the victim models on CIFAR-100. In Appendix H.3 we provide more detailed comparison, showing that the calibration attacks consistently produced higher degree of miscalibration compared to the original unaltered SA. ECE begins to saturale at 500 iterations, but even at 100 ileralions the victim models become heavily miscalibrated.
4.5 Qualitative Analysis
GradCAM Visualization. In Figure we show the coarse localization maps produced with GradCAM (Selvaraju et al. 2017). which highlights the most important regions that model relies on make prediction using the gradients from different layers of a network. We apply GradCAM to ResNet-50 models that are fine-tuned and tested on Caltech-101, based
Table 3: Comparison between the efficiency of the UCA and OCA.
ResNet
# datapoints
Avg. #q
Med. #q Avg. Conf.
CIFAR-100
90% -10% UCA
186.0=44
8.5
5.7
77.9 99.01am
90% +10% DCA
186.0244
36-511.
31.0217
80% -10% UCA
94.0+4.2
0.5go.
4.3
67.910.4
80% +10% DCA
94.0
9.8.1.7
7.0
91.3
Caltech-10r
90% 100% UCA
160.7=7.
39.4g11.8
31.7=a.5
80 80.2
on the standard attack settings discussed in [Dction[] We choose images where the attacks led large change in predicted confidence (at.
90% 10% OCA
160.7=7
225.6455.5
212 8403.3
98. 98.8
tiCA
53.3425.0
17.5
16.0-19.1
63.142.
80% 10% OCA
53.3+24.0
30.0+10.5
21.7=11.5
89.5
GTSRB
least 10%). Figure shows several represen- lalive images. We can that the coarse lo- calization maps have minimal to noticeable changes after the adversarial images are pro- duced, especially in the case of the overcon- fidence attacked images. This analysis shows that it could be difficult to identify the attacks based on gradient visualization methods.
90% -10% UCA
48.0
11.7.0.1
8.241.4
77.5
+100 DCA
48 48.0
98.5140
62.8moi7
19.1
80% 100% UCA
30.0
9.2=a.9
6.0
70.5
80% 10% DCA
30.0
18.6
12.8
91.049.1
VIT
# datapoints
Avg. #q
Med. #q Avg. Conf."
CIFAR-100
-10% UCA
392.7233.3
20.0g16
9.3
76.1
90% +10% DCA
392.7
883.74120.
883.7+1202
97.1
80% 10% UCA
138.715.
8.821.7
5.5
66. 10.3
80% +10% OCA
138.745.1
55.8414
20.7
90.0ga.1
Caltech-101
90% -10% UCA
472.0
261.7456.a
198.0451.0
81.140.
t-SNE. Appendix H.4 provides detailed t-SNE visualization on the effect of different forms of calibration attacks. The visualization shows that the overconfidence attack causes the representations for different classes to be split. apart as much as possible, while the undercon- fidence attack causes a more jumbled represen- tation with most data points falling closely to the decision boundary. The visualization shows that the attacks achieve their intended goals.
90% 10% DCA
472.0
700.
700.72518.
700.7
93. 23.219.
-10% UCA
222.0=17.s
149.5
66.3=9.a
70.0+0.8
80% +0% DCA
222.0=17.3
323.94125.5
269.3
86.0.0.2
GTSRB
90% 100% UCA
184.0
31.02o.5
12.721.9
76.1
90% +100 DCA
1810227
235.3
180.0ga2.0
96.5.0.
80% 100% UCA
79.0gus.
16.1
7.7
67. 67.3
80% 10% OCA
79.0+28.6
140.1445.2
41.7+43.7
89.6+0.8
9

---

Published in Transactions on Machine Learning Research (11/2024)
4.6 Confidence Certification
We follow the method in Kumar et al. (2020) to compare the lower bound of expected confidence of a smoothed classifier between base imagos and their under- and overconfidence attacked counterparts. We find that the lower bound is not significantly different between adversarially attacked images and their originals, though the accuracy of the smoothed classifier is affected. This means that methods for determining certified lower bounds are not strongly affected by calibration attacked samples (similar to conventional adversarial samples), but limitations in efficiency and accuracy of these methods means that establishing defence method against such attacks still paramount. detailed discussion is in A Appendix H.7
4.7 Attack Transferability
Determining the transferability of calibra- tion attacks across different model architec- tures is of interest since althongh one can effectively optimize attacks against a single model, using an ensemble model could po- tentially be form of defence if transfer- ability is limited. We investigate how our allacks transfer to different. victim models. Specifically in our experiments, the attacks are constructed on CIFAR-100 images with regular settings based on ViT but used on (i) ResNet and (ii) ensemble victim models combining prediction of both ResNet-50 and VIT (Table 4 We focus on the transfer of the UCA attack, given that OCA has little room to skew confidence due to the already high accuracy of victim models. The attacks show transferability, although in general they become less effective when applied on different architectures. This is expected because the attacks are constructed without considering the target victim models' properties. Most notably, the simple ensemble of two typical victim models (ViT and ResNet) is still greatly subject to the attacks (the output distributions ResNet and ViT are averaged to get the ensemble), so using ensembles as a defence does not fully protect against calibration attacks. To further explore transferability, attack methods need to be explicitly redesigned (Xie et al. 2018 Wang et al. 2021), which could be further studied in the future.
Table 4: Attack transfer VIT to ResNet and to ensembled victim models (with ViT target) on CIFAR-100.
ECE
KS
Avg.-Conf.
Pre-atk
.051..00
.035g.006
.914
UCA SQ VIT - ResNet UCA PGD VIT-> ResNet
.104g.003 .003
..01
.780+.012
.1964.020
.194g.021
.7294.011
Pre-atk
.0284.005
.015 000s
.920
UCA SQ Ensemble UCA PCD Ensemble
.011
.086
.008
.7082.012
.018
.014 ..08
.637
4.8 Influence of Model Scale on Attack Effectiveness
To help understand the influence of varying model sizes on attack performance, we per- form study models of different sizes over different architectures across four attack vari- ants. Specifically, for ResNet, we compare ResNet-18, ResNet-50, and ResNet-112. For ViT, we included Base bott-base-patchh6---1 224- in21k) and Large models (wil-large-patch 16 224-in21k). The model training closcly follows the settings detailed in Appendix D.1. We fine- tune the models for each variant and average the results. We ran attacks with the same set- tings used to produce Tables I and 3.
Table 5: Attack effectiveness of MMA (black-box setup) across different model sizes on CIFAR-100.
Avg #q Med. #q
ECE
KS
Avg. Avg.Conf.
ResNet-18 (Aec: 1844
Pre
.077
.016 no5
.870
Post 52.9414
36.0ga0
.531
.471 001
.464.003
RexNet-50 (Acc: 082) Pre
.352..08
.035 o0e
.961.008
Post 72.9428
41.5.2a
6064
.497 0022
.502
ResNet- 152 (Arc: 1883.002)
Pre
051
.046
.929
Post 80.5438
47.043.
2004
475+.002
.472.008
ViT-lase (Aec: .035 Pre
0644..00
.054g aos
.882
Post 104.8475
62.7.47
.616
.564 on0
.431
In Table 5 we present the results of the strongest variant of calibration attack (i.c., MMA). It shows that the proposed calibration at- tacks are effective among different model sizes.
ViT-large (Ace: .93610001 Pre
10371.009
.022 012
.911 1.011
Post 122.1+20
76.3+3.2
.31+.012
.508 n20
.450 024
10

---

Published in Transactions on Machine Learning Research (11/2024)
The model sizes do not have major impact on the susceptibility to calibration attacks, as the overall ECE and KS scores do not follow clear pattern along with varying model sizes. However, when models scale up, the numbers of queries increase in general, suggesting that attacking complex models requires more rounds of attacks most cases. In Appendix H.11 we present detailed results the black-box and white-box sctups across three architectures under the four types attacks, which agree with our aforementioned observations.
5 Defending Against Calibration Attacks
We compare a wide range of recalibration and de- fence methods under the sctup of the maximum miscalibration attacks, which, as shown above, are among the most effective calibration attacks.
Table 6: Effectiveness of calibration methods and adver- sarial defences. The best performances of post-attack ECE (PsECE) and KS (PsKS) marked bold, and the second best are marked with underlines.
Specifically, for post-calibration methods, we in- clude Temperature Scaling (TS) (Guo et al. 2017),
WideResNet
Avgllg Med
Ace PrECE PSECE PrKS
PsKS
C-PAR-100
Multi-domain Temperature Scaling (MD-TS) (Yu
61.
86.5
450
117
.137
2010
Rebufi,
et al. 2022), and calibration with splines (Splines)
Pang, 22
30.5
64.5
6688
186
214
.187
.105
Guptt al. 2021). For training- based regular- include two effective models, DCA Liang et al. 2020) and SAM (Forct et al. 2021 Regarding adversarial defence methods, we test the top-3 SOTA models under the loo at- tack for CIFAR-100, using Wet on the Ro- bustBench leaderboard (Croce al. 2021), which are only available for CiFAR-10 and CIFAR-100, hence rum over CIFAR-100 to compare with our previous baselines. We further include a re- cent. post-process defence called Adversarial At- tack Against Attackers (AAA) (Chen et al. 2022 in addition to the most common defences
ResNet-50
CIFAR-100
ization methods
TS
74.1
40.0
.880
.034
.007
MD-TS
Splines
6.6
83.0
8766
0220
.019
.573
DCA
68.0
31.0
.866
(149
4604
.0R1
.402
SAM
83.8
11.5
882
.033
.009
.014
.006
AAA
7.7
31.5
880
(338
22G
.011
.123
AT
65.9
60.0
.750
.035
.022
.279
CAAT
.842
.051
Caltech-101
TS
19L7
276.0
.970
.014
311
.005
.322
MD-TS
178.6
280.0
.970
.025
319
.017
.321
Spiines
4.5
150.0
.970
.019
.010
DCA
1806
SAM
AAA
1.1
20.0
.664
.061
100
.08
AT
25.7
194.0
.918
.038
07B
.018
068
CAAT
127.5
206.0
.972
.117
261
.012
266
CS
179.4
254.0
.970
.026
.065
.017
.067
GTSRB
TS
160.6
the form of rGsebased adversarial training (AT)
MD-TS
146.5
110.5
.023
Spiines
0.7
22.0
.972
.018
.126
.007
.123
(Madry et al., 2018)) We also included the two defences proposed for calibrations attacks: CAAT
DCA
131.2
97.0
976
.117
389
.011
.372
SAM
117.4
87.0
.778
.012
384
.003
.311
AAA
3.1
51.0
.92
.023
.071
.014
.059
and CS, which are introduced in Section 3.3 (The
experiment setup is described in Appendix and the detailed description these basclines are in Appendix E.2
CS
151.2
111.0
.72
.019
097
.020
.055
VIT
Avgay Med#e
Ace P:ECE
PsECE
PrKS
P&KS
CIFAR-I00
TS
117.7
73.0
.838
.014
.010
Results and Analyses. shoos the exper- iment results of query efficiency, accuracy, and the ECE and KS errors, before (PrECE and PrKS) and after the attacks (PsECE and PrKS), using different recalibration and defence models. The methods with the best performances in terms of post-attack FCF. (PsECE) and KS (PsKS) are marked in bold, and the second best are under- lined.
MD-TS
97.1
38.0
.908
.026
.542
.021
.514
Splines
9.9
130.5
933
.023
.016
.016
.358
DCA
125.6
60.0
.944
.124
565
.011
.519
SAM
123.0
66.0
.942
.072
.064
.561
AAA
0.8
42.0
.002
102.4
56.0
8022
0216
537
.010
CS
97.0
38.0
.008
.044
.197
.033
.142
Calfoch-
TS
154.8
280.0
.972
.030
.13
.723
266
MD-TS
Splines
DCA
140.9
254.5
.976
345
.025
.315
SAM
145.9
278.0
.962
.77
.458
.170
.450
AAA
0.3
2115
.972
.189
.96
.189
.98
AT'
48.2
188.0
.946
.122
.20
.132
231
CAAY
136.2
341.0
.986
.048
.049
.318
CS
143.3
We can that CS is overall the strongest meth- ods at maintaining low calibration errors on post- allack datapoints. It showed the best post-atlack calibration performance in five out of six setups on PsECE, and ranked among top in all the other setups. The other top calibration performances are distributed among A AAA, Splines, and AT. Know-
TS
132.9
$0.0
.950
.038
.483
.033
.410
MD-TS
131.5
81.5
.940
.017
.36
.012
.422
Splines
1.7
16.0
.950
.040
115
.D41
.067
DCA
132.8
92.0
.500
.052
5066
.057
476
SAM
AAA
AT CAAT
66.9
121.0
.930
132
320
.130
.317
118.9
$5.0
.932
066
448
.055
.411
CS
1.30.5
81.5
.960
022
.092
105
189
11

---

Published in Transactions on Machine Learning Research (11/2024)
ing the property of calibration attacks and organizing defences accordingly is helpful to ensure better defence results.
Overall, from the ECE and KS scores, we can see that there are still significant limitations on recalibration and defences for calibration attacks, which invites more future research. Simple and widely used calibration models like TS are effective on clean data prior to the attacks, but they offer very little benefit post-attack. Training-baas based models like DCA and SAM also tend bring few benefits after being attacked the post-attack ECE and KS errors not substantially different compared to the vanilla models. Lastly, we conclude that although the defence methods from the leaderboard are generally the most adversarially resistant, their inherent high levels of miscalibration even before the attacks render them unsuitable for calibratioon-sesitivee tasks.
6 Conclusions
We highlight and perform comprehensive and dedicated study on calibration allacks, which aim to trap victim models into being heavily miscalibrated, hence endangering the trustworthiness of the models and any follow-up decision-making processes based on confidences. We propose four forms of calibration attacks and demonstrate their severity from different perspectives. We also show calibration attacks are difficult to detect compared to standard attacks. An investigation was conducted study the effectiveness of wide range of adversarial defences and calibration methods, including the defences that are specifically designed for calibration attacks. From the ECE and KS scores, recalibration and defences in handling calibration attacks. We hope this paper helps attract more attention to the attacks against confidence and hence mitigate their potential harm. We provides detailed analyses to help understand the characteristics of the attacks for future work.
see that there are still limitations these
12

---

Published in Transactions on Machine Learning Research (11/2024)
References
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square Attack: query-effiiient black-box adversarial attack via random scarch. ECCV, 2020.
Stavros Autifakos, Nicky Kern, Bernt Schiele, and Adrian Schwaninger. Towards improving trust in context- aware systems by displaying system confidence. In Proceedings of the 7th International Conference on Human Computer Interaction with Mobile Devices & Services, MobileHCT '05, pp. 94,4, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595930892. doi: 10. 1114451085777.118577788
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated Gradients Give a False Sense of Security: Circunventing Defenses to Adversarial Examples. In ICML, 2018.
Yoshua Bengio, Yann Lecun, and Geoffrey Hinton. Deep Learning for AI. Commun. ACM, 64(7):58-65, June 2021. ISSN 0001-0782. doi: 10. 10.1145/3448250.
Dor Bernsohn, Gil Semo, Yaron Vazana, Gila Hayat, Ben Hagag, Joel Niklaus, Rohit Saha, and Kyryl Truskovskyi. LegalLens: Leveraging LLMs for legal violation identification in unstructured text. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2129 229-2 St. Julian's, Malta, March 2024. Association for Computational Linguisties.
Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Practical Black-bos Attacks on Deep Neural Networks using Efficient Query Mechanisms. In ECCV, 2018.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning Imbalanced Datasets with Label-Distibattion-Aaar Margin Loss. In NewrIPS, 2019.
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In Symposium on Securily and Privacy. IEEE, 2017. doi: 10.1109/sp.2017..0
Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. ZOO: Zeroth Order Optimization Based Black-Box Attacks to Deep Neural Networks without Training Substitute Models. In Workshop on Artificial Intelligence and Security, pp. 15 26, 2017. ISBN 9781450352024.
Sizhe Chen, Zhehao Huang, Qinghua Tao, Yingwen Wu, Cihang Xie, and Xiaolin Huang. Adversarial attack attackers: Post-process to mitigate black-box score-based query attacks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In NewrIPS Datasets and Benchmarks Track, 2021.
Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. A case for humans-in-the-loop- De- cisions in the presence of erroncous algorithmic scores. In Proceedings of the 2020 CHI Conference on Human Pactors in Computing Systems, CHT '20, pp. 1-12, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450367087 doi: 10.1145/331383/ 333/33333831.3376633
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.
Guneet S. Dhillon, Kamyar Asizzadenesheli, Jeremy D. Bernstein, Jean Kossaifi, Aran Khanna, Zachary C. Lipton, and Animashree Anandkunar. Stochastic activation pruning for robust adversarial defense. In ICLR, 2018.
Shehzaad Dhuliawala, Vilém Zouhar, Mennatallah El-Assady, and Mrinmaya Sachan. A diachronic perspec- tive on user trust in AI under uncertainty. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5567-5584 Singa- pore, December 2023. Association for Computational Linguistics. doi: 10.18653/v| http//2..emnlppoee main.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16
Words: Transformers for Image Recognition at Scale. ICLR, 2021.
T. Dyer, L. Dillard, M. Harrison, T. Naunton Morgan, R. Tappouni, Q. Malik, and S. Rasalingham. Diagnosis of normal chest radiographs using autonomous deep-lcarning algorithm. Clinical Radiology, 76(6):
1..ee--733.e1,, 2021. ISSN 0009-9260. doi: htlps://dbi.orr/10000 1aorg//10.100 20006//.raa..221..000
Cornelius Emde, Francesco Pinto, Thomas Lukasiewiez, Philip Torr, and Adel Bibi. Certified calibration: Bounding worst-case calibration under adversarial attacks. Tn The Second Workshop on New Fronfiers in Adversarial Machine Learning, 2023.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning Generative Visual Models from Few Training Examples: An Ineremental Bayesian Approach Tested on 101 Object Categorics. CVPR Workshop, 2004.
Pierre Forct, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware Minimization for Efficiently Improving Generalization. In ICLR, 2021.
Ido Galil and Ran El Yaniv. Disrupting deep uncertainly estimation without harming accuracy. NewrIPS, 2021.
Jacob Gildenblat and contributors. PyTorch library for CAM methods, 2021.
Ian J. Goodfellow, Jonathon Shiens, and Christian Szegedy. Explaining and Harnessing Adversarial Exam- ples. CoRR, abs/1412.6572, 2015.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On Calibration of Modern Neural Networks. In ICML, 2017.
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. In ICLR, 2021.
Paula Harder, Franz-Josef Pfreundt, Margret Keuper, and Janis Keuper. Spectraldefense: Detecting adver- sarial attacks on enns in the fourier domain. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2021. doi: D..1009/JCNN55233..2 2TN.0255337.2222..000. 2021.9533443
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in wol-world images: The German Traffic Sign Detection Benchmark. In IJONN, 2013.
Olakunle Ibitoye, Rana Abou Khamis, Ashraf Matrawy, and M. Omair Shafiq. The threat of adversarial attacks on machine learning in network security A survey. CoRR, abs/1911 abs/1911.02261,, 2019.
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. Calibrating predictive model estimates support personalized medicine. Journal of the American Medical Informatics Association JAMTA, 19: 263 74, 03 2012. doi: 10.1136/ 1036//mmajjl--- 2011-0009100
Benjamin Kompa, Jasper Snoek, and Andrew L. Beam. Second opinion needed: communicating uncertainty in modical machine learning. NPJ Digital Medicine, 4, 2021.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
Alex Krizhevsky, Πya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In NeurIPS, volume 25. Curran Associates, Inc., 2012.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Meelis Kull, Miquel Perellé-Nieto, Markus Kängsepp, Telmo de Menezes Silva Filho, Hao Song, and Peter A. Flach. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. In NewrIPS, 2019.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified Uncertainty Calibration. In NewrIPS, 2019.
Aounon Kumar, Alexander Levine, Soheil Feizi, and Tom Goldstein. Certifying confidence via randomized smoothing, NeurIPS, 2020.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable Calibration Mcasures for Neural Networks from Kernel Mean Embeddings. In ICML, 2018
Yann LeCun, Y. Bengio, and GeofTrey Hinton. Deep Learning. Nature, 521:436 44, 05 2015. doi: 10.1038/ nature14539.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In NewrIPS, 2018.
Gongbo Liang, Yu Zhang, Xiaoqin Wang, and Nathan Jacobs. Improved Trainable Calibration Method for Neural Networks Medical Imaging Classification. In BMVC, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV), 2021.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi Wijewickrema, Grant Schocnebeck, Michael E. Houle, Dawn Song, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimen- sionality. In TCLR, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant Adversarial Attacks. In ICLR, 2018.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Ann Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the Calibration of Modern Neural Networks. ArXiv, abs/2100.007998, 2021.
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Yaniv Ovadia, Emily Fertig, J. Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. NewrIPS, 2019.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. AAAI, 2015.
Tianyu Pang, Min Lin, Xiao Yang, Junyi Zhu, and Shuicheng Yan. Robustness and Accuracy Could Be Reconcilable by (Proper) Definition. In ICML, 2022.
Kanil Patel, William Beluch, Dan Zhang, Michael Pfeiffer, and Bin Yang. On-manifold Adversarial Data 2021 201.10911..2.....111.....
Augmentation Improves Uncertainty Calibration. In ICPR, 2021. doi:
Coby Penso, Lior Frenkel, and Jacob Goldberger. Confidence calibration of medical imaging classification system that is robust to label noise. IEEE Transactions on Medical Imaging, 43(6) 2060, 2024. doi:
John C. Platt. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likeli- hood Methods. In Advances in Large Margin Classifiers, pp. 61-74. MIT Press, 1999.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Yao Qin, Xuezhi Wang, Alex Beutel, and Ed Chi. Improving Calibration through the Relationship with Adversarial Robustness. In NewrIPS, volume 34, pp. 14358-1488 14369. Curran Associates, Inc., 2021.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A Python toolbox to benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop, ICML, 2017.
Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel. Foolbox Native: Fast. adver- sarial attacks to benchmark the robustness of machine learning models PyTorch, TensorFlow, and JAX.
Journal of Open Source Software, 5(53):2607, 2020. doi: 10.21105/joss.0200
Sylvestro-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Data augmentation can improve robustness. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Amy Rechkemmer and Ming Yin. When confidence meets accuracy: Exploring the effects of multiple performance indicators on trust in machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors Computing Systems, CHI '22, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450391573. doi: 10.1145/ 0..114/3499100 34211122511119
Kui Ren, Tianhang Zheng, Zhan Qin, and Xue Liu. Adversarial Attacks and Defenses in Deep Learning. http://bb..rrg/...011/.../...
Engineering, 6(3):346 360, 2020. ISSN 2095-8099. doi:
Ramprasaath R. Sclvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient- Localization. Tn ICCV, 2017. doi: 10.1109/ICCV .2017.74
Madhu Shashanka, Min-Yi Shen, and Jisheng Wang. User and entity behavior analytics for enterprise security. In 2016 IEEE International Conference on Big Data (Big Data), pp. 1867 1874, 2016. doi: http//BiDDaaa.11666
David Stutz, Matthias Hein, and Bernt Schiele. Confidence-Calibratte Adversarial Training: Generalizing to Unseen Allacks. JCML, 2020.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect. In NewrIPS, 2020.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. In NearIPS, 2019.
Christian Tomani and Florian Buettner. Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibration. In AAAI, 2021.
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. JMLR, 9(8):22579 2605, 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasæ Kaiser, and Illia Polosukhin. Attention is all you need. In 1. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Xiaosen Wang, Jiadong Lin, Han Hu, Jingdong Wang, and Kun He. Boosting adversarial transferability through enhanced momentum. In British Machine Vision Conference, 2021.
Chaowei Xiao, Zhongzhu Chen, Kun Jin, Jiongxiao Wang, Weili Nie, Mingyan Liu, Anima Anandkumar, Bo Li, and Dawn Song. Densepure: Understanding diffusion models towards adversarial robustness. In ICLR, 2023.
Cihang Xic, Zhishuai Zhang, Jianyu Wang, Yuyin Zhou, Zhou Ren, and Alan Loddon Yuille. Improving transferability of adversarial examples with input diversity. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2734, 2018.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Cihang Xie, Yuxin Wu, Laurens der Maaten, Alan L. Yuille, and Kaiming He. Feature Denoising for Improving Adversarial Robustness. In CVPR, 2019.
Yaodong Yu, Stephen Bates, Yi Ma, and Michael Jordan. Robust calibration with multi-domain temperature scaling. In NeurIPS, 2022.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In ICML, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In KDD, pp. 694 699, 2002.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, Seplember 19-22, 2016. BMVA Press, 2016.
Huimin Zeng, Zhenrui Yue, Yang Zhang, Lanyu Shang, and Dong Wang. Manipulating out-domain un- certainty estimation in deep neural networks via targeted clean-label poisoning. In CIKM, 2023. ISBN 9798400701245.. doi: 10.1145 04.1445/553770036111
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond Empirical Risk Minimization. ICLR, 2018.
Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, pp. 295 295-305, New York, NY, USA, 2020. Association for doi: 10.1145/335109933322222
Computing Machinery. ISBN 9781450369367.
Roland S. Zimmermann, Wieland Brendel, Florian Tramer, and Nicholas Carlini. Increasing confidence in adversarial robustness evaluations. In NeurIPS, 2022.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Appendix
A Detailed Summary of Related Work
In the field of calibration, a great. deal of current research is devoled to the creation of new calibration methods that can be applied to create better calibrated models while possessing as minimum overhead in applying them possible. Methods are generally divided into either post-calibration or maining-basod Post-calibration methods can be applied directly to the predictions of fully trained models at test time, and methods of this class include temperature scaling (Guo et al. 2017). More traditional methods of this type include Platt scaling (Platt, 1999p, isotonic regression (Zadrozny & Elkanj 2002), and histogram binning (Zadrozny & Elkan, 2001]. All of these three methods are originally formulated for binary classification settings, and work by creating a function that maps predicted probabilities based on their values more in tune with the modcl's level of performance. Although they are casy to apply, they often come with the limitation of needing a large degree of validation data to tune, especially with isotonic regression, and performance can struggle when applied to more out of distribution data.
The second class of methods are training-based methods, which typically add bias during training to ensure that a model learns to become better calibrated. Often times these methods help by acting as a form of regularization that can punish high levels of overconfidence late into training. In computer vision, Mixup (Zhang et al.] (018)) is a commonly usod method of this type that serves as an effective regularizer by convexly combining random pairs of images and their labels and helps calibration primarily due to the use of soft, interpolated labels (Thulasidasan et al.] 2019 Other methods work by adding a penalty to the loss function, like in the case of MMCE,
RKHS kernel-based measure of calibration that is added as penalty on top of the regular loss during training so that both are optimized jointly (Kumar et al., 2018). Similarly, [Tomani& & Bueltner (2021) create a new loss term called adversarial calibration loss that directly minimizes calibration error using adversarial examples. Given the effectiveness of many of these mothods in regular testing scenarios, we desire to illustrate how well a diverse range of these methods can cope against attacks targeting model calibration and whether they possess limitations that require them to be overhauled to deal with an attack scenario.
With respect to adversarial attacks, attacks in this field are wide ranging. Well known white-box attacks include the basic Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015). This method works by finding adjustments to the input data that maximizes the loss function, and uses the backpropagated gradi- ents to produce the adversarial examples. Projected gradient descent (PGD) (Madry et al., 2018) is popular iterative-based method that similarly uses gradient information, and has been shown to be a universal first- order adversary, and thus is the strongest form of attacks making using of gradient and loss information. In the black-box space of attacks, ZOO (Chen et al., 2017) is example of popular score-based attack that uses zeroth order stochastic coordinate descent to attack the model, and avoids training a substitute model. The authors make use of attack-space dimension reduction, hierarchical attacks and importance sampling to make the attack more query efficient, which is required as black-box attacks generally need a lot of queries to run compared to white-box methods
A broad range of defences against adversarial attacks have been developed, but among the most popular and effective is adversarial training (Goodfellow et al., 2015), where during training the loss is minimized over one of or both clean and generated adversarial examples. Adversarial training however greatly increases training time due to the need to fabricate adversarial examples for every batch. Gradient masking (Carlini & Wagner 2017) is a simple defence based on obfuscating gradients so that attacks cannot make use of gradient information to create adversarial examples, although it can easily be circunvented in many cases for white-box models (Athalye et al.) 2018), and black box attacks do not need gradient information in the first place. It is by and large difficult for adversarial defences to keep pace with the broad range of attacks and to be provably robust against a large number of them. Although the main topic of this work is calibration, we do focus on modelling adversarial defences and their effectiveness against these attacks.
18

---

Published in Transactions on Machine Learning Research (11/2024)
B Miscalibration Effects on Downstream Tasks
In addition to the discussion of the technical details of calibration attacks for the victim models and de- fences, it is important to reemphasize why maintaining good calibration under such attacks is critical. Well-calibrated models are a crucial component for wrustworthy complex systems. As an example, in real-life deployment, proper confidence scores are essential for determining the instances that need further exami- nation by authority mechanisms such as expert cause additional strain an authority system or any downstream processes, creating risk of significantly slowed-down decision-making due to an increased load of cases to be examined. As another example, by attacking misclassified datapoints with an overconfidence approach, test cases may be erroneously missed by downstream systems such as braking in an autonomous vehicle before a stop sign.
committce of them. Undcrconfidence attacks could
In terms of specific real-world applications, skewed confidence scores or uncertainties have been shown to have dramatic impacts on downstream tasks and human decision-making, which many previous works have highlighted. Previous studies such as those of Penso al. (2024); Jiang al. (2012); Kompa et al. (2021)
have shown the effect of miscalibration on down-stream authority mechanisms. Penso et al. (2024) discuss how medical imaging, if skipping expert review for the confident but incorrect model predictions, can have disastrous consequences, stressing the importance of being well-calibrated on incorrect predictions. Kompa et al. (2021) advocate for medical AI models being able to abstain from providing a diagnosis when there is 8 significant uncertainty for patient, and to defer to additional human expertise to make a more informed diagnosis. This relics on a model that can give well-calibrated estimates.
Moreover, prior works have examined the role of skewed confidence affecting the level of trust humans have in system, which influences applications where humans and Al decision-making occurs symbiotically Zhang et al. (2020); Rechkemmer & Yin (2022). Antifakos et al. (2055), through a study on context-aware mobile phones, show how providing explicit confidence improves user trust in a system. [hhag et al. (2020) demonstrate when a model produces high confidence scores, users are more prone to rely on the Al for decision-making These factors show why miscalibrated oulputs can have great influence on downst processing and authority mechanisms.
As more specific examples in the healtheare domain, chost abnormality detection Dyer ct al. [2021] tag radio- graphs with higl-confidence normality to help radiologists make decisions. In child maltreat hotlines, screening models Ar-Aaga et al. (2020) are deployed where confidence is provided to suggest the likelihood that a child on the referral will experience adverse welfare-rclated outcomes. Call workers use confidence, along with other information, to make their decisions. The study discusses how the tool could be "glitching" out by providing incorrect confidence, compromising operability. Tn the legal domain, Legall Beensshh let al. (2024) detects legal violations. Entities with high confidence are further processed with downstream models. In enterprise security, anomalous behaviour detectors Shashanka et al. (0166) use confidence to send alerts about likely malicious activity.
Dhuliawala et al. 2023) conduct user studies that show that confidently incorrect predictions heavily under- mine user trust in human-Al collaboration settings, requiring only a few incorrect instances with inaccurate confidence estimates to damage user trust and performance, creating long-term effects with recovery being very difficult once humans have been exposod to miscalibratod confidence estimates. They observe that both confidently incorrect stimuli and unconfidently correct stimuli reduce trust, with the former being more sig- nificant, which justifies our formulation of MMA as having high potential for creating mistrust by optimizing the creation of both of these stimuli.
C Details for Maximum Miscalibration Attacks
Proof of Proposition 3.1
Proof. Let a classifier have non-zero accuracy. We cannot expect to reach the error of 100% since ps = 0 cannot be the case (for the top predicted class) nor can P(y; = ((x))) = be true for all yi- However, to achieve the highest calibration error on a set of data points in this scenario, one can first isolate the misclassified data points and if the classifier is made to output confidence scores of 100% all of them, using
19

---

Published in Transactions on Machine Learning Research (11/2024)
the calibration attack for example, it would create a total calibration error of 100% on this set of misclassified data points. With regard to the correctly classified points, where accuracy is 100%, one can create the largest difference between the accuracy and average confidence by making the average confidence on this set as low as possible. Since confidence scorcs can only range from 1/K to 1, the largest possible difference between the average confidence score and the accuracy of 100% 1 Again, if Pk = 1/K, and every P& (x,) = 1/K; while yi = (x, then this makes the calibration error: P(y: = ((x)) (x,) = pk) - Ps = 1/K. It is not possible to create a higher level of calibration error since if pk > 1/K on some number of the correctly classified datapoints, then P(y- j(xx) (x(xx)) = p&) will still be 1, while pk 1/K will lead to less calibration error on that subset of datapoints. With errors on both mutually exclusive subsets of data maximized, the theoretically highest miscalibration will be created on the full data.
To derive the upper bound of the ECE value that can be achieved by MMA, if we assume q is the accuracy of a K-way classifier F on the dataset D = {(x,: bn))im then post successful attack all of the datapoints will fall into one of two bins representing average confidence scores of and 1/K, assuming the confidence range of each bin is 1%. The accuracy in these bins would be 0 and 1, respectively. And the proportion of data points falling into each respective bin is 1 g and q. Based on the ECE formula the maximum error would bc:
= TbbEm1onn/K19 /11% aace (100/10) - con (bin((00/0)0)) +
Thbir 100% (ace(bin - con (bhmt0000) ) N
Emess
N
= 11 1/K| //K++ + (1 - 9) * 00
= 9/9
D Details of Experimental Setup
Metrics. To assess the degree of calibration error caused by each attack, we use two metrics, the popular binning- based Expected Calibration Error (ECE) (Pakdaman Naeini et al. 2015), and KS error (Gupta
et al. 2021). which are formulated in detail in Section F
Datasets.
The datasets we use in our study are CIFAR-100, Caltech-101, and the German Traffic Sign Recognition Benchmark (GTSRB). CIFAR-100 and Caltech-101 are both popular image recognition bench- mark datasets, containing various objects divided into 100 classes and 101 classes respectively. Given the importance of calibration in safety critical applications, we include a common use case of autonomous driving with the GTSRB datasot, which consists of images of traffic signs divided 43 classes. CIFAR-100 has 50,000 images for training, and 10,000 for testing. Caltech-101 totals around 9000 images. GTSRB is split inlo 39,209 training images and 12,630 test. images
Models. ResNet-50 He et al. 2016 is the primary model we train and test on due to it being standard model for image classification. results on image classification tasks, so we also experiment with the popular Vision Transformer (VTT) architecture (Dosovitskiy et al., 2021]. Both of these models are the versions with weights pretrained on ImageNet (Deng et al. 2009). We use the VIT VIT__B_B __166 variant of ViT, and the pretraining dataset used for each model is ImageNet. 1K for ResNet and ImageNet 21K for ViT, and are fine-tuned on the target datascts. Pretrained models are advantagcous to study given they can increase performance over training from randomly initialized weights and present a more practical use-case. The specific details behind our training procedures and our various model hyperparameters can be seen in Section E
Non-convolutional attention-based networks have recently attained great
Attack Scttings. Regarding the SA version of the attacks, for the loo and le norm attacks we use the default SA settings for and p, which are = 0.05 and 0.05 for los and = 5.0 and p= = 0.1 for 42 For our primary results we run the attacks on a representative 500 test cases from the test set of each dataset. Each attack is ran for 1000 iterations, far less than the default 10,000 in Andriushchenko et al. (2020), but since there is no need to change the label, less iterations are required, bolstering the use-case and threat for this form of attack.
20

---

Published in Transactions on Machine Learning Research (11/2024)
The settings for the PGD version of the attacks differ due to the accommodations that need to be made to prevent the PGD algorithm from changing the label while still being able to have a. large effect on the confidence. In terms of general settings, we again use E 0.05 as the adversarial noise value for an lo0 norm. We use an a attack step size value of 5/255. For our whitc-box results we use 10 iterations of the attack. In addition to these settings, some were made to the attack algorithm as simply preventing PGD from changing the class label while trying to calculate the adversarial noise often leads to poor performance practice as many updates are prevented. Instead, & dropout factor is added to the (h * w c) adversarial noise matrix after each attack iteration that only applies select portion of the updates, lessening the effect of updates that are too strong and have high chance of flipping the label. The value for the dropout is dependent on whether it is the overconfidence or underconfidence attack. The most effective values in our experiments were found to be a dropout value of 0.95 for the underconfidence atlack, and 0.2 for the overconfidence attack.
The results in our experiment section are obtained on three runs of cach model with different random seeds.
E Specific Training Details
E.1 General Settings
As mentioned previously, for our general atlack implementation we use SA, which works by using random- ized search scheme to find localized square-shaped perturbation at random positions which are sampled in such a way as to be situated approximately at the boundary of the feasible set. We still use the original sampling distributions, however we remove the initialization (initial perturbation) for each attack since it is prone to changing the predicted labels. Naturally, we use the untargeted versions of the attacks, whereby the perturbations lead to increases in the probabilites of random non-predicted classes for the underconfidence attack, since we only care about the probability of the top predicted class.
The details of the training procedure for each of the models and datasets is as follows: For CIFAR-100 and GTSRB, we use the predefined training and test sets for both but use 10% of the training data for validation purposes. For Caltech-101, which comes without predetermined splits, we use an 80:10:10 train validation/test split. For all of the datasets, we resize all images to be 224 by 224. We also normalize all of the data basod on the ImageNet channel means and standard deviations. We apply basic data aug- mentation during training in the form of random cropping and random horizontal flips to improve model generalizability. The hyperparameters we used for training the ResNet-50 models include: a batch size of 128, with Cosine descent (SGD) optimizer. For ViT, the settings are the same, except we also use gradient clipping with the max norm set to 1.0. We conduct basic grid search hyperparameter tuning over a few values for the learning rate 1...0.00.0000,,000001 and training duration (in terms of epochs). Generally, we found that a learning rate of 0.01 worked best for both types of models. The training times vary for each datasct and model. For the ResNet-50 models we trained for 15 epochs on CIFAR- 100, 10 epochs on Caltech-101, and 7 epochs on GTSRB. Likewise for ViT, we trained for 10 epochs on CIFAR-100, 15 epochs on Caltech-101, and 5 epochs on GTSRB. The results reported in Sections 3 and 5 are shown for models on the epoch at which they attained the best accuracy on the validation set. All of the training occurred on 24 GB Nvidia RTX-3090 and RTX Titan GPUs. Finally, we use 15 bins to calculate the ECE.
scheduler, 0.9 momentum, 5e-4 weight decay, and a stochastic gradient
E.2 Defence Training Settings
In this section, describe each of the defences we used in Section 5 and the settings we use to train them (if applicable).
Temperature Scaling (TS) (Guo et al., 2017). TS is a post-process recalibration technique applied to the predictions of an already trained model that reduces the amount of high confidence predictions without affecting accuracy. TS works by re-scaling the logits after the final layer of the neural network to have a higher entropy by dividing them by temperature parameter T, that is tuned by minimizing negative log likelihood (NLL.) loss on the validation set. Temperature scaling only works well when the training and test
21

---

Published in Transactions on Machine Learning Research (11/2024)
distributions are similar (Kumar et al. 2019), but by reducing overconfidence it may have an advantage against overconfidence attacks.
Multi-domain Temperature Scaling (MD-TS) (Yu et al., 2022) MD-TS is a method based on TS but is designed to be more robust in situations when data comes from multiple domains, as in this case with images corrupted using different types of calibration attacks. Tt modifies the original TS method by first finding the ideal temperature across each domain, then training a linear regression classifier using the feature embeddings of each datapoint and the corresponding ideal temperatures based on the respective domain, yielding classifier that can dynamically calculate an ideal temperature for each instance at test time. We modify this domain for our task of defence by creating three different domains for the base images and their underconfidence attacked and overconfidence attacked counterparts. We select 500 validation instances and find the temperature for each domain and conduct the rest of the method as in its original incarnation. The feature embeddings are as before, using the penultimate layer outputs of model before the classification layer. We experiment with converting the feature embeddings to Fourier domain using the fast Fourier transform before feeding them to the classifier, similar to the principle behind the detection method in HHadder et al.) (2021) to make it easier to identify adversarially attacked datapoints, though we find that the conversion brings little benefit over the basc variation of the method.
Calibration of Neural Networks using Splines (Spline) (Gupta et al., 2021). Spline is another post- process recalibration technique that uses a recalibration function to map existing neural network confidence scores to better calibrated versions by fitting a spline function that approximates the empirical cumulative distribution. It is lightweight, and often performs better than TS.
Difference between confidence and accuracy (DCA) (Liang et al., 2020). DCA is training- based calibration method that adds an auxiliary loss term to the cross-entropy loss during training that penalizes any difference between the mean confidence and accuracy within a single batch, inducing a model to not produce confidence scores that are miscalibrated. We set the weight of DCA to 10 based on the recommendation by Liang et al. 2020). Training settings are kept the same as described in the general settings.
Sharpness Aware Minimization (SAM) (Forct et al., 2021). SAM is technique that improves model generalizability by simultancously minimizing loss value and loss sharpness. It finds parameters that lie in neighbourhoods having uniformly low loss by computing the regularized "sharpness-award" " gradient. The motivation behind using this technique as a defence is that models with parameters that lie in uniformly low loss areas may present additional difficulty in creating adversarial examples, and may be more regularized as a whole. We use a neighbourhood size p == 0.05. We kept the training settings the same as we described in the general sottings.
RobustBench (Croce et al., 2021). To understand how state-of-the-art adversarial defences work against our attack, we take the top performing (in terms of adversarial robustness) WideResNet (Zagoruyko & Komodakis 2016) defences on the popular RobustBBenc defence model benchmark for CIFAR-100 under the I.x 6 = 8/255 attack model. We only choose the WideResNNe models given their closer similarity to the primary model we study in this work, ResNet-50. The defences we choose are those of Gowal et al. (2020) (ranked first), Rebuffi et al. (2021) (ranked third) and [ang et al. (2022) (ranked fifth). These defences use a combination of adversarial training and ensembling to produce models that are robust against a wide range of conventional adversarial attacks. In addition, they use different lechniques, like combining larger models, using Swish/SiLU activations and model weight averaging, and data augmentation to significantly improve robust accuracy.
Adversarial Training (AT). AT is among the most common and effective defences against a wide range of adversarial attacks where models are trained on adversarially attacked images. We implement our version similar to Madry et al. (2018) and Xie et al.] (2019), where we run PGD-based adversarial training, given how this form of defence has been shown to be effective across a wide range of attacks due to PGD being close to universal first-order lox attack. We train exclusively on images attacked with an n-step lxo PGD attack each batch, with the number of steps chosen depending the model and dataset. Since we already test RobustBench models that often make use of AT with large amount of steps, we specifically tune our AT models to have less steps to compromise less on accuracy and miscalibration. We wish to see whether
22

---

Published in Transactions on Machine Learning Research (11/2024)
more lightly-tuned AT can still provide major benefits given calibration attacks are not as severe. For the PGD attack, we attack each image in a batch using an norm of 0.1. We use an attack stepsize relative to of 0.01 / 0.3, with random starts. The number of attack iterations ran for each batch was carefully chosen to balance performance and adversarial robustness. We used 15 iterations on all of the ResNet models, while for ViT we generally required much fewer, with three for the CTFAR- models, and five for the remaining two datasets. In terms of remaining training details, we keep them largely the same as described in the general settings, although the training durations were sometimes varied by few epochs to optimize accuracy. We use the Foolbox implementation of the PGD attack (Rauber et al. 2020: 2017).
Adversarial Attack Against Attacks (AAA) (Chen et al., 2022). A recent adversarial defence specifically tuned towards black box score based methods like Square Attack, this is a post processing method that works on an already trained neural network's logits that uses a function that misleads the attack methods towards incorrect attack directions by slightly modifying the output logits. The method is shown to be very effective against score-based query methods at low computational cost, and is purported to maintain good calibration, which makes it of particular interest in this case as defence against. calibration attacks.
Calibration Attack Adversarial Training (CAAT). Our novel form of adversarial training that uses calibration attacks to generate adversarial examples rather than the regular attack algorithm. Although the general methodology is still the same as PGD-based adversarial training, the primary difference is that for each minibatch, both the underconfidence PGD calibration attack and its overconfidence version are applied to the images and the loss between the two sets of images is added. As this uses calibration attack, the labels of these images are unaffected. The settings we use for the attacks are the same as those described in D for the white-box version. Regarding the settings for each model and dataset, they are largely similar to those of regular AT, although the number of attack iterations is kept consistent at 10, even for VIT. The number of training epochs are the same as those we use for regular fine-tuning.
Compression Scaling (CS). This is a novel post-process defence that does not require training and is specifically designed to maintain the regular confidence score distrriuution and thereby preventing reme miscalibration while undergoing a calibration attack. Since calibration attacks does not flip the original label, for any given classifier, the strongest effect of calibration attacks will be reducing the confidence score correctly classified "easy" datapoints while making the model more overconfident on difficult, misclassified datapoints. This creates a shift in the distribution where for a given high performing classifier the average confidence will drop dramatically while the accuracy remains high, and some misclassified datapoints will shift to a higher confidence level. In any case, a distribution that was originally skewed towards high confidence scores is now essentially shifted lower. Therein lies the goal of CS, to essentially shift back the distribution by scaling such that it lies in high confidence space as before. If we assume that already low confidence correctly classified datapoints will be more affected by calibration attack than one that is much higher confidence, and if we assume that incorrectly classified datapoints will have lower confidence then due to the relative inefficiency of the overconfidence attacks they will likely not reach extremely high confidence levels unless the attack is ran for a very large amount of iterations, then the relative ordering between many of the datapoints is still preserved even if the distribution is shifted, meaning the misclassified datapoints may still get mapped to the lower end of the confidence scale. The advantage of this method is that it largely does not incur a lot of calibration error even on clean data while being among the most effective and consistent defence methods against calibration attack. In addition, if one wants to do rownstream decision making then one can still filter out the bottom p percentage of images with confidence score. For the number of bins, we mostly choose 3 or 4 as this lcads to the smallest error post attack. We find the scaling factor by iterating through a large range of possible values so that the new desired confidence score for the datapoint is then achieved within the new confidence range.
Binning Details. In our two defence algorithuss, the range of possible confidence scores are first split into equally sized bins. In our case, we divide confidence scores into 15 bins and chose the top 3 (or 4) highest confidence bins the compressed bins as mentioned above.
23

---

Published in Transactions on Machine Learning Research (11/2024)
F Calibration Metric Formulation
Here we formulate the two calibration metrics that we use in our experiments. As Equation 2 is an idcalized representation of miscalibration that is intractable, approximations have been developed which are grouped into the more common binning- metrics, and non-binning based metries.
Expected calibration error (Pakdaman Naeini et al. 2015 is the most widely used calibration metric in research. It is a binning-bae based metric where confidence scores the predicted classes are binned into M number evenly spaced bins, which is a hyperparameter that must be carcfully chosen. In each bin, the difference wetween the average confidence score and accuracy of all data points within the bin is calculated, representing the bin-wise calibration error. Afterwards, the weighted sum over the error in each bin consti- tutes the expectation of the calibration error of the model. The equation for ECE is as follows given Bn are the data points in the th bin, and Rm the number of data points in that bin.
M
ECE = Σ
Tem
[acc((m)) - conf (Bm))
(8)
ECE can underestimate the levels of miscalibration due to being sensitive to the number of bins (Ovadia et al. 2019) and by having underconfident and overconfident data points overlapping in one bin (Nixon el al. [2019] Sennogorov-Smmrnne Calibration Error (Gupta et al. 2021) an alternative evaluation metric, that instead of binning, leverages the Kolmogorov-Smirnov statistical test for comparing the equality of two distributions. The error is det by laking the maximum difference between the cumulative probability distributions of the confidence scores and labels. Specifically, the first step is to sort the predictions according to the confidence score on class k, i.e, Px, leading to the error being defined as:
KS error = mar h; hul:
where, hg = hn 0,
(9)
h, h;-1 + 1(y: = K)/N,
h, = h;-1 p& (2.)//
G Adversarial Attack Detection Details
Local Intrinsic Dimensionality (LID) (Ma et al., 2018). This detection method exploits the estimated Local Intrinsic Dimensionality (LID) characteristics across different layers of model of set of adversarial examples, which are found to be notably different than that of clean datapoints or those with added random noise. First, a training set is made up of clean, noisy, and adversarial examples, and a simple classifier (logistic regression) is trained to discriminate between adversarial and hon-adversarial examples. For each training miníbatch, the input features to the classifier are generated based on the estimated LJD across different layrr for all of the datapoints. The hyperparameters for this mothod are batch size and the mumber of nearest neighbours involved in estimating the LID. We choose consistent batch size of 100 in line with previous work such (Harder et al. 2021), and for each case test the possible nearest neighbors from the following list {10, 20, 30, 40, 50, 60. 70, 80, 90} and report the results for the best value, which vary for different datasets and models. We use the implementation from
Lee et al. 2018
Mahalanobis Distance (MD) (Lee et al., 2018). The premise behind this method is to use a set of training datapoints to fit class-condiiional Gaussian distribution based on the empirical class means and empirical covariance of the training datapoints. Given test datapoint, the Mahalanobis distance with respect to the closest class-conditional distribution is found and taken the confidence score. A logistic regression detector is built from this which determines whether a datapoint is adver- sarial. The main hyperparameter for this method is the magnitude of the noise used, which we vary wetween {0.0, 0.01, 0.005, 0.002, 0.0014, 0,001, 0.0005) for each case and pick the value that results the highest de- toction accuracy. In addition, calculating the mean and covariance is necessary to use the method, which we utilize the respective training set to do for each dataset. We use the implementation of MD from (Lee)
et al. 2018
24

---

Published in Transactions on Machine Learning Research (11/2024)
SpectralDefense (Harder et al., 2021). This detection method makes use of Fourier spectrum analysis to discriminate between adversarial and clean images. The spectral features from Fourier coefficients, which are computed via two-dimensional discrete Fourier transformation applied to each feature map channel, are found for each image, and a detector based on logistic regression is trained using the Fourier coofficients. The magnitude Fourier spectrum based detector (InputMPS) is the version we use our experiments.
H Additional Analysis and Results
In this section we provide additional results with white-box attacks, more details on the analyses described in Scction 4.4, and qualitative analysis of the properties of our attacks, as well as quantitative analysis under a common real world issue of imbalanced data distributions. Apart from the white-box results, the remaining analyses are conducted using our black-box setup.
H.1 White-box Calibration Attack
The results for the white-box variation of our attacks be found in Table 7 on the three datasets and across our two tested models, similar to how we presented our black-box results. For each scenario, we show the ECE, KS error and average confidence. We used 10 attack steps to generate the results for an e noise value of 0.05.
Much like the SA results, the PGD attack manages to create significant. miscalibration compared to before the attack with only a small number of attack steps. The results are less severe worse
Table 7: Results of white-box PGD variant of calibration attack.
ResNet
than for SA where the level of miscalibration achieved despite the base PGD attack being far more effective at affecting classification accuracy. We believe this is because the modifica- tions that are made to ensure that the calibration attack algo- rithm does not cause the predicted class to change greatly. This reduces the effectiveness of PGD as the most effective gradient updates that cause a great swing in the confidence score can- not be used since they are likely to change the predicted class, and instead much less significant updates that do not change the confidence great deal serve as the primary noise that gets added to the adversarial images.
ECE
KS
Avg. Conf.
CIFAR (Accuracy:
0.881:100002)
Pre-atk
0.052:00.00 000
0.035=0.006
0.91610.006
UCA
0.213:10.003
0.175=2.007
0.74710.011
OCA
0.02:100.033
0.070=10001
0.9511100022
NNA
0.187:20.00 008
0.161=0.000
0.716:0.007
RCA
0.187:10.016
0.13620.013
0.75010.015
Coltech- 101 (Accuracc::
0.96610:001)
Pre-atk
0.035:10.002
0.031 10.004
0..001
UCA
0.388++00019
0.380+0.022
0.509+0.022
OCA
0.018:+0.008 003
0.019=0.002
0.984+0.001
NNA
0.375:10.022
0.376:0.022
0.5011
RCA
0.353-+0.019
0.352-0.021
0.619+0.022
GTSRB
(Aocuracy: 0.972=0)
Pre-atk
0.019:10.000
0.00810.002
0.98=0.002
UCA
0.233::0.00 020
0. 0.232=0.016
0.75240.011
GCA
0.020:10.002
0.019=0.003
0.99110.003
H.2 Detailed Setup and Comparison of Efficiency of Underconfidence Overconfidence Attacks.
NNA
0.226:10.000
0.207
0.75010.008
BCA
0.217:0.011
0,218=0.009
0.763+0.008
VIT
CIFAR-10 100 (Accuracy:
0.935++0.002)
To understand which form of attack is most query efficient when the amount of change in confidence is the same, for each attack type we identify all of images in the test set that are around a given confidence level. We use the corresponding attack to made the model produce either an increase of 10% in confidence, or a decrease of 10%. We choose two base confidence levels of 80% and 90% and find all the data points within 1% of each. When an attack causes change at or past the set threshold for the given goal probability, the attack stops and the number of queries is recorded. The results can be seen in Table 3 The consistent pattern we observe for both base confidence levels is that it takes notably fewer queries to create undorconfidence than overconfidence, and the former attack is more effective at affecting the average confidence.
Pre-atk
0.08142.0000
0.051
0.88240.001
UCA
0.277:10.000 001
0.274=0.001
0.67110.006
OCA
0.045::0.00 003
0.017=0.002
0.928440002
NNA
0.260::0.00 006
0. 0.262=0.007
0.675+4.005
BCA
0.236:10.013
0. 0.230
0.69910.013
Coltenh- 102 (Accuracy:
0.961
+00024)
Pre-atk
0.137:10.039
0.136:1.066
0.0051
UCA
0.489++0.071
0.089
0.492
OCA
0.086++00045
0.082+0.049
0.879-40.073
MNA
0.458:10.070
0.488
0.473.10094
RCA
0.435:1:00048
0.005
0.507
GTSRR
(Aceuracy:
0.947-0.006)
Pre-atk
0.0:00:000006
0.0262000117
0.922
UCA
0.321::0000 0-17
0.315
0.65120.052
OCA
0.037410001 013
0.020=0.011
0.93610.021
NNA
0.302:10.0337 0.037
0. 0.302=0.036
0.645:10.043
RCA
0.292:40.30
0. 0.092
0.657
25

---

Published in Transactions on Machine Learning Research (11/2024)
H.3 Detailed Analysis on Varying Epsilon and Attack Iterations
Epsilon. The e parameter plays major role in adversarial attacks, as it controls how much noise can be added when creating perturbations. Although setting higher e value for an attack lets it casier and more efficient for the algorithm to create adversarial examples, it potentially cause the visual changes to images more perceplible, so a small is preferable while still being able to produce good adversarial examples. In the case of calibration attack, there is no need to go as far as flipping a label, so lower e-bounds have the potential to create some miscalibration. To provide further details on our results in Figure figure as mentioned previous we lested on CIFAR-100 using ResNet-50. The five different. e values we use are {0.005, 0.01, 0.05, 0.1, 0.25) after being attacked using all four of the attacks with the other settings the same as in Appendix D with the results averaged over three models. In addition to the miscalibration being strong for most of the attacks at low e values, we can see that MMA consistently outperforms the rest across the different values. UCA does not have much change with higher e, but it is largely because the models has already almost reached the peak level of attacking the confidence with low epsilon values, and as such does not have a large effect on ECE. As middle figure largely displays the same trends as ResNet, revealing that the results are not architecture dependant. The rightmost figure uses ResNet and goes over the same values as before, except MMA is run over both Caltech-101 and GTSRB models. Again the trends are similar, although the increase in ECE is not as severe as for CIFAR-100.
/ for the leftmost
Iterations. Expanding on the results in the right- most figure in Figure 2 the number of iterations of MMA is varied from 100, 500, 1000, to 5000, whilst attack both ViT and ResNet models trained and tested on CIFAR-100 with the same settings as in Appendix D We note how the ECE begins sat- urate at close to 500 ilerations, after which the ben- efits of running the attack longer are minor, though even at 100 iterations the ResNet model becomes heavily miscalibrated despite the standard e value of 0.05 being used. In our tests showing the effec- tiveness of the original SA versus its calibration at- tack version, seen in Figure g we specifically com- pare over accuracy and ECE between the maximum miscalibration attack and the regular untargeted SA across the four aforementioned iteration values for both ResNet and ViT on CIFAR-100. As expected, SA greatly reduces the accuracy even with a small number of iterations. Nevertheless, in terms of ECE, the calibration attacks consistently produce higher amounts of miscalibration compared to the original SA algorithm across the different iteration amounts.
35
Ga arstor Arast
30
10:5
00:00
120
12.00
2550
0000
138
04-
0.-
01-
Aztck Rericions
Atackk Rerocions
Figure 4: The contrast between the effects on accuracy and ECE between the original version of the Square Attack algoritim and the maximun variation of the calibration attack algorithm at 1000 attack iterations. (Top) ResNet- 50 results. (Bottom) ViT results.
H.4 t-SNE Visualizations
To help visualize the effect of each of the attack types in latent space and to confirm they are having the expected effects, we run t-SNE analysis dvan der Maaten & Hinton 2008) on the representations of ResNet- 50 right before the classification layer. The datasets we use throughout this study, with their large number of classes, are not ideal for visualization purposes. Instead, we create a binary subset using CIFAR-100 by taking all of the images from two arbitrary classes, bicycles and trains. We create separate training set and test set to perform this procedure independently, and fine-tunc a ResNet-50 model on the training set. The specific details are similar to those described in Section E for CIFAR-100 ResNet. We train the model for 5 epochs with a learning rate of 0.005. The attack settings are the same as in Section for the Lo version, and we only run the attacks for 500 iterations. We run the t-SNE analysis on a balanced slice of 200 images from the new test set for easy visualization purposes, before and after all four attack types. The model achieves 95% accuracy the full test set.. Figure shows the graphs. It can be seen the effect on the representations
26

---

Published in Transactions on Machine Learning Research (11/2024)
liting Correc
0.5
24-
02-
0.2
12
Comeet Hoor Wong
Cans
14-
0.4
Cornec Cass lrong
Clos Weems
Figure 5: t-SNE visualization of the effect of different forms of calibration attacks ResNet model trained and tested on binary subset from CIFAR-100, with the test set (consisting of 200 data points) results being displayed. In the order from top left to bottom right, the plots for the pre-attack (vanilla model), and the UCA, OCA, RCA, and MMA variations can be seen.
for the adversarially attacked data is as expected. OCA causes the representations for both class predictions, even incorrect ones, to be split apart as much as possible, while UCA causes more jumbled representation between the two classes with most falling closely to the decision boundary. MMA has a similar effect to the underconfidence attack, except the misclassified images are pushed far away from the decision boundary make appear if the model is clusters for each prediction type to form, random data points pushed to be more overconfident underconfident than they originally were. With these results, we see visually confirm that the attacks possess their intended behaviour.
confident in its decisions. Lastly, RCA causes two distinct random
H.5 Imbalance Ratio
110
0.6-
0.85
0.5
Aug. Querits
Acturacy
0.80
0.75
Post
0.3
PrevAzaad
0.79
0.65
0.2
0.60
70-
0.0
0.2
0.4
0.5
0.8
1.5
0.0
0.2
0.4
0.5
0.8
1.0
0.9
02
0.4
0.5
0.8
3.0
Imbalance Ratio
Imbalance Ratio
Imbalance Ratio
Figure 6: Graphs comparing the vulnerability of ResNet and ViT models trained with different imbalance ratios CIFAR-100 to the maximum miscalibration attack at 1000 iterations and an of 0.05, and their corresponding overall trends in average queries and accuracy.
Dataset imbalance has a profound effect on how model learns and how well it performs, with detrimental effects occurring when imbalance ratios are very high. With how common imbalanced data distributions are in real world scenarios, we believe it is worth studying the influence of imbalance ratio and its relationship with robustness against calibration attacks
additional point of analysis. We choose CTFAR-100
27

---

Published in Transactions on Machine Learning Research (11/2024)
primary dataset for this analysis, and we follow the procedures in Tang et al. (220)) and Cao et al. (2019) to create training sets with long-tail imbalance. This is form of imbalance where the datapoint sizes in the classes follow an exponential decay. We use the variable p to denote the ratio between datapoint sizes of the class with the smallest datapoint size, and those of the one chosen to be the biggest. We create training sets with P values of 0.01, 0.02, and 0.01 (for 1:100, 1:50 and 1:10 ratios of smallest to biggest class). We then train 3 ResNet-50 and 3 ViT models on each imbalanced set. The training details are again the same as those described in the general settings Section E although the training times are different. 15 epochs is used to training the 1:100 ratio models, while 10 epochs is used for the rest. We subject the models to the maximum miscalibration attack using the same settings as in Section D for CIFAR-100 (test data is balanced), and calculate the resulting average and deviation of the pre and post attack ECE, average number of queries, and accuracy. The graphs displaying the results can be seen in Figure 6 Unsurprisingly, the higher the imbalance ratio, the lower the accuracy is on the balanced data. In terms of robustness, the more balanced the data the more resistant is against getting miscalibrated from the attacks, for both the ResNet and ViT architectures. This is similar to the trends in the inherent miscalibration present before the atlacks, although the calibration differences between the different ratio models are not severe, and ViT at the 1:500 imbalance ratio is the best calibrated beforehand but becomes the worst after the attack. The trend in the number of queries it takes for successful attack is reversed for ResNet and ViT, with ViT requiring more queries the more balanced the data is, while ResNet is vice-versa. Overall, dataset imbalance does not create favourable conditions for robustness, though the use of imbalance data techniques could potentially remedy some of these issues.
H.6 GradCAM Visualization Details
Given the effectiveness of the attacks at leading a model to produce highly miscalibrated outputs, for both base styles of attacks, we endeavour to explore whether they also lead to any changes in where the model focuses on in an image when making its decision, and especially with OCA. Knowing this can lead to further insights as to how models are affected by calibration attacks. To accomplish this analysis, we use MradCAM (Selvaraju et al. 2017), a popular visualization method that produces a coarse localization map highlighting the most imporlant regions in an image that the model uses when making ils prediction by making use of the gradients from the final convolutional layer (or a specific layer of choice) of network. We apply GradCAM to our ResNet-50 models fine-tuned on Caltech-101 to images from the Caltoch-101 test set before and after the underconfidencce and overconfidence atlacks at the standard attack settings used in Section D and using the GradCAM implementation of Gildenblat & contributors (2021). Since the method calculates relative to a specific class, we do so in-terms of the original predicted class for each image. Figure 3 shows the results with some representative images. We specifically choose images where the attacks led to large change in predicted confidence (at least 10%). On the whole, we have observed that the coarse localization maps have minimal to no noticeable changes after the adversarial images are produced, especially in the case of the overconfidence attacked images. This leads us to believe the primary mechanism of the attacks changing the model confidence is in the final classification layer as opposed to the convolutional layers. This analysis also shows that might be difficult to identify these attacks are occurring based on these types of gradient visualization methods.
H.7 Certified Confidence Scores
In this section, we follow recent work in the domain of providing provable guarantees on the robustness of confidence scores by examining the effect on smoothed classifier and its bounds when handling calibration attacked data, particularly in the case of OCA. Given the lack of label flipping, we expect that adversarial examples generated by calibration attack lie close to the original in the data manifold, thereby having a minimum effect on the certified confidence. To test this hypothesis, we closely follow Kumar et al. (2020) on certification to provide lower bounds on the confidence of a smoothed classifier. A strong lower bound can be produced by using the probability distribution of the confidence scores of a Gaussian cloud around the input image using Neyman-Pearseen lemma to calculate this for a given cortified radius. We select a sub- sample of 100 random datapoints from CIFAR-100 and use our base ResNet-50 models that are attacked using the underconfidence and overconfidence tacks using our standard settings, and average the expected
28

---

Published in Transactions on Machine Learning Research (11/2024)
confidence score lower bound produced by the smoothed model across different Radii. We utilize the version of the method that uses CDF information and use a smoothing faction value of a = 0.25, failure probability a = 0.001 and 100,000 Monte- Monte-Carlo datapoints for the estimation. The results for the certified smoothed model can be seen in Figure 7 We observe that the average lower bound for the underconfidence attack is lower than the base version and the overconfidence attack, although it is not. dramatically different due the huge error bounds. Nevertheloss, it appears that the smoothed model works well at being robust, particularly on the overconfidence attacked datapoints, but despite the expected confidence, the average certified accuracy for each set of data was 74,, 59%, and 76% respectively, so the smoothed classifier struggles at performing more accurately on the perturbed datapoints. Overall, this method of guarantee could be a useful tool for counteracting calibration attacks without using specific defences by determining how much confidence scores can reasonably be affected, particularly when using classifiers for safety-crilical tasks that rely on confidence scores.
Expected Confidencce Scare Lower Bound
2.0 0.8
Bass Undertzefieence
Oseecorfitrrcee
0.6-
0.0
75
Radius
Figure Expected Confidence Score lower bounds for set of CIFAR-100 images (original and attacked using UCA and OCA) based on ResNet-50 classifier.
H.8 Results on Swin Transformers
We perform additional experiments using the Swin Transformer (Liu et al. 2021 architec- ture further validate our attack results on additional architectures. Table g depicts the results all types of calibration attack: UCA, OCA, MMA, and RCA the Swin-Large variant of the model. The trends of the results on Swin Transformer agree with the existing results in Table I Specifically, UCA, MMA and RCA all bring both ECE and KS to high values. As observed previously, OCA successfully raises average confidences to high levels, but given the high accuracy of this victim model, incrcasing confidence will not have drastic effect calibration.
Table 8: Results of UCA, OCA, NMA, and RCA with the Swin-Large model on CIFAR-100. Accuracy of the victim model included.
Avg qq
Med. qq
ECE
KS
Avg. Conf.
Swig-Large (Ace: 0432.005)
Pre-atk
.266 0006
.010+-003
.943-+.005
UCA
177.819.9
96.023.5
.4552.006
.30:..005
.5344.088
DCA
31.4::9
1.0:0.0
.099 002
.0104.001
.804.000
MHA
137.5+99.
9..24885
.461 0617
4344-007
5366+0007
RCA
138.8214.4
102.245.3
.4332.011 011
.0941.006
55574.003
H.9 Detailed Analyses on Recalibration and Defences
In this section, we provide more detailed analyses on the results of the different defences seen in Table 6
The RobustBench models compromise substantially accuracy, and have high level of miscalibration clean data. They do largely avoid getting extremely miscalibrated as result of the attacks compared to the defenceless models, except the top model on the leaderboard. Nevertheless, their high inherent miscalibration means they are unfavourable in situations where the model must be well calibrated.
In terms of the calibration methods, TS tends to be among the best methods at reducing calibration error prior to the attacks, but after the attacks offers very little benefit compared to the vanilla models. The MD-TS method is similar with its prior calibration being solid, but post-attack only brings minor benefits
29

---

Published in Transactions on Machine Learning Research (11/2024)
over TS in most cases. It appears that this is due in part to not finding the ideal temperature parameter for each image due to the difficulty of identifying the correct image domains, as in, recognizing which form of attack occurred. The Splines method is similar in its pre-attack calibration benefits to TS, but differs greatly in its performance post-attack. In some cases, like with CIFAR-100 using ResNet, it is casily the worst performing defence method. In other cases, particularly for Caltech-101 and GTSRB ViT, it is able to keep ECE at relatively reasonable values post-attack. This discrepancy shows that finding an ideal recalibration function has the potential to be a strong defence. The training-based DCA and SAM methods tend to bring few benefits after being attacked, even when they improve the calibration clean data, the post-attack ECE and KS errors are not substantially different. compared to the vanilla models.
The performance of the regular adversarial defence techniques is mixed. For robustness, AAA in most cases achieves the lowest post-attack ECE. Even in the best cases like Caltech-100 101 ResNet, ECE tends to be at least double compared pro-attack, and most cases we still observed multiple-fold increases. This technique is also among the poorest calibrated on clean data. Regarding AT, our approach docs not compromise accuracy and miscalibration on clean data. It brings notable robustness, especially compared to the calibration methods, but it is not among the strongest.
Lastly, CS is the strongest methods at maintaining low calibration error on post-attack datapoints. Moreover, the technique tends to have better calibration error on clean data compared to AAA. It shows how it is key that high confidence values are retained to have decent calibration after the attacks. Altogether, despite some promising results with the defences, as a whole there are still limitations particularly with the strongest adversarial defences. The compromise of poor ECE on clean data for better calibration robustness against the attacks that observe, well as the general inconsistent performance means that further refinement defences is warranted.
H.10 Results of l2 calibration attacks
Table Results of the based calibration attacks on three datasets.
BVX thg
Auy.
Accurncy: 0.881100000
Pro-Attuck
06551:0.005
0.988560005
1004
182 7
91.08110
0:30900009
0:350-00000
0.506--000S
DCA
4166333
1.000.0
0:129-10077
0::31-00007
0.96--00000
1904
137.315
9271106
0:195-1001
00.002
0.8001-00003
BCA
125.3.1.33
00.149
0:11110006
0.350:0.012
06814160011
USilecn 101
Accuracy: 0.060200.00 001
Pie Attac
0:001:0008
0,081:00004
0336650.011
DCA
203. 24
100,020177
0.16821002
0.157:01005
0180025000122
DC4
60. 96444
1.005.00
0.019200004
00007110005
03802000012
1964
40.811.5
2272:10.
0.143:10000
0.:01:::005
0.866500055
RCA
33.645.3
206.0288.3
0.120:0000
0,121:0005
0.848:0.005
Accursey: 0 0.97210..00 000000
Pre- Attacle
0.019100006
0:008:0.:022
DCA
2911 C42224
1667516666
0.700200322
0::75:0122
0.770=0000
DC4
19.543.3
10040.0
0,02200002
01021-0.012
(3975-0000
1914
91.11-1.1
1013-918
0.339-00006
0:325-0.031
RCA
97.51
211.011-111
0.30310014
00.01
0.794166011
VIT
Accursrys: 0.955 0.0002
Pre- Attacia
0001100000
0,051:00055
0.800-00001
UKA
100.6:7.11
111:2:11.55
0.88310014
0.55615.011 0.312:0.015
DKA
051.3248844
(81.55488.4
0.022100022
0,221:0.055
0.95850.003
1964
111-947.4
131.5417.7
0.105-10000
03381-0.03
0.590--00007
BCA
108-2:6.9
157.
0.511100000
0.351-0.007
06111-6/001
CSilacnn 193
1224
Pire- Attock
0:37740059
0:3650000
0.855-00003
258-74:47.2
0.50120007
0.251:01577
0.729:0.001
DCA
28.26:16.0
100.0.0
0.30220046
0.200:0::4
01800500033
1954
31.5e:2.2
206.5652.0
0,23423008
0.254:0155
0.740:000000
RC4
23.06-8.1
2015-224.2
0339629038
0.30500ds
0.76650001
Accurneyys
6.01110006
0992166001
258-51227.8
188.7.0155
0:001100122
0:062.00011
0886160.15
20.2151.3
0.092110006
0,001:0:122
0368850015
30

---

Published in Transactions on Machine Learning Research (11/2024)
H.11 Full Attack Scale Results
In this section, we present the results of the four attack types across both black-box and white-box selups at various model scales of ResNet and ViT. Furthermore, we include four variants of Swin Transformers for cach attack scenario. We included the Tiny hathh--iing-attchh-iinddoooo window7-224); Base of Swin Transformers in our comparison trained with a similar setup used to obtain the results described Appendix H.8
Small (swin-small-patche ). and Large (swin-large caarge-patch4 patch4-window/ window 7-224) variants
Table 10: Attack effectiveness of MMA (black-box setup) across different model sizes Swin Transformers on CIFAR- 100.
Avg #q Med. #q
ECE
KS
Avg.Conf.
Swin-Small (Acc: 299)
Pre
.036+.007
.024#0008
.23+t.001
Post 120.3+5.7
72.5+411
5111.005
.494.007
4944.003
Bwin-Base (Acc: .9392.009)
Pre
034+.001
.012+.005
.40+.001
Post
153.4+14.2
101+12.3
.64+.015
.437+.011
.527.0014
Swin- Swin-Large (Acc: .945..005)
Pre
026+.006
.010+.003
.955+.005 .33++0007
Post
137.5+9.8
99.2+..55
461+.007
.34++0007
Table 11: Attack effectiveness of UCA (black-hox setup) across different model sizes CIFAR-100.
Avg #q
Med. #q
ECE
KS
Avg. Conf.
ResNet-18
(Accuracy: .8442.04))
Pre 60.5+1.5
.037=.003
.016:.005
.870:.001 .4662.00
Post
35.3+1.5
.531=.005
.470=.006
ReaNet-50 (Aocuracy: .8812.002)
Pre
0552_.006
.3551.006
.16.1.006
Post 74.3+3.44
1..7+1.5
.540=005
.479+.001
.455+.005
R5s
(Accurocy: 888:+.002)
Pre
.051=005
.0461.004
.294.000
Post 90.415.5
48 0.0
007
47611.004
.433..008
ViT-base (Acouracy: .9522. 002)
Pre
.064-0006
.054+.005
.82++0004
Post 118.5224
62.023.1
.572-0007
.5532 004
.04+..00
ViT-large (Acouracy: 096220.001)
Pre
.007
022.1.012
921:1.021
Post
154.0=6.1
76.8+44.5
5336-007
.514+.015
.44+.019
Swin- Swin-Tiyy (Accuracy: .8834.002)
Pre
.040=.088
.031#.013
.94+.0011
Post 100.116.5
54.5.22
.002
.44-1.003
55111.006
Swin-Small (Accuracy: .8092.000
Pre
.036-.007
.024+.008
.23+-0011
Post 139 73.742.0
.507=0001
.564.005
4964.002
Base (Accuracy:
.9922.009
Pre
.034=.001
.012+..05
.440+0004
Post 191.-18.2 104.7414.5
A64=.012
.4382.110
.555.0015
Swin- Swin-Large (Accuracy: 9432.005)
Pre
.066=006
.100=.003
.455.005
Post 177.8.9.9
96.13.5
456_.006
430.1.005
534_1.008
31

---

Published in Transactions on Machine Learning Research (11/2024)
Table 12: Attack effectiveness of RCA (black-box setup) across different model sizes on CIFAR-100.
Avg #q Med. #q
ECE
KS
Avg. Conf.
R8
(Aocuracy: 8542.004)
Pre
0007_.00
.16.1.005
.70.1.001 .455t .06
Post 56.3+19
10.3+1.5
.479=.015
+207
RexNet-50 Pro
(Aecwracy: .8112.002)
.052.000
.0331.006
.9611.006
Post 68.9+4.6
42.7+ 1.2
.558-001
.461+.003
-51++0033
152
(Aecaracy: +.022
Pre
0551-005
.046+0004
.29++0005
Post 81.0=155
48.022.0
5122.003
451=.002
1964.008
ViT-base (Accuracy: .9552. 002)
Pre
004
.54.1.003
.82.1.004
Post 106.4+3.0
7..3+1.5
.599-002
.505=.003
.41
ViT-large (Accuracy:
.9642.001)
Pre
037_
022.1.012
.2111.021
Post
122.5+11.6
83.5+5.1
552=.005
.479+.001
.479+0008
Swin- Tiny (Accuracy: 002)
Pre
0040=008
.031+.013
914+-011
Post 101.4..
62. 22.99
.44= 018
4171.009
5264 .00
Swin-Small (Accuracy: .8992. 008)
Pre
.036-0007
.024+.008
.23++0001
Post 124.625.4
79.223.4
483=.008
.4322
.519+.006
Swin- (Accuracy: 009)
Pre
.0341.001
.012.1.03
94041.004
Post 161.6.25.5 116.6.12..
.440..011
.15.1.009
54881.010
Large (Acouracy: .943..005)
Pre
.026=0006
.010=.003
.955+.005
Post
138.81144
102.215.3
001
.0911.066
.57.1.003
Table 13: Attack effectiveness of DCA (black-box setup) across different model sizes on CIFAR-100.
Avg #q Med. #q
ECE
KS
Avg. Conf.
ResNet-18 (Accuracy: 88442.004)
Pre
.03.1.003
.01611.055
.80.1.001 .3661.003
Post
21.6114
1.04.0
.82.1.003
.0831.003
ResNet-50 Pre
50 (Accurccy: 188140002
05244.000 006
.0351.006
.9144. 006
Post 16.04.8
1.04.0
.124:.002
.1244.002
.9964.000
ResNet-152 (Acewaoy
883+.002)
Pre
.051+.005
.0462.004
.999+.005
Post 10.3+2.6
1.0..00
.077+.001
.077#.001
.961+.002
VeT-base (Accuracy: .95+.002)) 002)
Pre Post 524.7+88.7
.064+-006
.054+.005
.82+-0004
510.5+114.3
.043+-007
.043+.006
0011+-001
VET-large (Aocuracy: .9622.001 001)
Pre
.037+-009
.022+.012
0211
Post 345.0196.1 205.751117.7
.0301.006
.03911.00
.962.L.088
Swin-Ting (Accuracy: 88811.002)
Pre
.40.1.008
.3111.013
.9141.011
Post 27.319.8
1.0..0
.0741.004
.0751.004
.9581.00 003
Swin-Smail (Acourccy: Pre
8894.
8094.008)
.0364.007
.0244.008
.9234. 001
Post
35.247.3
1.0..0
.0654.005
.0652.005
.965+.002 002
Swin- Base (Accuracy: 99992.000
Pre
.034+.011
.012+.005
.940+.004
Post
34.6+4.33
1.0+.0
.039+-004
.039+.005
1978+.003
Suin- Large (Accuracy: .4844005)
Pre
.026+-006
.010+-003
.945+-005
Post
34.4+4.9
1.0+.0
-039+-002
.040++0001
.980+-004
32

---

Published in Transactions on Machine Learning Research (11/2024)
Table 14: Attack effectiveness of MMA (white-box setup) on different model sizes CIFAR-100.
ECE
KS
Avg. Conf.
ResNet-18 18
Pre
.037+-003
.016+.005
.870+-001
Post
.158+.006
.144+-004
.726+-005
ResNet-50
Pre
.052+.006
.035+.006
916+-006
Post
-187+-008
161++007
.746+0007
ResNet-152
Pre
.0514.005
.0461.004
.9591.005
Post
.1574.011
.1374.007
.7724.002
ViT-hase
Pre
.644+-006
.054+.005
.82+-0004
Post
.20++.006
.262+0007
.675+0005
ViT-large
Pre
.0374.009
.0224.012
.9111.021
Post
.2794.054
27744.056
.6221.056
Swin-Tiny
Pre
.040+-008
.031+.013
.14+-011
Post
.078+.005
.059+.015
.42:+-018
Swin-Small
Pre
.0641.007
0244.008
923:E.001
Post
.117:.000
.1014.008
.81=1.005
Swin-Base
Pre
.0344.001
.0124.005
.4001.00
Post
.160+.035
.1544.032
.793+.024
Swin-Large
Pre
.026:1.006
.010+.003
9451.005
Post
.13::.009
.133:.012
$271.013
Table 15: Attack effectiveness of UCA
(white-box setup) on different model sizes CIFAR-100.
ECE
KS
Avg. Conf.
ResNet-18
Pre .037+.003
.016:.005
.870=-001
Post
.232+t002
1176+-006
.740+.003
ResNet-50
Pre .0521.0000 1622
.0352.0000 006
9161.0006
Post
.213+.003
.175+.007
.747+.011
ResNet-152
Pre
.0511.005
.0461.004
.9291.005
Post
.210=.012
.158+.003
.781+.007
bas-
Pre
.0641.000 .006
.0541.005
.8821.004
Post
.277+.001
.244+.004
.671+.006
ViT-large
Pro
.0374.009
0022:.0122
92111001
Post
.299:..088
.285+.003
668++.069
Swin-Tiny
Pre
.040:1.008
.0311.013
.144.0011
Post
.142:.023
.088 021
8541.017
Swin-Small
Pre
.036:..07
.0241.008
.2311.011
Post
.179::.004
.128:.008
82941.055
Swin-Basc
Pre
.034:..01
.012+.005
.404.004
Post
.217:.0199
.175:.023
8124.020
Swin-Large
Pre
.026:E.006
.010+.003
.955..005
Post
.191:.015
.1581.01
8164.017
33

---

Published in Transactions on Machine Learning Research (11/2024)
Table 16: Attack effectiveness of RCA (white-box setup) on different model sizes CIFAR-100.
ECE
K$
Avg. Conf.
ResNet-18 18
Pre
.037+.003
.016=005
.870+.001
Post
.179+-016
.138+0006
.762+-003
ResNet-50
Pre
.052+.006
0355-000
916+.006
Post
117++006
.156+.013
.759+-015
ResNet-152
Pre
.0512.000
.0462.004
.9292.003
Post
.188:.004
.144=.008
.789 016
ViT-hase
Pre
.044+0006
.05++0055
.882+-004
Post
.236+.013
.239+.015
.999+.013
ViT-large
Pre
.0372.009
0222.012
.912.021
Post
.2632.047
2592.051
6862.057
Swin-Tiny
Pre
.040+0008
.031 +003
.94+-011
Post
1103+.0140 40
+067+.018
885+.018
Swin-Small
Pre
.036:..07
.0242.008
.23=.001
Post
.1 .148:.004
.112=.015
832=.010
Swin-Base
Pre
.034+.001
.012+.005
940+ 004
Post
.175t.022
.152=.019
8166.013
Swin-Large
Pre
026..0006
.010=.003
945=..05
Post
.15:t.014
.136..014
842+.020
Table 17: Attack effectiveness OCA (white-bor box setup) on different model sizes on C1FAR-100.
ECE
KS
Avg. Conf.
ResNet-18
Pre .0374.003
.016+0005
880++.001
Post
.079++000
.079++0006
933++0002
ResNet-50
Pre .052+.000
.035+.006
.16++.006
Post
.072+.003
.070+-001
.91++0002
ResNet-152
Pre
.0511.005
.0461.004
.9991.005
Post
.075+.001
.075+.002
.59++.001
bas-
Pre
.0642.000 000
.0541.005
888:1.004
Post
.045+=003
.017+.002
.28++.002
ViT-large
Pro
.0374.000 .00
00221.012
92111.021
Post
.039:.011
0021+0008
9289
Swin-Tiny
Pre
.040:1.008
.031#.013
.144.0011
Post
.070-.004
.066+.005
.949-0003
Swin-Small
Pre
.036:.007
.0241.008
.2311.011
Post
.057:.008
.056:.008
.9551.001
Swin-Basc
Pre
.034:t.011
.012+.005
.40+.004
Post
.037:1.005
.0361.008
.94410006
Swin-Large
Pre
.0261.006
.010+.003
.945..005
Post
.0381.003
.0374.002
.9884.004
34


Reviews:
[{"review_id": "NY8ZRiC1nz", "reviewer": "Reviewer_RYXa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Generally interesting question posed by the paper: how can we attack (and defend) against attackers that care about some sort of uncertainty/confidence of the model\n- Clean writing and presentation\n- Fairly thorough experimental setup, including several baselines, adversarial example detection, white- and black-box attack, etc.\n\nWeaknesses:\n- Generally, I am not convinced by the set up of the calibration attacks studies in this paper. Mainly, I believe that uncertainty estimates such as confidence are only useful in a decision-making context. While ECE is a useful measure for calibration, having a low ECE is only relevant in practice if the confidence is actually used for something. This could be abstention, deferral to another system or human, OOD detection or similar detection settings, human rater studies, etc. Without a downstream use-case (i.e., decision problem), the confidence does not matter much. In classification, without a downstream use of the confidence, the actual classification task does not care about the confidence being calibrated or not. I feel this paper misses the point to some extent, focusing on confidence attacks without any downstream task, using ECE as a proxy. Instead, I would expect experiments on downstream tasks or potentially alternative approaches to calibration and measuring calibration (e.g., from the conformal prediction literature [1] or a Bayesian perspective).\n\n[1] https://arxiv.org/abs/2107.07511\n\n- The attacks themselves do not introduce any new tools or improve over existing besides changing the objectives. It is well known that many attacks, especially PGD, are widely applicable. For example, [2] already had an attack targeting confidence (but still expecting mis-classification) and highlighting some of the optimization problems that come with a different objective. Btw, [2] also has a similar adversarial training routine that would be an appropriate baseline.\n\n[2] https://arxiv.org/abs/1910.06259\n\n- Regarding the CS defense I am wondering why this procedure does not impact the calibration on clean examples? Table 4 shows that pre attack ECE is low with CS, but if we increase the confidence of low confidence examples, ECE should increase more significantly. In any case, it severely changes the calibration property on clean examples which should be a no-go in many practical applications where we care about calibration.\n- Where the models used for attacking and as defenses calibrated in any way before the experiments? I couldn’t find that information in appendix C but would expect that all models where calibrated using e.g. vector or temperature scaling prior to experiments.\n- Tables 1 and 5 suggest that black-box attacks are actually more successful in worsening the ECE. Did I understand that correctly? If so, why? White-box attacks should be significantly more powerful and the difference in ECE is significant. Is this an optimization problem?\n- The proposed AT variant does not improve over baseline AT. Only the accuracy-ECE trade-off is better – the proposed method has worse robustness in terms of ECE but slightly better accuracy. This trade-off is not explored.\n- Detecting white-box examples is known to be flawed. [2] and other works repeatedly showed that the considered methods are not robust in the white-box setting when considering appropriate attacks. At this point, the experiments merely show that confidence attacks are easier to detect; or they might show that the confidence attacks are not tuned enough to be successful. In any case, these experiments do not add a lot of insights.\n\nConclusion:\nWhile I like the general direction, I think the paper focused on the wrong experimental setup by studying impact on ECE of such attacks rather than actually relevant downstream tasks (e.g., abstention, detection, deferral, human-AI teams, other decision making setups). Moreover, the experiments have some flaws and do not provide many insights beyond general intution about adversarial attacks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a family of black-box and white-box adversarial attacks on the calibration of a neural network model. Specifically, they consider crafting adversarial examples causing overconfidence, underconfidence, random confidence or cause maximum miscalibration across test examples. They also consider adversarially training models against such attacks as well as a post-calibration defense based on temperature scaling.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UnjDXjGCvH", "reviewer": "Reviewer_nBsh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper is well-structured and easy to comprehend.\n- The authors comprehensively explore four types of calibration attacks.\n\nWeaknesses:\n- The impact of calibration attacks on subsequent decision-making processes is not clearly evaluated. \n- Additional evaluations across different model architectures, such as Swin Transformer, are necessary.\n- The influence of varying model sizes on attack effectiveness is not investigated.\n- Further exploration across different tasks, particularly within the realm of natural language processing (NLP) tasks and models, is needed", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study investigates calibration attacks, which manipulate model confidence levels while preserving predicted labels. Four types of attacks are identified and evaluated across different model architectures. The findings underscore the need for heightened awareness and continued research efforts to enhance the resilience of models against such nuanced adversarial threats.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6LHLIfOR3A", "reviewer": "Reviewer_9Y8b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The authors make novel contributions to calibration attacks. In particular, they properly formulate four different types of calibration attacks. They also invented two defense methods against calibration attacks, and this contribution to defense should be recognized.\n2. The paper contains ample experiments that answer various questions regarding the properties of calibration attacks. First, different models, datasets, and metrics combinations are considered for each attack type. Also, the paper asks many research questions and performs experiments to demonstrate the answer.\n\n### Weaknesses\n1. As far as I know, this is the first paper that tries to bring the calibration attack to formal research. However, I don’t think that the paper justifies enough about the importance of the problem. I’d suggest adding more details in Section 3.4 with more illustrative examples. For example, provide more specific details about how real-world authority mechanisms work with well-calibrated inputs to show why miscalibration can be an existing real-world threat.\n2. The miscalibration's transferability (across different model architectures) should be experimentally verified. If it does not show enough transferability, using multiple models simultaneously could be an easy defense against calibration attacks.\n3. Though the writing quality is generally good, the paper writing can be improved further. Please see **Requested Changes** for more details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper mainly studies the calibration attack, a specific type of adversarial attack that tries to make the target model miscalibrated rather than to induce misclassifications. Specifically, modern machine learning applications exploit the model's confidence to improve its trustworthiness, and therefore, the miscalibrated confidence will undermine the reliability of real-world machine learning applications.\n\nThe authors start with formalizing the objective of calibration attacks. In a calibration attack, the attack does not want to change the prediction but wants to make the target model less calibrated for adversarial examples. Here, the definition of class-wise calibration follows the conventional definition that a well-calibrated model will match the model's confidence to the probability that the prediction is correct. Therefore, the attacker will aim to make the model’s confidence on the adversarial example far from such probability. The authors propose four different objective functions that can yield miscalibration: *Underconfidence Attack* (UCA), *Overconfidence* Attack (OCA), *Maximum Miscalibration Attack* (MMA), and *Random Confidence Attack* (RCA). Also, the author proposes two defenses against calibration attacks: *Calibration Attack Adversarial Training* (CAAT) and *Compression Scaling* (CS).\n\nThe authors demonstrate the properties of calibration attacks, e.g., attack performance, sneakiness against detection, qualitative analysis, etc., with a set of experiments. Their simple implementation is effective enough to produce miscalibrated predictions of the target model. Also, the existing adversarial example detection algorithms do not work well against calibration attacks. The authors also experiment with possible defenses against calibration attacks, even though the proposed defense methods show limited performances.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please supplement Section 3.4 with more detailed real-world scenarios and emphasis on the threat.\n2. If possible, do some additional experiments that show the transferability of miscalibration across different model architectures. For example, does UCA output miscalibrated inputs that show underconfidence across many different architectures?\n3. Writing improvements\n   * The authors should provide more details about the definition of metrics. This will make the result more understandable based on what each metric means.\n   * In Tables 1, 2, and 3, highlight the key results with boldface (as the authors did in Table 4). This helps the readers to focus on the important part more quickly.\n   * In Figure 2, use subfigure captions rather than referring them to subfigure 1, 2, etc.\n   * Section 5 also contains experimental results. Why is Section 5 separated from Section 4? If this is for the distinction between attack-related experiments and defense-related experiments, you may change the title of Section 4. In my opinion, Section 5 does not have enough content to be an independent section, so I’d suggest merging Section 5 into the subsections of Section 4.", "broader_impact_concerns": "I don’t see a particular broader impact concern regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, reviewers were supportive of acceptance, subject to the two main suggestions above: more discussion of the significance of the problem, demonstrating the downstream impact of the attacks. Both would undoubtedly strengthen the paper; we believe though that even without the latter, the paper could be of sufficient interest as it is. Nonetheless, it is important that the authors add a discussion on this point.

**Requested changes**

- add discussion of the significance of the problem, in Section 3.4 or elsewhere. This can be a summary of the response to Reviewer 9Y8b.

- add discussion on the focus of ECE versus downstream metrics.

- add experimental results on the transferrability of the attacks across models, per the response to Reviewer 9Y8b.

- add experimental results on Swin Transformers, per the response to Reviewer nBsh.

- typo: page 3, "nubmer"

- formatting: x-axis for Fig 1 and legend for Fig 2 are quite small.

- formatting: in Equation 3, e.g., suggest to use \rm font for the subscript.
Decision: Accept (poster)