Extreme Miscalibration and the Illusion of Adversarial Robustness (2024)

Abstract: Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify. Adversarial Training (AT) is often used to increase model robustness. However, we have discovered an intriguing phenomenon: deliberately or accidentally miscalibrating models masks gradients in a way that interferes with adversarial attack search methods, giving rise to an apparent increase in robustness. We show that this observed gain in robustness is an illusion of robustness (IOR), and demonstrate how an adversary can perform various forms of test-time temperature calibration to nullify the aforementioned interference and allow the adversarial attack to find adversarial examples. Hence, we urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations to ensure that any observed gains are genuine. Finally, we show how the temperature can be scaled during training to improve genuine robustness.

heading: 1 Introduction
text: Deep learning Natural Language Processing (NLP) models are able to perform well in a range of tasks (Manning et al., 2014). However, these NLP models are susceptible to adversarial attacks, where perturbing clean input text samples slightly (accidentally or maliciously by an adversary) can lead to a NLP model misclassifying the perturbed input (Jia and Liang, 2017). However, the emergence of the Adversarial Training (AT) paradigm (Goodfellow et al., 2015) has shown some success in training models to be more robust to these small adversarial perturbations. Here, the traditional training process is adapted to minimize the empirical risk associated with a “robustness loss” as opposed to the risk
∗Equal contribution. This paper reports on the work VR did whilst at Amazon Web Services. VC holds concurrent appointments as an Amazon Scholar and as a faculty at EPFL. This paper describes the work performed at Amazon.
associated with the standard loss for clean input samples. The robustness loss is the standard loss applied to the worst-case (loss maximizing) adversarial sample for each training sample. In NLP, due to the discrete nature of the text, this adversarial training min-max formulation is particularly challenging as the inner maximization is computationally expensive (Yoo and Qi, 2021). Nevertheless, a variety of approaches have been proposed in the robustness literature, ranging from augmentation of the training set with adversarial examples for a specific model, to sophisticated token-embedding space optimizations for the inner maximization step (Wang et al., 2019b; Goyal et al., 2023).
Although many NLP AT methods appear effective in boosting model robustness, we argue that, in some cases, the observed improvement is merely an illusion of robustness (IOR) that can be easily circumvented. In the computer vision literature, Athalye et al. (2018) show that certain modeling decisions can give rise to such an illusion via gra-
2500
dient obfuscation (Papernot et al., 2017). We build on these findings by showing that IORs can also emerge easily in NLP AT due to highly miscalibrated models. Concretely, we:
• Argue that the extreme class confidence of miscalibrated models (Guo et al., 2017) gives rise to an illusion of robustness by disrupting the adversarial attacks’ search processes.
• Demonstrate this by intentionally creating highly over- and underconfident models at inference time, and by applying an existing adversarial training technique from the literature. These models appear to be up to three times more robust than the baseline. However, we reveal this robustness to be an illusion that largely evaporates when naive model calibration is applied.
• Further introduce a test-time adversarial temperature optimization algorithm that practically nullifies the perceived robustness gains and demonstrate its efficacy on a range of encoder models, classification datasets, and adversarial attacks.
• Finally use the above insights to improve true robustness to unseen attacks by increasing the training temperature, demonstrating its efficacy alone and in combination with other AT methods. In contrast to Papernot et al. (2015), we do not perform model distillation but only training-time temperature scaling as the adversarial defense.
In light of our findings, we urge the community to perform test-time temperature scaling during all robustness evaluations to ensure that the observed robustness is genuine and not merely an illusion. To our knowledge, we are the first to demonstrate this phenomenon in NLP models and propose trainingtime temperature scaling as an adversarial defense.

heading: 2 Background and Related Work

heading: 2.1 Adversarial Attacks
text: An untargeted adversarial attack is able to fool a classification system, F() with trained parameters θ̂, by perturbing an input sample, x to generate an adversarial example x̃ to cause a change in the predicted class,
F(x; θ̂) ̸= F(x̃; θ̂). (1)
Traditional adversarial attack definitions (Szegedy et al., 2014) require the perturbation to be imperceptible as per human perception. In NLP it can be
challenging to measure imperceptibility. Following notation in Raina and Gales (2023) the distance between the original, clean sample and the adversarial example is limited as per a proxy distance measure G(x, x̃) ≤ ϵ.
A plethora of adversarial attack approaches have been proposed for efficiently discovering adversarial examples for NLP models (Alzantot et al., 2018; Li et al., 2018; Gao et al., 2018; Wang et al., 2019c; Ren et al., 2019; Jin et al., 2019; Tan et al., 2020; Garg and Ramakrishnan, 2020; Li et al., 2020; Tan and Joty, 2021). Many of the popular attack approaches are implemented in the TextAttack library (Morris et al., 2020).

heading: 2.2 Adversarial Training
text: Standard supervised training methods seek to find model parameters, θ̂, that minimizes the empirical risk (for a dataset of x ∼ p(x)), characterised by a loss function,
θ̂ = argmin θ E x∼p(x) [L(x, θ)]. (2)
Adversarial Training (AT; Goodfellow et al., 2015) adapts the objective to minimize the empirical risk associated with the worst-case adversarial example, x̃, such that we are minimizing a robust loss,
θ̂ = argmin θ E x∼p(x)
  max
x̃: G(x,x̃,)≤ϵ, x̃∈A
L(x̃, θ)   .
(3) It is too computationally expensive to perform the inner maximization step to find textual adversarial examples in each step of training. A group of AT methods speed-up this optimization step by finding adversarial examples in the token embedding space, which allows for faster gradientbased approaches: PGD-K (Madry et al., 2018), FreeLB (Zhu et al., 2020), TA-VAT (Li and Qiu, 2020), InfoBERT (Wang et al., 2020). However, limited success of these approaches has been attributed to perturbations in the embedding space being unrepresentative of real textual adversarial attacks. Hence, AT methods such as Adversarial Sparse Convex Combination (ASCC; Dong et al., 2021) and Dirichlet Neighborhood Ensemble (DNE; Zhou et al., 2020) identify a more sensible embedding perturbation space, which they define as the convex hull of word synonyms. Nevertheless, the simplest and most common AT approach in NLP is to augment the training set with textual
adversarial examples x̃ for each clean sample x by applying the attack to a model trained in the standard manner (Equation 2).

heading: 2.3 Model Calibration
text: Modern deep learning models are often miscalibrated, where the model’s confidence in the predicted class does not reflect the ground truth correctness likelihood (Guo et al., 2017). A model with a predicted class confidence Pθ̂(ĉ|x), is defined as perfectly calibrated when
P ( ĉ = c∗|Pθ̂(ĉ|x) = p ) = p, ∀p ∈ [0, 1], (4)
where ĉ = F(x; θ̂) is the predicted class and the true (label) class is c∗. Any deviation from this indicates miscalibration. Typical single-value summaries for the calibration error are the Expected Calibration Error (ECE) and the Maximum Calibration Error (MCE) (Naeini et al., 2015).

heading: 2.4 Obfuscated Gradients
text: Computer vision literature has demonstrated that a ‘false sense of security’ to adversarial attacks on image classification systems can arise due to ‘obfuscated gradients’ (Athalye et al., 2018; Papernot et al., 2017; Tramèr et al., 2018). Obfuscated gradients block an adversary’s search process for an adversarial example and hence give the sense that the system is robust to the adversarial attack — however, this is not the case, as adversarial examples still exist, but specific gradient-based mechanisms used by adversaries to find the adversarial examples are not effective when obfuscated gradients are present. In computer vision, obfuscated gradients in typical image systems can arise due to various reasons, including ‘shattered gradients’, ‘stochastic gradients’, exploding/vanishing gradients and any form of gradient masking (Athalye et al., 2018; Papernot et al., 2017; Tramèr et al., 2018). However, to our knowledge, no previous work has explored how the illusion of robustness phenomenon emerges in NLP systems.
In this work, we observe that in various adversarial training regimes designed for NLP tasks, there is often the risk that systems become extremely miscalibrated. Extreme miscalibration effectively results in ‘obfuscated gradients’, which in turn results in the Illusion of Robustness. Hence, in summary, both computer vision and NLP systems can suffer from the Illusion of Robustness due to obfuscated gradients, but the practical scenarios which
cause these obfuscated gradients differ for image and NLP classification systems. As a result, the methods used to mitigate the root causes behind obfuscated gradients for image classification and NLP classification systems, also differ.

heading: 3 The Illusion of Robustness
text: Certain modeling approaches can lead to an illusion of robustness (IOR). In computer vision, it is shown that obfuscating gradients (Athalye et al., 2018) through shattered, stochastic or exploding/vanishing gradients; or masking gradients (Papernot et al., 2017; Tramèr et al., 2018) can lead to such an IOR - the model appears robust to adversarial attacks. We build on these works and argue that (un)intentional extreme model miscalibration is an example of a realistic cause of IOR.
The robustness gains observed for traditional NLP AT approaches (Equation 3), may not always be due to inherent robustness gains, but can be a consequence of extreme model miscalibration. This miscalibration can induce extreme confidence predictions, such that the model’s predicted class confidence Pθ̂(ĉ|x) is either very high (overconfident) or very low (underconfident). Figure 1 (using a standard NLP model, test dataset and adversarial attack described in Section 3.3) demonstrates that highly miscalibrated models with extreme confidence values in the predicted class (around 1.0 for overconfident models or 1/C, where C is the number of classes for underconfident models) are significantly more “robust” to adversarial attacks.
This apparent increase in robustness of extremely miscalibrated models can be explained. For both underconfident and overconfident models, the predicted class confidence has very little variance for different input sequences, x,
Ep(x)[Pθ̂(ĉ|x)− Ep(x)[Pθ̂(ĉ|x)]]2 < ζ, (5) where ζ is some small variance. The narrow confidence distribution makes it challenging for an adversary to identify an appropriate search direction for adversarial examples. To illustrate this, consider a miscalibrated model with extremely high confidence in the predicted class probability, Pθ̂(ĉ|x) ≈ 1.0, then for most search directions d that are not in an adversarial direction d ̸= d̃ (where x̃ = x+ d̃) the model has very little sensitivity,1 i.e.,
dT∇xPθ̂(ĉ|x) ≈ 0. (6) 1Note that these strict mathematical operations are not
Consequently, any whitebox adversarial attack approach looking to exploit gradients or even a blackbox attack approach measuring the sensitivity of the predicted probability, has a small confidence range to observe. The implication is that the impact of any proposed perturbation gives a very noisy signal to its actual effect on the output. As a result, the adversarial attack search process will converge extremely slowly or outright fail to find the desired adversarial perturbation direction d̃. We verify this hypothesis empirically in Appendix E.7, corroborating related computer vision literature on gradient obfuscation (Athalye et al., 2018). We first demonstrate how to induce an IOR by explicitly miscalibrating models at test-time, before discussing how this can happen unintentionally during training.

heading: 3.1 Explicit: Test-time Temperature Scaling
text: Let θ̂ be a model trained using the standard training objective, as in Equation 2. For this model with predicted logits, l1, . . . , lC for C output classes, the probability of a specific class is typically estimated by the Softmax function, Pθ̂(c|x) = exp (lc)∑ i exp (li)
. However, we can intentionally miscalibrate the model and increase the model confidence at test time by using a temperature, T = Td, to scale the predicted logits,
Pθ̂(c|x;T ) = exp (lc/T )∑ i exp (li/T ) . (7)
A design choice of Td ≪ 1.0 concentrates the probability mass in the largest logit class to create an overconfident model, whilst conversely Td ≫ 1.0 creates an underconfident model. Hence, explicitly setting a design temperature T (d) at inference time can be used to serve highly miscalibrated models, which can disrupt an adversary’s attack search process (Equation 6), whilst maintaining the simplicity of the standard training objective (Equation 2).

heading: 3.2 Implicit Overconfidence: Grad. Norm.
text: Having shown how to induce an IOR by explicitly scaling a model’s temperature, we now discuss how certain implementation strategies and algorithmic features in adversarial training (AT) procedures can also implicitly induce model overconfidence.
We first examine the recently proposed Danskin Descent Direction approach for adversarial training (DDi-AT; Latorre et al., 2023). Latorre et al. (2023)
defined for the input text space and are simply representative of equivalent discrete textual space perturbations.
adapted the standard AT paradigm (Equation 3) to identify optimal gradient update directions for increased robustness, showing promising results in computer vision. We describe the NLP-specific implementation in Appendix A.
In preliminary experiments, we observed that DDi-AT creates highly overconfident models without compromising clean accuracy, such that a DDiAT model almost always predicts with near 100% confidence in its predicted class, Pθ̂(c|x) ≈ 1.0 (Table 1). Further ablations, detailed in Appendix A.3, revealed that the gradient normalization step in the DDi algorithm was responsible for this model overconfidence. In the following section, we verify this by applying gradient normalization to other AT schemes that may use it during training.

heading: 3.3 Experiments
text: We first present the experimental setup for all experiments in this paper before showing how illusions of robustness can arise when evaluating highly overconfident or under-confident models for robustness.
Data. Experiments are carried out on six standard NLP classification datasets. For IOR experiments we use Rotten Tomatoes (Pang and Lee, 2005); the Twitter Emotions Dataset (Saravia et al., 2018); and the AGNews dataset (Zhang et al., 2015). We observe the same general trends across all datasets, and therefore present the results on Rotten Tomatoes here and include the others in Appendix E.1.
Models. We follow existing adversarial robustness literature and use Transformer (Vaswani et al., 2017) encoders, which are state-of-the-art on many classification tasks. 2 Specifically, we consider the base variants of DeBERTa (He et al., 2020), RoBERTa (Liu et al., 2019) and BERT (Devlin et al., 2019). We observe the same general trends across all models, and therefore present the results for DeBERTa here and the others in Appendix E.2.
Adversarial attacks. We experiment with four common adversarial attacks. BERT-based Adversarial Examples (bae) (Garg and Ramakrishnan, 2020) is a word-level blackbox attack, where the adversary only has access to the model inputs and predictions. We also include Textfooler (tf ) (Jin et al., 2019) and Probability Weighted Word Saliency
2Appendix C demonstrates the superior performance of encoder-only models relative to generative language models for many classification tasks.
(pwws) (Ren et al., 2019), more powerful wordlevel attacks. Finally, we include the DeepWordBug (dg) (Gao et al., 2018) attack as a whitebox, character-level adversarial attack. We use the TextAttack implementations and their default settings (Morris et al., 2020). To measure the impact of each adversarial attack, we report the adversarial accuracy, which is the accuracy of the target model on adversarial examples found by the attack.
Explicit temperature scaling. We create underand over-confident models, ↓conf and ↑conf, by scaling the temperature as in Section 3.1.
AT approaches. We consider a range of AT methods: Danskin Descent Direction (ddi-at; Latorre et al., 2023), PGD-K (pgd; Madry et al., 2018) and FreeLB (freelb; Zhu et al., 2020 as embeddingspace AT schemes, and ASCC (ascc; Dong et al., 2021) as a text-embedding combined AT approach. We further create variants of the baseline, pgd and ascc that use gradient normalization during training, named baseline∗, pgd∗, and ascc∗, respectively. Finally, we consider the most common NLP AT method: augmenting the training set with adversarial examples, using DeepWordBug as the representative attack (dg-aug). Hyperparameters are detailed in Appendix D.

heading: 3.3.1 Results
text: Model Confidence. We see from Table 1 that the ↑conf, baseline∗, ddi-at, pgd∗, and ascc∗ models are significantly more confident than the models
with no intervention, whilst the ↓conf model is far less confident, as intended. Note that the differences in the confidence are more pronounced for the adversarial examples (pwws is used to attack the test set). The differences in confidence between the original and gradient normalization variants of the baseline, pgd, and ascc models verifies our hypothesis that using gradient normalization during training is one cause of extreme overconfidence.
Robustness. Table 2 presents the adversarial robustness of each model as measured by the adversarial accuracy under the different adversarial attacks. While the regular AT approaches (dg-aug, pgd, ascc, freelb) increase robustness to some extent, the gains in robustness of the highly overconfident models (↑conf, ddi-at, baseline∗, pgd∗, ascc∗) appear to outstrip them by two- to three-fold.
We argue that the apparent increase in robustness of the extreme confidence models (↓conf, ↑conf, baseline∗, ddi-at, pgd∗ and ascc∗) in Table 2 is due to the attack search process being disrupted, but the models are still susceptible to adversarial examples. We know this must be true for the explicitly temperature-scaled models since the predicted class never changes from the baseline’s, only its confidence. We further empirically find that extreme confidence results in a noisier search for the adversarial attack. We discuss this in greater detail in Appendix E.7 due to space limitations.

heading: 4 Piercing the Illusion
text: Having demonstrated how to induce an IOR from the developer or defender’s perspective, we now discuss how to nullify it as an adversary. Although the following approaches involve modifying aspects of the model’s output at test-time, these modifications are only used to create/find adversarial examples, which can then be applied to the original, un-modified model served by the model developer.

heading: 4.1 Naive Test-Time Temperature Calibration
text: Highly miscalibrated models (Section 3) interfere with an adversarial attack’s ability to find meaningful search directions due to the little sensitivity in the predicted probabilities. An adversary aims to mitigate this disruption to the attack search process. The natural solution, then, is to calibrate the model so that the confidences are in a sensible range and can now be exploited by adversarial attacks.
A strong indicator of model miscalibration (Section 2.3) can be given by the Negative Log Likelihood (NLL; Hastie et al., 2017). Thus, assuming access to the output model logits l1, . . . , lC and a labelled validation set of data {xi, c∗i }i, an adversary can apply test-time temperature calibration (Guo et al., 2017). This works regardless of how the model was miscalibrated (implicitly or explicitly). Now, the adversary optimizes an adversarial temperature, Ta to minimize the Negative Log Likelihood (NLL) of the validation set samples,
Ta = argmin T
∑
i
− logPθ̂(c∗i |xi;T ), (8)
where Pθ̂(c ∗|x;T ) is the confidence of the true class after temperature scaling as in Equation 7. Due to the continuous nature of the transformation and the need to optimize a single parameter, Ta, we use standard gradient descent optimization.3
Other than temperature optimization, an adversary can attempt other post-training model calibration approaches such as Histogram Binning (Zadrozny and Elkan, 2001), isotonic regression (Zadrozny and Elkan, 2002) and multi-class versions of Platt scaling (Niculescu-Mizil and Caruana, 2005; Platt and Karampatziakis, 2007). However, we empirically find temperature calibration to be the most practical and effective for an adversary seeking to mitigate a model’s IOR. We discuss this in greater detail in Appendix E.6.
3Inspired by https://github.com/gpleiss/ temperature_scaling/tree/master.

heading: 4.2 Adversarial Temperature Optimization
text: While simple, the naive calibration approach has two shortcomings:
1. The adversarial temperature, Ta is not directly tuned to minimize adversarial robustness, as it only considers the likelihood of clean examples in a validation set.
2. Learning the adversarial temperature, Ta to minimize the NLL (Equation 8) uses a gradient descent–based optimization algorithm that is sensitive to hyperparameters and does not guarantee an optimal solution.
Hence, we now outline an algorithm that directly optimizes the adversarial temperature Ta to minimize a model’s adversarial robustness at test time. We define the adversarial accuracy, Q() as a function of the temperature parameter,
Q(T ) = 1 J
∑
j
I [ F(x̃j(T )) = c∗j ] , (9)
where x̃j(T ) represents the adversarial example generated from an adversarial attack on the given model, θ̂ with the logits scaled by a temperature T as in Equation 7. Figure 1 illustrates that as the temperature parameter is swept from large to small values (increasing model confidence), the adversarial accuracy, Q() behaves almost as a convex function of temperature, T , such that, Q(αT1 + (1−α)T2) ≤ αQ(T1) + (1−α)Q(T2), where 0 ≤ α ≤ 1. The optimal adversarial temperature Ta minimizes the adversarial accuracy Q(T ),
Ta = argmin T
Q(T ). (10)
Hence, Ta can be found efficiently over the nondifferentiable convex function, Q(), using search methods such as golden section search (Kiefer, 1953). We use the Brent-Dekker method, an extension of golden section search that accounts for a parabolic convergence point (Brent, 1971).

heading: 4.3 Experiments
text: Setup. We maintain the experimental setup as Section 3.3 and supplementary results for different models and datasets are provided in Appendix E.
IOR mitigation. We experiment with the two proposed approaches to mitigate the disruption of the adversarial attack search processes and remove
the IOR: naive temperature calibration (cal) and adversarial temperature optimization (opt). The learning rate is set to 0.01 with a maximum of 5000 iterations for cal. For opt, we use DeepWordBug to attack the validation set when optimizing for Ta.
Evaluation. We first modify the model by scaling the predicted logits by Ta and then the adversarial attacks are run on the modified model to find adversarial examples. The original, unmodified model is lastly evaluated on these adversarial examples.

heading: 4.3.1 Results
text: Table 3 shows the impact of the different adversarial approaches (cal and opt) to learn Ta on the adversarial robustness of the models. For the overconfident models, ↑conf, baseline∗, ddi-at, pgd∗ and ascc∗, simple temperature calibration (cal) is sufficient to cause a significant drop in model robustness.4 For the ↓conf model, the temperature optimization approach (opt) is necessary to significantly reduce robustness, illustrating its efficacy at nullifying IORs.
4Appendix E.4 discusses the relationship between the calibration error and the model confidence.
It is worth analyzing why simple calibration (cal) is effective in removing IOR in the overconfident models (↑conf) but not for the underconfident models (↓conf). This observation can perhaps be explained by considering the search space for the calibrating temperature: the cal method is naive temperature calibration, where a temperature parameter Ta (divisor of logits) is learnt on a validation set by minimizing the negative log likelihood. The solution for Ta differs when calibrating low-confidence and high-confidence models. For low-confidence models, Ta must lie between 0 and 1 (as the logits have to be scaled up), and for highconfidence models, Ta only needs to be greater than 1. Hence, the solution space for Ta is more constrained when calibrating low-confidence models, meaning it is more challenging for gradient based search methods to find the solution space for Ta. Therefore, cal struggles more when calibrating the low confidence models.
Critically, our results highlight the risk of some AT approaches, whether by design or as an implementation detail, giving the illusion of robustness even if they do not yield genuinely robust models.

heading: 4.4 Discussion
text: Although adversarial temperature optimization is more effective in nullifying the IOR, it is significantly slower than the naive calibration approach. Therefore, naive temperature calibration is favorable in computational resource–scarce settings and should always be run at minimum in robustness evaluations. Another consideration is detectability. When evaluating an API model, adversarial temperature optimization will require sending many similar queries as part of the attack, which has a higher chance of being detected and blocked by the API. In contrast, naive temperature calibration only requires querying the API for predictions on clean examples, which will appear far more innocuous.
One might even argue that to expose an IOR, it is unnecessary for an adversary to modify the model with adversarial temperature scaling to find adversarial examples. Instead, we could find adversarial examples for another model (e.g., baseline) and transfer to the target model. We test this hypothesis in Appendix E.3. We find that although the transfer attack from baseline to ddi-at reduces the adversarial accuracy, it is unable to bring it down to the same values as baseline, as achieved by our proposed temperature scaling approaches.

heading: 5 Raising the Training Temperature for Genuine Robustness
text: Section 3 showed that it is easy to unintentionally develop AT schemes that do not yield true robustness gains but instead induce IOR. The success of temperature scaling at interfering with an attack’s search process poses the natural question: Can a similar effect be induced in training such that it cannot be nullified, thus improving true robustness?
We now present a simple modification to standard training that boosts the true robustness of NLP models to unseen attacks. We consider an attack to be unseen when its adversarial examples are not used in adversarial training. In our experiments, this refers to all AT methods other than dg-aug.

heading: 5.1 Method
text: Since scaling the temperature down at test time reduces an adversarial attack’s efficacy, intuitively, we would like to “bake” this behavior into a model’s weights such that it cannot be neutralized at the logit layer by the approaches from Section 4.
We propose to do this by increasing the temperature during training, bringing the probabilities of the different classes closer together. This encourages the model’s parameters to compensate by pushing the logits of the different classes further apart (Figure 3 in the Appendix). This increases the distance to the class boundary in the logit space and makes it more difficult for an adversarial attack to change the predicted class (Robey et al., 2023).
Adversarial robustness can also be viewed as type of generalization, and Agarwala et al. (2020) find that model generalization depends strongly on the training temperature, where larger temperatures yield stronger results for vision models. Therefore, our method can also be viewed as flattening the loss landscape, which has been shown to improve generalization for adversarial robustness in computer vision (Stutz et al., 2021). Future work will aim to rigorously understand the observed empirical robustness gains of high temperature training.

heading: 5.2 Experiments
text: We maintain the setup from Section 3.3, with the addition of three datasets from the General Language Understanding Evaluation benchmark (GLUE; Wang et al., 2019a): the Corpus of Linguistic Acceptability; Question-answering NLI; and the Microsoft Research Paraphrase Corpus. We further include gradient normalization (∗) in all ex-
periments, as we found this to stave off decreases in clean accuracy as we increase training temperature (ablation is detailed in Appendix F.7). To ensure the robustness gains are not IORs, we apply the test-time calibration described in Section 4. Appendix F.8 demonstrates the extent of IOR when such test-time temperature scaling is not applied.

heading: 5.2.1 Results
text: Results in Table 4 show that a high training temperature not only boosts genuine adversarial robustness for a model trained with the standard objective (baseline∗⊕T ), but can also be combined with popular adversarial AT schemes for a further boost. We observe that the high training temperature approach improves the adversarial accuracy against unseen attacks for all of the adversarial training approaches experimented upon (⊕T ). This demonstrates that high temperature training is complementary to existing AT methods and consistently encourages a gain in genuine robustness. Interestingly, we observe that for dg-aug∗, using a high temperature during adversarial training further improves adversarial accuracy for the unseen attacks but causes a slight drop in adversarial accuracy for the (seen) dg attack compared to regular AT. This suggests that although a high training temperature successfully increases robustness to unseen attacks, it may not yield further robustness gains against seen attacks.
Additionally, Figure 2 presents the change in clean and adversarial accuracy of a model trained with the standard training objective and different temperatures T used during training.5 We further observe a consistent robustness profile where robustness peaks at similar temperatures for all tested attack types. This is particularly useful for a model developer with access to only one type of adversarial attack. The training temperature can be tuned for optimal robustness on that specific attack form with the confidence that the robustness gains will transfer to the other unseen/unknown attack forms. Finally, we observe a decrease in accuracy at extremely large training temperatures, an indication that overly large temperatures may excessively smooth the predicted probability distribution and make it too challenging for the model to learn. However, this is easy to avoid by sweeping the temperature with a single attack since there is a consistent robustness profile for each dataset.

heading: 6 Conclusion
text: NLP models are susceptible to adversarial attacks, where small changes in the input cause the model to predict incorrectly. Many adversarial training (AT) approaches have been proposed to induce robustness to adversarial attacks. In this work, we argued that the observed robustness gains may not be due to true increases in model robustness. We demonstrated how AT schemes can unknowingly create
5Appendix F.5 contains a detailed breakdown for each training temperature, adversarial attack, and dataset.
highly miscalibrated models that disrupt common adversarial attack search methods by obfuscating gradients, yielding up to three-fold perceived robustness gains. However, this is merely an illusion of robustness (IOR). We proposed simple methods an adversary could use to circumvent such gains. Specifically, we showed how to perform test-time temperature scaling to mitigate disruptions to the adversarial attack search processes and pierce the IOR. Hence, we strongly recommend all adversarial robustness evaluations incorporate adversarial temperature scaling to ensure any observed robustness gain is genuine and not an illusion. Finally, we proposed a practical training-time modification to increase a model’s genuine robustness to unseen adversarial attacks and demonstrated its efficacy alone and in combination with other AT methods.

heading: 7 Limitations
text: • Empirical results are presented for stateof-the-art encoder-based Transformer models. Although Appendix C demonstrates that encoder-based models are more appropriate for many NLP classification tasks than the recently popularized generative Large Language Models (LLMs), it would be useful to investigate how susceptible these larger, decoderbased LLMs are to the IOR.
• We evaluate common Adversarial Training (AT) baselines to illustrate the IOR phenomenon. However, future work would benefit from conducting an in-depth study that includes other recently proposed approaches for adversarial robustness, e.g., contrastive learning based approaches (Rim et al., 2021) and Textual Manifold Defence (Nguyen Minh and Luu, 2022), where all inputs are mapped to a robust manifold. This would help the community understand the extent to which these proposed approaches are improving true robustness and the extent to which they may be unknowingly creating an IOR.
• In Section 5, we used a constant temperature during training for simplicity. However, varying the temperature over the course of training may further increase the effectiveness of high temperature training. Future work will aim to study the impact of temperature schedules.

heading: 8 Risks and Ethics
text: This work presents results on the topic of adversarial training. The contributions in this work encourage the development of truly robust systems and therefore there are no identified ethical concerns.

heading: A Danksin’s Descent Direction for NLP

heading: A.1 Original Theory
text: Latorre et al. (2023) demonstrate that the standard formulation and implementation of AT (as in Equation 3) is potentially flawed. Specifically, solving the inner maximization to find the worst-case adversarial example x̃, can give a gradient direction (in standard stochastic gradient descent approaches), that can in fact increase the robust loss (the new worst-case adversarial example, x̃, with the updated model parameters, θ, can give a robust loss that is greater than before the update step), i.e. worsening the adversarial robustness of the model. This flaw is attributed to the reliance on a single adversarial example, as a parameter gradient step to reduce the model’s sensitivity to a particular adversarial example does not guarantee reduction in the model’s sensitivity to all adversarial examples (the model may now be less robust to other adversarial examples) for a specific sample x. The paper argues that their exist multiple solutions to the inner-maximization for the robust loss and the optimal parameter gradient direction depends on all of those solutions. Thus, Equation 3 can theoretically be adapted to selecting the adversarial example that maximises the gradient direction in each gradient update step for a batch size of K samples,
θi+1 = Φ ( θi,γ ∗ = − ∇θg(x1:K , θi, ˆ̃x1:K)
||∇θg(x1:K , θi, ˆ̃x1:K)||2
) ,
g(x1:K , θi, ˆ̃x1:K) = 1
K
∑
k
L(ˆ̃xk, θi),
ˆ̃xk = argmax x̃∈S∗(θi,xk) ||∇θ=θiL(x̃, θ)| |2,
(11)
where Φ(θ,γ) is the first-order stochastic gradient descent (SGD) algorithm used to update θ as per descent direction γ, e.g. in standard SGD, Φ(θ, γ) = θ + βγ, where β is the step-size (learning rate). Further S∗(θi,xk) represents the set of all maximizers of the robust loss,
S∗(θ,x,G) = argmax x̃:
G(x,x̃,)≤ϵ, x̃∈A
L(x̃, θ). (12)
This set of (robust loss) maximizers, S∗(θ,x,G) can theoretically be infinite. However, if assume we have access to a finite set with M adversarial examples, such that they define,
S∗(M)(θ,x) = {x̃(1), . . . , x̃(M)}, (13)
then Latorre et al. (2023) propose an efficient algorithm termed, Danskin’s Descent Direction (DDi), that provides a method to approximate the steepest direction, γ∗ as though as if we are still selecting from the infinite set S∗ 6, despite only having access to S∗(M). The optimization problem over an infinite set in Equation 11 can be solved by finding an optimal linear combination, α ∈ △M of the gradients of the loss, ∇θg for each different adversarial example. Note that △M defines the M -dimensional simplex (on which α lies). If we let ∇θg(θ, S∗(M)1:K (θ)) be the matrix with columns ∇θg(x1:K , θi, x̃(m)1:K)) for m = 1, . . . ,M , then
γ∗ = − ∇θg(θ, S ∗(M) 1:K (θ))α ∗
||∇θg(θ, S∗(M)1:K (θ))α∗||2 ,
α∗ = argmin α∈△M ||∇θg(θ, S∗(M)1:K (θ))α||22. (14)

heading: A.2 DDi-AT for NLP classification
text: The challenge with NLP is that generating strong textual adversarial examples as per Equation 13 can be extremely slow. Hence to increase speed, we generate adversarial examples in the token embedding space, such that we follow Equation 14, but adapt Equation 11 to,
g(x1:K , θi, ˆ̃ h1:K) =
1
K
∑
k
L(ˆ̃hk, θi),
ˆ̃hk = argmax h̃∈S∗(θi,hk)
∣∣∣|∇θ=θiL(h̃, θ) ∣∣∣ |2, (15)
where hk = {hk,1, . . . ,hk,L} represents the sequence of token embeddings for tokens xk = {xk,1, . . . ,xk,L}. We can create our proxy finite set of maximizers, S∗(M) (Equation 13) by using a computer-vision style Projected Gradient Descent (PGD) attack (Madry et al., 2019) in each token embedding space with initialisations of the PGD attack at different points to create multiple adversarial examples,
S∗(M)(θ,h) = {PGD(1)(θ,h), . . . ,PGD(M)(θ,h), }. (16) In this work we refer to DDi gradients applied to PGD AT as, DDi-AT.

heading: A.3 Gradient Normalization and
text: Overconfidence
It is shown in Table 1 that the use of the DDi gradients with the PGD AT approach (ddi-at) gives rise
6Theorem 3 in the paper justifies the conditions to certify that the approximation is the steepest descent direction
to a highly overconfident model, which is responsible for the IOR. This section aims to determine the route cause of this overconfidence in the DDi gradient update algorithm. Equation 11 indicates that in the DDi gradient update algorithm global gradient normalization is applied. Note that this is different to standard training algorithms where either no normalization is applied or gradient clipping is used where global gradient normalization is only applied if the global gradient norm is larger than a threshold (Pascanu et al., 2012). Table 5 demonstrates that the use of the global gradient normalization in DDi-AT is responsible for the overconfidence and thus IOR. Interestingly, Table 6 reveals that gradient normalization can also induce overconfidence for the standardly trained baseline model.

heading: B Dataset Descriptions
text: We conduct experiments across six standard NLP classification datasets to ensure our findings are robust (statistics summarised in Table 7). Rotten Tomatoes (rt; Pang and Lee, 2005) is a binary sentiment classification task for movie reviews. The Emotion Dataset (emotion; Saravia et al., 2018) categorizes Twitter tweets into one of six emotions: love, joy, surprise, fear, sadness or anger. The remaining three datasets are sourced from the the General Language Understanding Evaluation
(GLUE) benchmark (Wang et al., 2019a).7 The Corpus of Linguistic Acceptability (cola) dataset comprises English acceptability judgments sourced from books and journal articles on linguistic theory. Each instance consists of a word sequence annotated to indicate if it is grammatically correct. The Question-answering NLI (qnli) dataset assesses the task of sentence pair classification, where one sentence is a question and the other a context. The goal is to ascertain whether the context sentence contains the answer to the question. The Microsoft Research Paraphrase Corpus (mrpc) consists of pairs of sentences automatically extracted from online news sources. Human annotations identify if the sentences in each pair are semantically equivalent. Finally we consider the popular AGNews dataset (Zhang et al., 2015), consisting of articles from 2000 news sources classified into one of four topics: business, sci/tech, world or sports. There are a combined 120,000 training samples and 7600 test samples.

heading: C Generative LLMs
text: With the advent of powerful generative Large Language Models, their use has become increasingly ubiquitous. However, we found that such popular generative models were not suitable in this work for the following reasons:
1. The encoder-only models used in this work are state-of-the-art when fine-tuned on classification tasks. Some recent studies (Periti et al., 2024; Zhong et al., 2023) have also found that often (fine-tuned) encoder-only models can be better than the popular generative LLMs for specific classification tasks. We show a comparison of performance in Table 8 below to a SOTA generative LLM (with zero-shot and few-shot prompting). We also present a comparison of the performance from Zhong
7For datasets where the provided test set is not labeled, we used the validation set.
et al. (2023) between BERT-based encoder models and ChatGPT (GPT3.5) on some of the datasets covered in our work (Table 9).
2. As a consequence of the competitive if not superior performance of finetuned encoderbased models on classification tasks, in many industry applications of NLP, such models (BERT-based models) are used extensively due to being light-weight (far fewer parameters than popular generative LLMs), costeffective and still able to perform extremely well at many classification tasks.
3. The adversarial attack and defence literature that we are contributing to focuses on encoder models. Therefore, matching their setting allows us to build upon existing attacks and defences.

heading: D Hyperparameter selection
text: We train the Transformer baseline models using standard hyper-parameter settings (He et al., 2020): initial learning rate of 1e− 5; batch size of 8; total of 5 epochs; 0 warm-up steps;8 ADAMW opti-
8We follow TextDefender (Li et al., 2021a) in setting no warm-up steps. Further, empirically validation accuracy remained the same with warm-up of 50 and 100 steps.
mizer, with a weight decay of 0.01 and parameters β1 = 0.9, β2 = 0.999, ϵ = 1e− 8.
The Adversarial Training (AT) baseline approaches are trained with the same hyperparameters as for the baseline model and AT specific hyperparameters are as described in Li et al. (2021b). The default hyperparameters for each baseline (pgd, ascc and freelb) are: 5 adversarial iterations; adversarial learning rate of 0.03; adversarial initialisation magnitude of 0.05; adversarial maximum norm of 1.0; adversarial norm type of l2; α for ascc is 10.0; and β for ascc is 40.0. For DDi-AT, DDi gradients are applied to the PGD AT approach, with M = 3 gradients and K = 3 PGD iteration steps.

heading: D.1 DDi-AT Ablation
text: The main results report DDi-AT results for DDi gradients applied to PGD AT with K = 3 PGD steps to find each adversarial example (in the embedding space) during training and M = 3 adversarial examples (refer to Section A.2). Table 10 gives the impact on adversarial accuracy (with and with out adversarial temperature calibration) of varying K and M . It appears that with greater iteration steps, K, the model presents a smaller IOR and a greater true robustness as the robustness accuracy does not degrade as much after calibration.

heading: E Further Experiments for Illusion of Robustness and Mitigation

heading: E.1 Experiments on Other Datasets
text: Equivalent results are presented for Twitter Emotions (6 emotion classes) in Table 11 and for the AGNews dataset (4 news classes) in Table 12.

heading: E.2 Experiments on Other Models
text: The illusion of robustness is presented for an overconfident, underconfident and DDi-AT DeBERTa model in the main paper in Table 2. The same trends are observed for other popular Transformerencoder (base) models: RoBERTa (Table 13); and BERT (Table 14).

heading: E.3 Transferability Defence against IOR
text: One might even argue that to expose an IOR, it is unnecessary for an adversary to modify the model with adversarial temperature scaling to find adversarial examples. Instead, adversarial examples can be found for another model (e.g., baseline) and transferred to the target model. This follows from Demontis et al. (2018) where it is shown that similar architectures can be susceptible to the same adversarial examples. We test this hypothesis. It is clear from Table 15 that although the transfer attack from baseline to ddi-at is effective in reducing the adversarial accuracy, it is unable to bring the adversarial accuracy down to the same values as baseline, as achieved by our proposed temperature scaling approaches in Table 3.

heading: E.4 Calibration Error
text: In Table 16 we verify that the calibration approaches are effective in calibrating the models. We report the metrics: Expected Calibration Error (ECE) and Maximum Calibration Error (MCE).
E.5 IOR in AT Approaches
The main results demonstrate that highly miscalibrated systems have an illusion of robustness (IOR), where an adversary’s temperature calibration can mitigate this illusion of robustness. Considering the rotten tomatoes dataset and the DeBERTa model, Table 17 demonstrates that standard AT approaches considered in this work can also suffer from the IOR, when global gradient normalization is included in the training algorithm (Note that Table 6 shows that gradient normalization can be a source of model overonfidence). Nevertheless, Table 18 demonstrates that when global gradient normalization is excluded from the training algorithm, the baseline AT approaches considered in this work no longer present IORs as calibration does not degrade their adversarial accuracy.

heading: E.6 Alternative Calibration Approaches
text: In the main results, temperature calibration was implemented to detect adversarial examples based on two central considerations: 1) Temperature calibration effectively facilitates the adversarial attack search, especially for obviously mis-calibrated models; and 2) Temperature calibration preserves the rank order of logits, thereby ensuring transferability of adversarial examples from the calibrated to the original uncalibrated model. To broaden the analytical scope, alternative calibration techniques are examined. The goal is to assess their potential in mitigating the disruption to the adversarial attack search processes and to determine the potency of the resulting adversarial examples on the uncalibrated model. Binning-based calibration is deemed unsuitable due to its intrinsic non-differentiability, which could prevent the adversarial search process. Hence, the multi-class version of Platt Scaling is
explored as a viable calibration strategy and subsequently contrasted against the benchmark temperature calibration approach from the main results. The performance of the calibration results is shown in Table 19, where it is evident that the Platt scaling approach is far less stable than temperature calibration and can in fact excessively enhance the illusion of robustness.
For automatic calibration, standard training hyperparameters were employed. Specifically, the temperature calibration protocol was set at 5,000 iterations with a learning rate of 0.01. Similarly, the Platt scaling protocol was also designed for 5000 iterations with a learning rate of 0.01. A point to note for practical implementation: adversaries might need to refine calibrator hyperparameters to minimize the Expected Calibration Error (ECE) on a specified validation set. However, ECE determination is nuanced, largely due to its sensitivity to chosen bin widths, as highlighted in Table 16 for instances of underconfidence.

heading: E.7 Extreme Miscalibration Leads to Masked Gradients
text: Section 3 argues that for heavily miscalibrated systems, the ‘gradients’ of the output probabilities with respect to the input are extremely noisy. Therefore, of-the-shelf adversarial attack methods, that use these gradients to select which tokens in the input sequence to attack, receive noisy signals and fail to operate. In this section, we demonstrate that extreme miscalibration does indeed cause noisy gradients for off-the-shelf-adversarial attacks. Note that these noisy gradients are referred to as obfuscated gradients or gradient masking by Athalye et al. (2018).
We consider two systems: the standard baseline
system from the main paper and the heavily miscalibrated, overconfident system, ↑conf in the main paper. Experiments are on the rt dataset and we consider specifically the PWWS attack and Textfooler attack. These off-the-shelf adversarial attack approach rank all tokens wi in the input sequence x by their influence on the output of the model (N.B. this is considered an approximation for the gradient of the output with respect to each input token). The PWWS attack refers to this influence as saliency, whilst the Textfooler attack calls it importance. To assess the impact of heavy miscalibration on the rank ordering, Table 20 reports the Spearman Rank Correlation between the rank of all input tokens (in the first iteration of the attack) as per the two models: baseline and ↑conf. The average correlation and standard deviation are given over the entire dataset. The average rank correlation is 0.28 for PWWS and 0.29 for Textfooler, which is very low and demonstrates that by simply having heavy miscalibration there is a significant impact on the attack mechanism. Further, the standard deviation is also large, suggesting that for many input sequences, the correlation is even lower.

heading: F Further Experimental Results for High Temperature Training for Genuine Robustness
text: Here ST∗ will refer to a standardly trained baseline model (Equation 2), with gradient normalization during training whilst AT will refer to an adversarially trained model (Equation 3), as indicated by the naming convention given in Table 21.

heading: F.1 Class Margin Explanation
text: The success of high temperature training for adversarial robustness can perhaps be explained by considering the size of the class margin (Robey et al., 2023). A high temperature smooths the probability distribution across classes, such that the probabilities of the different classes are closer together. To minimize the cross entropy loss during the training, the model’s parameters learn to compensate for this smoothing by pushing the logits of the different classes further apart (we see this in Figure 3, where the range of logits substantially increases with higher training temperatures). Intuitively, this can be viewed as increasing the distance to the class boundary in the logit space and thus making it more difficult for an adversarial attack to change the predicted class, giving rise to the observed increase in adversarial robustness. Future work will aim to rigorously understand and explain the observed robustness gains of training with a high temperature.

heading: F.2 High Temperature Standard Training
text: For each dataset, Figure 4 presents the change in clean and adversarial accuracy of a standardly trained baseline ST model trained as per a standard training objective (Equation 2), with different temperatures T used during training. We present the detailed breakdown of the clean and adversarial accuracies for each training temperature, for each adversarial attack and each dataset, in Appendix F.5.
We first observe a general increase in adversarial accuracy (robustness) with the training temperature and then a decrease in the accuracy with extremely
large training temperatures,9 across all datasets. In some datasets (e.g., qnli and mrpc) there is a slight decrease in robustness before the sudden rise in robustness. Nevertheless, there exists a consistent robustness profile for each dataset, where robustness peaks at similar temperatures for all tested adversarial attack types (bae, tf, pwws, and dg). This is particularly useful, as a model developer, with access to only one form of adversarial attack, can tune the training temperature for optimal robustness on that specific attack form, yet be confident that the robustness gains will also transfer to the other unseen/unknown attack forms.
A further observation is that increasing temperature can lead to a small drop (between 1% and 4%) in clean accuracy. This is perhaps expected as the model can be viewed as being trained in a mode further from the optimal hyper-parameter setting. However, across all the datasets, the optimal temperature (aligned with the peak in adversarial accuracy) results in a maximal drop in clean accuracy of 1% (apart from for the emotion dataset). Given the gains in adversarial accuracy can be between 4% and 14%, this trade-off for clean accuracy can be acceptable. Further, a model developer can choose to operate at a different operating point, by selecting a training temperature that gives a smaller drop in clean accuracy (and settle for a less significant gain in the model robustness).
9The drop in robustness for extremely large training temperatures may be attributed to large temperatures excessively smoothing the predicted probability distribution during training, which makes it too challenging for the model to learn, as is reflected by the significant decrease in clean accuracy.

heading: F.3 High Temperature Adversarial Training
text: Here we explore the impact of combining the high temperature training approach with popular NLP AT methods. We consider four popular adversarial training approaches: dg-aug∗, PGD∗, FreeLB∗ and ASCC∗. Table 22 and Table 23 give the baseline ST∗ results and AT results combined with the tem-
perature training approach on the rt and emotion datasets, respectively.
Although more significant for rt than emotion, for both datasets, combining with the high training temperature approach improves the adversarial accuracy for all adversarial attack forms (bae, tf, pwws, and dg) for the different adversarial training approaches PGD∗, FreeLB∗, and ASCC∗. This demonstrates that high temperature training is complementary with such adversarial training approaches and thus consistently encourages a gain in robustness. Interestingly, we observe that for dg-aug, high temperature training is able to consistently improve adversarial accuracy for bae, tf, and pwws adversarial attacks, but can cause a drop in adversarial accuracy for the dg attack. It should be emphasized that dg in this context behaves as a seen attack form, as the training uses augmentation with dg adversarial examples, whilst the other attacks (bae, tf, and pwws) can be considered unseen attack forms that the model developer has no knowledge of during training. This suggests that for augmentation-based NLP adversarial training approaches, a high training temperature does not necessarily increase robustness to seen attack forms, but is successful in boosting robustness to unseen attack forms.

heading: F.4 Transferability of High Temperature Training
text: It is shown that training with a high temperature leads to a consistent gain in adversarial robustness to unseen adversarial attack forms. However, an adversary may attempt to exploit attack transferability when looking to attack the target model trained with high temperature. To explore this notion of a transfer attack, with the rt dataset, Table 15 shows the impact of finding adversarial examples for the source baseline ST∗ model and assessing their efficacy on the target baseline ST∗ ⊕ T model. It is evident from the significant increase in the adversarial accuracy for all the attack forms (bae, tf, pwws, and dg), that a transfer attack is not able to degrade the observed robustness gains for models trained with high temperatures.

heading: F.5 Detailed Performance Breakdown
text: Figure 4 presents the adversarial accuracy of baseline ST∗ models trained with different training temperatures. In this section, for reference, we provide the detailed breakdown (average across 3 seeds and standard deviation) of performances for the different training temperatures for each dataset: rt (Table 25), emotion (Table 26), cola (Table 27), qnli (Table 28), and mrpc (Table 29). These results are given for the DeBERTa model as in the main paper.
In Table 30, we further include results on a 6th dataset AGNews (Zhang et al., 2015), where there are four news classes, 96k training samples, 24k validation samples and 7.6k test samples. For this dataset, it can be observed that a high training tem-
perature is not a successful method unless a fraction
of the dataset (10k training samples) is used during training. Future work is necessary to understand the nature of this specific dataset or other similar datasets that led to such a different behaviour for the temperature training approach.

heading: F.6 Reproducing with Other Models
text: The main paper presents results using the DeBERTa model. Here we repeat the core experiments on other popular baseline models: BERT (Table 32) and RoBERTa (Table 33). The results here are presented for the rt dataset.

heading: F.7 High Temperature Training GradNorm Ablation
text: We find that including gradient normalization during training with a high temperature is beneficial in preventing a decrease in clean accuracy, whilst maintaining the gains in genuine adversarial robustness. This is demonstrated in the results in Table 34, where calibration is used at test-time to ensure there is no IOR.
F.8 IOR From High Temperature Training In the main paper we present high temperature training as an effective method to induce genuine adversarial robustness. Here we demonstrate the need to apply the test-time calibration methods of Section 4 to ensure there is no IOR. From Table 35, we see an approximately two-fold increase in adversarial accuracy when no calibration is applied before evaluation.

heading: F.9 Other Ablations
text: Augmentation based adversarial training approaches, such as dg-aug∗ in the main paper, have twice as many training steps (due to there being double the training set size). To match the standard training setting, in Table 36 we evaluated the training with high temperature approach combined with dg-aug∗ at half the number of training steps. Similarly, in Table 37 we consider the inverse setting, where we double the number of training iterations for the baseline∗ model (in standard training), as well as linearly scaling the learning rate scheduler across the increased number of iterations.

heading: G Algorithms

heading: G.1 Method for Temperature Scaling Optimization to Mitigate IOR
text: d e f o p t i m i z e _ t e m p ( s e l f , f a c t o r = 1 0 ) : ’ ’ ’ r e t u r n t e m p e r a t u r e o p t i m i z e d t o be s u c c e s s f u l i n dg a t t a c k ’ ’ ’
adv_temp = 1 acc = s e l f . e v a l _ a t t a c k ( adv_temp )
# check whe the r we need t o i n c r e a s e o r d e c r e a s e temp t e s t _ t e m p _ a c c = s e l f . e v a l _ a t t a c k ( adv_temp * f a c t o r ) i f t e s t _ t e m p _ a c c < acc :
l e f t = adv_temp r i g h t = 1 e6 # assumes t h i s a s a maximum
e l s e : l e f t = 1e −10 # 0 c a u s e s f l o a t i n g p o i n t e r r o r s r i g h t = adv_temp
# s e a c h f o r o p t i m a l temp ( minima adv acc ) u s i n g B r e n t a l g o r i t h m ( f a s t e r c o n v e r g e n c e o f go ld en s e c t i o n a l g o r i t h m ) opt_ temp = s c i p y . o p t i m i z e . b r e n t ( s e l f . e v a l _ a t t a c k , b r a c k =( l e f t , 0 . 5 * ( l e f t + r i g h t ) , r i g h t ) , m a x i t e r =10)
r e t u r n opt_ temp

heading: G.2 Base Class Definition with High Temperature Training for Genuine Robustness
text: c l a s s B a s e C l a s s i f i e r ( nn . Module ) : d e f _ _ i n i t _ _ ( s e l f , model_name = ’ b e r t −base − uncased ’ , n u m _ l a b e l s =2 , p r e t r a i n e d =True , t e m p e r a t u r e = 1 ) :
s u p e r ( ) . _ _ i n i t _ _ ( ) s e l f . model_name = model_name s e l f . t e m p e r a t u r e = t e m p e r a t u r e i f p r e t r a i n e d :
s e l f . model = A u t o M o d e l F o r S e q u e n c e C l a s s i f i c a t i o n . f r o m _ p r e t r a i n e d ( model_name , n u m _ l a b e l s = n u m _ l a b e l s ) s e l f . t o k e n i z e r = Au toToken i ze r . f r o m _ p r e t r a i n e d ( model_name )
e l s e : c o n f i g = AutoConf ig . f r o m _ p r e t r a i n e d ( model_name , n u m _ l a b e l s = n u m _ l a b e l s ) # r e t u r n s c o n f i g and n o t p r e t r a i n e d w e i g h t s s e l f . model = A u t o M o d e l F o r S e q u e n c e C l a s s i f i c a t i o n . f r o m _ c o n f i g ( c o n f i g ) s e l f . t o k e n i z e r = Au toToken i ze r . f r o m _ p r e t r a i n e d ( model_name ) s e l f . c o n f i g = AutoConf ig . f r o m _ p r e t r a i n e d ( model_name , n u m _ l a b e l s = n u m _ l a b e l s )
d e f f o r w a r d ( s e l f , i n p u t _ i d s =None , a t t e n t i o n _ m a s k =None , i n p u t s _ e m b e d s =None ) : l o g i t s = s e l f . model ( i n p u t _ i d s , a t t e n t i o n _ m a s k = a t t e n t i o n _ m a s k , i n p u t s _ e m b e d s = i n p u t s _ e m b e d s ) [ 0 ] l o g i t s = l o g i t s / s e l f . t e m p e r a t u r e r e t u r n l o g i t s

heading: Future Work
text: Future work is necessary to understand the nature of this specific dataset or other similar datasets that led to such a different behaviour for the temperature training approach