A Curious Case of Searching for the Correlation between Training Data and Adversarial Robustness of Transformer Textual Models (2024)

Abstract: Existing works have shown that fine-tuned textual transformer models achieve state-of-the-art prediction performances but are also vulnerable to adversarial text perturbations. Traditional adversarial evaluation is often done only after fine-tuning the models and ignoring the training data. In this paper, we want to prove that there is also a strong correlation between training data and model robustness. To this end, we extract 13 different features representing a wide range of input fine-tuning corpora properties and use them to predict the adversarial robustness of the fine-tuned models. Focusing mostly on encoder-only transformer models BERT and RoBERTa with additional results for BART, ELECTRA, and GPT2, we provide diverse evidence to support our argument. First, empirical analyses show that (a) extracted features can be used with a lightweight classifier such as Random Forest to predict the attack success rate effectively, and (b) features with the most influence on the model robustness have a clear correlation with the robustness. Second, our framework can be used as a fast and effective additional tool for robustness evaluation since it (a) saves 30x-193x runtime compared to the traditional technique, (b) is transferable across models, (c) can be used under adversarial training, and (d) robust to statistical randomness. Our code is publicly available at https://github. com/CaptainCuong/RobustText_ACL2024.

heading: 1 Introduction
text: Pre-trained transformer models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have recently demonstrated superior performance in various downstream NLP classification tasks. However, they are also vulnerable to adversarial text attacks (Sun et al., 2020; Jin et al., 2020), which aim to generate adversarial examples by applying imperceptible perturbations to input texts such that the resulting examples cause a target text classifier to make incorrect predictions (Goodfellow et al., 2015). This makes the robustness of
transformer models against adversarial attacks crucial, especially in high-stake domains such as banking, law, and content moderation (Rodríguez Cardona et al., 2021; Sanz-Urquijo et al., 2022; Ashley, 2019) where susceptibility to such attacks can result in detrimental consequences such as giving out high-risk loans, wrongful indictments and enabling hate speech and disinformation. Thus, ML practitioners must ensure their models are robust against text perturbations before deploying them to the public. To achieve this, existing works have proposed several ways to benchmark and analyze the robustness of perturbations of transformer models. In general, they often take a model-first approach–i.e., assuming that the model itself, such as its architecture or loss function formulation, is mainly responsible for its adversarial vulnerability and aiming to understand what kinds of changes in a model would shift its adversarial robustness (Mao et al., 2022; Zhang et al., 2022b,a; Han et al., 2024). Particularly, this approach iteratively makes a controlled alternation in the model–e.g., changing the architecture type, experimenting with novel attention layers, adding noises to the embeddings, etc., and then fine-tune the new model on the same finetuning dataset, followed by generating adversarial examples and benchmarking the model using the generated examples. Although this model-first approach has resulted in several useful insights in practice, it assumes that adversarial robustness can only be evaluated only after a model has already been fine-tuned and adversarial examples
13475
have been generated. This approach tends to isolate the effects of the fine-tuning dataset and hence does not provide many insights on how such training data affect a model’s robustness, such as “how the distribution of fine-tuning texts’ embeddings and labels affect a model’s robustness?”, “how do the unique vocabulary and lengths of fine-tuning texts correlate with a model’s robustness?”, etc. Exploring the relationship between fine-tuning data and model robustness promises to open up data-centric research directions to improve model robustness, introduced in detail in Sec. 8.5. Therefore, in this paper, we propose a global interpretation framework, as shown in Fig. 1, to investigate whether there is a strong direct correlation between finetuning data and model robustness and interpret the features of fine-tuning data that have the greatest influences.
To this end, we take a different approach from model-first analysis and propose to analyze the adversarial robustness from a data-first approach. To do this, we extract 13 different features that comprehensively capture several important properties of not individual training examples but of the fine-tuning dataset as a whole. Then, via regression analysis, we attempt to correlate them with the adversarial robustness of the models to be fine-tuned on the dataset measured as the average attack success rates (ASRs) of 4 representative text perturbation methods on an unseen test set.
To demonstrate one application of such novel analysis, we also try utilizing our interpretation framework to estimate the adversarial robustness of transformer classifiers even before they are fine-tuned and without the need to generate adversarial examples, only by analyzing their fine-tuning dataset. This approach is 30-193 times faster than the traditional method, as shown in Fig. 1.
Contributions of our paper are as follows. • To the best of our knowledge, this is the first
paper to analyze and investigate a comprehensive correlation between fine-tuning data and model robustness with a taxonomy of 13 dataset-level indicators, • As an application, we demonstrate that this novel analysis also enables a Random Forest predictor to effectively evaluate the adversarial robustness of BERT and RoBERTA with the averaged mean absolute errors (MAEs) ranging in 0.025–0.176 for both in-domain and out-of-domain prediction, • Our framework can also be used as a fast tool to evaluate the robustness of transformer-based text classifiers, which (i) is 30x-193x faster than the usual procedure, (ii) can be used under an adversarial training setting, (iii) transferable between transformer-based models, and (iv) robust to statistical randomness.

heading: 2 Problem Formulation
text: We propose to develop a function Gfθ (D) parameterized by θ that can effectively approximate the adversarial robustness of a pre-trained transformer-based classification model f when it is fine-tuned by an input training dataset D. In other words, Gfθ (D) estimates the difference between predictions on examples of a clean, unseen test set D∗ (D ∩D∗=∅) that is drawn from the same distribution with D and is sufficiently large and on their corresponding adversarial examples. Let’s denote R(f,D,D∗) such adversarial robustness, we have:
R(f,D,D∗) = 1|D∗| ∑
x∈D∗ d(fD(x), fD(x+ δ)),
(1) where δ is an adversarial perturbation and d(·) is a metric such as attack success rate as often adopted in existing literature. Since D∗ is sufficiently large, we assume to observe only a small variance among the adversarial robustness measured on different randomly sampled D∗, drawn from the same distribution as D. Hence, we simplify the adversarial ro-
bustness to be estimated as R(f,D)≈R(f,D,D∗) with any arbitrary D∗.
To train Gfθ , which is specific to the model type f such as BERT or RoBERTa, we can then formulate this as a regression prediction problem and minimize the L2 loss for an arbitrary fine-tuning dataset D as follows.
minimize θ LD = ||Gfθ (D)−R(f,D)||22, (2) where LD is then the loss for one fine-tuning corpora D. To effectively train G that can approximate adversarial robustness for any unseen fine-tuning corpus, we will need to optimize such loss function not for one but N training corpus Q={Q1,Q2, ..QN}, resulting in the final objective with mean square error (MSE) loss function:
L = 1 N
∑ Q∈Q LQ (3)
In this work, we want to evaluate Gfθ in two prediction scenarios, namely interpolation and extrapolation. In (1) interpolation or in-distribution evaluation, we want to validate Gfθ on a fine-tuning corpus that is similar to one of the corpus included in Q that the model Gfθ has been trained on. This is also the standard evaluation setting in a typical machine learning problem. In (2) extrapolation or out-of-distribution evaluation, we want to validate Gfθ on a dataset that is very different from corpus included in Q–e.g., training on sentiment classification datasets and evaluating on a non-sentiment dataset such as Q&A or fakenews detection.

heading: 3 Method
text: Overview. Our goal is to create a regression dataset that includes (1) the features of the several smaller datasets and (2) their adversarial robustness–i.e., attack success rate (ASR) of a transformer-based model f fine-tuned on each of them. Then, we can use regression ML algorithms to predict such adversarial robustness and then analyze the influence of those features on the robustness of the model. Fig.
Algorithm 1 Data Preparation Pseudo-code Input: Set of text corpusD, training sample size N Output: Final datasets Q to be used for training/validation/testing Initialize: Q ← ∅, i← 0 1: for corpus d inD do 2: Randomly sample Sdtest∈S 3: end for 4: for i in [1..N] do 5: Randomly a sample corpus d fromD. 6: Randomly sample Sitrain, S i val from d such that
7: Sitrain∩Sdtest=∅; Sival∩Sdtest=∅; Sitrain∩Sival=∅ 8: Q ←Q ∪ (Sitrain, Sival, Sdtest) 9: end for
10: return Q
2 illustrates the entire framework of five sequential phases.

heading: 3.1 Phase 1: Data Preparation.
text: Our dataset preparation pipeline starts with set D, which includes 9 diverse and publicly available NLP classification corpus. Different from a typical ML problem, in this work, each training example is a dataset and not a single text. Hence, we proposed a data splitting strategy as shown in Algorithm 1. For each text corpus d ∈ D, we first randomly sample a test set of size K to be used for calculating the attack success rate–i.e., adversarial robustness, as prediction labels in Phase 3 (Fig. 2) (Alg. 1, Ln. 1–3). To sample one instance in our final dataset, we first randomly pick a text corpus d ∈D and randomly sample from it a small train and validation set of size 9 ∗ K and K to achieve a 9:1 ratio between train and validation set, then pair them with the fixed test set previously sampled for d (Alg. 1, Ln. 5–7). We repeat such process N (Alg. 1 Ln. 4–8) times to sample N total triplets of non-overlapping train, validation, and test sets.

heading: 3.2 Phase 2: Feature Engineering.
text: This phase extracts a total of 13 features that capture different aspects of each fine-tuning dataset (Fig. 2) for robustness prediction afterward. The features are categorized into 4 aspects, namely Embedding Distribution, Label Distribution, Weak Model’s Learnability, and Dataset Statistics. Within each aspect, we develop several quantitative predictive indicators as summarized in Fig. 3. Our goal is to investigate the influence of these features on the adversarial robustness of the fine-tuned transformer-based models.
• Embedding Distribution. Inspired by (Yu et al., 2018) which shows the influence of input space on the adversarial robustness of transformer-based models, we propose to use several indicators that
summarize how closely the included texts are distributed in the embedding space. They are (1) mean distance among classes (MD), (2) Fisher’s discriminant ratio (F), (3) Calinski-Harabasz Index (CHI), (4) Davies-Bouldin Index (DBI) and (4) number of clusters (# of clusters). To do this, we use the Universal Sentence Encoding (Cer et al., 2018) to encode the sentences in each fine-tuning dataset into embedding vectors. • Label Distribution. Fine-tuning datasets with a skewed or peaked label distribution can lead to biased predictions, especially for complex transformer-based models that are prone to overfitting to the majority class and lead to poor generalization. Thus, we adopt several indicators to quantify the skewness and peakedness of input labels, including (1) Pearson Median Skewness (PMS), (2) Kurtosis (Kurt). Furthermore, we include the number of labels as a feature so that robustness prediction may be tailored to specific tasks. • Surrogate Model’s Learnability. Inspired by (Zhang et al., 2022b), we assume that the predictive performance of a weak model on the finetuning dataset can also inform about potential predictive biases that will also transfer to transformerbased models. We coin this feature Misclassification Rate (MCR). Intuitively, a surrogate model with good predictive performance makes it more likely that a fine-tuned transformer model will also achieve similar or even better generalization. Conversely, a surrogate model with poor predictive performance provides a quick sanity check for potential biases in the fine-tuning dataset–e.g., inconsistent, noisy, or skewness in labels, which will eventually lead to poor generalization of the fine-tuned transformer-based model. Particularly, we use a character-based CNN classifier that is smaller than a typical transformer-based model as the surrogate model. Such a model is more computationally efficient during training and inference, and more powerful than traditional ML classifiers such as Naive Bayes or Decision Tree. • Token-Based Statistics. The length of input text and typos affect the robustness of the transformerbased model (Jia and Liang, 2017; Sun et al., 2020). Hence, we examine the influence of some summary statistics of the dataset on the robustness of the transformer-based model, namely the (1) average number of tokens (avg. # tokens), (2) the minimum number of tokens (min # tokens), (3) the maximum number of tokens (max # tokens), and (4) the number of unique tokens (# unique tokens).
These statistics reflect the types of texts where a fine-tuned transformer-based model has observed and thereby informs its performance when dealing with unseen, adversarial examples. 3.3 Phase 3: Extract Adversarial Robustness as Regression Labels. Phase 3 aims to predict the adversarial robustness of the model after fine-tuning the datasets prepared in Phase 1 (Fig. 2). After extracting the features of the fine-tuning data Strain, we fine-tune a transformer-based classifier f(·) on the training dataset Strain and validate on Sval. Then, the adversarial robustness of f(·) will be extracted by averaging the attack success rates of four text perturbation methods used to attack f(·). They include one character-level attacker DeepWordBug (Gao et al., 2018) and three word-level attackers BERTAttack (Li et al., 2020), PWWS (Ren et al., 2019), and TextFooler (Jin et al., 2020). These four attackers are both standard benchmark text perturbation methods in the literature and represent diverse attack methods in practice.

heading: 3.4 Phase 4: Regression Analysis through Adversarial Robustness Estimation.
text: Phase 4 aims to train a regression classifier Gfθ (·) that inputs the engineered features of a fine-tuning dataset and predicts the adversarial robustness, measured by ASR, of a corresponding fine-tuned transformer-based architecture f , for f is either BERT or RoBERTa (Fig. 2). Phase 1, 2, and 3 have provided us with a tabular training dataset total of N data points, each of which contains the engineered features of each small fine-tuning dataset Strain and its corresponding ASR on unseen Stest of a fine-tuned transformer-based model. We adopt three popular ML models for predictor Gfθ , namely Gradient Boosting, Linear Regression, and Random Forest. These predictors are computationally efficient and achieve competitive predictive performance compared to advanced deep models on tabular datasets (Grinsztajn et al., 2022).

heading: 3.5 Phase 5: Evaluation and Analysis.
text: To evaluate and gain meaningful insights into the trained predictor Gfθ , we report results and carry out analyses as follows. • Runtime: We compare the runtime of our framework over the conventional adversarial robustness measurement approach which requires both finetuning a model and generating adversarial examples. • Prediction Performance: We evaluate our framework under two inference scenarios, namely interpolation and extrapolation. Interpolation is the process of estimating ASRs within the domain of observed data points while extrapolation, conversely, is a prediction of ASRs on out-of-domain data. Although extrapolation evaluation is more challenging, it is more practical as we want to evaluate how well our regression predictor Gfθ performs on a corpus that it does not see during training. • Feature Analysis: We adopt the Permutation Feature Importance and Accumulated Local Effects technique to estimate and analyze the influence of engineered features on Gfθ ’s ASR predictions–i.e., how their values correlate with the predicted adversarial robustness, and their importance rankings. • Prediction under Adversarial Training: We evaluate our adversarial robustness predictor under adversarial training setting (Goodfellow et al., 2015). Adversarial training is a popular technique that helps improve a model’s robustness by training a model with additional adversarial perturbations. This means that a good predictor Gfθ is expected to consistently output smaller ASRs, and hence informing a more robust model, under this setting. • Prediction Transferability: Transformer-based models are well-known for robustness transferability. To put it another way, their robustness is quite the same. Thus, we expect our robustness predictor to work on other untrained transformer models at an acceptable level. • Prediction Consistency: Since ASR is a statistical metric, randomness is inevitable. We examine whether these statistical labels affect the performance of our robustness prediction.

heading: 4 Related Work
text: Adversarial Attacks in NLP. The general framework for adversarial attacks on a sentence includes two steps: (1) choosing which words in the sentence a target text classifier is most vulnerable to and (2) replacing them with a candidate such that the prediction label crosses the original prediction.
Thus, most of the attack methods differ in how they come up with new replacements, with the majority of them using word-level perturbation strategies such as via word-substitution ( (Li et al., 2020; Jin et al., 2020; Ren et al., 2019)) or character-level attack such as via swapping and deleting characters within an original word ( (Gao et al., 2018; Li et al., 2019)). While one can choose a set of random words in a sentence to perturb, existing works also propose several optimization schemes such as greedy search or genetic algorithm to select the optimal words to perturb and also their replacements. Although these mechanisms help maximize the changes in the target classifier’s behaviors while still preserving the sentence’s original semantic meaning, the fact that they work with discrete NLP domain induces a substantial additional computational cost due to the need to continuously ping the target model for fine-tuning their perturbations, often on one token at a time. Interpreting the Adversarial Robustness of Models. (Zhang et al., 2022a) claimed that a lack of model robustness is caused by non-robust features. As a result, they improved text classification models by including a bottleneck layer in their architectures to eliminate the effects of low-quality features. Moreover, (Han et al., 2024) attributed the non-robust transformer models to outliers, and presented a resilient framework called transformerRKDE by replacing the dot-product attention with attention deriving from robust kernel density estimators. In addition to these works that focus more on model architectures, works such as (Jia and Liang, 2017) focused more on drawing the relationship between specific linguistic patterns and the adversarial robustness, but only on unseen test sentences during inference. Distinguished from these works, we emphasize and analyze the role of the fine-tuning dataset during model training on adversarial robustness and isolate the effects of the model architecture and inference inputs.
5 Experiment Setup1
Datasets. We include 9 diverse publicly available classification corpus in the set D, namely AG News (Zhang et al., 2015), Amazon Reviews Full, Amazon Reviews Polarity (Keung et al., 2020), DBpedia (Lehmann et al., 2015), Yahoo Answers, Yelp Reviews Full, Yelp Reviews Polarity (Zhang et al., 2015), Banking77 (Casanueva et al., 2020),
1We refer the readers to the supplementary materials for implementation and reproducibility details.
and Tweet Eval Review (Barbieri et al., 2020) Target Models. We focus on studying the adversarial robustness of encoder-only transformer language models (LM) BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), which are often the standard baseline for text classification tasks. Moreover, we also report experiment results on decoder-only LM GPT2 (Radford et al., 2019), encoder-decoder LM BART (Lewis et al., 2020) transformer models, and ELECTRA (Clark et al., 2020), which is an encoder-only model but trained with an additional discriminator. Interpolation and Extrapolation Evaluation. For interpolation, we employ overlapped k-fold crossvalidation of 80%:20% split and with k=200 to train and validate our framework on Q. For extrapolation, data points are split based on their original dataset. For example, we have a list of datasets Dl and split them into three sets D1, D2, and D3 such that Dl =
⋃{D1,D2,D3} and ∅ = ⋂i,j∈{1,2,3} and i ̸=j{Di,Dj} for training, validation and testing purposes, and to be more specific, |D1| = 5, |D2| = 2, and |D3| = 2. The train, val, and test sets of the extrapolation prediction include the data points respectively sampled from datasets in D1, D2, and D3. With this strategy, the train,
val, and test sets have different contexts and ranges which are useful for extrapolation testing purposes. Evaluation Metrics. We employ standard evaluation metrics of regression prediction problems including root mean square error (RMSE), R squared (R2), mean absolute error and percentage error (MAE, MAPE), explained variance score (EVS).
6 Results, Analyses, and Discussions2
Finding 1: Fine-tuning data have a strong correlation with Model Robustness. Table 1 shows the results of ASRs under both interpolation and extrapolation settings. Random Forest predictor achieves the best results, followed by Gradient Boosting and Linear Regression in most cases except for extrapolation prediction on RoBERTa. Regarding to MAE, Random Forest scores are as low as 0.025 and 0.037 for interpolation prediction on BERT and RoBERTa. It also achieves reasonable extrapolation prediction with MAE of only around 0.045 and 0.044 on BERT and RoBERTa. Requiring only one initial training, our framework shows to be effective at benchmarking the adversarial robustness of BERT and RoBERTa with only a lightweight Random Forest predictor. Finding 2: Embedding distribution and tokenbased statistics features are among the most influential indicators of adversarial robustness. Finding 1 demonstrates that our engineered features are highly informative about the fine-tuned model’s robustness. Fig. 5 further summarizes the order of influence of each feature in the case of Random Forest, which is the best regression predictor we found in Finding 1. We only show features that have an average influence score twice greater than their variance.
Overall, embedding distribution and token-based statistics are the two groups of most influential features. In interpolation, CHI, FR, and # of unique tokens have a significant influence on the adversarial robustness of BERT (Fig. 5a), whereas such feature set of RoBERTa also includes MD (Fig. 5b). We also observe a similar pattern in predicting the adversarial robustness of BERT (Fig. 5c) and RoBERTa (Fig. 5d) in extrapolation prediction. Finding 3: CHI, FR, # unique tokens and # classes have clear correlations with ASR. Fig. 6 provides the correlation between notable features discussed in Finding 2 and how they influence the ASR prediction on average. These results show
2We refer the readers to Section 8 for more details.
that the distances among classes in the embedding space–i.e., class separation sub-group (Fig. 3), are highly indicative of the adversarial robustness of the fine-tuned models. When the embedding among classes disperses in the space and is not concentrated, FR feature has a low value and CHI feature has a high value, which correlates to a greater robustness against adversarial examples. The opposite also holds as well, as illustrated in Fig. 4. Furthermore, token-based statistics of the dataset such as # of unique tokens and # of classes also contribute to the influence on adversarial robustness. As # of classes increases, the embedding space becomes denser and clusters among prediction labels show more overlaps, the less robustness observed in the fine-tuned models. Moreover, a large # of unique tokens often informs a diverse fine-tuning dataset, which makes the pre-trained transformer-based models more generalizable and hence more difficult to attack. Error Analysis. 3 The top three error-inducing features in ASR prediction are DBI, # of classes, and MR. Unlike MD, FDR, and CHI, DBI lacks robustness and fails to accurately represent embedding concentration because it is based on the distance to the nearest cluster compared to the original cluster. The increasing # of classes makes the decision boundary more complicated and greatly affects ASR, but when a saturation threshold is crossed, this phenomenon no longer occurs. This explains our observation that although there is a strong correlation between the # of classes and ASR shown in Finding 3, our predictor has poor performance when the # of classes is greater than 10. CNN calculates the misclassification rate (MR) of the surrogate model, leveraging its focus on local
3We refer the readers to experiment setup for error analysis in supplementary materials.
structures, whereas the transformer model relies on global dependencies. Consequently, in some cases, while the MR of CNN may vary significantly, the transformer’s adversarial robustness remains relatively consistent.

heading: 7 Another Tool for Robustness Analysis
text: The proposed approach saves significant runtime in estimating adversarial robustness with reasonable accuracy. The advantage of our method lies in skipping the adversarial example generation of four attacking methods used for evaluating adversarial robustness, making our inference time 30x–193x faster than the traditional approach when evaluating adversarial robustness on 100 examples (Fig. 7). For example, when inferring the robustness of a transformer model fine-tuned on 900 test samples shown in Fig. 7, our method takes 153.02s including feature extraction (152.48s) and robustness inference by Random Forest (0.18s). Conversely, the traditional method takes 4807.83s including Finetuning PLM (130.51s) and Adversarial Generation (4677.32s). As a result, our proposed framework is 31.4 times faster.
Thanks to the accurate predictions discussed in Finding 1 of Section 6 and fast runtime speed, our framework can be used as a additional tool for quickly pinpointing adversarial robustness. Generalization between transformer-based text classifiers. We perform robustness predictor training on 3 models and test on the remaining one. The
results of Table 2 show that R2 and RMSE range from 0.62-0.81 and 0.06-0.10, respectively. This indicates the transferability between transformerbased text classifiers of our robustness predictor. Support adversarial training. We perform adversarial robustness prediction ability of the best performing Random Forest predictor in the case of adversarial training. Specifically, we predict the robustness of BERT and RoBERTa on a fine-tuning dataset that includes both original and perturbed texts. Fig. 8 summarizes the results. Our Random Forest framework consistently outputs lower ASRs, thus informing more robust BERT and RoBERTa models under both interpolation and extrapolation. This shows that our engineered features can capture nuanced changes in the text embedding space of the fine-tuning datasets and inform the Random Forest predictor to respond accordingly even without observing any adversarial examples during training. Robustness to statistical randomness. Because ASR is a statistical metric; inevitably, the robustness predictor itself is not robust to randomness. Evaluations for the prediction of Random Forest in Table 1 also show its consistency in that results just vary from 0.00-0.01 and 0.00-0.03 in interpolation and extrapolation settings.

heading: 8 Further Discussion

heading: 8.1 Confounding Factors
text: This work assumes two main factors affecting adversarial robustness: training data and a model’s architecture. Most existing works analyze a model’s adversarial robustness after it has been fine-tuned on the training data, thus mixing the two factors,
making the analysis of how much training data (alone) correlates with adversarial robustness difficult (Xu et al., 2019; Zheng et al., 2023). Therefore, in this work, we separate the two factors and focus only on using training data as the input to measure the adversarial robustness for a specific model’s architecture without fine-tuning such a model (fixed second variable–i.e., model architecture, and vary the first variable–i.e., training data).
Another possible confounding factor that affects both the training data and a model’s adversarial robustness is the label and data curation process. For example, some malicious actors might intentionally poison the training data to affect the training features, labels, and a model’s adversarial robustness. We assume that all of our training data is clean (we use official, published sources of all datasets) and we do not expect any confounding factors might affect our analysis.
There might be other confounding factors that we might overlook. Within the pioneering nature of our work on this research topic, we hope to see future works that explore those confounding factors, for example, from the perspective of causality.

heading: 8.2 Contextual Features
text: More nuanced, contextual, and semantic features of the training data will be useful for adversarial robustness prediction. In fact, our proposed framework already leverages the more nuanced aspects of fine-tuning data such as context and semantics by representing the original text using the Universal Sentence Encoder (USE) during the feature engineering step in Sec. 3.2. However, there might be other more complex features that would only be captured using complex neural network models, which then might inhibit interpretability and increase the runtime. For instance, if we aim to duplicate our training process but opt for an efficient neural network like CNN, we would have to train it with training examples. Each of them would consist of 900 sentences, where the average sentence length is 60 tokens. This results in an input size as large as 900 × 256 for each training example, with 256 as the word embedding size. This can be also considered as an image of 900× 256 dimension, which is much larger compared to images of 28×28 dimension in the MNIST dataset. If we are not using CNN but a simple 1-hidden layer neural network with only 10 neuron units, we would need over 138M model parameters. We would expect a much longer runtime compared to our approach.

heading: 8.3 Relationship between compactness of input embedding and adversarial robustness
text: (Pal et al., 2024) introduces the theoretical concept of input data concentration, demonstrating that a robust classifier emerges when the input embedding is concentrated. On the other hand, (Si et al., 2021) employs augmentation techniques to enhance the density and concentration of the input embedding. Consequently, classifiers trained on such embeddings exhibit greater robustness. However, the approach of (Si et al., 2021) lacks an explicit explanation of the underlying rationale and fails to establish a direct correlation between input embedding concentration and adversarial robustness. Our research complements the findings of (Si et al., 2021) by revealing that sparser input embeddings lead to greater model robustness, while denser inputs result in decreased robustness as shown in Finding 3 in Section 6.

heading: 8.4 Generalization to comprehensive transformer architecture
text: We prove that our framework can be extended for all types of transformers including encoderonly (BERT, RoBERTa, ELECTRA), decoder-only (GPT2), and encoder-decoder (BART) since the RMSE of interpolation ASR prediction is good for not only encoder-only transformers (range from 0.031 to 0.070) but also decoder-only and encoderdecoder transformers (0.025 and 0.028 respectively), shown in Table 3.

heading: 8.5 Future Directions
text: Thanks to the high accuracy (about 0.025 in RMSE) of the proposed robustness predictor, it can also be considered an influence function (Koh and Liang, 2017) for robustness. Like other applications of influence function (Chhabra et al., 2023; Guo et al., 2021; Ladhak et al., 2023), robustness predictor is promising to be used for selecting or pruning data, robustness attribution, and data debugging to make the model more robust.

heading: 9 Conclusion
text: In this paper, we introduce an approach to correlate the adversarial robustness of transformer models fine-tuned on new downstream datasets. By learning a lightweight regression-based robustness predictor on a taxonomy of 13 features of a fine-tuning dataset, we empirically demonstrate that our framework can effectively predict the model robustness in both interpolation and extrapolation settings with a significant speedup in inference.

heading: Limitations
text: Although we try our best to demonstrate that our robustness evaluation toolkit can be used in practice, there are still limitations in how we design the framework. One such is that the process of finetuning a target transformer model does not take too much time and can be incorporated as additional signals to our algorithm. Such signals may help improve the robustness prediction performance and still ensure fast runtime. However, this approach will introduce confounding factors, and hence cannot help fully interpret the influence of fine-tuning data on model robustness, which is the main focus of this work.
Like any other “first work”, this research direction is in its infancy. Its novelty will come with early limitations that cannot be fully resolved in one single work, and thus call for further investigations from the community. At this stage, in practice, we recommend this as an additional fast interpretable toolkit to understand and evaluate the robustness of transformer models.

heading: A Reproducibility

heading: A.1 Notations
text: Following are the symbols used throughout our work.
• X : embedding space
• X: input sentence
• Y: labels
• N : total number of samples in train data
• Cn: naive classifier
• T : a data set with pairs of a text and a label, (x, y)
• T : tokenizer
• M: NLP classifier
• A: attack success rate
• F : features of train data
• P: ASR predictor

heading: A.2 Feature Engineering
text: Embedding Distribution. While the mean distance between classes, Fisher’s discriminant ratio, and Calinski-Harabasz Index based on the labels of the inputs are used to measure the separation between classes, the number of clusters and the Davies-Bouldin Index is used to measure the density or sparseness of the embedding space. For mapping input text into multidimensional space, we use a pre-trained transformer-based Universal Sentence Encoder (Cer et al., 2018). ■ Regrading indicators for class separation, let denote the vectors mapped from input sentences to an embedding space by the Universal Sentence Encoder (Cer et al., 2018) X = {xi}Ni=1, xi ∈ R1×512 and Y = {yi}Ni=1 are their labels. Vectors with the same label will be classified into the same clusters. C = {Ci, Ni,mi}Ki=1 is an array of clusters in the embedding space where Ci, Ni, and mi are a set of indexes in the ith cluster, the number of vectors of the ith cluster, and the center of the ith cluster respectively. K is the number of clusters or possible labels in the training dataset. Since mi is the center of the ith cluster, the following formula holds:
mi = 1
Ni
∑ j∈Ci xj
Similarly, {C,N,m} represents the cluster covering all vectors, the number of vectors, and the global centroid. Hence, we have,
m = 1
N
∑ j∈C xj
Let denote ri, r, dij the average distance between each point of the ith cluster and the centroid of that cluster, also known as cluster diameter or intracluster distance, the global diameter and the distance between ith and jth cluster centroids, also known as inter-cluster distance.
ri = 1 Ni
∑ i∈Ci (xi −mi)(xi −mi)T
r = 1N
∑ i∈C (xi −m)(xi −m)T
dij = (mi −mj)(mi −mj)T The formulas for the Mean Distance between classes, Fisher’s Discriminant Ratio, and CalinskiHarabasz Index are expressed as follows:
• Mean Distance between classes (MD): This indicator calculates the average distance between the means of different classes in the input space. A larger value indicates that the means of different classes are further apart, which implies a higher degree of separation between classes.
MD = 2×
∑
i,j
dij
N(N − 1)
• Fisher’s Discriminant Ratio (FDR): This metric measures the ratio of the variance between classes to the variance within classes. A larger value indicates a higher degree of separation between classes.
FR = SB SW ,
where
SW =
K∑
i=1
Si
Si = ∑ Ni × ri
SB = K∑
k=1
Nk(mk −m)(mk −m)T
• Calinski-Harabasz Index (CHI): The CalinskiHarabasz index also known as the Variance Ratio Criterion, is the ratio of the sum of between-clusters dispersion and of intercluster dispersion for all clusters, the higher
the score, the better the performances.
CHI = SW SB × N −K K − 1
We next describe in detail the formula for the features listed in Fig. 3. Features are divided into 4 main groups, namely Embedding, Distribution of Labels, Learning Ability of a Surrogate Model, and Dataset Statistics. ■ For the indicators for clustering, the notations are the same as the case for class separation except that the vectors are clustered based on the HDBSCAN (Malzer and Baum, 2020) algorithm instead of being based on their labels. Hence, K now is the number of clusters obtained from the HDBSCAN (Malzer and Baum, 2020) algorithm.
• Number of clusters: This indicates how vectors in the high-dimensional space are distributed. There are K clusters of vectors. • Davies-Bouldin Index (DBI): The score is defined as the average similarity measure of each cluster with its most similar cluster, where similarity is the ratio of within-cluster distances to betweencluster distances. Thus, clusters that are farther apart and less dispersed will result in a better score. The minimum score is zero, with lower values indicating better clustering. The Davis-Bouldin Index is defined as:
DBI = 1
k
k∑
i=1
max i ̸=j Rij ,
where
Rij = ri + rj dij
Label Distribution. Let denote Y as a random variable representing possible labels in the training dataset Strain.
• Pearson Median Skewness (PMS): The sign indicates the direction of the skewness. The coefficient compares the distribution of the sample to that of a normal distribution. The greater the value, the greater the deviation from the normal distribution. A value of 0 denotes that there is no skewness at all. A big negative value indicates that the distribution is skewed. A high positive value indicates that the distribution is biased to the right.
PMS= 3(Ȳ −Md)
s ,
where Ȳ , Md, and s are respectively mean, median, and variance of the distribution of labels.
• Kurtosis (Kurt): Kurtosis is a measure of the peakedness or flatness of a distribution. A distribution with kurtosis equal to 3 is considered to be mesokurtic (i.e., having a normal distribution), while a distribution with kurtosis greater than 3 is considered to be leptokurtic (i.e., having a sharper peak and fatter tails) and a distribution with kurtosis less than 3 is considered to be platykurtic (i.e., having a flatter peak and thinner tails).
Kurt = E [ (Y − Ȳ)4 ]
(E[(Y − Ȳ)2])2

heading: Surrogate Model’s Learnability.
text: • Misclassification Rate (MCR): We create a naive classifier Cn : X → Y that turns text, X , into predicted classes, Y , and analyze the performance of this classifier because we think misclassification is related to its robustness. (Sun et al., 2020) shows that typos affect the robustness of the transformerbased model, so we choose the character-based CNN (Zhang et al., 2015) model for Cn because it can exploit character-level properties. Suppose classfier Cn turns a set T of text xi with true class yi into a predicted class ŷi = Cn(xi). Misclassification Rate is expressed by the following formula:
MCR = |{ŷi ̸= yi|(xi, yi) ∈ T}|
|T |
Token-Based Statistics. We use a tokenizer T , namely Bert-base-cased tokenizer (Devlin et al., 2019), to convert a sentence into an array of tokens. The notation X is a text set in the training dataset Strain. Tokenizer T converts each text xi ∈ X into a list of Mi tokens {tij}Mij=1, T : ti → {tij}Nij=1. Denote Y the collection of lists of tokens, so T turns X into Y . The formulas for those syntactic features are illustrated as follows:
Avg. # tokens = 1
|X|
|X|∑
i=1
Mi
# unique tokens = |{t|t ∈ ∪|X|i=1{tij}Mij=1 & t exists once}|
Min # tokens = min({Mi}|X|i=1) Max # tokens = max({Mi}|X|i=1)
(4) In addition, the total number of classes is number of possible classes of dataset D from which the subdataset Strain is sampled

heading: A.3 Evaluation Metrics
text: Denote Â and A the predicted ASR made by P and the actual ASR. The metrics we use to evaluate ASR predictors include the following:
• Root Mean Squared Error (RMSE): This is the square root of the mean square error (MSE), the average of the squared differences between the predicted values and the actual values.
RMSE =
√√√√ ∑N
i=1
( Ai − Âi )2
N , (5)
where N is the number of predicted values. • Mean Absolute Error (MAE): This is the average
of the absolute differences between the predicted values and the actual values. It is less sensitive to outliers than MSE, but may not penalize large errors as heavily.
MAE = 1
N
N∑
i=1
∣∣∣Ai − Âi ∣∣∣ (6)
• R-squared (R2): This is measured by the ratio between the mean squared error of a regression model and the variance of the target variable. It ranges from 0 to 1, with higher values indicating better performance.
R2 = 1− ∑N
i=1(Ai − Âi)2∑N i=1(Ai − Ā)2 , (7)
where Ā = ∑N
i=1 Ai N
• Mean Absolute Percentage Error (MAPE): This is the average of the absolute percentage differences between the predicted values and the actual values. It is commonly used in forecasting applications.
MAPE = 1
N
N∑
i=1
∣∣∣∣∣ Ai − Âi
Ai
∣∣∣∣∣ (8)
• Explained Variance Score (EVS): This is the proportion of variance in the target variable that is explained by the model relative to the total variance. It ranges from 0 to 1, with higher values indicating better performance.
EV S = 1− V ar(A− Â) V ar(A) (9)

heading: A.4 Experiment Setup
text: Hardware Specifications. We use one GPU NVIDIA RTX A6000 and 32 CPUs AMD Ryzen Threadripper PRO 5975WX 32-Cores for our experiment.

heading: A.5 Error Analysis
text: We examine how features impact errors in ASR prediction. Initially, we employ the Random Forest model to predict ASR for test samples. Subsequently, test samples exhibiting errors surpassing the 70th percentile are categorized as False; otherwise, they are labeled as True. Then, logistic regression is applied to distinguish outlier test samples. Ultimately, the absolute magnitude of parameter weights from the logistic regression model is utilized to gauge feature significance. Essentially, these parameter weights indicate the degree of error influencing ASR prediction.

heading: Sampling Datapoints.
text: The dataset list, Dl, that we have used includes AG News, Amazon Review Full, Amazon Review Polarity, DPedia, Yahoo Answers, Yelp Review Full, Amazon Review Polarity, Banking 77, EmojiTweetEval. In Dl, we divided the datasets into two groups: Da = { AG News, Amazon Review Full, Amazon Review Polarity, DPedia, Yahoo Answers, Yelp Review Full, Amazon Review Polarity } and Db = { Banking 77, Emoji-TweetEval }. The datasets in Da have label counts of 2, 4, 5, 10, and 14 and are confined to a few settings, such as Yelp reviews, news articles, Yahoo inquiries, etc while those in Db are 77 and 22. Due to the lack of diversity within each group of label counts, datasets in Da are kept, whereas datasets in Db are slightly adjusted to boost contextual variety for each group of label count. When a sub-dataset (Sitrain, S i val, S d test) in Q introduced in Algorithm 1 is sampled, if it is in the Db, the number of classes of that sub-dataset will also be randomly converted to 2, 4, 5, 10, or 14. For example, to convert a 22-label Emoji-TweetEval dataset into a 4-label dataset, the labels from 1 to 5, from 5 to 10, from 11 to 15, and from 16 to 20 will be converted into the new labels 1, 2, 3, 4, while samples with residual labels 21, 22 will be discarded.
After that, we sampled 500 data points of train data features and attack success rates in 72 hours. BERT and ROBERTA hyperparameters. The tokenizer has a maximum length of 512 words. The learning rate, weight decay, and warmup step are 5e-4, 0.01, and 500, respectively. We train 5 epochs in that, from our observations, it is enough for the model to converge and get good inference results on the test dataset. Character-level CNN. A tokenizer converts each character in a sentence into a one-hot vector in this
model. It can only be 1024 characters long. There are six CNN layers, the kernel size of the first one and the five following are 3 and 7 respectively. The first and final CNN layer is followed by a 3 kernel size 1D pooling layer. HDBSCAN. The minimum cluster size is five. The metric is Euclidean. Gradient Boosting Regressor. The learning rate, maximum bin, and number of estimators are 0.05, 400, and 5000, respectively.

heading: Future Work
text: Within the pioneering nature of our work on this research topic, we hope to see future works that explore those confounding factors, for example, from the perspective of causality