SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Laura F. Nern
Yahoo Research
laurafee.nern@yahooinc.com
Harsh Raj
Delhi Technological University
harsh77711raj@gmail.com
Maurice Georgi
Hyundai Mobis
georgmaurice@gmail.com
yash.sharma@bethgelab.org
As large-scale training regimes have gained popularity, the use of pretrained models for downstream tasks has become common practice in machine learning. While pretraining has been shown to enhance the performance of models in practice, the transfer of robustness properties from pretraining to downstream tasks remains poorly understood. In this study, we demonstrate that the robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation, regardless of the protocol used for pretraining. We prove (i) a bound on the loss that holds independent of any downstream task, as well as (ii) a criterion for robust classification in particular. We validate our theoretical results in practical applications, show how our results can be used for calibrating expectations of downstream robustness, and when our results are useful for optimal transfer learning. Taken together, our results offer an initial step towards characterizing the requirements of the representation function for reliable post-adaptation performance.<sup>1</sup>
Recently, there has been a rise of models (BERT [31], GPT-3 [7], and CLIP [43]) trained on large-scale datasets which have found application to a variety of downstream tasks [6]. Such pretrained representations enable training a predictor, e.g., a linear "head", for substantially increased performance in comparison to training from scratch [10, 11], particularly when training data is scarce.
Generalizing to circumstances unseen in the training distribution is significant for real-world applications [2, 60, 29], and is crucial for the larger goal of robustness [27], where the aim is to build systems that can handle extreme, unusual, or adversarial situations. In this context, adversarial attacks [51, 5], i.e., inducing model failure via minimal perturbations to the input, provide a stress test for predictors, and have remained an evaluation framework where models trained on large-scale data have consistently struggled [53, 21, 22], particularly when, in training, adversaries are not directly accounted for [23, 40].
With that said, theoretical evidence [8] has been put forth that posits overparameterization is required for adversarial robustness, which is notable given its ubiquity in training on large-scale datasets. Thus, given the costs associated with large-scale deep learning, we are motivated to study how adversarial robustness transfers from pretraining to downstream tasks.
<https://github.com/lf-tcho/robustness_transfer
Figure 1: Overview. We distinguish between a robust (red) and non-robust (green) representation by its sensitivity to adversarial attacks, and use both for linear separation of the representation space, e.g., binary classification. This illustrates the dependence of the classifier on the representation's robustness, i.e., the predictor can only be robust if  $x_{adv}$  and  $x$  can be linearly separated from  $x'$  in representation space.
For this, considering the representation function to be the penultimate layer of the downstream predictor, we contribute theoretical results which bound the robustness of the downstream predictor by the robustness of the representation function. Fig. 1 illustrates the intuition behind our theory, and shows that a linear classifier's robustness is dependent on the robustness of the pretrained representation. Our theoretical results presented in Sec. 3 and Sec. 4 are empirically studied in Sec. 5, where we validate the results in practice, and in Sec. 6 to show how the theory can be used for calibrating expectations of the downstream robustness in a self-supervised manner [4]. In all, our theoretical results show how the quality of the underlying representation function affects robustness downstream, providing a lens on what can be expected upon deployment. We present the following contributions:
- we theoretically study the relationship between the robustness of the representation function, referred to as adversarial sensitivity (AS) score, and the prediction head. In Thm. 1, we prove a bound on the difference between clean and adversarial loss in terms of the AS score. In Thm. 2, we prove a sufficient condition for robust classification again conditioned on the robustness of the representation function.
- we evaluate the applicability and consistency of our theory in applications in Sec. 5.1 and Sec. 5.2. Furthermore, in Sec. 6, we illustrate the utility of the AS score to predict expected robustness transfer via linear probing.
- in Sec. 6.2, we study how linear probing compares to alternatives, like full finetuning, in robustness transfer for robust performance downstream.
Transfer Learning: While empirical analyses of pretraining abound [33, 12, 62, 42, 3, 48], a number of theoretical analyses have also been contributed [56, 52, 19, 14, 49, 35, 38]. The majority of the theoretical studies aim to understand the benefit of transfer learning with respect to improved performance and data requirements compared to training from scratch. Therefore, the referenced studies include results on sample complexity [52, 19, 14, 49] and generalization bounds [38] for fine-tuning and linear probing in various scenarios. In this work, we primarily focus on the linear probing approach and perform a theoretical analysis to investigate its efficiency in transferring adversarial robustness.
Robustness: There is a direct connection between the mathematical formulation of adversarial robustness and the definition of a local Lipschitz constant of the network [25], but computing this quantity is intractable for general neural networks. However, there exists a line of work providing more practical approaches in certifiable robustness [54, 44, 15, 37, 36, 46], which can provide guarantees on the detection of adversarial examples. Relative to empirical approaches [23, 40, 18, 48, 55], certifiable methods have comparatively struggled to scale to datasets such as ImageNet [17]. Furthermore, likely due to computational expense, adversarial training has not yet been adopted by models trained on large-scale datasets [31, 7, 43]. Still, a number of works have incorporated adversarial training into
pretraining, in particular, self-supervised contrastive learning [32, 30, 9, 20]. With that said, our work is focused on the analysis of robustness irrespective of the pretraining approach.
Regarding analysis, [59] studied the trade-off between accuracy and robustness, and showed that, under a set of separation conditions on the data, a robust classifier is guaranteed to exist. Their work focuses on proving the robustness of a network itself and therefore requires Lipschitzness of the model. Further, [8] showed that overparameterization is needed for learning such smooth (i.e. Lipschitz) functions, which is critical for robustness. On the other hand, we specifically study the relationship between the robustness of the representation function and the prediction head, a scenario consistent with the practice of pretraining for transfer learning.
Transfer Learning & Robustness: The question of how robustness is transferred from pretraining to downstream task has been studied empirically in the prior literature [50, 58, 28, 63]. Notably, while [50] observe that robust representations underly robust predictors, [63] also observe that non-robust representations underly non-robust predictors. An empirical analysis of adversarial robustness transfer for the special case of linear probing is included in ,e.g., [50]. Our work not only derives theoretical results and concrete proof for the empirical phenomena reported in these studies but also provides further empirical investigation.
3 Robustness Bound
Our first theoretical result concerns the discrepancy between the standard and adversarial loss of a neural network  $f$ . The considered loss function  $\ell$  is evaluated on a dataset of input samples  $\mathbb{X} = \{x_1, \ldots, x_n\} \subset \mathcal{X} \subset \mathbb{R}^d$  and labels  $\mathbb{Y} = \{y_1, \ldots, y_n\}$ , where  $x_i$  is sampled from  $\mathcal{X}$  with probability distribution  $\mathcal{P}(\mathcal{X})$ .
For the network, we consider a model  $f: \mathbb{R}^d \to \mathbb{R}^c$  with a final linear layer, i.e.  $f(x) = f_{W,\theta}(x) = W g_\theta(x)$ . The function  $g_\theta: \mathbb{R}^d \to \mathbb{R}^r$  can be considered as a representation function for the input parameterized by  $\theta$ , and  $W \in \mathbb{R}^{c \times r}$  is the weight matrix of the linear layer. Note that this composition is both standard practice [10, 11], and has been extensively studied [56, 52, 19] in the literature.
For a given data pair  $(x_{i},y_{i})$ , we are interested in the stability of the model's prediction of  $y_{i}$  with respect to (w.r.t.) perturbations of the input  $x_{i}$  as our robustness measure. Specifically, for  $\delta \in \mathbb{R}^d$ , we find the maximum loss  $\ell(f(x_i + \delta), y)$  under a specified constraint  $\| \delta \| < \epsilon$  based on some norm  $\| \cdot \|$ . Note that the defined measure is commonplace in the literature on adversarial attacks [51, 23].
We introduce the following notation for the standard loss  $\mathcal{L}(f_{W,\theta})$  and the adversarial loss  $\mathcal{L}_{\mathrm{adv}}(f_{W,\theta})$ ,
$$
\mathcal {L} \left(f _ {W, \theta}\right) := \mathbb {E} _ {\mathcal {P} (\mathcal {X})} \left[ \ell \left(f _ {W, \theta} (x), y\right) \right],
$$
$$
\mathcal {L} _ {\mathrm {a d v}} (f _ {W, \theta}) := \mathbb {E} _ {\mathcal {P} (\mathcal {X})} \left[ \max  _ {\| \delta \| <   \epsilon} \ell (f _ {W, \theta} (x + \delta), y) \right].
$$
We aim to bound the effect of adversarial attacks in terms of their impact on the representation function, i.e.  $\| g_{\theta}(x_i) - g_{\theta}(x_i + \delta)\| _2$ . To obtain such a result for a general loss function  $\ell$ , the variation in the loss must be bounded in relation to the proportional change in  $f(x_{i})$ . This condition is formalized by the concept of Lipschitzness,
Definition 1. A function  $\ell : \mathbb{R}^d \to \mathbb{R}$  is  $C$ -Lipschitz in the norm  $\|\cdot\|_{\alpha}$  on  $\mathcal{R} \subset \mathbb{R}^d$ , if
$$
\left| \ell (r) - \ell \left(r ^ {\prime}\right) \right| <   C \| r - r ^ {\prime} \| _ {\alpha},
$$
for any  $r,r^{\prime}\in \mathcal{R}$
The following theorem is valid for any loss function  $\ell$  that satisfies Defn. 1 w.r.t.  $L_{1}$ ,  $L_{2}$  or  $L_{\infty}$  norm.
Theorem 1. We assume that the loss  $\ell(r, y)$  is  $C_1$ -Lipschitz in  $\|\cdot\|_{\alpha}$ , for  $\alpha \in \{1, 2, \infty\}$ , for  $r \in f(\mathcal{X})$  with  $C_1 > 0$  and bounded by  $C_2 > 0$ , i.e.,  $0 \leq \ell(r, y) \leq C_2 \forall r \in f(\mathcal{X})$ . Then, for a subset  $\mathbb{X}$  independently drawn from  $\mathcal{P}(\mathcal{X})$ , the following holds with probability of at least  $1 - \rho$ ,
$$
\mathcal {L} _ {\mathrm {a d v}} (f _ {W, \theta}) - \mathcal {L} (f _ {W, \theta}) \leq L _ {\alpha} (W) C _ {1} \frac {1}{n} \sum_ {i = 1} ^ {n} \max _ {\| \delta \| <   \epsilon} \| g _ {\theta} (x _ {i} + \delta) - g _ {\theta} (x _ {i}) \| _ {2} + C _ {2} \sqrt {\frac {\log (\rho / 2)}{- 2 n}},
$$
$$
L _ {\alpha} (W) := \left\{ \begin{array}{l l} \| W \| _ {2} & ,   i f \| \cdot \| _ {\alpha} = \| \cdot \| _ {2}, \\ \sum_ {i} \| W _ {i} \| _ {2} & ,   i f \| \cdot \| _ {\alpha} = \| \cdot \| _ {1}, \\ \max  _ {i} \| W _ {i} \| _ {2} & ,   i f \| \cdot \| _ {\alpha} = \| \cdot \| _ {\infty}. \end{array} \right.
$$
The proof of the theorem can be found in Appx. A.1. Note that the norm used to define the perturbation  $\delta$  is arbitrary and only needs to be consistent on both sides of the bound.
Proof sketch: First, we apply Hoeffding's inequality, which requires a fixed function  $f_{W,\theta}$ , independently drawn samples  $\{x_1,\ldots ,x_n\}$ , and an upper bound  $C_2$  on the loss function. Next, we bound the average losses utilizing the  $C_1$ -Lipschitzness in  $\| \cdot \|_{\alpha}$  of the loss function. Further, the Cauchy-Schwarz inequality is used to separate the norms of  $W$  and the representation function  $g_{\theta}$ .
Intuition: Thm. 1 shows that the effect an  $\epsilon$ -perturbation has on the representation function  $g_{\theta}$  indeed upper bounds the difference between the standard and adversarial loss. Thus, we characterize the relationship between how vulnerable the linear predictor and the underlying representation is to adversarial examples. Further, Thm. 1 formalizes the impact the weight matrix  $W$  has on robustness transfer and thereby provides a theoretical argument for regularization in this context.
The statement of Thm. 1 is formulated in terms of the expectation of the losses over  $\mathcal{P}(\mathcal{X})$ . In particular, the second summand on the right hand-side (RHS) of the bound illustrates the finite-sample approximation error between the expectation and the average over  $n$  samples in  $\mathbb{X}$ . The following lemma provides a version of the theorem for the average difference of the losses over a fixed finite sample  $\mathbb{X}$ , which can directly be concluded from the proof of Thm. 1 in Appx. A.1. We use this lemma to study our theoretic result in the experiments in Sec. 5.1.
Lemma 1 (Finite-sample version of Thm. 1). We assume that the loss  $\ell(r, y)$  is  $C$ -Lipschitz in  $\|\cdot\|_{\alpha}$ , for  $\alpha \in \{1, 2, \infty\}$ , for  $r \in f(\mathcal{X})$  with  $C > 0$ . Then, the following holds for any dataset  $(\mathbb{X}, \mathbb{Y})$ ,
$$
\frac {\frac {1}{n} \sum_ {i = 1} ^ {n} \max _ {\| \delta \| <   \epsilon} \ell (f _ {W , \theta} (x _ {i} + \delta) , y _ {i}) - \ell (f _ {W , \theta} (x _ {i}) , y _ {i})}{L _ {\alpha} (W) C} \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \max _ {\| \delta \| <   \epsilon} \| g _ {\theta} (x _ {i} + \delta) - g _ {\theta} (x _ {i}) \| _ {2}.
$$
Lemma 1 shows that the normalized difference between average clean and adversarial loss can be bound by the average robustness of the representation function on the RHS, which we denote as the representation function's adversarial sensitivity (AS) score on the dataset  $\mathbb{X}$ .
3.1 Valid Loss Functions
To elaborate on the applicability of the result, we discuss exemplary scenarios in which the assumptions of our theory are fulfilled in practice. As a first example, we consider a classification task where we use the softmax cross-entropy loss. While the cross-entropy loss itself is not Lipschitz continuous, since the logarithm is not, the softmax cross-entropy loss is indeed Lipschitz. We define the softmax cross-entropy loss as follows,
$$
\ell_ {C E} (f (x), y) = - \log (p _ {y} (x)) = - \log \left(\frac {\exp \{f (x) _ {y} \}}{\sum_ {i = 1} ^ {c} \exp \{f (x) _ {i} \}}\right),
$$
where  $p_{y}(x)$  denotes the y-th entry of the softmax function applied to the final layer,  $f(x)_y$  is the y-th entry of the model output with  $f(x)_y = W_y g(x)$  and  $W_y$  being the y-th row of the matrix  $W$ . In Lemma 3 (in Appx. A), we prove that  $\ell_{CE}$  is 2-Lipschitz in  $\| \cdot \|_{\infty}$ .
Further, the theory is also valid for regression tasks, where the mean square error (MSE)
$$
\ell_ {M S E} (f (x), y) = \| f (x) - y \| _ {2}
$$
is used to measure the loss. This loss function is 1-Lipschitz in  $\| \cdot \| _2$ , since, by the reverse triangle property, it holds that
$$
\left| \left| \left| f (x _ {1}) - y \right| \right| _ {2} - \left| \left| f (x _ {2}) - y \right| \right| _ {2} \right| \leq \left| \left| f (x _ {1}) - f (x _ {2}) \right| \right| _ {2}.
$$
Additional losses which satisfy Defn. 1 include the logistic, hinge, Huber and quantile loss [13].
Figure 2: The graphic shows two classes of datapoints and a linear classifier's decision boundary (black). The line  $M_0$  (green) is the optimal margin defining the boundary between a successful or failed adversarial attack. Line  $M$  (red) represents the margin derived from Thm. 2 which is an upper bound on  $M_0$ . Four datapoints from the blue class are highlighted: 1 is wrongly classified, 2 is a non-robust example, 3 is robust but does not fulfill equation (1), and 4 is a robust classification fulfilling the bound in (1).
4 Robustness Classification Criterion
In Sec. 3, we presented a theoretical result independent of any particular downstream task and valid for various loss functions. Here, we focus on classification tasks, and, in this context, we provide an additional statement that studies the robustness of a linear classifier w.r.t. adversarial attacks. We refer to a classifier as robust if there does not exist a bounded input perturbation that changes the classifier's prediction.
In particular, we consider the classifier
$$
c (x) = \operatorname {a r g m a x} _ {i} f (x) _ {i},
$$
where  $f(x) = Wg_{\theta}(x)$  is the model as defined in Sec. 3. The following theorem provides insights on how the robustness of the underlying representation function  $g_{\theta}$  impacts the robustness of a linear classifier. Specifically, it formally describes how the robustness of the representation, i.e. the AS score on a datapoint  $x$ , determines the set of weight matrices which yield a robust classifier.
Theorem 2. The classifier  $c(x)$  is robust in  $x$ , i.e.,  $c(x + \delta) = c(x)$  for  $\delta \in \mathbb{R}^d$  with  $\| \delta \| < \epsilon$ , if for  $x \in \mathcal{X}$  with  $c(x) = y$  it holds that
$$
\left\| g _ {\theta} (x + \delta) - g _ {\theta} (x) \right\| _ {2} \leq \min  _ {j \neq y} \frac {\left| f (x) _ {y} - f (x) _ {j} \right|}{\left\| W _ {y} - W _ {j} \right\| _ {2}}. \tag {1}
$$
The proof of the theorem can be found in Appx. A.2.
Proof sketch: To prove robustness, we need to show that  $f(x + \delta)_y > f(x + \delta)_j$  if  $f(x)_y > f(x)_j$ . The proof consists of rewriting this condition and applying the Cauchy-Schwarz inequality to separately bound in terms of the difference in  $W$  and  $g_{\theta}$ .
Intuition: Thm. 2 shows that the effect an  $\epsilon$ -perturbation has on the representation function  $g_{\theta}$  determines the necessary class separation by the linear predictor for robust classification to be ensured. Specifically, the RHS of Thm. 2 can be understood as a normalized margin between class logits for  $x$ , where the classification margin refers to the difference between the classification score for the true class and maximal classification score from all false classes. Thus, the theorem states that if the margin is larger than the effect of an  $\epsilon$ -perturbation on the representation function, then the classifier is robust. Note that if a datapoint does not satisfy the bound, i.e. LHS is greater than the RHS, the robustness of the classifier is ambiguous. In this case, the RHS margin is not the exact separation, or the optimal margin, between robust and non-robust classification. Fig. 2 provides intuition for this interpretation of the result in a binary classification scenario.
5 Empirical Study of Theory
The following experiments serve to validate the theoretical results in practical applications. For the finite sample version of Thm. 1, Lemma 1, we consider a classification task in Sec. 5.1, and a regression task in Appx. B.2. In Sec. 5.2, we study the robustness condition established in Thm. 2.
In the following sections, we train a linear predictor on top of a pretrained representation function for a specific downstream task, i.e., we perform transfer learning via linear probing. To study robustness transfer, we use robustly pretrained representation functions and examine if the transferred robustness is consistent with the theoretical bounds. As our robust representation function, we use models robustly pretrained on CIFAR-100 [1, 34], and ImageNet [47, 17]. Both models are available on RobustBench [16] and are adversarially trained using the softmax cross-entropy loss and  $L_{\infty}$ -attacks. For our downstream tasks, we consider CIFAR-10 [34], Fashion-MNIST [57] and Intel Image<sup>2</sup>. Further experimental details can be found in Appx. B.1.
5.1 Classification - Cross-Entropy Loss
Here, we empirically evaluate Lemma 1, the finite sample version of Thm. 1, on classification tasks. As per Sec. 3.1, for softmax cross-entropy, we can apply Lemma 1 with Lipschitz constant  $C = 2$  and mapping function  $L_{\infty}(W)$ .
In Tab. 1, we present the left-hand side (LHS), the scaled difference between clean and robust cross-entropy (CE) loss, and the right-hand side (RHS), the representation function's AS score, of Lemma 1. Note that, for LHS, the attack on  $f$  maximizes the CE loss, while for RHS, the attack on  $g_{\theta}$  maximizes the impact on the representation function, i.e.  $\| g_{\theta}(x_i) - g_{\theta}(x_i + \delta)\|_2$ . See Appx. B.1 for further details.
The results in Tab. 1 validate our theoretical predictions in practice, as the LHS is indeed always upper bounded by the RHS. Furthermore, we see signs of correlation between the LHS and RHS, i.e. significant changes on one side of the bound are accompanied by similar changes on the other. Tab. 2 further highlights the connection between our result and weight norm regularization, as it confirms that a smaller weight norm, i.e., smaller  $L_{\alpha}(W)$ , corresponds to a smaller decrease in robustness, i.e., a smaller difference between the losses.
Table 1: For the CIFAR-100 and ImageNet pretraining datasets, the LHS and RHS, i.e., the AS score, of Lemma 1 are computed for each of the CIFAR-10, Fashion-MNIST and Intel Image downstream tasks. Additionally,  $L_{\infty}(W)$  and the absolute difference between clean and adversarial CE loss are provided.
CIFAR-10	ImageNet
CIFAR-10	F-MNIST	Intel Image	CIFAR-10	F-MNIST	Intel Image
Diff. CE	0.74	0.8	0.47	4.14	9.4	0.14
L∞(W)	4.4	4.6	3.1	1.9	2.0	1.4
LHS	0.08	0.09	0.08	1.11	2.33	0.05
AS score	0.98	2.11	1.05	15.7	53.2	2.75
Table 2: The same datasets and metrics are considered as in Tab. 1. Here, however, L2 weight matrix regularization  $(\lambda = 0.01)$  is employed for all downstream tasks.
CIFAR-10	ImageNet
CIFAR-10	F-MNIST	Intel Image	CIFAR-10	F-MNIST	Intel Image
Diff. CE	0.46	0.51	0.38	3.57	7.3	0.14
L∞(W)	2.1	2.3	2.2	1.4	1.5	1.2
LHS	0.11	0.11	0.09	0.81	2.43	0.06
AS score	0.98	2.11	1.05	15.7	53.2	2.75
$^{2}$ https://www.kaggle.com/datasets/puneet6060/intel-image-classification
5.2 Classification Criterion
Here, we focus on empirically analysing Thm. 2, which specifies an upper bound on the representation's sensitivity against  $\epsilon$ -perturbations, where the condition ensures robustness of the classifier against local perturbations for each datapoint.
In Tab. 3, we observe that indeed: if condition (1) is fulfilled, the classification is always robust, as can be seen from "rob. fulfilled". With that said, we do observe instances of robust classification which do not fulfill Eq. (1), which we proved ensures robustness, as can be seen from comparing "robust accuracy" to "prop. fulfilled". As with Lemma 1, we see that tighter results would be necessary for identifying all transferred robustness. Furthermore, as discussed in Sec. 4, since datapoints fulfilling our bound are even further away from the classification boundary than necessary, we also find that fulfilling the bound is predictive of robustness to stronger attacks, as can be seen in Appx. B.3.
For the scenario of pretraining on ImageNet and transferring to CIFAR-10 or Fashion-MNIST, almost no datapoint from the test set fulfills condition (1). This observation is accompanied by a steep drop from clean accuracy to robust accuracy, which suggests this is likely a consequence of the large distribution shift between ImageNet and the lower-resolution images of CIFAR-10  $(32 \times 32)$  and Fashion-MNIST  $(28 \times 28)$  resized to ImageNet resolution  $(256 \times 256)$ . Note that this distribution shift is evident regardless of how the images are scaled before applying the ImageNet representation function, see Appx. B.5. In Tab. 1, the correlation between distribution shift and loss of robustness is also captured prior to transfer learning by a large AS score, and Appx. B.5 provides further insight into this connection.