SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Amur Ghose\*1,2, Apurv Gupta3, Yaoliang Yu1,2, Pascal Poupart1,2
<sup>1</sup>David R. Cheriton School of Computer Science, University of Waterloo,
$^{2}$ Vector Institute,  $^{3}$ Columbia University
a3ghose@uwaterloo.ca, apurvgupta1996@gmail.com,
yaoliang.yu@uwaterloo.ca,ppoupart@uwaterloo.ca
The construction of adversarial examples usually requires the existence of soft or hard labels for each instance, with respect to which a loss gradient provides the signal for construction of the example. We show that for batch normalized deep image recognition architectures, intermediate latents that are produced after a batch normalization step by themselves suffice to produce adversarial examples using an intermediate loss solely utilizing angular deviations, without relying on any label. We motivate our loss through the geometry of batch normed representations and their concentration of norm on a hypersphere and distributional proximity to Gaussians. Our losses expand intermediate latent based attacks that usually require labels. The success of our method implies that leakage of intermediate representations may create a security breach for deployed models, which persists even when the model is transferred to downstream usage. Removal of batch norm weakens our attack, indicating it contributes to this vulnerability. Our attacks also succeed against LayerNorm empirically, thus being relevant for transformer architectures, most notably vision transformers which we analyze.
Adversarial examples  $x_{adv}$  are commonly defined as data instances which lie  $\epsilon$  away in some norm (usually  $L_{\infty}$ ) from an actual data instance  $x_{real}$ . To humans,  $\epsilon$  is small and  $x_{real}, x_{adv}$  share labels, yet to a classifier neural network, they do not. Since their discovery [Szegedy et al., 2013, Goodfellow et al., 2014], adversarial examples have spurred research in both attacking (i.e., generating adversarial data instances) [Akhtar and Mian, 2018, Tramer et al., 2020] and defending [Yuan et al., 2019] neural networks against them. Common methods for generating adversarial images use loss gradients. A pioneering attack is Fast Gradient Sign Method (FGSM) [Goodfellow et al., 2014], possibly strengthened via Projected Gradient Descent (PGD) [Madry et al., 2017] - an iterative form of FGSM. Defenses against these attacks have been derived, including adversarial training [Shrivastava et al., 2017], where adversarial images are fed to the network for training. Adversarial training is expensive but powerful [Salman et al., 2019] and scalable [Wong et al., 2020, Shafahi et al., 2019]. Defenses can give illusory security by obfuscating gradients [Athalye et al., 2018]. An unsaid aspect of defense is hiding model parameters - changing attacks from the white-box (model parameters known) scenario to black-box (models only accessible via outputs) scenario, which makes them harder to attack.
We consider the family of intermediate-level attacks, exemplified by members such as Intermediate Level Attack Projection (ILAP) [Huang et al., 2019] and its variants [Li et al., 2020b]. In this, an initial direction in the latent space of the hidden intermediate layers is determined via FGSM or other attacks. Gradient steps then maximally perturb the intermediate hidden representation to find a suitable  $x_{adv}$ , unlike directly using the label. Such methods can outperform the method used for
*Corresponding author: Amur Ghose, email: a3ghose@uwaterloo.ca
the initial direction itself and be more transferable across different models. ILAP uses FGSM to set initial directions, but once the direction arises, the layers after the intermediate layer play no role.
We now ask: Given an unlabeled data point  $x$  along with full access to a deep network up to  $K$  layers, where  $K$  is less than the network depth, could we craft an adversarial example out of  $x$  without any other information - e.g., the label of  $x$ , any  $x' \neq x$  - so long as the deep network utilized batch normalization? We answer this question positively when  $x$  is an image for several popular families of image recognition architectures, exploiting the hyperspherical geometry of latent spaces to propose an angular adversarial attack. The label access of the black box case is absent - we do not allow access to hard labels (true label of  $x$ ) or soft labels (logits obtained at penultimate layers). We adapt the method advanced in ILAP to remove the need for label-based FGSM, and proceed instead with an angular loss based on the geometry of batch normalization. Our concrete contributions are :
- To provide a label-free attack that relies on only gradient access upto intermediate layers
- Ablation of our method to show it is not brittle to choices of intermediate layers, (small variations between using layers  $i$ ,  $i + 1$  if  $i$  is sufficiently deep)
- To show prominent vision models - ResNets and EfficientNets - fall to our attack. Removing batchnorm in ResNets via Fixup neuters the attack - suggesting it is playing a role in this.
- To show that the attack empirically works even for transformer architectures (ViT) that employ Layernorm over batchnorm.
- To show that these attacks persist in the transfer learning setting, where the models are frozen up to a certain layer and then fine-tuned for downstream usage.
- To improve the supervised attack case when labels are available by using our loss in conjunction with the loss from the true label.
This implies that releasing the first  $K$  layers of a model publicly as a feature extractor, without any label access, can create adversarial attacks, which has security implications. Weaknesses persist even when the unseen, later layers of the network are tuned on different datasets (e.g. CIFAR10/100) - adversarial examples can be crafted for CIFAR without using the changed, later layers only using this first  $K$ . We exploit the radial geometry of batch normalization and assume the norm of the neural representations concentrates around a constant, allowing us to exploit the geometry formed. This indicates batch normalization may make networks not just less robust to conventional attacks [Galloway et al., 2019], but open up new attacks entirely. We ablate our method and compare to scenarios of naive baselines and having access to  $x' \neq x$  to showcase its competitiveness.
1.1 Relation to Related Work
We tie together two areas: analyzing batch normalization from a geometric standpoint under convergence assumptions, and adversarial attacks with partial knowledge of architectures. Batch normalization has been analyzed from the standpoint of deep learning theory and model optimization [Santarkar et al., 2018]. While its detrimental effects on model robustness [Galloway et al., 2019, Benz et al., 2020] are known, most analysis focuses on how batch normalization makes the model weaker to already-existing attacks. We propose that batch normalization might impact the network by making it more vulnerable to novel attacks. We are to our knowledge the first to connect this with the concentration of norm phenomenon implied by results such as the dynamics of batch normalization at high depths [Daneshmand et al., 2021]. These tie into results such as neural collapse [Papyan et al., 2020], which find that later layers in deep networks have predictable geometric patterns.
Research in adversarial examples has focused in specific settings such as white/grey/black box settings denoting levels of model access/knowledge. Even in the black box case, it is often assumed we can feed an input  $x$  into the model  $M$  under attack and retrieve  $M(x)$ , the model's prediction of the label of  $x$ . We consider the case of partial model knowledge and attacking a model with no label outputs. This is relevant with the presence of large-scale pretraining, where models are fed large training sets of data unrelated to any one domain to capture the overall distribution of a particular type of modality, such as GPT-3 [Brown et al., 2020] with natural language. Deployed downstream, these public models may fine tune on a smaller dataset, and most of their weights are not fine-tuned. Public models may act as feature extractors that distill large amounts of data beyond dataset scopes. They output not labels, but representation vectors, and our attack targets this case. Previous works do use latent representations to construct adversarial examples [Park and Lee, 2021, Creswell et al.,
2017, Yu et al., 2021] through moving in latent space towards points of differing labels or maximizing deviations in latent space [Kos et al., 2018]. Along with the related no-box [Li et al., 2020a, Zhang et al., 2022b] setting (assuming no model knowledge), these methods require extra parts, such as representations of instances of different labels as targets to move towards, surrogate models, or some soft labels, often with gradient access - all of which we do not require. They are also ad hoc in their choice of layers to attack, but we give a definite answer of which layers and architectures are the most vulnerable - batch norm layers situated deeper in the network. The closest case to our attack are label-free, single-stage latent-based attacks [Zhou et al., 2018, Inkawich et al., 2019, Ganeshan et al., 2019, Inkawich et al., 2020b, Zhang et al., 2022a, Wang et al., 2021, Inkawich et al., 2020a] which we outperform. These methods use the latent to boost the supervised scenario and not exploit batch normalization or two-step optimization in an unsupervised setting as we do.
1.2 Setup - FGSM and PGD Attacks
We describe FGSM aka Fast Gradient Sign Method [Goodfellow et al., 2014] for generating an adversarial example. The task is to find  $x'$  which differs from  $x$  in the label assigned by a classifier  $C$ , which outputs a distribution over  $K$  classes ( $K$  positive numbers that sum to 1).
$$
\begin{array}{l} \| x ^ {\prime} - x \| \leq \epsilon \quad \text {a n d} \\ \underset {i} {\arg \max } [ C (x) ] _ {i} \neq \underset {i} {\arg \max } [ C \left(x ^ {\prime}\right) ] _ {i} \tag {1} \\ \end{array}
$$
Generally, we only restrict attacks to cases where  $\arg \max_{i}[C(x)]_{i}$  is the true label, i.e.,  $C$  is correct. The norm  $\|\cdot\|$  used to define the attack is almost always the  $L_{\infty}$  norm, though other choices such as  $L_{2}$  [Carlini and Wagner, 2017] have been tested as well. We will assume the norm to be  $L_{\infty}$  for the description. Computing  $x'$  (possibly with projection to a valid domain) proceeds as follows:
- Apply  $C$  to  $x$  to obtain  $C(x)$ .
- Compute gradient  $\nabla L$  wrt  $x$  where  $L$  is the loss between  $C(x)$  and the true label vector  $v_{l}$ .
- Compute  $x_{i}^{\prime} = x_{i} + \epsilon \times \mathrm{sign}(\nabla L)_{i}$ , i.e., move by  $\pm \epsilon$  based on sign of gradient.
We understand the sign function to map to  $-1, 0, 1$  respectively as the argument is  $< 0$ ,  $0$ , or  $>0$ . FGSM uses gradient information, which may vary rapidly around  $x$ . A stronger variant, the Projected Gradient Descent (PGD) attack [Madry et al., 2017] uses multiple steps near  $x$  taking gradients at each step. Under PGD, there is a step size  $\alpha$ , typically  $<< \epsilon$ , which iteratively updates  $x_i^t$  as:
$$
x _ {i} ^ {t} = x _ {i} ^ {t - 1} + \alpha \times \operatorname {s i g n} (\nabla L) _ {i} ^ {t - 1}
$$
With a clamping step on  $x_{i}^{t}$  that enforces the constraint of  $\| x' - x \| \leq \epsilon$ , this step after initializing  $x_{i}^{0}$  at  $x_{i}$  yields much stronger adversarial examples than FGSM at higher  $t$ .
1.3 Intermediate Level Attack Projection (ILAP)
These methods - FGSM and PGD - rely on the input and the final layer's output loss with respect to the true label. We can consider a neural network  $N$  of depth  $D$  as:
$$
N (x) = N _ {i + 1, D} (N _ {1, i} (x))
$$
With 1 based indexing,  $N_{j,k}$  is the neural sub-network via composing layers from depth  $j$  to  $k$ . Let  $z_i = N_{1,i}(x)$  be the latent representation at depth  $i$ . Consider  $x_{adv}$  from any baseline method, and let
$$
z _ {i} ^ {a d v} = N _ {1, i} (x _ {a d v}); z _ {i} ^ {o r i g} = N _ {1, i} (x); \Delta_ {i} ^ {a d v} = z _ {i} ^ {a d v} - z _ {i} ^ {o r i g}
$$
We then seek an alternate adversarial example,  $x_{i}^{ILAP}$  which seeks to work with the loss function:
$$
L = \left\langle \Delta_ {i} ^ {a d v}, \Delta_ {i} ^ {I L A P} \right\rangle \text {w h e r e} \Delta_ {i} ^ {I L A P} = z _ {i} ^ {I L A P} - z _ {i} ^ {\text {o r i g}} \text {a n d} z _ {i} ^ {I L A P} = N _ {1, i} \left(x _ {i} ^ {I L A P}\right)
$$
The  $L$  can then be plugged into either FGSM or PGD. Put simply, this loss encourages latent space movement directionally similar to FGSM/PGD.  $L$  increases the inner product with  $\Delta_i^{adv}$  and later iterations use it over the true loss. This alternate adversarial example is highly transferable - e.g., adversarial examples from ResNet18 have higher success rates vs different models such as DenseNet, when created by an ILAP process than directly using FGSM. It can even beat FGSM on the source model itself. [Huang et al., 2019] ILAP demonstrates that latent representations provide ammunition for adversarial attacks, but requires an initial adversarial example  $x_{adv}$  created with full model access and label. To remedy this, we craft an attack needing neither. We show that if the correct layers are attacked with an angular version of the same loss, the initial adversarial example is unnecessary.
2 Geometry of Batch Normalization
Given a minibatch of inputs  $x_{i}$  of dimensions  $n$  with mean  $\mu_{ij}$  and standard deviation  $\sigma_{ij}$  at index  $j$ , consider batch normalization [Ioffe and Szegedy, 2015] or just BN as:
$$
\left[ \mathrm {B N} \left(x _ {i}\right) \right] _ {j} = \frac {x _ {i j} - \mu_ {i j}}{\sigma_ {i j}} \tag {2}
$$
Such normalization (with affine shift) has become commonplace in neural networks. During training, the empirical average of the minibatch is used and during testing, computed averages (mean/standard deviation) are used. By batchnorm, we mean only the batchnorm layer without an affine shift. Layer normalization [Ba et al., 2016] is an alternate form of normalization that normalizes over all neurons in a hidden layer. For space reasons, we discuss batch normalization in the main text and layer normalization in Appendix A. Empirically, our attack succeeds on layer norm and architectures using it, such as transformers [Vaswani et al., 2017] and vision transformers [Dosovitskiy et al., 2020].
Converged regime: Suppose in training, the sample means/standard deviations in a pre-BN layer converge to sample statistics. Post-batchnorm, representation vector  $Z$  satisfies, at every index  $j$ :
$$
\mathbb {E} \left(Z _ {j}\right) = 0, \quad \mathbb {E} \left(Z _ {j} ^ {2}\right) = 1 \tag {3}
$$
i.e., for a layer of dimensionality  $d$  and denoting the entire latent vector as  $Z$ ,  $E(Z) = 0$ ,  $E(\|Z\|^2) = d$  by linearity of expectation. The above convergence is with respect to the training set, but carries over to the test set (approximately) under the I.I.D assumption when learning takes place. Suppose  $\|Z\|^2$  was concentrated about its expected value - i.e., nearly all  $\|Z\|$  values were  $\approx \sqrt{d}$ . We first discuss the implications of such a scenario and what attacks they allow. We assume that in this latent space, the inner product determines 'similarity'. Two latents  $Z_a, Z_b$  from different instances  $X_a, X_b$  will be closer as  $\langle Z_a, Z_b \rangle$  rise. Under this metric, the most dissimilar point to  $Z_a$  is parallel to  $-Z_a$ . We will formulate our attack assuming that  $\|Z\| = \sqrt{d}$  i.e., a hyperspherical latent space. After we have formulated our attack, we consider scenarios of such spaces and concentrations of norm.
2.1 Angular Attack Based on Converged Batchnorm
Let  $N_{1,i}$  be the network up to layer  $i$  with dimensionality  $d$ . If layer  $i$  is a batch norm layer (without affine shift),  $z = N_{1,i}(x)$  lies (with converged batchnorm and  $\| Z\|$ 's norm concentrated) approximately on a shell of radius  $\sqrt{d}$ . The natural distance metric between two  $z, z'$  is angular distance:
$$
\frac {\langle z , z ^ {\prime} \rangle}{\| z \| \| z ^ {\prime} \|}
$$
where  $\langle z,z^{\prime}\rangle$  is at a local maximum on the hypersphere  $\| z\| =$  constant when  $z = z^{\prime}$  and no gradient exists. We need another gradient initially, and propose the following attack algorithm. Given a real example  $x^0$  with latent  $z^0 = N_{1,i}(x^0),z^t = N_{1,i}(x^t)$  we consider the loss  $L_{\mathrm{init}}^{t} = -\| z^{t}\| =$ $-\| N_{1,i}(x^t)\|$  for the first  $t_\mathrm{init}$  iterations. We then iteratively generate  $x^{t}$ :
$$
x ^ {t} = x ^ {t - 1} + \alpha \times \operatorname {s i g n} \left(\nabla L _ {\mathrm {i n i t}} ^ {t - 1}\right) \quad ; \quad 1 \leq t \leq t _ {\mathrm {i n i t}}
$$
After  $t_{\mathrm{init}}$  iterations, we modify the angular loss to  $L_{\mathrm{radial}}$ , using the deviations obtained so far.  $L_{\mathrm{radial}}$  is angularly defined, denoting  $z_i^t$  the latent at depth  $i$  and iteration  $t$ :
$$
L _ {\mathrm {r a d i a l}} = - \frac {\langle z _ {i} ^ {0} , z _ {i} ^ {t} \rangle}{\| z _ {i} ^ {0} \| \| z _ {i} ^ {t} \|}; z _ {i} ^ {0} = N _ {1, i} (x ^ {0}), z _ {i} ^ {t} = N _ {1, i} (x ^ {t})
$$
We then iteratively update as follows (graphically depicted in Figure 1):
$$
x ^ {t} = x ^ {t - 1} + \alpha \times \operatorname {s i g n} \left(\nabla L _ {\text {r a d i a l}} ^ {t - 1}\right) \quad ; \quad t _ {\text {i n i t}} <   t \leq t _ {\text {r a d i a l}}
$$
The figure geometrically depicts how intermediately moving from a point to another in latent space lowers the radial norm ( $A_{i}$ , left). Directly lowering the radial norm via a path from  $A_0$  to  $A_{1}$  (right) can create an initial deviation. But extending this to  $C$  as in the figure ends up with a lower angular deviation than maximizing angular deviation directly and ending up at  $A_{adv}$ . Methods which directly increase the  $L_{2}$  deviation starting from an initial method would result in points such as  $C$ .
Figure 1: L:  $A, C$  are latent image representations.  $A_{i}$  lies on chord  $AC$ . Targeted latent space movements lower norm. R: Initial radial loss towards  $O$  forms  $A_{1}$  with implied movement towards  $C$ .  $A_{2}, \ldots, A_{adv}$  follow. Implied movement is towards  $C'$  - further than  $C$  from  $A$  in  $\theta$  (angles).
Both losses depend on the layer  $i$ . With access till layer  $D$ , multiple candidate  $i$  values exist. For simplicity, we only use  $D, D - 1$  and replace  $\mathrm{sign}(\nabla L_{\mathrm{init}})$  with  $\frac{1}{2} [\mathrm{sign}(\nabla L_{\mathrm{init},D - 1}) + \mathrm{sign}(\nabla L_{\mathrm{init},D})]$  where the subscript  $L_{\mathrm{init},D - 1}$  denotes setting  $i$  as  $D - 1$ , and similarly we set  $i = D$  and average the sign for  $\mathrm{sign}(\nabla L_{\mathrm{radial}})$ . This averaged term becomes  $\Psi$  in our pseudocode (Algorithm 1). We clamp  $x^k$  iterates to satisfy  $\| x^k - x^0 \| \leq \epsilon$ . Our attack resembles ILAP, with the initial direction  $\Delta_i^{adv}$  from an unsupervised radial loss without labels. We then maximize the angular deviation from the original latent, and do not "follow the leader" on the original angular deviation after  $t_{init}$ . The intuition is as follows. The surrogate loss  $-||z^t||$  moves along a chord in the hypersphere's interior from  $z^0$  towards  $-z^0$ , the most dissimilar point, while latent representations are on the surface. We use the last 2 layers for simplicity - stronger attacks might exist using all layers. Concentration of norm forms the hypersphere, and the 2-step process is key. 1-step methods, e.g. single-stage works discussed under related works, use methods such as random initialization and angular minimization, but perform worse (see Appendix P). The concentration is in  $L_2$ , but the adversarial example is in the  $L_{\infty}$  metric.
Algorithm 1 Angular attack algorithm
Input: Neural Network  $N$ ; access of model parameters of  $N$  till layer  $D$ ,  $N_{1,i}$ : sub-network of  $N$  upto layer  $i$ , a starting real sample  $x^0$ , perturbation  $\alpha$
Result:Adversarial example  $x^{t_{\mathrm{radial}}}$
$t_{\mathrm{init}} \gets \mathrm{Num}$  iterations with loss  $L_{\mathrm{init}}$ ;  $t_{\mathrm{radial}} \gets \mathrm{Num}$  iterations with loss  $L_{\mathrm{radial}}$
$$
k \leftarrow 1, n \leftarrow t _ {\mathrm {i n i t}} + 1
$$
while  $k\leq t_{\mathrm{init}}$  do
$$
L _ {\text {i n i t}, i} ^ {k - 1} = - \left\| N _ {1, i} \left(x ^ {k - 1}\right) \right\|; \quad \Psi = \operatorname {s i g n} \left(\nabla L _ {\text {i n i t}, D - 1} ^ {k - 1}\right) + \operatorname {s i g n} \left(\nabla L _ {\text {i n i t}, D} ^ {k - 1}\right)
$$
$$
x ^ {k} \leftarrow x ^ {k - 1} + \frac {\alpha}{2} \times \Psi ; \quad x ^ {k} \leftarrow \operatorname {c l a m p} \left(x ^ {k}, \epsilon\right); \quad k \leftarrow k + 1
$$
while  $n \leq t_{\mathrm{radial}} \mathbf{d}\mathbf{o}$
$$
L _ {\text {r a d i a l}, i} ^ {n - 1} = - \frac {\left\langle N _ {1 , i} \left(x ^ {0}\right) , N _ {1 , i} \left(x ^ {n - 1}\right) \right\rangle}{\| N _ {1 , i} \left(x ^ {0}\right) \| \| N _ {1 , i} \left(x ^ {n - 1}\right) \|}; \quad \Psi = \operatorname {s i g n} \left(\nabla L _ {\text {r a d i a l}, D - 1} ^ {n - 1}\right) + \operatorname {s i g n} \left(\nabla L _ {\text {r a d i a l}, D} ^ {n - 1}\right)
$$
$$
x ^ {n} \leftarrow x ^ {n - 1} + \frac {\alpha}{2} \times \Psi ; \quad x ^ {n} \leftarrow \operatorname {c l a m p} \left(x ^ {n}, \epsilon\right); \quad n \leftarrow n + 1
$$
Return:  $x^{t_{\text{radial}}}$
Moving from  $z^0$  along  $-\nabla \| z^t \|$  does not move linearly towards the origin when the optimization is imperfect - e.g. using the gradient sign. We manipulate  $x^i$ , not  $z^i$  - it may not be  $\exists x'$  such that  $N_{1,i}(x') = \gamma z^0$ ,  $\forall \gamma, 0 \leq \gamma < 1$  (the ray joining origin-  $z^0$  may lack points with pre-images in  $x$ ). Another issue is  $\alpha$  movement direction-wise (ignoring sign averaging) - e.g. let  $Z_i \in \mathbb{R}^2$ ,  $N_{1,i}(x) = x$  - the representation as identity map. Let  $x = (1,\epsilon)$  lie on the hypersphere of radius  $\sqrt{1 + \epsilon^2}$ . The sign of  $\nabla L_{\mathrm{init}}$  is  $(-, -)$  and  $x^0 = (1 - \alpha, \epsilon - \alpha)$  - not necessarily collinear with  $(0,0) - (1,\epsilon)$ . Batch
normalization leaks mean/variance to adversaries. Suppose we had a perfect network  $N$  (could find  $x$  mapping to any  $z$ , i.e., was onto), and were assured the latent space was a zero mean hypersphere and we get adversarial instances if we move the latent from  $z$  to  $-z$ . This would be insufficient if all latents were translated by some vector  $c$ , i.e., the mean of the hypersphere was nonzero. To find the point opposite  $z$  on the new hypersphere  $(c - (z - c))$ , we need  $c$ . Batch norm gives us this  $c$ . This weakness outweighs knowing  $z'$  arising from  $x'$  of a different label, as we show in the results.
2.2 When Does the Norm Concentrate?
We assumed that after a batch normalization step,  $\| Z \|$  concentrates. Consider sufficiency conditions. Let  $Z_{ij}$  be the  $j$ -th entry of the  $i$ -th latent  $Z_{i}$  formed from  $X_{i}$  with expectation 0, variance 1 (batch norm). Consider independent instances  $X_{i}$  and assume no instances indexed by  $i$  were used for training to impact model parameters, which affects the other latents and makes them dependent - i.e. consider test/Validation sets. Then,  $Z_{ij}, Z_{i'j'}$  are independent if  $i \neq i', \forall j' = j$ . Now:
$$
\left\| Z _ {i} \right\| ^ {2} = \sum_ {j = 1} ^ {d} Z _ {i j} ^ {2}
$$