SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: CIFAR-100 Class-Incremental Learning Split CIFAR-100 is trained with 10 episodes of 10 classes in the standard CIL evaluation suite [72]. Average accuracy and forgetting is reported over the 10 episodes according with [10]. *Sequential fine-tuning results are taken from [72].
Top-1 Test Error
Dataset	InCA	Open-InCA	Query only Open-InCA	In. LP
CUB-200	9.1	9.5	12.1	16.2
DTD	17.8	17.1	19.2	18.9
Aircrafts	15.8	18.1	38.6	50.6
MIT-67	10.1	9.4	9.1	9.7
Oxford Flowers	0.3	0.4	0.4	0.6
Oxford Pets	4.7	4.0	5.4	6.1
Stanf. Cars	8.4	8.4	22.8	29.2
Stanf. Dogs	8.1	5.7	5.3	5.3
Average	9.3	9.1	14.1	17.1
Table 6: Open-InCA adapter performance We compare InCA, Open-InCA, "querytraining" Open-InCA and Intermediate Linear Probing (In. LP). We observe that Open-InCA is comparable with InCA and that "querytraining" significantly out-performs In. LP.
B Intermediate Representation Signatures
The parallel training of InCA results in the synthesis of tens of models that can run inference with insignificant per-adapter marginal costs. As a result we have the ability to glean highly useful information about the network's different representations and study the network inner representations effectively. This is especially important for recent non-convolutional based architectures that do not have as many inductive biases explaining some of their behavior. In this section we present results showing information we retrieve from the performance of InCA adapters.
Figure 4: (repeated) Partial fine-tuning vs. InCA Vertical dashed lines indicate the top InCA layer; curves show final test accuracy for different partial tuning training runs. Each mark indicates a run where all of the pre-trained model parameters are trained up to a "freeze point" in the network's layers. Note partial tuning performance saturates in close proximity to the optimal InCA adapter layer. This is aligned with our hypothesis that full fine-tuning attempts to surface existing representations already in the network. In that case, performance improves until the tuning approach unlocks the capacity to utilize an existing relevant representation and performance plateaus afterwards.
B.1 Partial fine-tuning and adapter performance
Below we present in detail the experiments discussed in Sec. 5, in particular regarding partial fine-tuning and InCA. The experiments illustrate the relationship between InCA adapters at different layers with partial tuning. We tune the pre-trained model starting from different "freezing points". In particular for neural network  $f(x) = g_{1} \circ \ldots \circ g_{l}$ , for each freezing point  $g_{m}$  we consider its position  $m$  and all of the preceding layers and apply layer freezing to  $g_{1}, \ldots, g_{m-1}$  (i.e. not updating gradients for those layers). Back-propagation is then only applied for optimizing  $g_{m}, \ldots, g_{l}$  including the network's prediction "head". In Figure 4 we show the dynamics of partial tuning, where we optimize the pre-trained network (ViT-L/16) in different runs with each run having a different freezing point. We compile the final Top-1 Test accuracy of each freezing run to create a partial tuning "curve" for a single dataset. We compare the partial tuning performance curve of each dataset with the corresponding top layer of the InCA adapter trained on that dataset and observe that they are highly aligned, with datasets that prefer later InCA layers plateauing in their test accuracy earlier (at a later freezing point). In particular what we observe is that partial tuning performance plateaus roughly at the same layer where InCA identifies the top adapter representation. This is also the point at which partial fine-tuning is capable of harnessing that representation for the downstream task. Overall this gives further evidence that "your representations are in the network" and fine-tuning simply surfaces existing representations that are already identified by InCA. When drawing the vertical lines of the top InCA adapters, we refer to output layers, e.g., the adapter at block 19 means the adapter corresponding to the final output of block 19, or the first input of block 20.
B.2 Task Layer Affinities
In InCA we select top-performing adapters that "listen" to different intermediate representations of a neural network. In our work we observe that one is able to achieve strong and diverse transfer learning by utilizing intermediate representations, and that for challenging tasks it is often required to use intermediate representations to achieve top results. Indeed the best representation layer for an adapter tends to be highly robust to hyper-parameter variables of the optimization. Even more intriguingly, we find that this representation affinity is preserved across different pre-trainings and even architectures. This is, certain tasks have a strong "affinity" to a certain range of representation layers even for different architectural circumstances. The majority of the architectures we consider have some pre-trained component on one of the ImageNet datasets (aside from the CLIP ViT-L/14 model). At the same time, the fact that different architectures give rise to similarly helpful representations gives strong clues about the effect of different architectures as compared with the pre-training task during learning over a large diverse dataset such as ImageNet.
Figure 5: Best-performing representation for InCA adapter for Aircraft (Top-left) MIT-67 (Top-right) and Stanf. Cars (Bottom).
In detail, we look at the best-performing InCA adapter for a fixed task on different architectures. The pre-trained models we consider consist of 2 different pre-trainings of the ViT-L/16 architecture (ViT- original and DeiT), the 384-resolution pre-training of DeiT with the resolution adjusted ViT-L/16, CLIP's ViT-L/14 architecture, the SWIN-L architecture, and the convolutional based ConvNext-Base architecture. All of the vanilla ViTs we consider each have 24 residual transformer blocks so that comparing between blocks is directly aligned. SWIN-L and ConvNext follow the "Stage" breakdown of blocks, namely SWIN-L has (2,2,18,2) stage breakdown that conveniently also adds up to 24 blocks (hence aligned in the figure) and lastly, ConvNext follows a  $(3 - 3 - 26 - 3) +$  head stage block composition, which we rescale in the figures to fit on the same 24 block range. In addition in the plots we also present the test error gap of each architecture with using the InCA adapter applied on its final block representation. Tasks that prefer earlier layers such as Stanf. Cars and Aircraft have a large gap from the performance of the last layer representation adaptation and such later layers lead to sub-optimal results.
We remark that the work of InCA sheds light on the inner representations learned by neural networks showing in some aspects performance is invariant to the architecture and more based on the pretraining dataset. We leave this topic for further research and find it to be an intriguing topic of study.
C Efficiency Results
InCA is highly efficient especially for large models, which is based on the isolated adapter architecture that does not modify the backbone. We delineate the efficiency aspects as follows:
- Training memory efficiency The use of a frozen pre-trained model makes the training much more efficient and scalable since not all of the intermediate computations need to be stored as done in standard training or as required by methods that compute gradient information using inner-layers of the network. As soon as any intermediate layer requires a gradient, all subsequent activations must be held in GPU memory after the forward pass. This means methods like LoRA, FitBit and VPT all require storing of all of the activation maps for all of the layer operations in the network since they update parameters based on gradients from the very early layers in the network.
- Fast optimization Unlike typical parameter efficient methods that insert some form of trainable parameters in the network, InCA adapters are trained with "direct gradient" information coming from an isolated loss. Essentially each adapter corresponds to a very shallow neural network trained directly via back-propagation. This makes the training dynamics fast as direct gradient information about the loss easily reaches all of the adapter parameters. On the other hand, to update inserted parameters in the backbone, the gradient information is indirect and needs to be back-propagated through the backbone, with the risk of information loss and making the optimization more challenging, as we and the authors [34] observe regarding prompt tuning.
- Efficient multi-task inference As we present in Table 4 the unchanged backbone execution enables efficient and parallel inference efficiency as multiple tasks can be evaluated at once.
C.1 Computational Efficiency of InCA Compared with VPT
In Table 7 we observe that InCA is an order of magnitude more efficient to train than VPT. For the results in the table we consider the VPT-Deep adaptation method, that is trained with 50 prompt tokens in each layer. We report calculated training times in GPU-hours of a standard Nvidia-T4 GPU using a ViT-L/16 architecture and accuracy numbers based on the datasets of Table 1 with the DeiT pre-training. For larger architectures such as ViT-H/14 ("ViT Huge") the difference in training-time is even more striking, as InCA maintains good per-run training time of 2.5, VPT-Deep requires staggering 55.8 GPU-hours per-run for a single GPU. On ViT-H/14 this is exacerbated as we must reduce the batch-size of VPT significantly to fit training on a common-place single GPU (Nvidia-T4). We measure in terms of training InCA and VPT-Deep for the same number of epochs. This however, is inaccurate as InCA trains an order of magnitude faster on a per epoch basis (see Figure 6).
Method	Mean Test Err.	Max. Full-FT gap	Training time per run (GPU hrs.)	# Hparam. per dataset	Train time per dataset (GPU hrs.)
InCA	10.2	2.4	2.0	2 (parallel)	4.0 (2.4*)
VPT Deep [34]	12.3	6.8	5.8	24	139.6
Table 7: Computation costs of adaptation We adapt ViT-L/16 to CUB-200 downstream classification with the same number of training epochs. We evaluate the training and computational costs of a single run and training VPT-Deep and InCA for one training dataset. *Training with 2 learning rates in parallel leads to training time decreasing from 4.0 to 2.4 GPU-hours.
We attribute the difference in training time of InCA to:
1. InCA does not require back-propagation through the whole model which gives  $\sim 50\%$  speed improvement alone.
2. InCA is robust to hyper-parameters and we optimize it using just 2 learning rates, compared with the hyper-parameter set of VPT (in our experiments we use 24 hyper-parameter configurations per dataset while using the full configuration presented in [33] takes even longer). In addition, with "one-to-many" training, we train the two hyperparameters of InCA in parallel and report the specific training time in Table 7 denoted by  $(^{*})$  for parallel hyper-parameter training.
3. InCA does not increase the number of propagated tokens in the transformer (e.g. in VPT with 100 propagated tokens the attention matrix doubles, from  $\sim 4\times 10^4$  to  $\sim 9\times 10^4$  entries).
C.2 Optimization dynamics of InCA and VPT
In Fig. 6 we conduct an experiment where we train InCA and state of the art prompting method VPT-Deep [34] for different numbers of epochs and report the final test accuracy. We observe that InCA trains order of magnitude faster than prompting and reaches within  $95\%$  relative test accuracy after 3 training epochs.
Figure 6: Optimization Speed for training InCA and prompt tuning (VPT-Deep) on the Aircrafts dataset. We train each method until completion with varying numbers of epochs and report the relative final test accuracy to 50 epoch training. The shallow adapter architecture and direct gradient signal in InCA makes the training of the adapter an order of magnitude faster (in terms of gradient updates) than prompt tuning approaches. Both methods use batch-size 32 and take the same number of gradient steps in each corresponding run, under the optimal learning rate.
Empirically, we consistently observe that cross-attention as opposed to a linear or MLP-3 architecture enables InCA to better harness the existing model. We present a theoretical result asserting that using the cross-attention layer for aggregation as opposed to linear averaging, or even full-concatenation followed by a large dimensional linear layer is capable of learning over a strictly broader set of data distributions.
We give the precise statement in Theorem D.1 and intuitively argue that cross-attention with learned queries has the ability to sift through irrelevant pieces of the representation that may be at variable positions in different data samples.
Recall in the settings considered thus far, the extracted activation of an image data-point can be viewed as  $\mathbf{x}_i\in \mathbb{R}^{d\times T}$  or  $T$  tokens e.g.  $\mathbf{x}_i = [x_i^1,x_i^2\dots x_i^T ]$ , with  $x_{i}^{j}\in \mathbb{R}^{d}$ . We argue that in many scenarios, task-pertinent information is a property of individual tokens (e.g.  $x_{i}^{j}$ ) within a data-point  $\mathbf{x}_i$  and not a property of the overall feature map. We present the theorem below. To this end we define a Token-Separability (TS) notion of a dataset.
Definition 1 (Token-separable Dataset). A dataset  $\mathcal{D} = \{(\mathbf{x}_1, y_1), \ldots, (\mathbf{x}_n, y_n)\}$  with  $\mathbf{x}_i = [x_i^1, x_i^2, \ldots, x_i^T] \in \mathbb{R}^{d \times T}$  and  $y_i \in \{-1, 1\}$  is said to be linearly-token-separable if there exists a scalar  $c > 0$  and  $w \in \mathbb{R}^d$  satisfying  $\| w \|_2 = 1$ , such that for each data point  $(\mathbf{x}_i, y_i) \in \mathcal{D}$  there exists a token  $x_i^{j_i} \in \mathbf{x}_i$  with
$$
y _ {i} \left(\left\langle x _ {i} ^ {j _ {i}}, w \right\rangle + b\right) \geq c. \tag {1}
$$
We define  $(w_{\mathcal{D}},b_{\mathcal{D}})$  and  $c_{0}$  as the maximum margin solution and maximum margin respectively, i.e.  $c_{0} = \max_{\{\| w\| = 1,b\in \mathbb{R}\}}\min_{\mathcal{D}}\{y_{i}(\langle x_{i}^{j_{i}},w\rangle +b)\}$  for  $\mathcal{D}$  with  $(w_{\mathcal{D}},b_{\mathcal{D}})$  corresponding to the selected  $c_{0}$ .
Intuitively,  $\mathcal{D}$  is a TS-dataset if each of its data points contain a token that leads to linear separability (the same  $w$  shared among all points  $\mathbf{x}_i\in \mathcal{D}$ ). One can further distinguish between aligned-TS Datasets, where the index  $j_{i}$  of the linearly separating token is consistent among the  $n$  data points, or permutable-TS where  $j$  is dependent on  $i$ . Further TS datasets can be generalized to  $k$ -token separable datasets where  $k$  tokens are responsible for separability in each  $\mathbf{x}_i$ , for this theoretical contribution we don't make an assumption on whether the dataset is aligned or permuted, but consider the setting provided by Definition 1 (i.e. not the  $k$ -separable generalization). We present an analytical statement for the advantage of cross-attn, the theorem is provided for binary classification via a scalar prediction, but can conventionally extend to  $C$ -class classification. For binary classification we define a prediction via the standard scalar binary aggregator as  $\sigma (u) = \mathrm{sign}(\sum_i u_i)$  that converts a vector into a binary prediction.
Theorem D.1. Let  $\mathcal{D}$  be a binary-class, token-separable dataset with max-margin  $c_{\mathcal{D}}$  and max-margin solution  $(w_{\mathcal{D}}, b_{\mathcal{D}})$  consisting of  $n$  data points. Suppose that  $\mathcal{D}$  is distributed such that for  $\mathbf{x}_i = (x_i^1, \ldots, x_i^T)$  with  $(\mathbf{x}_i, y_i) \in \mathcal{D}$  are normalized for separating token  $x_i^{j_i} \in \mathcal{D}$  and that the rest of the tokens correspond to "noise",  $x_i^k \sim N(0, I / d)$ ,  $\mathbb{E}\| x_i^k\|_2^2 = 1$ . Furthermore assume
$$
c _ {\mathcal {D}} \geq \max  \left(\sqrt {\frac {3 2}{d} \left(\log (1 / \delta) + \log (2 n T)\right)}, 2 | b _ {\mathcal {D}} |\right). \tag {2}
$$
Then there exists a cross-attention classifier
$$
f (x; q, \{W \}, b) = \sigma \left(\sum_ {l = 1} ^ {T} \operatorname {c r o s s - a t t n} (\mathbf {x}, q) _ {l} + b\right) \tag {3}
$$
that separates  $\mathcal{D}$  with probability at least  $1 - \delta$ . In contrast, every fixed member  $g(\mathbf{x};w,b) = \sigma\left(\sum_{l=1}^{T} \mathbf{x}_i \cdot w + b\right)$  of the linear classifier family will fail to separate  $\mathcal{D}$  with probability at least  $\frac{1}{\sqrt{2\pi}} \frac{s}{s^2 + 1} \exp(-s^2 / 2)$  where  $s = \frac{\sqrt{d}}{\sqrt{T-1}} c_{\mathcal{D}}$ .
As stated above, the failure probability of the simple linear classifier  $g$  depends on  $s$  which satisfies  $s \sim \sqrt{d / T}$ . For existing architectures  $d, T$  tend to have a similar order of magnitudes, e.g. for ViT-B/16,  $d = 768$ ,  $T = 196$  which makes the failure probability non-negligible. Before presenting the proof, we make the following observation: in InCA, we use the same cross-attn layer with latent  $q$ , which we show can be simplified via reparameterization.
Observation D.2 (Query Collapse Reparameterization). A single-head cross-attention parameterization with latent  $[q]$  is equivalent to the following simplified layer: cross-attn( $\mathbf{x}, q$ ) =  $\sum \text{softmax}(q^*\mathbf{x}) \odot \mathbf{W}\mathbf{x}$  with  $q^* \in \mathbb{R}^d$ .
This can be derived by decomposing the attention score which is the input to softmax.
$$
a _ {j} = \left\langle \mathbf {W} _ {q} q, \mathbf {W} _ {k} x ^ {j} \right\rangle = \left(\mathbf {W} _ {q} q\right) ^ {\top} \left(\mathbf {W} _ {k} x ^ {j}\right) =
$$
$$
q ^ {\top} \mathbf {W} _ {q} ^ {\top} \mathbf {W} _ {k} x ^ {j} = (q ^ {*}) ^ {\top} x ^ {j}.
$$
Where  $q^{*} = q^{\top}\mathbf{W}_{q}^{\top}\mathbf{W}_{k}$  and  $q^{*}\in \mathbb{R}^{d}$ , hence the cross-attn layer simplifies, which is used in the proof.
As our proof shows, the cross-attn layer can operate on a large data bandwidth, e.g.  $\mathbf{x} \in \mathbb{R}^{d \times T}$  while still being selective in finding task specific representations. Empirically we also observe that increasing the number of heads of cross-attn also improves the performance of the InCA. This is in part because it enables the learned latent query parameter  $q$  to identify more useful token patterns, and since  $q$  is fixed using more heads remain stable (as opposed to when  $q$  is a data input). We now present the proof of the claim.
Proof of Theorem D.1
Proof. The proof of the theorem has two parts A) the positive condition on the cross-attn layer and B) the negative condition on the linear layer (a non-separability probability lower bound). We start with A) and consider separability of positive and negative data samples in turn. First we simplify and write an equivalent cross-attention binary classifier expression for the cross-attn classifier.
Positive result for the cross-attention model We consider a "single-head" cross-attention layer and by Observation D.2 we can write the cross-attn layer as follows
$$
\operatorname {c r o s s - a t t n} \left(\mathbf {x} _ {i}; q, \left\{\mathbf {W} \right\}\right) = \sum_ {j = 1} ^ {T} \operatorname {s o f t m a x} \left(\left\langle x _ {i} ^ {j}, q ^ {*} \right\rangle\right) \cdot \mathbf {W} _ {v} x _ {i} ^ {j}. \tag {4}
$$
Note that softmax is a function of the entire vector  $\{\langle x_i^j,q^*\rangle \}_{j\in [1,T]}$ , however we write it in the form above to illustrate the summed terms. For simplicity of notation, we drop the asterisk and write  $q^{*}\in \mathbb{R}^{d}$  as  $q$ . Combining cross-attn with the binary aggregator, we have aggregation over the output vector of the cross-attn layer.
$$
\begin{array}{l} f (\mathbf {x} _ {i}; q, \{W \}, b) \\ = \sigma \Bigg(\sum_{l = 1}^{d}\big(\operatorname {cross - attn}(\mathbf{x}_{i};q,\{W\})\big)_{l} + b\Bigg). \\ \end{array}
$$
Define  $S(\mathbf{x}_i,q)\in \mathbb{R}^{1\times T}$  as the computed softmax argument,
$$
S \left(\mathbf {x} _ {i}, q\right) = S = \operatorname {s o f t m a x} \left(\left[ \left\langle x _ {i} ^ {1}, q \right\rangle , \dots \left\langle x _ {i} ^ {T}, q \right\rangle \right]\right). \tag {5}
$$
Substituting into the classifier, we have
$$
\begin{array}{l} f (\mathbf {x} _ {i}; q, \{\mathbf {W} \}, b) = \sigma \left(\sum_ {l = 1} ^ {d} \left(\sum_ {j = 1} ^ {T} S _ {j} \cdot \mathbf {W} _ {v} x _ {i} ^ {j}\right) _ {l} + b\right) \\ = \sigma \left(\sum_ {j = 1} ^ {T} S _ {j} \sum_ {l = 1} ^ {d} \left(\mathbf {W} _ {v} x _ {i} ^ {j}\right) _ {l} + b\right). \\ \end{array}
$$
Let  $u = \sum_{l=1}^{d}(\mathbf{W}_v)_{[l,:]}$  be the sum of the rows of  $\mathbf{W}_v$ . Note  $x_i^j$  can be pulled out from the inner summation to give
$$
f (\mathbf {x}; q, u, b) = \sigma \left(\sum_ {j = 1} ^ {T} S _ {j} \langle u, x _ {i} ^ {j} \rangle + b\right).
$$
Thus the cross-attn classifier presented is equivalent to the parameterization above. Next we consider the two terms in the sum, namely  $S_{j}$  and  $\langle u,x_i^j\rangle$ . We will be deriving their distribution in the case where a data point  $(\mathbf{x}_i,y_i)$ , has prediction labels  $y_{i} = 1$  and  $y_{i} = -1$  separately. We start with  $y_{i} = 1$  and consider
$$
S _ {k} = \frac {\exp \left(\langle x _ {i} ^ {k} , q \rangle\right)}{\sum_ {j = 1} ^ {T} \exp \left(\langle x _ {i} ^ {j} , q \rangle\right)}. \tag {6}
$$
Take  $j_{i}$  to be the separating token for sample  $\mathbf{x}_i$ . By the assumption of the theorem for  $k \neq j_{i}$  the tokens correspond to isotropic noise of expected squared norm 1, i.e.  $\mathbf{x}_i^k \sim N(0, I / d)$ . For a fixed  $u \in \mathbb{R}^d$  with  $\| u \|_2 = 1$ , we take  $\eta_k$  to be the distribution of the dot product,
$$
\eta_ {k} = \langle u, \mathbf {x} _ {i} ^ {k} \rangle = \sum_ {l} (u) _ {l} \cdot \left(\mathbf {x} _ {i} ^ {k}\right) _ {l}. \tag {7}
$$
For  $k \neq j_{i}$  this is a sum of independent Gaussians and each coordinate is distributed as  $\sim N(0, \frac{u_i^2}{d})$ . As such we have
$$
\begin{array}{l} \langle u, \mathbf {x} _ {i} ^ {k} \rangle \sim N \left(0, \sum_ {l} \frac {1}{d} \cdot u _ {l} ^ {2}\right) = N (0, \| u \| _ {2} ^ {2} / d) \\ = N (0, 1 / d) \\ \end{array}
$$
since  $\| u\| _2 = 1$ . Next we consider  $k = j_{i}$ . By the hypothesis we have that
$$
y _ {i} \left(\left\langle \mathbf {x} _ {i} ^ {j _ {i}}, w _ {\mathcal {D}} \right\rangle + b _ {\mathcal {D}}\right) \geq c _ {\mathcal {D}}.
$$
With positive label  $(y_{i} = 1)$  this gives  $\langle \mathbf{x}_i^{j_i},w_{\mathcal{D}}\rangle +b_{\mathcal{D}}\geq c_{\mathcal{D}}$  . Note that since  $c_{\mathcal{D}}\geq 2|b_{\mathcal{D}}|$  we have that  $\langle \mathbf{x}_i^{j_i},w_{\mathcal{D}}\rangle \geq c_{\mathcal{D}} / 2$  . Since  $w_{\mathcal{D}}$  is the maximal margin solution, we have  $\| w_{\mathcal{D}}\| = 1$  and  $c_{\mathcal{D}} > 0$  Take  $q$  to be of the form  $q = t\cdot w_{\mathcal{D}}$  for  $t\in \mathbb{R}^+$  and  $u = w_{\mathcal{D}}$  , then
$$
t \cdot \eta_ {j _ {i}} = \left\langle x _ {i} ^ {j _ {i}}, q \right\rangle = t \left\langle x _ {i} ^ {j _ {i}}, w _ {\mathcal {D}} \right\rangle \geq t \cdot c _ {\mathcal {D}} / 2. \tag {8}
$$
For  $\eta_{k}$ ,  $k \neq j_{i}$ , separating  $t$ , we have that  $\langle x_i^k, q \rangle = t \cdot \eta_k$ . Define  $M = \max_{k \neq j_i}(|\eta_k|)$ .  $M$  is a random variable distributed as the maximum of  $T - 1$  i.i.d. Gaussians distributed according to  $N(0,1/d)$ . We bound  $M$  by investigating an upper bound of the Gaussian CDF. Recall that the moment generating function of a Gaussian random variable  $X \sim N(0,1)$  is given by  $M_X(r) = \mathbb{E}[e^{rX}] = e^{\frac{1}{2} r^2}$ . Then note that for any  $s > 0$  we have
$$
\mathbb {P} (X \geq r) = \mathbb {P} \left(e ^ {s X} \geq e ^ {s r}\right) \leq e ^ {- s r} M (s) = e ^ {- s r + \frac {1}{2} s ^ {2}}
$$
where the inequality is an application Markov's inequality. Setting  $s = r$  this gives the tail bound
$$
\mathbb {P} (X \geq r) \leq \exp (- r ^ {2} / 2). \tag {9}
$$
For our settings with  $\eta_k\sim N(0,1 / d)$
$$
\mathbb {P} \left(\eta_ {k} \geq r\right) \leq \exp \left(- d r ^ {2} / 2\right). \tag {10}
$$