SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For a two-sided bound, by symmetry of the distribution we have
$$
\mathbb {P} \left(\left| \eta_ {k} \right| \geq r\right) \leq 2 \exp \left(- d r ^ {2} / 2\right). \tag {11}
$$
Therefore a union bound results in
$$
\begin{array}{l} \mathbb {P} (M \geq r) = \mathbb {P} \left(\bigcup_ {k \neq j _ {i}} | \eta_ {k} | \geq r\right) \\ \leq \sum_ {k \neq j _ {i}} \mathbb {P} (| \eta_ {k} | \geq r) \\ \leq (T - 1) \cdot 2 \exp (- d r ^ {2} / 2) \\ \leq 2 T \exp (- d r ^ {2} / 2). \\ \end{array}
$$
We can bound the bulk of the distribution of  $M$  as
$$
\mathbb {P} (M <   r) \geq 1 - 2 T \exp (- d r ^ {2} / 2). \tag {12}
$$
Taking  $r = c_{\mathcal{D}} / 4$ , then with probability at least  $1 - 2T\exp (-d(c_{\mathcal{D}} / 4)^2 /2) = 1 - 2T\exp (-d(c_{\mathcal{D}})^2 /32)$  we have
$$
M = \max  _ {k \neq j _ {i}} | \eta_ {k} | <   \frac {c _ {\mathcal {D}}}{4} \tag {13}
$$
$$
\max  _ {k \neq j _ {i}} (\langle q, x _ {i} ^ {k} \rangle) <   \frac {t c _ {\mathcal {D}}}{4}. \tag {14}
$$
With high probability, Eq. (13) holds. This implies that for  $j_{i}$ ,
$$
\begin{array}{l} S _ {j _ {i}} = \frac {\exp (\langle x _ {i} ^ {j _ {i}} , q \rangle)}{\sum_ {j = 1} ^ {T} \exp (\langle x _ {i} ^ {j} , q \rangle)} \\ = \frac {1}{1 + \sum_ {j \neq j _ {i}} ^ {T} \exp (\langle x _ {i} ^ {j} , q \rangle - \langle x _ {i} ^ {j _ {i}} , q \rangle))} \\ \geq \frac {1}{1 + \sum_ {j \neq j _ {i}} ^ {T} \exp (\langle x _ {i} ^ {j} , q \rangle - t c _ {\mathcal {D}} / 2))} \\ \geq \frac {1}{1 + \sum_ {j \neq j _ {i}} ^ {T} \exp (- t c _ {\mathcal {D}} / 4))} \\ = \frac {1}{1 + (T - 1) \exp (- t c _ {\mathcal {D}} / 4))} \\ = 1 - \frac {(T - 1) \exp (- t c _ {\mathcal {D}} / 4))}{1 + (T - 1) \exp (- t c _ {\mathcal {D}} / 4))}. \\ \end{array}
$$
Note that  $c_{\mathcal{D}} > 0$  and  $T$  is fixed. Nonetheless, the probability bound is independent of  $t$  which may take arbitrarily values, e.g. for any  $\epsilon > 0$ , take  $t = 4 / c_{\mathcal{D}} \log (T / \epsilon)$ , which gives
$$
S _ {j i} \geq 1 - \epsilon . \tag {15}
$$
Since  $S_{k}\geq 0$  for each  $k$  and  $\sum_{k = 1}^{T}S_{k} = 1$  we have for  $k\neq j_{i}$
$$
S _ {k} \leq \sum_ {j \neq j _ {i}} S _ {j} = 1 - S _ {j _ {i}} \leq \epsilon . \tag {16}
$$
We consider the classifier prediction
$$
f (\mathbf {x}; q, u) = \sigma \left(\sum_ {j = 1} ^ {T} S _ {j} \langle u, x _ {i} ^ {j} \rangle + b\right) \tag {17}
$$
where  $b$  is a bias parameter we can choose. Recall  $u = w_{\mathcal{D}}$ . Focusing on the inside of the sign function
$$
\begin{array}{l} \sum_ {j = 1} ^ {T} S _ {j} \langle w _ {\mathcal {D}}, x _ {i} ^ {j} \rangle = S _ {j _ {i}} \langle w _ {\mathcal {D}}, x _ {i} ^ {j _ {i}} \rangle + \sum_ {k \neq j _ {i}} S _ {k} \eta_ {k} \\ \geq S _ {j _ {i}} \left\langle w _ {\mathcal {D}}, x _ {t} ^ {j _ {i}} \right\rangle - \sum_ {k \neq j _ {i}} S _ {k} \max  _ {j \neq j _ {i}} (| \eta_ {j} |) \\ \geq (1 - \epsilon) c _ {\mathcal {D}} / 2 - \epsilon \cdot c _ {\mathcal {D}} / 4 \\ = (1 - (3 / 2) \epsilon) c _ {\mathcal {D}} / 2 > \frac {c _ {\mathcal {D}}}{4} \\ \end{array}
$$
provided that  $\epsilon < 1/3$ . If we take  $b = -\frac{c_{\mathcal{D}}}{4}$  we have that for  $q = tw_{\mathcal{D}}$  and  $u = w_{\mathcal{D}}$
$$
f \left(\mathbf {x} _ {i}; q, u, b\right) = \sigma \left(\sum_ {j = 1} ^ {T} S _ {j} \langle w _ {\mathcal {D}}, x _ {i} ^ {j} \rangle + b\right) = 1 = y _ {i}. \tag {18}
$$
Next we address the case where  $y_{i} = -1$ . We consider the classifier prediction
$$
f (\mathbf {x}; q, u, b) = \sigma \left(\sum_ {j = 1} ^ {T} S _ {j} \langle u, x _ {i} ^ {j} \rangle + b\right) \tag {19}
$$
with  $u = w_{\mathcal{D}}$ . Again for  $k \neq j_i$  we have that
$$
\max  _ {k \neq j _ {i}} \left(\left\langle u, x _ {i} ^ {k} \right\rangle\right) <   c _ {\mathcal {D}} / 4. \tag {20}
$$
On the other hand for  $j_{i},y_{i} = -1$  we have that
$$
\begin{array}{l} y _ {i} \left(\left\langle x _ {i} ^ {j _ {i}}, w _ {\mathcal {D}} \right\rangle + b _ {\mathcal {D}}\right) \geq c _ {\mathcal {D}} \\ \Longrightarrow \langle x _ {i} ^ {j _ {i}}, w _ {\mathcal {D}} \rangle + b _ {\mathcal {D}} \leq - c _ {\mathcal {D}} \\ \Longrightarrow \langle x _ {i} ^ {j _ {i}}, w _ {\mathcal {D}} \rangle \leq - c _ {\mathcal {D}} / 2 <   0 \\ \end{array}
$$
where we have used the hypothesis that  $c_{\mathcal{D}} \geq 2|b_{\mathcal{D}}|$  in the last line. We consider the term inside the classifier. We note that
$$
\begin{array}{l} \sum_ {k = 1} ^ {T} S _ {k} \langle u, x _ {i} ^ {k} \rangle <   \sum_ {k \neq j _ {i}} ^ {T} S _ {k} \langle u, x _ {i} ^ {k} \rangle \\ <   \sum_ {k \neq j _ {i}} ^ {T} S _ {k} \cdot (c _ {\mathcal {D}}) / 4 \\ \leq c _ {\mathcal {D}} / 4 \\ \end{array}
$$
where in the first inequality we have used the fact that  $S_{j_i}\langle u,x_i^{j_i}\rangle < 0$  and in the last inequality we have used the fact that  $\sum_{j = 1}^{T}S_{j} = 1$ . Therefore again with bias term  $b = -c_{\mathcal{D}} / 4$  we have that
$$
\sum_ {j = 1} ^ {T} S _ {j} \left\langle u, x _ {i} ^ {j} \right\rangle + b <   0 \tag {21}
$$
and  $f(\mathbf{x}_i; q, u, b) = -1 = y_i$ . Thus we have just shown that with probability  $1 - 2T\exp(-d(c_{\mathcal{D}})^2/32)$  the model  $f(x; q, u, b)$  with  $u = w_{\mathcal{D}}, q = tw_{\mathcal{D}}$ ,  $b = -c_{\mathcal{D}}/4$  gives the correct label for  $\mathbf{x}_i$ . Taking the union bound over all  $n$  points in  $\mathcal{D}$  we get with probability at least  $1 - 2Tn\exp(-d(c_{\mathcal{D}})^2/32) \geq 1 - \delta$  the model  $f(x; q, u, b)$  with  $u = w_{\mathcal{D}}, q = tw_{\mathcal{D}}$ ,  $b = -c_{\mathcal{D}}/4$  separates  $\mathcal{D}$ .
Negative result for the linear model We consider the linear classifier
$$
g (\mathbf {x}; w, b) = \sigma \left(b + \sum_ {j = 1} ^ {T} \langle w, x ^ {j} \rangle\right) \tag {22}
$$
where  $w \in \mathbb{R}^d$  is restricted to have unit norm  $\| w \| = 1$ . For an input  $\mathbf{x}_i$  under the aggregation, the term inside the sign function simplifies to
$$
\sum_ {j = 1} ^ {T} \langle w, x _ {i} ^ {j} \rangle = \langle w, \sum_ {j = 1} ^ {T} x _ {i} ^ {j} \rangle .
$$
We recall that the  $x_{i}^{k}$  for  $k \neq j_{i}$  are distributed according to  $N(0, \frac{1}{d} I)$ . Thus we have that
$$
\alpha_ {i} := \sum_ {k \neq j _ {i}} x _ {i} ^ {k} \sim N \left(0, \frac {T - 1}{d}\right). \tag {23}
$$
So the problem of classification is equivalent to learning a linear classifier over the separating tokens under the presence of Gaussian noise with distribution  $N(0, \frac{T - 1}{d})$ . Let  $i^{*}$  be the index corresponding to the input  $\mathbf{x}_{i^{*}}$  with smallest margin, i.e.
$$
i ^ {*} = \operatorname * {a r g m i n} _ {i} y _ {i} (\langle w, x _ {i} ^ {j _ {i}} \rangle + b _ {\mathcal {D}}).
$$
Then we have that  $y_{i^*}(\langle w, x_{i^*}^{j_{i^*}} \rangle + b_{\mathcal{D}}) \leq c_{\mathcal{D}}$ . We note that any linear classifier  $g(\mathbf{x}; w, b)$  with  $\| w \|_2 = 1$  will fail to classify  $\mathcal{D}$  whenever  $y_{i^*} \alpha_{i^*} < -c_{\mathcal{D}}$ . Thus we will lower bound the probability of  $\mathbb{P}(y_{i^*} \alpha_{i^*} < -c_{\mathcal{D}})$ . Note for a standard Gaussian random variable  $\eta \sim N(0, 1)$  as shown in [14] we have for  $r > 0$
$$
\mathbb {P} (\eta > r) \geq \frac {1}{\sqrt {2 \pi}} \frac {r}{r ^ {2} + 1} \exp (- r ^ {2} / 2). \tag {24}
$$
Set  $s = \frac{\sqrt{d}}{\sqrt{T - 1}} c_{\mathcal{D}}$ . Then by symmetry of the Gaussian distribution the above bound translates into the following bound for  $\alpha_{i*}$
$$
\mathbb {P} \left(y _ {i ^ {*}} \alpha_ {i ^ {*}} <   - c _ {\mathcal {D}}\right) \geq \frac {1}{\sqrt {2 \pi}} \frac {s}{s ^ {2} + 1} \exp \left(- s ^ {2} / 2\right).
$$
It follows that for any  $w \in \mathbb{R}^d$  such that  $\| w \|_2 = 1$  that the linear classifier  $g(\mathbf{x}; w, b)$  incorrectly classifies  $\mathcal{D}$  with probability at least
$$
\frac {1}{\sqrt {2 \pi}} \frac {s}{s ^ {2} + 1} \exp (- s ^ {2} / 2).
$$
E Further Results
We present additional experiments below. In Subsection E.1 we present per-dataset results for additional architectures and a discussion about ensembling InCA is given in Subsection E.2.
E.1 Per-dataset results for different architectures as presented in Table 2
Table 8 provides per-dataset results that are presented in aggregate in Table 2. Below we present the results for ConvNext-Base and ViT-L/16 (original pre-training) pre-trained models (with the results for ViT-L/16 DeiT and SWIN-L presented in Table 1 and Table 3 respectively).
E.2 Ensembling learned adapters
Because of "one-to-many" inference of InCA can take a set of independently learned adapters and ensemble them without a marginal increase to the inference cost. We follow non-parametric equalweight ensembling, by taking the output predictions of two adapters  $h_1(x), h_2(x)$  on a sample image  $x$ . Note that the adapters are computed with their relevant representations via a single forward pass, which makes the execution of  $h_1(x)$  and  $h_2(x)$  together only incrementally higher than computing just  $h_1(x)$ . The ensemble is defined as
$$
h ^ {*} (x) = \frac {h _ {1} (x) + h _ {2} (x)}{2}. \tag {25}
$$
Given the large combinatorial selection of  $k$  adapters from the  $l$  learned adapters we consider the case of ensembling with just two adapter members. After training we evaluate all  $m(m - 1) / 2$  such pairs and compare them with the top performing single layer predictor which we present in Figure 7. In the figure, we illustrate the representations and corresponding adapter pairs that lead to best performance and also present the computed ensemble gain which is the difference between the ensembled model accuracy and the top accuracy of any single adapter.
In addition to improving classification accuracy, ensembling can aid in improving robustness and out of distribution performance which we leave as a future work. Further directions of ensembling include ensembling performance when using adapters of different adapter architectures (e.g. an MLP-3 ensembled with an InCA adapter) or adapters that use representations from different neural networks [21].
E.3 Ablation on the number of queries
We apply an ablation to see the effects of using a different number of queries in the InCA adapter architecture. In particular, the InCA adapter is written as,
$$
v _ {\text {c r o s s}} (\mathbf {z}) _ {[ 1: m ]} := \text {c r o s s - a t t n} _ {\theta} ([ z ^ {1}, \dots , z ^ {T} ], [ q _ {1}, \dots q _ {m} ])
$$
$$
\operatorname {I n C A} _ {\theta} (\mathbf {z}) := \operatorname {h e a d} _ {\theta} \circ \operatorname {n o r m} \left(\operatorname {a v g - p o o l} \left(v _ {\text {c r o s s}} (\mathbf {z}) _ {[ 1: m ]}\right)\right).
$$
Top-1 Test Error for ConvNext-B
Dataset	Full fine-tuning	InCA	InCA (last)	Inter. LP	LP
CUB-200	9.3	9.3	9.3	13.0	13.0
DTD	16.7	17.4	17.4	18.6	18.6
Flood Depth	16.9	16.5	20.5	19.4	19.9
EuroSAT	0.9	1.6	2.2	2.8	3.1
Aircrafts	10.5	17.9	23.1	54.7	54.7
Herbarium	17.0	22.7	26.4	37.4	39.5
MIT-67	10.9	10.3	10.7	10.2	10.4
Oxford Flowers	0.5	0.4	0.4	0.6	0.6
Oxford Pets	5.2	4.6	5.6	5.9	6.0
Stanf. Cars	6.8	9.3	14.4	39.9	39.9
Stanf. Dogs	8.9	7.6	7.6	7.3	7.3
Ave. Top-1 Test Error	9.4	10.7 (-7.4)	12.8 (-12.6)	19.7 (-44.2)	20.0 (-44.2)
Top-1 Test Error for ViT-L/16 (ViT pre-training)
Dataset	Full fine-tuning	InCA	InCA (last)	Inter. LP	LP
CUB-200	11.7	10.9	10.9	12.2	12.2
DTD	18.3	18.9	20.1	19.9	20.1
Flood Depth	20.8	18.1	18.7	18.7	18.7
EuroSAT	0.8	1.1	1.9	2.5	3.5
Aircrafts	20.7	23.2	28.2	44.5	46.4
Herbarium	20.3	26.9	31.3	38.9	41.3
MIT-67	12.8	11.3	11.9	10.4	11.1
Oxford Flowers	0.6	0.3	0.4	0.3	0.4
Oxford Pets	5.5	5.3	5.4	6.5	6.5
Stanf. Cars	9.3	10.9	12.9	27.6	30.2
Stanf. Dogs	11.0	10.4	10.4	10.1	10.1
Ave. Top-1 Test Error	12.0	12.5 (-6.6)	13.8 (-11.0)	17.4 (-23.8)	18.2 (-25.7)
Table 8: Per-dataset Adaptation Top-1 Test Error on various architectures We test transfer learning performance of fine-grained datasets applied to different architectures and pre-trainings including, ViTs, SWIN, and convolutional networks. We report the per-dataset Top-1 test error for the 11 datasets presented in Table 2
For  $m > 1$  the output of tokens  $[q_1, \ldots, q_m]$  through the cross-attn layer are averaged, and we test whether using  $m > 1$  brings additional representational benefit to each adapter. We present the result in Table 9 and observe that using a different  $m$  does not have a consistent effect on the accuracy of the learned adapters, and in our experiments we use  $m = 1$  for InCA adapters to be most computationally efficient.
We present the optimization and augmentation details for training InCA, and note we use standardized procedures for augmentation and training (without extensive hyper-parameter optimization) of the different transfer learning methods we evaluate.
Augmentation Unless otherwise specified we train with input image size 224 and standard augmentation practice [62]. In particular, during training we resize to image-size 256 and apply random cropping, for testing we apply resizing and center cropping. For larger image resolutions we maintain the same resize-crop ratio of 0.875.
**Optimization** For the linear probing and InCA approaches, we train with the AdamW optimizer [50], cosine annealing learning rate scheduler [51] for 30 epochs and with weight decay  $1\mathrm{e} - 4$ . In each method we sweep over 2 learning rates  $\mathrm{lr} = \{1\mathrm{e} - 4,3\mathrm{e} - 4\}$ . For full fine-tuning, we also train with AdamW optimizer (weight decay  $1\mathrm{e} - 4$ ), cosine annealing for 30 epochs, but in addition, identify optimal learning rates for each pre-training and architecture separately. We first identify an
Figure 7: Optimal Representation pairings Optimal ensemble pairs of InCA of listeners at different locations of the network; Optimal ensembles can improve over any single layer. ViT-L/16 DeiT pre-training.
Top-1 Test Error for ViT-L/16 (DeiT pre-training)
# of InCA queries (m)
Dataset	1	2	4	16
CUB-200	9.1	9.5	9.6	9.5
DTD	17.8	18.4	19.2	19.1
Aircrafts	15.8	19.3	19.8	16.8
MIT-67	10.1	10.8	11.0	10.9
Oxford Flowers	0.3	0.3	0.3	0.4
Oxford Pets	4.7	4.7	4.5	4.4
Stanf. Cars	8.4	8.7	8.8	8.2
Stanf. Dogs	8.1	6.3	6.3	5.9
Table 9: Varying # of queries in the InCA adapter We run an ablation testing the effect of applying a different number of queries  $q_{1}, \ldots, q_{m}$  and then averaging when using the InCA adapter. We observe that in most cases  $m$  does not have a big effect on accuracy and that  $m = 1$  has sufficient representation capacity for the adapter.
architecture coarse-range learning rate based on performance on 5 datasets by sweeping over  $\mathrm{lr} = \{1\mathrm{e} - 2,1\mathrm{e} - 3,1\mathrm{e} - 4,1\mathrm{e} - 5,1\mathrm{e} - 6\}$  followed by a refined sweep with learning rates  $\mathrm{lr} = \mathrm{B}$ , 2B with B being the optimal coarse learning rate.
For the VPT baseline, we follow the details presented in the paper and train with VPT-Deep which was observed to outperforms VPT-Shallow. To train VPT, we use the SGD optimizer with momentum and cosine annealing for 100 epochs. For each dataset we run a sweep on the prompt length  $\{5,20,100\}$ , base learning rate  $\{0.25,0.1,0.05,0.01\}$ , and weight-decay  $\{1e - 2,1e - 4\}$  for a total of 24 runs with 100-epochs for each dataset. We compare the training cost of InCA and VPT-Deep in Table 7. In general we note that the shallow and small architecture of InCA or linear probing that are separate from the base model makes them straightforward to optimize, compared with adaptation methods that receive back-propagated gradients from a frozen intermediate layer of the network as shown in Fig. 2.
For the LoRA baseline [28] we apply a LoRA modified attention to each block's self-attention layer  $(W_{k}, W_{q}, W_{v})$  in ViT based architectures and to each block's WindowAttention for SWIN. For the low rank dimension we sweep over the best value among  $d = 5, 10, 50$ . For BitFit we follow the discussion in [6] and train all of the bias-parameters in the network in addition to full training of the head. Analogously for [45] we follow their procedure with LayerNorm which includes training each of the LayerNorm parameters  $(\gamma, \beta)$  for each layer along with training of the head of the pre-trained model. For all of the efficient training methods above we sweep over  $\mathrm{lr} = \{3\mathrm{e} - 5, 1\mathrm{e} - 4, 3\mathrm{e} - 4, 1\mathrm{e} - 3\}$  to identify the best learning rate for the dataset.
Broader Impacts Our method, InCA enables efficient and modular model adaptation that can be applied to any strong available pre-trained backbone. In that sense, InCA reduces the computational barriers to entry for training and evaluating over a large set of (potentially massive scale) models and optimization settings to identify a model to be used for downstream adaptation. This bridges the gap between cutting edge research in general visual representation learning and specific domain applications, especially since the best performing models are computationally expensive to adapt. Given that InCA operates well on fine-grained visual datasets, this can have positive applications in scientific domains such as medical imaging. In many scientific domains, the available datasets are known to be fine-grained yet also with sparse training data. In addition the ease of use and reduced computational costs associated with downstream adaptation with InCA makes it possible for domain experts without machine learning expertise to use InCA without access to large computational resources. This can enable domain researchers solve their domain problems by leveraging various public pre-trained models to achieve competitive results.