SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\sigma$  represents the sigmoid function;  $\rho_{\mathrm{min}}$ ,  $\rho_{\mathrm{max}}$  control the minimum and maximum dropping rate applied; and  $\mu$  and  $\tau$  control the shape of the schedule. We experimented with both increasing  $(\tau > 0)$  and decreasing  $(\tau < 0)$  schedules. In all cases we set  $\rho_{\mathrm{min}} = 0.2$  and  $\rho_{\mathrm{max}} = 0.8$ . Figure 16 shows that, by decreasing the dropping rate throughout training, one can improve the final accuracy, at fixed training cost. Conversely, increasing the token dropping rate harms performance.
C Model information
C.1 Positional embeddings
Extending ViTs to variable input sizes necessitates rethinking positional embeddings added to every token after embedding. We considered several variants of positional embeddings, and evaluated them based on (1) the best performance model using them achieve within training distribution of input sizes; and based on (2) how well these models perform when evaluated on image sizes outside of the training distribution. Results and discussion of these experiments can be found in Section 3.4.
Broadly, we considered positional embeddings that varied along three axes: (1) whether they were learned, parametric or fixed; (2) whether they were absolute or fractional; and (3) whether they are factorized.
Image credit: Matthew Henry burst.shopify.com/photos/dog-staying-warm
Figure 17: We use two different views of the same image, of resolutions  $96 \times 128$  and  $224 \times 160$ , and demonstrate different coordinate systems when using patch size 32.
Absolute and fractional coordinates A natural way of indexing token within an image is to select  $a$  priori a maximum possible image side length (shared for width and height) maxLen, and to assign to token integer coordinates  $(x,y)$  based on their original location within the image. Embedding coordinates defined in this way allow models to consume images with resolutions up to  $R = P \cdot \max \text{Len}$ . However, when learned absolute coordinate embeddings are considered, extreme values of  $x$  and  $y$  must also be observed during training, which necessitates training on images with varied aspect ratios and limits models generalisation.
To alleviate the necessity of observing extreme aspect ratios and image size during learning of positional embeddings, we also consider fractional coordinates, which are normalized to the actual size of the input image and are obtained by dividing the absolute coordinates  $x$  and  $y$  above by the number number of columns and rows respectively, i.e. the corresponding side length. Doing this allows the model to observe extreme token coordinates during training, which intuitively should help with generalization to higher resolutions. However, this is accomplished at the cost of obfuscating the input images aspect ratio.
Factorized embeddings We further consider whether coordinates  $x$  and  $y$  should be embedded independently or jointly. In case of independent embedding, the two coordinates  $x$  and  $y$  are embedded independently, and their embeddings are combined via addition or by stacking. For joint embeddings and embedding for each position  $(x,y)$  is obtained directly.
Learned, parametric and fixed positional embeddings Finally, we also explored the relative benefits of fixed, learned and parametric embeddings. For fixed embeddings we followed [57] and used sinusoidal positional embeddings, and learned embeddings were implemented as in [1].
For parametric positional embeddings we followed [8] and used Fourier embeddings. Specifically, coordinates  $(x,y)$  were mapped using a single linear layer before applying sin and cos activations to them, and stacking the results to obtained the positional embeddings.
Experiments Because not all combinations of the above embedding choices are equally promising or natural, we experimented only with subset of them shown in Table 3 and Figure 10a.
Table 3: Classification of positional embedding experiments from Figure 10a.
Name	Coordinates	Type	Factorized
Learned 1D (ViT)	Absolute	Learned	No, position in flatted token sequence
Learned 2D (Pix2struct)	Absolute	Learned	No
Factorized abs. (+)	Absolute	Learned	Yes, sum
Factorized abs. (stack)	Absolute	Learned	Yes, stack
Factorized abs. (×)	Absolute	Learned	Yes, product
Fourier abs.	Absolute	Parametric	No
Sinusoidal abs.	Absolute	Fixed	Yes, stack
Factorized frac. (+)	Fractional	Learned	Yes, sum
Fourier frac.	Fractional	Parametric	No
Sinusoidal frac.	Fractional	Fixed	Yes, stack
In sinusoidal and factorised embeddings experiments with fractional coordinates fractional coordinate embeddings were obtained from absolute coordinate embeddings via bilinear interpolation.
D Inference strategies
We performed various experiments to measure model quality for given runtime cost. The runtime can be tuned by changing the number of processed patches, or by using choosing different size of the model.
We firstly looked at how model quality changes in respect to decreasing area of the image compared to native resolution, presented in Figure 18a. We observed that on ImageNet [4] model retains most of the quality down to  $40\%$  of the image size. After that, the quality drastically decreases. On the other hand, increasing the size of the image have a diminishing return in quality. This can be directly compared with random token dropping as an alternative to resizing, which showed to be very ineffective way to decrease number of patches during inference - Figure 18b.
Please note that this highly depends on the native resolution of the images in the dataset - e.g. dataset with twice as big images than ImageNet can probably be safely resized to  $20\%$  of area.
A better way to quantify the performance is by giving a constant compute budget corresponding to number of patches. Figure 19a shows that resizing the image (while preserving aspect ratio) to 256 tokens retains most of the quality (within  $0.3\%$ ). This corresponds to 256x256 area (given patch size of 16). At 128 tokens (181x181 area) the quality difference reaches  $1\%$  and drops significantly after that.
Figure 18: (a) The effect of resizing the image. (b) Dropping random tokens is ineffective way to decrease number of patches compared to resizing the image. Data from NaViT-L/16. (c) Given number of patches as compute budget, it is beneficial to upscale the image.
(a)
Figure 19: (a) Quality on ImageNet in respect to number of patches (sequence length). (b) Runtime of models compared to the accuracy on ImageNet.
Here we also resized the image past its native resolution in case it already fit the given sequence length budget. We observed that it is beneficial to resize the image to the given sequence length past the native resolution to keep monotonic increase in quality, which is showed on Figure 18c.
Figure 19b presents the runtime of NaViT-L/16 and NaViT-B/16 for different sequence lengths. We can see that NaViT-L/16 at sequence length 128 is as fast as NaViT-B/16 with sequence length 512, while having almost  $1\%$  difference in quality.
E Cascades
Another strategy to be more compute efficient would be to assign more tokens (and thus FLOPs) to the examples deemed hard by the model. This is in particular interesting for bulk inference workloads, where one can amortize over large datasets and where only the total inference time matters.
Figure 20: Performance of a model cascade versus the average inference time. The labels at the select points denote the number of tokens at that scale.
To evaluate the feasibility of this approach, we consider two sequence lengths  $n_1$  and  $n_2$  with respective inference times  $t_1$  and  $t_2$ . Then, we (i) send all examples though the model with  $n_1$  tokens, and send only the  $\alpha \in (0,1)$ -fraction deemed hardest (those with the smallest maximum probability) to the model with  $n_2$  tokens. To have an input almost exactly fit into  $n_1$  or  $n_2$  tokens we perform and aspect ratio preserving resize. Hence, the total amortized inference time per-example is  $t_i + \alpha t_2$ , while the accuracy obtained by combining the accuracy of the first model on the  $1 - \alpha$  most-confident fraction of the data, and the performance of the more expensive model on the remaining data. By considering several pairs of models and varying  $\alpha$  we obtain the plot in Figure 20. As we can see this strategy is indeed useful and provides not only the best performing models at given compute budgets, but because  $\alpha$  is a real parameter one can obtain very fine-grained trade-offs.
F Calibration
To evaluate behaviour of the predicted uncertainties with scale, we compute the calibration error of a -B sized ImageNet-finetuned model (the -L model performs similarly). Note that these models were trained with sigmoid loss, i.e., the 1000 labels were predicted independently without enforcing that the probabilities should sum up to 1. As we varied the sequence of tokens per example between 128 and 1024, we obtained very stable calibration errors (top-1, using  $\ell_1$  and 30 buckets, i.e., the settings from [29]), which we present in Table 4.
Table 4: Expected calibration error on ImageNet-1K with varying sequence lengths.
Sequence Length	128	256	384	512	640	768	1024
Calibration Error	0.047	0.046	0.048	0.047	0.047	0.046	0.045
G Out of distribution evaluation
For ViT, we apply the "Crop" strategy from [17], namely an aspect-preserving crop of the central  $75\%$  of the image for ObjectNet and ImageNet-A, and square resize followed by a  $87.5\%$  central crop for the other datasets. We also apply a simple "Resize" strategy that does not crop the images. For NaViT, both the "Crop" and the "Resize" strategy do an aspect preserving resize of the target images.
Figure 21: Same evaluation as in Figure 11, but without any special preprocessing of the images before the evaluation. Employing a simple resize (square for ViT, aspect preserving for NaViT) results in much better performance on datasets that have images with an extreme aspect ratio. Same data as in table Table 5.
H Fairness Signal Annotation
In Figure 22, we demonstrate that using native image resolution improves the performance of fairness signal annotation. Prior research has shown that metrics, such as group calibration, are vulnerable to labeling errors, particularly for underrepresented groups. Moreover, this problem persists even when accounting for label noise during training [30]. Thus, reducing the labeling error of fairness signals has the potential of improving the reliability of bias mitigation and post-hoc auditing [31]. Nevertheless,
Table 5: Detailed results of out evaluation of pretrained models with a label-map (see Section 3.5). Same data as in Figure 11 and Figure 21.
Compute	ImageNet	ImageNet-A	ObjectNet
ViT	NaViT	ViT	NaViT	ViT	NaViT
Crop	B/32	7.9×101	54.4	63.5	14.7	32.7	32.9	41.2
1.9×102	60.2	65.6	22.0	37.0	38.0	43.5
3.9×102	63.7	67.2	26.6	40.1	41.7	45.1
7.9×102	65.5	68.2	30.7	42.5	44.2	45.7
B/16	2.6×102	66.4	68.5	37.3	52.3	47.2	48.4
6.6×102	61.7	66.0	27.0	44.2	41.3	45.9
1.3×103	70.3	71.2	48.8	57.0	52.8	50.7
2.6×103	68.7	70.1	43.5	55.1	50.5	49.8
L/16	5.4×102	70.7	73.6	51.5	65.5	53.3	55.0
1.3×103	66.4	71.1	39.2	58.6	47.6	52.1
2.7×103	73.9	75.1	60.4	68.9	57.7	57.9
5.4×103	73.0	74.6	57.6	67.9	56.2	57.1
Resize	B/32	7.9×101	51.7	64.0	12.6	26.7	15.9	31.6
1.9×102	57.9	66.2	17.2	30.1	20.0	33.8
3.9×102	61.6	67.4	20.7	32.6	23.5	34.6
7.9×102	63.6	68.6	22.8	35.0	25.3	36.1
B/16	2.6×102	65.4	69.2	27.4	43.9	28.7	38.7
6.6×102	60.4	66.5	20.4	36.8	23.6	35.3
1.3×103	69.5	72.5	36.2	51.5	34.2	42.1
2.6×103	67.7	71.0	32.5	48.6	32.0	40.5
L/16	5.4×102	70.0	73.9	38.5	59.9	35.3	45.6
1.3×103	65.1	71.5	28.2	51.6	29.3	41.5
2.7×103	73.2	76.0	47.3	65.5	39.8	48.8
5.4×103	72.3	75.3	44.3	64.1	38.8	48.2
Figure 22: Summary of results of evaluating the accuracy of annotators trained on fairness-related signals using either NaViT-L/16 or ViT-L/16. TOP: NaViT offers better representations that improve the accuracy of annotators. BOTTOM: Using native aspect ratios in NaViT results in a higher performance when compared to resizing images to squares.
we emphasize that while NaViT improves the annotation accuracy in these tasks, care must be taken in such situations since classifiers can be inaccurate and lead to a broad categorization of people that misidentifies real identities. We encourage readers to delve into the comprehensive work outlining such potential risks, e.g. [58, 59], for further insight. In assessing the technical capabilities of NaViT, our intent is not to promote or encourage their application in inappropriate contexts. Rather, our objective is only to illuminate these technical findings for scenarios where they may be considered beneficial, such as when measuring the level of diversity in a dataset or auditing/mitigating biases in predictive models. We strongly advocate for responsible AI use, maintaining that the benefits of technological advancements should not overshadow the importance of user safety and privacy. AI tools, including
ours, should always be deployed judiciously, with a keen awareness of potential risks and a commitment to avoiding harm.
I Evaluation on model-vs-human OOD datasets on different resolutions
Just like NaViT, human visual perception works across flexible aspect ratios and resolutions (just imagine how strange the world would look like if we could only see it through a  $224 \times 224$  pixel window!). We investigate how the ability to cope with variable resolutions affects performance on "model-vs-human", a benchmark of 17 challenging datasets [60]. For this purpose, we replicate the setup from Figure 6, but instead of evaluating ImageNet accuracy, we evaluate OOD accuracy on the model-vs-human benchmark.
Figure 23: OOD accuracy on "model-vs-human" datasets across different fine-tuning resolutions. A single NaViT model trained on varying resolutions (red) performs roughly on par as fine-tuning one NaViT model per test resolution (blue). The ViT baseline (orange) is mostly worse than NaViT models for lower resolutions and mostly better for higher resolutions.
This corresponds to testing JFT B/16 models finetuned on ImageNet at various resolutions. The test dataset has a fixed  $224 \times 224$  square resolution; thus we resize the test images to fit each model's finetuning resolution. Note that using square images has been standard practice designed for convolutional networks, but NaViT models no longer require square input, thus existing benchmarks are not tailored to those new possibilities. For datasets with multiple difficulty levels (such as different levels of blur), we average performance across levels while excluding levels that are too easy (not OOD) or too hard (human performance is close to chance), which follows the approach of (author?) [60] as explained in their "Appendix G Benchmark scores".
To ensure a fair comparison, we use models that are compute-matched for pretraining, and during fine-tuning, compute and data are identical for all models. The results of our comparison are shown in Figure 23. Overall, a single NaViT model trained on varying resolutions (red) performs roughly on par with fine-tuning one NaViT model per test resolution (blue).
The ViT baseline (orange) is mostly worse than NaViT models for lower resolutions and mostly better for higher resolutions. It may be worth noting that ViT models have a bit of an advantage in this comparison since they are fine-tuned on square images, whereas NaViT models are fine-tuned on flexible resolution images (preserving the image's aspect ratio) that have the same number of pixels, while not necessarily being square.
For the purpose of our comparison, we exclude the "cue-conflict" dataset from the OOD evaluation, since there is no objective ground truth class in the case of images with a texture-shape cue conflict.