An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels

## Abstract

Abstract This work does not introduce a new method. Instead, we present an interesting finding that questions the necessity of the inductive bias – locality in modern computer vision architectures. Concretely, we find that vanilla Transformers can operate by directly treating each individual pixel as a token and achieve highly performant results. This is substantially different from the popular design in Vision Transformer, which maintains the inductive bias from ConvNets towards local neighborhoods ( e.g . by treating each 16 × \times × 16 patch as a token). We mainly showcase the effectiveness of pixels-as-tokens across three well-studied tasks in computer vision: supervised learning for object classification, self-supervised learning via masked autoencoding, and image generation with diffusion models. Although directly operating on individual pixels is less computationally practical, we believe the community must be aware of this surprising piece of knowledge when devising the next generation of neural architectures for computer vision.

## 1 Introduction

The deep learning revolution can be characterized as a revolution in *inductive biases* for computer vision. Learning previously occurred on top of manually crafted features, such as those described in [, ], which encoded preconceived notions about useful patterns and structures for specific tasks. In contrast, biases in modern features are no longer predetermined but instead shaped by direct learning from data using predefined model architectures. This paradigm shift’s dominance highlights the potential of reducing feature biases to create more versatile and capable systems that excel across a wide range of vision tasks.

Beyond features, model architectures also possess inductive biases. Reducing these biases can facilitate greater unification not only across tasks but also across data modalities. The *Transformer* architecture [] serves as a great example. Initially developed to process natural languages, its effectiveness was subsequently demonstrated for images [], point clouds [], codes [], and many other types of data. Notably, compared to its predecessor in vision – ConvNet [, ], Vision Transformer (ViT) [] carries much less image-specific inductive biases. Nonetheless, the initial advantage from such biases is quickly offset by more data (and models that have enough capacity to store patterns within the data), ultimately becoming restrictions preventing ConvNets from scaling further [].

Of course, ViT is not entirely free of inductive bias. It gets rid of the spatial hierarchy in the ConvNet and models multiple scales in a plain architecture. However, for other inductive biases, the removal is merely half-way through: translation equivariance still exists in its patch projection layer and all the intermediate blocks; and *locality* – the notion that neighboring pixels are more related than pixels that are far apart – still exists in its ‘patchification’ step (that represents an image with 16$\times$16 patches on a 2D grid) and position embeddings (when they are manually designed). Therefore, a natural question arises: can we *completely eliminate* either or both of the remaining two inductive biases? Our work aims to answer this question.

**Table 1: Major inductive biases in vision architectures. ConvNet [, ] has all three – spatial hierarchy, translation equivariance, and *locality* – with neighboring pixels being more related than pixels farther apart. Vision Transformer (ViT) [] removes the spatial hierarchy, reduces (but still retains) translation equivariance and locality. We use Pixel Transformer (PiT) to investigate the *complete removal* of locality by simply applying Transformers on individual pixels. It works surprisingly well, challenging the mainstream belief that locality is a necessity for vision architectures.**
| inductive bias | ConvNet | ViT | PiT (ours) |
| --- | --- | --- | --- |
| spatial hierarchy | ✓ | ✗ | ✗ |
| translation equivariance | ✓ | ✓ | ✓ |
| locality | ✓ | ✓ | ✗ |

Surprisingly, we find locality can indeed be removed. We arrive at this conclusion by directly treating each individual pixel as a token for the Transformer and using position embeddings learned from scratch. In this way, we introduce zero priors about the 2D grid structure of images. Interestingly, instead of training divergence or steep performance degeneration, we obtain *better* results in quality from the resulting architecture. For easier reference, we name this PiT, short for Pixel Transformer. Note that our goal is not to promote PiT as an approach to replace ViT, but the fact that PiT works so well suggests there is *more* signals Transformers can capture by viewing images as sets of individual pixels, rather than 16$\times$16 patches. This finding challenges the conventional belief that ‘locality is a fundamental inductive bias for vision tasks’ (see Tab. 1).

In the main paper, we showcase the effectiveness of PiT via three different case studies: (i) supervised learning for object classification, where CIFAR-100 [] is used for our main experiments thanks to its 32$\times$32 input size, but the observation also generalizes well to ImageNet []; (ii) self-supervised learning on CIFAR-100 via standard Masked Autoencoding (MAE) [] for pre-training, and fine-tuning for classification; and (iii) image generation with diffusion models, where we follow the architecture of Diffusion Transformer (DiT) [], and study its pixel variant on ImageNet using the latent token space provided by VQGAN []. In all three cases, we find PiT exhibits reasonable behaviors, and achieving results better in quality than baselines equipped with the locality inductive bias. This observation is further generalized to fine-grained classification and depth estimation tasks in the appendix.

As a related investigation, we also examine the importance of two locality designs (position embedding and patchification) within the standard ViT architecture on ImageNet. For position embedding, we have three options: sin-cos [], learned, and none – with sin-cos carrying the locality bias whilst the other two not. To systematically ‘corrupt’ the locality bias in patchification, we perform a pixel *permutation* before dividing the input into 256-pixel (akin to a 16$\times$16 patch in ViT) tokens. The permutation is fixed across images, and consists of multiple steps that swap a pixel pair within a distance threshold. Our results suggest that patchification imposes a stronger locality prior, and (given PiT) translation equivariance is still indispensable for network designs.

Admittedly, PiT is not as practical as ViT, since treating each pixel as a token will lead to a sequence length much longer than previously adopted for images. This is especially limiting as Self-Attention operations in Transformers demand quadratic computations. In practice, patchification is still arguably the most effective idea that trades quality for efficiency, and locality is still *useful*.
Nevertheless, we believe our investigation delivers a clean, compelling message that locality is *not* a necessary inductive bias for model design. We believe this finding will be an integral piece of the community knowledge when exploring the next generations of architectures to process images.

## 2 Related Work

#### Locality for images.

To the best of our knowledge, most modern vision architectures [, ], including those aimed at simplifications of inductive biases [, ], still maintain locality in their design. Manually designed visual features before deep learning are also locally biased. For example, SIFT [] uses a local descriptor to represent each point of interest; HOG [] normalizes the gradient strengths locally to account for changes in illumination and contrast. Interestingly, with these features, *bag-of-words* models [, ] were popular – analogous to the *set-of-pixels* explored in our work.

#### Locality beyond images.

The inductive bias of locality is widely accepted beyond modeling 2D images. For text, a natural language sequence is often pre-processed with ‘tokenizers’ [, ], which aggregate the dataset statistics for grouping frequently-occurring adjacent characters into sub-words. Before Transformers, recurrent neural networks [, ] are the default architecture for such data, which exploit the temporal connectivity to process sequences step-by-step. For even less structured data (*e.g*. point clouds [, ]), modern networks [, ] will resort to various sampling and pooling strategies to increase their sensitivity to the local geometric layout. In graph neural networks [], nodes with edges are often viewed as being locally connected, and information is propagated through these connections to farther-away nodes. Such a design make them particularly useful for analyzing social networks, molecular structures, *etc*.

Figure: Figure 1: Overview of a Pixel Transformer (PiT), which is used to investigate the role of locality. Given an image, we simply treat it as a set of pixels. Besides, we also employ randomly initialized and learnable position embeddings without any information about 2D structure, therefore removing the remaining locality inductive bias from previous work (*e.g*., ViT []). These representations are then fed into the Transformer which performs set operations in its interleaved Self-Attention and MLP blocks (only showing one of each for clarity). We demonstrate the versatility of PiT with three case studies, covering both discriminative and generative tasks.
Refer to caption: x1.png

#### Other notable efforts.

We list four efforts in a rough chronological order, and hope it can provide historical context from multiple perspectives for our work:

- •
For ConvNets, relevant attempts have being made to remove locality. Notably, [] replaces all the spatial convolutional filters with 1$\times$1 filters in a ResNet []. It provides more interpretability to understand the decision making process of a ConvNet, but without inter-pixel communications, the resulting network is substantially worse in performance. Our work instead uses Transformers, which are inherently built on *set* operations, with the Self-Attention mechanism handling all-to-all communications; understandably, we attain better results.
- •
Before ViT gained popularity, iGPT [] was proposed to directly pre-train Transformers on pixels following their success on text [, ]. In retrospect, iGPT is a locality-free model for self-supervised next (or masked) pixel prediction. But despite the expensive demonstrations, its performance still falls short compared to simple contrastive pre-training [] for ImageNet linear classification. Later, ViT [] *re-introduced* locality (*e.g*., via patchification) into the architecture, achieving impressive results on many benchmarks including ImageNet. Since then, the community has moved on with 16$\times$16 patches as the default tokens for images. Even today, it is still unclear whether higher resolution or locality is the key differentiator between the two. Our work closes this understanding gap, pointing to resolution as the enabler for ViT, *not* locality with systematic analyses.
- •
Perceiver [, ] is another series of architectures that operate directly on pixels for images. Aimed at being modality-agnostic, Perceiver designs latent Transformers with cross-attention modules to tackle the *efficiency* issue when the input is high-dimensional. However, this design is not as widely adopted as plain Transformers, which have consistently demonstrated scalability across multiple domains [, ]. Through PiT, we show Transformers can indeed work directly with pixels, and given the rapid development of Self-Attention implementations to handle massive sequence length (up to a *million*) [, ], efficiency may not be a critical bottleneck even when all the pixels are counted.
- •
Our work can also be viewed as exploring sequence length scaling to the extreme. It’s to model individual pixels for images, and to model individual characters for text []. Longer sequence (or higher resolution) is generally beneficial, as evidenced in [, , ]. However, all of them stopped short of reaching the extreme case that completely gets rid of locality.

## 3 Inductive Bias of Locality

In this section, we provide in-depth discussions about the inductive bias of locality (or locality) in mainstream architectures. To be precise, locality is the inductive bias that neighboring pixels are more related than pixels farther apart. We cover ConvNets and ViTs.

### 3.1 Locality in ConvNets

In a ConvNet, the locality bias is reflected in the *receptive fields* of the features computed in each layer of the network. Intuitively, receptive fields cover the pixels involved in computing a specific feature, and for ConvNets, these fields are local.

Specifically, ConvNets consist of several layers, each containing convolutional operations using kernels (*e.g*., 7$\times$7 or 3$\times$3) or pooling operations – both of which are locally biased. For example, the receptive field in the first layer often corresponds to only a small local window. The field is progressively expanded as the network becomes deeper, but the window is still local and centered at the location of the pixel.

### 3.2 Locality in Vision Transformers

At the first glance, Transformers are locality-free. This is because the majority of Transformer operations are either global (*e.g*., Self-Attention), or purely within each individual token (*e.g*., MLP). However, a closer look will reveal two designs within ViT [] that can still retain the locality inductive bias: patchification and position embedding.

#### Locality in patchification.

In ViT, the tokens fed into the Transformer blocks are *patches*, not pixels. Each patch consists of 16$\times$16 pixels, and becomes the basic unit of operation after the first projection layer. This means the amount of computation imposed within the patch is drastically different from the amount across patches: the information outside the 16$\times$16 neighborhood can *only* be propagated with Self-Attention, and the information among the 256 pixels are always processed jointly as token. While the receptive field becomes global after the first Self-Attention block, the bias toward local neighborhood is already inducted in the patchification step.

#### Locality in position embedding.

Position embeddings can be learned [], or manually designed and fixed during training. A natural choice for images is to use a 2D sin-cos embedding [, ], which extends from the original 1D one []. As sin-cos functions are smooth, they tend to introduce locality biases that nearby tokens are more similar in the embedding space.(^1^11While sin-cos functions are also cyclic, it’s easy to verify that the majority of their periods are longer than the typical sequence lengths encountered by ViTs.) Other designed variants are also possible and have been explored [], but all of them can carry information about the 2D grid structure of images, unlike learned position embedding which does not make assumptions about the input.

The locality bias has also been exploited when the position embeddings are *interpolated* [, ]. Through bilinear or bicubic interpolation, spatially close embeddings are used to generate a new embedding of the current position, which also leverages locality as a prior.

Compared to ConvNets, ViTs are designed with much less pronounced bias toward locality. We push this further by completely removing this bias next.

## 4 Transformers on Pixels

We closely follow the standard Transformer encoder [] which processes a sequence of tokens. Particularly, we apply the architecture directly on an unordered set of pixels from the input image with learnable position embeddings. This removes the remaining inductive bias of locality in ViT [], and for reference, we name it Pixel Transformer (PiT, see Fig. 1). Conceptually, PiT can be viewed as a simplified version of ViT, with 1$\times$1 patches instead of 16$\times$16.

Formally, we denote the input sequence as $X=(x_{1},...,x_{L})\in\mathbb{R}^{L\times d}$, where $L$ is the sequence length and $d$ is the hidden dimension. The Transformer maps the input sequence $X$ to a sequence of representations $Z=(z_{1},...,z_{L})\in\mathbb{R}^{L\times d}$. The architecture is a stack of $N$ layers, each of which contains two blocks: multi-headed Self-Attention block and MLP block:

$$ $\displaystyle\hat{Z}^{k}$ $\displaystyle=\texttt{SelfAttention}\big{(}\mathrm{norm}(Z^{k-1})\big{)}+Z^{k- 1},$ $\displaystyle Z^{k}$ $\displaystyle=\texttt{MLP}\big{(}\mathrm{norm}(\hat{Z}^{k})\big{)}+\hat{Z}^{k},$ $$

where $Z_{0}$ is the input sequence $X$, $k\in\{1,...,N\}$ indicates $k$-th layer in the Transformer, and $\mathrm{norm}(\cdot)$ is a normalization layer (typically LayerNorm []). Both blocks use residual connections [].

#### Pixels as tokens.

The typical input in computer vision to the network is commonly an image of RGB values, $I\in\mathbb{R}^{H\times W\times 3}$, where $(H,W)$ is the size of the original image. We follow a simple solution and treat $I$ as an unordered set of pixels $(p_{l})_{l=1}^{H\cdot W},p_{l}\in\mathbb{R}^{3}$. Thus, PiT simply projects each pixel into a $d$ dimensional vector via a linear projection layer, $f:\mathbb{R}^{3}\rightarrow\mathbb{R}^{d}$, resulting the input set of tokens $X=\left(f(p_{1}),...,f(p_{L})\right)$ with $L=H\cdot W$. We append the sequence with a learnable [cls] token []. Additionally, we learn an content-agnostic position embedding for each position. The pixel tokens are then fed into the Transformer to produce the set of representations $Z$.

$$ $X=\big{[}\texttt{cls},f(p_{1}),...,f(p_{L})\big{]}+\texttt{PE},$ $$

where $\texttt{PE}\in\mathbb{R}^{L\times d}$ is a set of learnable position embeddings.

**Table 2: Specifications of PiT size variants. In empirical evaluations, we utilize ViT [] with the same configuration for head-on comparisons.**
| model | layers ($N$) | hidden dim ($d$) | MLP dim | heads | param (M) |
| --- | --- | --- | --- | --- | --- |
| PiT-T(iny) | 12 | 192 | 768 | 12 | 5.6 |
| PiT-S(mall) | 12 | 384 | 1536 | 12 | 21.8 |
| PiT-B(ase) | 12 | 768 | 3072 | 12 | 86.0 |
| PiT-L(arge) | 24 | 1024 | 4096 | 16 | 303.5 |

PiT removes the locality inductive bias and is permutation equivariant at the pixel level. By treating individual pixels directly as tokens, we assume no spatial relationship in the architecture and let the model learn it from data. This is in contrast to the design of the convolution kernel in ConvNets or the patch-based tokenization in ViT [], which enforces an inductive bias based on the proximity of pixels. In this regard, PiT is more versatile – it can naturally model arbitrarily sized images (no need to be divisible by the stride or patch size), or even generalize to irregular regions [].

Besides the removal of locality, using each pixel as a separate token has additional benefits. Similar to treating characters as tokens for language, we can greatly reduce the vocabulary size of input tokens to the Transformer. Specifically, given the pixel of three color channels in the range of $[0,255]$, the maximum size of vocabulary is $255^{3}$ (as pixels take discrete integer values); a patch token of size $p{\times}p$ in ViT, however, can lead to a vocabulary size of up to $255^{3{\cdot}p{\cdot}p}$. If modeled in a non-parametric manner, this will heavily suffer from out-of-vocabulary issues.

Of course, PiT also has downsides, with the biggest one being computationally expensive (or even prohibitive) for modeling long sequences. However, given the rapid development of techniques that handle massive sequence lengths for large language models (up to a *million*) [, ], it is entirely possible that soon, we can train PiTs on all the pixels directly (*e.g*., a standard 224$\times$224 crop on ImageNet ‘only’ contains 50,176 pixels). Therefore, the goal of our paper is to empirically verify the effectiveness and potential of PiT at a smaller scale – which we do next, and leave the engineering effort of practical deployment for the future.

## 5 Experiments for PiT

In this section, we verify the effectiveness of PiT with three case studies: supervised learning, self-supervised learning with MAE [], image generation with DiT []. We use four variants of PiT: Tiny (T), Small (S), Base (B) and Large (L) with the specifications shown in Tab. 2. Unless otherwise specified, we use the ViT [] variants of the *same* configuration as our baselines. In sum, our experiments show that PiT can indeed learn strong vision representations with no inductive bias on locality.

**(a) CIFAR-100 classification.**
| model | Acc@1 | Acc@5 |
| --- | --- | --- |
| ViT-T/2 | 83.6 | 94.6 |
| PiT-T | 85.1 | 96.4 |
| ViT-S/2 | 83.7 | 94.9 |
| PiT-S | 86.4 | 96.6 |
| ViT-B/2 [] | 72.6 | - |

### 5.1 Case Study #1: Supervised Learning

In this study, we train and evaluate PiT *from scratch* without any pre-training [, ]. Our baselines are ViTs with patch sizes 2$\times$2.

#### Datasets.

We use two datasets: CIFAR-100 [] with 100 classes and 60K images combined and ImageNet [] with 1K classes and 1.28M images for training, 50K for evaluation. While CIFAR-100 is suitable for exploring the effectiveness of PiT due to its intrinsic image size of $32\times 32$, ImageNet has many more images which helps us to further confirm our findings.

#### Evaluation metrics.

For both datasets, we train our models on the train split and report the top-1 (Acc@1) and top-5 (Acc@5) accuracy on the val split.

#### Implementation details.

For CIFAR-100, due to the lack of optimal settings even for ViT, we search for the recipe and report results using model sizes Tiny and Small. We use the augmentations from the released demo of [] to train from scratch, as we found more advanced augmentations (*e.g*., AutoAug []) not helpful in this case. All models are trained using AdamW [] with $\beta_{1}{=}0.9$, $\beta_{2}{=}0.95$. We use a batch size of 1024, weight decay of 0.3, drop path [] of 0.1, and initial learning rate of 0.004 which we found to be the best for all the models. We use a linear learning rate warm-up of 20 epochs and cosine learning rate decay to a minimum of 1e-6. Our training lasts for 2400 epochs, compensating for the small size of dataset.

On ImageNet, we closely follow the from-scratch training recipe from [, ] for ViT and report results using PiT-S and PiT-B. Due to the limit in computation, images are crop-and-resized to $28{\times}28$ as the low-resolution inputs by default.
Globally average pooled outputs are used for classification. The training batch size is 4096, initial learning rate is $1.6{\times}10^{-3}$, weight decay is 0.3, drop path is 0.1, and training length is 300 epochs. MixUp [] (0.8), CutMix [] (1.0), RandAug [] (9, 0.5), and exponential moving average (0.9999) are used.

#### Main comparisons ( Tab. 3 ).

While our baselines for both ViT variants (ViT-T and ViT-S) are well-optimized on CIFAR-100 (*e.g*., [] reports 72.6% Acc@1 when training from scratch with ViT-B, whilst we achieve 80+% with smaller sized models), PiT-T improves over ViT-T by 1.5% of Acc@1; and when moving to the bigger model (S), PiT shows an improvement of 1.3% of Acc@1 over the small model (T) while ViT seems to be saturated. These results suggest compared to the patch-based ViT, PiT is potentially learning new, data-driven patterns directly from pixels.

Figure: (a) Acc@1 with fixed sequence length.
Refer to caption: x2.png

Our observation also transfers to ImageNet – albeit with a significantly lower resolution our results are significantly lower than the state-of-the-art [, ] (80+%), PiT still outperforms ViT in both settings we have experimented.

#### PiT vs . ViT: a tale of two trends.

If position embeddings are learned, PiT is simply a version of ViT with 1$\times$1 patches. Therefore, it is crucial to study the performance trend when varying the patch sizes in ViT. There are three variables in concern: sequence length ($L$), input size ($H{\times}W$) and patch size ($p$). They have a deterministic relationship: $L=H{\times}W/(p^{2})$. Thus we have two ways to study the Acc@1 trend w.r.t. patch size $p$:

- •
Fixed *sequence length.* We show the trend on ImageNet with a fixed $L$ in Fig. 2(a). The model size is ViT-B. In this plot, the input size varies (from 224$\times$224 to 14$\times$14) as we vary the patch size (from 16$\times$16 to 1$\times$1). The last data point is equivalent to PiT-B. If we follow this trend, then PiT is the *worst*. This means sequence length is not the only deciding factor for Acc@1 even for classification – a task where a single label is assigned to the entire image. Input size, or the amount of information fed into the model is arguably a more important factor, especially when the size is small. It’s only when the input size is sufficiently large (*e.g*., 112$\times$112), the additional benefit of further enlarging the size starts to diminish. This also means when the amount of information from the input is not enough, PiT– or any architecture that follows this design (*e.g*., iGPT []) would not work well.
- •
Fixed *input size.* Our finding resides in the other trend, when we fix the input size (therefore the amount of information), and vary the patch size on ImageNet in Fig. 2(b). The model size is ViT-S. Interestingly, we observe an *opposite* trend here: it is always helpful to decrease the patch size (or increase the sequence length), aligned with the prior studies that claim sequence length is highly important. Note that the trend holds even when it ultimately reaches PiT – a model without any design for locality. So PiT performs the *best* in accuracy compared to ViTs.

With these two trend figures in Fig. 2, our study *augments* the observations made from previous studies, as they mainly focused on regimes where the input size is sufficiently large [, ], and presents a more complete picture.

To see what PiT has learned, we show visualizations of the attention maps, position embeddings from PiT in Appendix 0.A.

**(a) Tiny-sized models.**
| model | pre-train | Acc@1 | Acc@5 |
| --- | --- | --- | --- |
| PiT-T |  | 85.1 | 96.4 |
|  | ✓ | 86.0 | 97.1 |
| ViT-T/2 | ✓ | 85.7 | 97.0 |

### 5.2 Case Study #2: Self-Supervised Learning

In this subsection, we study PiT with self-supervised pre-training and then fine-tuning for supervised classification. In particular, we choose MAE [] due to its efficiency that only retains 25% of the sequence length for the encoder, and its effectiveness for fine-tuning based evaluation protocols.

#### Datasets.

We use CIFAR-100 [] due to its inherent size of $32\times 32$ for images. This allows us to fully explore the use of pixels as tokens on the original resolution.

#### Evaluation metrics.

We first perform pre-training on the train split. Then it serves as the initialization in the fine-tuning stage (also trained on train). Again, we use image classification on CIFAR-100 as the downstream task and report the top-1 (Acc@1) and top-5 (Acc@5) accuracy on the val split.

#### Implementation details.

We follow standard MAE and use a mask ratio of 75% and select tokens randomly. Given the remaining 25% visible tokens, the model needs to reconstruct masked regions using pixel regression. Since there is no known default setting for MAE on CIFAR-100 (even for ViT), we search for recipes and report results using PiT-T and PiT-S. The same augmentations as in [] are applied to the images during the pre-training for simplicity. All models are pre-trained using AdamW with $\beta_{1}{=}0.9$, $\beta_{2}{=}0.95$. We follow all of the hyper-parameters in [] for the pre-training of 1600 epochs except for the initial learning rate of 0.004 and a learning rate decay of 0.85 []. Thanks to MAE pre-training, we can fine-tune our model with a higher learning rate of 0.024. We also set weight decay to 0.02, layer-wise rate decay to 0.65, and drop path to 0.3, $\beta_{2}$ to 0.999, and fine-tune for 800 epochs. Other hyper-parameters closely follow the scratch training recipe for supervised learning (see Sec. 5.1).

Figure: Figure 3: Qualitative results for case study #3 (image generation). These 256$\times$256 samples are from PiT-L trained on ImageNet, following the same architecture design and generation protocol as that of DiT [].(^3^33The models were not prompted to generate using any of the people images/classes from ImageNet (scuba diver, baseball player, bridegroom).) PiT generations include fine features and of reasonably similar quality as the DiT generations with locality [].

#### Results.

As shown in Tab. 4, we find that for PiT too, self-supervised pre-training with MAE improves accuracy compared to training from scratch. This is true for both PiT-T and PiT-S that we experimented with. Notably, the gap between ViT and PiT, with pre-training, gets larger when we move from Tiny to Small models. This suggests PiT can potentially scale better than ViT.

### 5.3 Case Study #3: Image Generation

We switch to image generation with Diffusion Transformer (DiT) [] in this section, which has a modulation-based architecture different from vanilla ViT, and operates on the latent token space from VQGAN [, ] that shrinks the input size by 8$\times$. Dataset-wise, we use ImageNet for class-conditional generation, and each image is center-cropped to 256$\times$256, resulting in an input feature map size of 32$\times$32$\times$4 (4 is channel dimension). PiT-L is fed with this feature map, same as its baseline DiT-L/2 [].

#### Evaluation metrics.

The generation quality is measured by standard metrics: Fréchet Inception Distance (FID) [] with 50K samples, sFID [], Inception Score (IS) [], and precision/recall [], using reference batches from the original TensorFlow evaluation suite of [].

#### Implementation details.

We followed the settings for DiT training, with a larger batch size (2048) to make the training faster (the original recipe uses a batch size of 256). To make the training stable, we perform linear learning rate warm up [] for 100 epochs and then keep it constant for a total of 400 epochs.
We use a maximum learning rate of 8e-4, with no weight decay applied.

#### Qualitative results.

Sampled generations from PiT-L are shown in Fig. 3. The sampling takes 250 time steps, with the latent diffusion outputs mapped back to the pixel space using the VQGAN decoder. A classifier-free guidance [] scale of 4.0 is used. All generations are detailed and reasonable compared to the DiT models with the locality inductive bias [].

**Table 5: Results for case study #3 (image generation). We use reference batches from the original evaluation suite of [], and report 5 metrics comparing locality-biased DiT-L/2 and PiT-L. With the exception of the last two rows, we use classifier-free guidance [] (scale 1.5) during the generation process (250 steps). Last row is from [], compared to which our baseline is significantly stronger (8.90 FID [] with DiT-L, *vs*. 10.67 with DiT-XL and longer training). Overall, our finding generalizes well to this new task with a different architecture and different input representations.**
| model | FID ($\downarrow$) | sFID ($\downarrow$) | IS ($\uparrow$) | precision ($\uparrow$) | recall ($\uparrow$) |
| --- | --- | --- | --- | --- | --- |
| DiT-L/2 | 4.16 | 4.97 | 210.18 | 0.88 | 0.49 |
| PiT-L | 4.05 | 4.66 | 232.95 | 0.88 | 0.49 |
| DiT-L/2, no guidance | 8.90 | 4.63 | 104.43 | 0.75 | 0.61 |
| DiT-XL/2 [], no guidance | 10.67 | - | - | - | - |

#### Quantitative comparisons.

We summarize qualitative comparisons between DiT-L/2 and PiT-L in Tab. 5. First, our baseline is strong despite the change of training recipe: compared to the reference 10.67 FID [] with a larger model (DiT-XL/2) and longer training ($\sim$470 epochs), our DiT-L/2 achieves *8.90* without classifier-free guidance. Our main comparison (first two rows) uses a classifier-free guidance of 1.5 with 250 sampling steps. With PiT operating on the latent ‘pixels’, it outperforms the baseline on three metrics (FID, sFID and IS), and is on-par on precision/recall. With extended training, the gap is bigger (see Appendix 0.B).

Our demonstration on the image generation task is an important extension of PiT. Compared to the case studies on discriminative benchmarks from Sec. 5.1 and Sec. 5.2, the task has changed; the model architecture is changed from standard ViT to a conditioned one; the input space is also changed from raw pixels to latent encodings from the VQGAN tokenizer. The fact that PiT works out-of-box suggests our observation generalizes well, and locality-free architecture can be used across different tasks, architectures, and operating representations.

Figure: (a) $\delta{=}2$
Refer to caption: x5.png

## 6 Locality Designs in ViT

Finally, we complete the loop of our investigation by revisiting the ViT architecture, and examining the importance of its two locality-related designs: (i) position embedding and (ii) patchification.

#### Experimental setup.

We use ViT-B for ImageNet supervised classification. We adopt the exact same hyper-parameters, augmentations, and other training details from the scratch training recipe from []. Notably, images are crop-and-resized to 224$\times$224 and divided into 16$\times$16 non-overlapping patches.

#### Position embedding.

Similar to the investigation in [], we choose from three candidates: sin-cos [], learned, and none. The first option introduces locality into the model, while the other two do not. The results are summarized below:

| PE | sin-cos | learned | none |
| --- | --- | --- | --- |
| Acc@1 | 82.7 | 82.8 | 81.2 |

Our conclusion is similar to the one drawn by [] for self-supervised representation evaluation: learnable position embeddings are on-par with fixed sin-cos ones. Intrestingly, we observe only a minor drop in performance even if there is no position embedding at all – ‘none’ is only worse by 1.5% compared to sin-cos. Note that without position embedding, the classification model is fully *permutation invariant* w.r.t. patches, though not w.r.t. pixels – will show evidence next.

#### Patchification.

Next, we use learnable position embeddings and study patchification. To systematically reduce locality from patchification, our key insight is that neighboring pixels should no longer be tied in the same patch.
To this end, we perform a pixel-wise permutation before diving the resulting sequence into separate tokens. Each token contains 256 pixels, same in number to pixels in a 16$\times$16 patch. The permutation is shared, *i.e*., stays the same for all the images – including the ones for testing.

The permutation is performed in $T$ steps, each step will swap a pixel pair within a distance threshold $\delta\in[2,\inf]$ ($2$ means within the 2$\times$2 neighborhood, $\inf$ means any pixel pair can be swapped). We use hamming distance on the 2D image grid. $T$ and $\delta$ control how ‘corrputed’ an image is – larger $T$ or $\delta$ indicates more damage to the local neighborhood and thus more locality bias is taken away. Fig. 4 illustrates four such permutations.

Fig. 5 illustrates the results we have obtained. In the table (left), we vary $T$ with no distance constraint (i.e., $\delta=\inf$). As we increase the number of shuffled pixel pairs, the performance degenerates slowly in the beginning (up to 10K). Then it quickly deteriorates as we further increase $T$. And at $T=25K$, Acc@1 drops to 57.2%, a 25.2% decrease from the intact image. Note that in total there are $224\times 224/2=25,088$ pixel pairs, so $T=25K$ means almost all the pixels have moved away from their original location. Fig. 5 (right) shows the influence of $\delta$ given a fixed $T$ (10K or 20K). We can see when farther-away pixels are allowed for swapping (with greater $\delta$), performance gets hurt more. The trend is more salient when more pixel pairs are swapped ($T=20K$).

Overall, pixel permutation imposes a much more significant impact on Acc@1, compared to changing position embeddings, suggesting that *patchification is much more crucial for the overall design of ViTs*, and underscores the value of our work that removes the patchification altogether.

#### Discussion.

As another way to remove locality, pixel permutation is highly destructive. On the other hand, PiT shows successful elimination of locality is possible by treating individual pixels as tokens. We hypothesize this is because permuting pixels not only damages the locality bias, but also hurts the other inductive bias – *translation equivariance*. In PiT, although locality is removed altogether, the Transformer weights are still shared to preserve translation equivariance; but with shuffling, this inductive bias is also largely removed. The difference suggests that translation equivariance remains important and should not be disregarded, especially after locality is already compromised.

Figure: Figure 5: Results of pixel permutation for ViT-B on ImageNet. We vary the number of pixel swaps $T$ (left) and maximum distance of pixel swaps $\delta$ (right). Pixel permutations can drop accuracy by 25.2% (when 25K pairs are swapped), compared to the relatively minor drop (1.6%) when the position embedding is completely removed. These results suggest pixel permutation imposes a much more significant impact on performance, compared to swapping position embeddings.
Refer to caption: x9.png

## 7 Conclusion and Limitations

Through our explorations, we have demonstrated that Transformers can directly work with individual pixels as tokens. This is surprising, as it allows for a clean, potentially scalable architecture without *locality* – an inductive bias that was presumably fundamental for vision models. Given the spirit of deep learning that aims to replace manually inducted priors with data-driven, learnable alternatives, we believe our finding is of great value to the community, especially when designing the next-generation of models for the domain of 2D images and beyond.

However, the practicality and coverage of our current demonstrations remains limited. Given the quadratic computation complexity, PiT is more of a method for investigation, and less for applications. And even with the additional tasks in Appendix 0.C, the study is still not comprehensive. Nonetheless, we believe this work has sent out a clear, unfiltered message that locality is *not fundamental*, and patchification is simply a *useful* heuristic that trades-off efficiency *vs*. accuracy.

## Appendix 0.A Visualizations of PiT

To check what PiT has learned, we experimented different ways for visualizations.
Unless otherwise specified, we use PiT-B and ViT-B models trained with supervised learning on ImageNet classification, and compare them side by side.

#### Mean attention distances.

In Fig. 6, we present the mean attention distances for PiT and ViT across three categories: late layers (last 4), middle layers (middle 4), and early layers (first 4). Following [], this metric is computed by aggregating the distances between a query token and all the key tokens in the image space, weighted by their corresponding attention weights. It can be interpreted as *the size of the ‘receptive field’* for Transformers. The distance is normalized by the image size, and sorted based on the distance value for different attention heads from left to right.

As shown in Fig. 6(a) and Fig. 6(b), both models exhibit similar patterns in the late layers, with the metric increasing from the 8th to the 11th layer. In the middle layers, while ViT displays a mixed trend among layers (see Fig. 6(d)), PiT clearly extract patterns from larger areas in the relatively later layers (see Fig. 6(c)). Most notably, PiT focuses more on local patterns by paying more attention to small groups of pixels in the early layers, as illustrated in Fig. 6(e) and Fig. 6(f).

#### Mean attention offsets.

Fig. 7 shows the mean attention offsets between PiT and ViT as introduced in []. This metric is calculated by determining the center of the attention map generated by a query and measuring the spatial distance (or offset) from the query’s location to this center. Thus, the attention offset refers to the degree of spatial deviation of the ‘receptive field’ – the area of the input that the model focuses on – from the query’s original position. Note that different from ConvNets, Self-Attention is a *global* operation, not a local operation that is always centered on the current pixel (offset always being zero).

Interestingly, Fig. 7(e) suggests that PiT captures long-range relationships in the first layer. Specifically, the attention maps generated by PiT focus on regions far away from the query token – although according to the previous metric (mean attention distance), the overall ‘*size*’ of the attention can be small and focused in this layer.

#### Figure-ground segmentation in early layers.

In Fig. 8, we observe another interesting behavior of PiT. Here, we use the central pixel in the image space as the query and visualize its attention maps in the early layers. We find that the attention maps in the early layers can already capture the *foreground* of objects. Figure-ground segmentation [] can be effectively performed with low-level signals (*e.g*., RGB values) and therefore approaches with a few layers. And this separation prepares the model to potentially capture higher-order relationships in later layers.

Figure: (a) PiT in late layers.
Refer to caption: extracted/5665349/vis/comp/PiT-B_avg-att-dist_8-12.png

Figure: (a) PiT in late layers.
Refer to caption: extracted/5665349/vis/comp/PiT-B_avg-att-offset_8-12.png

Figure: Figure 8: Figure-ground segmentation in early layers of PiT. In each row, we show the original image and the attention maps of two selected early layers (from first to fourth). We use the central pixel in the image space as the query and visualize its attention maps. Structures that can capture the foreground of objects have already emerged in these layers, which prepares for the later layers to learn higher-order relationships.
Refer to caption: x10.jpeg

**Table 6: Extended results for image generation. We continue the training from 400 epochs (main paper) to 1400 epochs, and find the gap between DiT and PiT becomes larger, especially on FID.**
| epochs | model | FID ($\downarrow$) | sFID ($\downarrow$) | IS ($\uparrow$) | precision ($\uparrow$) | recall ($\uparrow$) |
| --- | --- | --- | --- | --- | --- | --- |
| 400 | DiT-L/2 | 4.16 | 4.97 | 210.18 | 0.88 | 0.49 |
| PiT-L | 4.05 | 4.66 | 232.95 | 0.88 | 0.49 |  |
| 1400 | DiT-L/2 | 2.89 | 4.43 | 242.13 | 0.85 | 0.54 |
| PiT-L | 2.68 | 4.34 | 268.82 | 0.85 | 0.55 |  |

## Appendix 0.B Extended Results on Image Generation

In the main paper (Sec. 5.3), both image generation models, DiT-L/2 and PiT-L, are trained for 400 epochs.
To see the trend for longer training, we followed [] and simply continued training them till 1400 epochs while keeping the learning rate constant.

The results are summarized in Tab. 6. Interestingly, longer-training also benefits PiT more than DiT. Note that FID shall be compared in a *relative* sense – a 0.2 gap around 2 is bigger than 0.2 around 4.

## Appendix 0.C Generalization to Other Tasks

To further examine the generalization of our observation, we tried PiT on two more tasks: (i) *fine-grained classification* on Oxford-102-Flower [], which requires nuanced understanding; and (ii) *depth estimation* on NYU-v2 []. Given the computation budget, we resize images to either 32$\times$32 (former) or 48$\times$64 (latter), and follow standard protocols to train and evaluate models. The results again shows PiT holds more effectiveness over ViT in quality:

|  | *fine-grained classification* | *depth estimation* |  |
| --- | --- | --- | --- |
|  | Acc@1 ($\uparrow$) | Acc@5 ($\uparrow$) | RMSE ($\downarrow$) |
| ViT-S/2 | 45.8 | 68.3 | 0.80 |
| PiT-S | 46.3 | 68.9 | 0.72 |

## Appendix 0.D Texture vs . Shape Bias Analysis

As a final interesting observation, we used an external benchmark(^4^44[https://github.com/rgeirhos/texture-vs-shape](https://github.com/rgeirhos/texture-vs-shape)) which checks if an ImageNet classifier’s decision is based on texture or shape. ConvNets are heavily biased toward texture ($\sim$20 in shape bias). Interestingly, we find PiT relies *more* on shape than ViT (57.2 *vs*. 56.7), suggesting that even when images are broken down into sets of pixels, Transformers can still sift through potentially abundant texture patterns to identify and rely on the sparse shape signals for tasks like object recognition.

## Appendix 0.E Additional Notes for Training PiT

While for some cases (*e.g*., DiT []), the training recipe can be directly transferred to PiT; for some other cases, we do want to note more potential challenges during training. Below we want to especially highlight the effect of *reduced learning rates* when training a supervised ViT from scratch.

We take CIFAR-100 as a representative example. As shown in Fig. 9, we find for PiT, the training becomes unstable if we maintain the same learning rate from ViT. It is especially vulnerable toward the end of the schedule. When the initial learning rate is reduced from $2e^{-3}$ to $1e^{-3}$, the training is more stable and leads to better accuracy. Similar observations are also made on ImageNet.

Figure: Figure 9: Training curves of different learning rates (PiT-T, AdamW, batch 1024, CIFAR-100). We find with a reduced learning rate compared to ViT training, the curve is more stable, and ultimately leads to better accuracy.
Refer to caption: extracted/5665349/fig/training_curve_lr.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 6
presentation: 4
questions: See Weaknesses.
soundness: 3
strengths: 1. This paper is well-written.
2. The topic is of great interest to the computer vision community.
3. The experiments are extensive.
summary: The study investigates whether the inductive bias of locality is necessary for modern computer vision models. It explores the effectiveness of using vanilla Transformers to directly treat each individual pixel as a token for various computer vision tasks. The findings show that pixel-based Transformers can perform competitively in supervised and self-supervised learning, as well as image generation, challenging the belief that locality is essential. The research examines how removing locality impacts architectures like Vision Transformers (ViT) and compares results with patch-based models. Although computationally intensive, the study demonstrates that locality-free Transformers can still learn effective visual representations, which may inform future neural network designs.
weaknesses: In general, I agree to recommend the acceptance of this paper. However, I think it is a borderline paper, due to the following reasons.
1. The finding is not surprising: removing the locality of visual backbones, and the models still work, at least to some extent. I think this conclusion can be expected by most of the researchers within the community (btw, I appreciate the efforts of authors to make these observations clear).
2. As also recognized by the authors, the practicality of this conclusion is questionable, given its limited computation efficiency.
3. The down-sampling algorithm (e.g., bicubic resampling) has already introduced some inductive biases of locality. It can be seen as a convolutional layer with fixed kernels. This effect may be particularly notable when the resolution is relatively small (e.g., 28x28 on ImageNet).

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 3
strengths: + The paper is well-written and has great clarity on motivation and experiment designs.
+ The authors extensively test the model on various tasks and benchmarks.
+ It's interesting to see that the patchification design to be explicitly re-checked.
summary: This paper presents a study of locality in ViT, through decreasing the patch size down to 1x1. The paper shows that smaller patch will always get better performance and 1x1, which does not encode locality in each patch, gets the best performance. The work contributes to understandings of whether locality / patchification is necessary. The authors conduct experiments on supervised learning, self-supervised learning, and image generation. The results validate the hypotheses and statements.
weaknesses: - The work focuses on using smaller side of ViTs. I wonder if scaling up, does 1x1 patch still help in performance? Could it be that for ViT/L or ViT/H the patchification does not matter that much among 1x1, 2x2, or 4x4? 

- This paper proposes a finding, but it seems no good solution is available for now. The growth of #tokens is quadratic, 512 x 512 will give you 262144, and 1048576 for 1024 x1024. Further more, video will even give more tokens. Flash Attention and other efficient version will still have a tough time taking these many tokens as inputs.

- The authors discuss a series of related works. Another important and emerging line of work that uses grouping operation is missing though. Images as set of points[1] is strongly relevant work that breaks the image into 1x1 pixel representation and uses fixed centers clustering / pooling for representation learning. The slot attention-based backbone [2], although do not use 1x1 patches, made the grouping operator (instead of key-axis attention in ViT) work on ImageNet scale and should also be discussed. Do the authors think grouping can be a more handy way for forming "flexible and irregular-shaped patches" automatically, and enjoy the sweet spot of lower computation while having smaller patch sizes?

[1] Image as Set of Points

[2] Perceptual Group Tokenizer: Building Perception with Iterative Grouping

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 4
questions: The concept of locality-free in this work is conditional and needs to be clearly differentiated from random pixel-level training. All discussions regarding locality-free should specify these conditions; otherwise, the statements may be too bold. While the paper identifies a problem, it does not explore potential solutions or the feasibility of addressing them. The authors should address these questions during the rebuttal to improve the rating score.
soundness: 4
strengths: 1. The questions raised are interesting. The locality-free concept is essential for bridging the gap between computer vision and natural language processing when language models are adopted to solve visual recognition tasks.
2. The observations are compelling and support the general conclusions on the potentials of locality-free modeling under certain conditions. The study performs extensive experiments across various scales, revealing consistent and widespread findings.
3. The experiments are well designed to demonstrate the potential of locality-free modeling for images.
summary: This work poses an intriguing question regarding locality-free modeling of images using Transformers. The new modeling perspective centered on the locality-free concept presents an interesting direction for applying Transformers in computer vision, given their success in machine translation tasks. The study conducts extensive experiments across various scale tasks, revealing consistent and general observations. These findings lead to broader conclusions regarding the effectiveness of locality-free modeling under certain conditions.
weaknesses: The conditions underlying the observations and conclusions require greater emphasis and detail. While "locality-free" suggests that input patches or tokens can be fragmented, it is still essential to preserve the relationships among adjacent input vectors or scalars. Developing a dictionary or tokenizer for image pixels is challenging, which hinders the advancement of locality-free models. The conclusions presented summarize valuable insights, but many of these observations, especially pixel-based modeling, probably have been considered during the design of Vision Transformers (ViTs). The limitations in their efficient training contribute to their performance issues. It is also quite challenging to represent image pixels semantically in a linguistic manner. If the authors could propose an effective training scheme or a pixel dictionary design for locality-free approaches, this work would become even more insightful and beneficial.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Rather than questions, I have a few suggestions to improve writing.

L411 Intrestingly, -> Interestingly

L201: in range [0,255],  the vocab size is 256^3 not 255^3. Please revise all the writing to ensure the correct information.

L44-46: The authors should clarify **why** translation equivariance still exists in ViT. Translation equivariance in neural networks, particularly in convolutional neural networks (CNNs), is the property where shifting the input (e.g., an image) results in a corresponding shift in the output features. This means the network recognizes that the content has moved spatially without altering the actual information content, which is useful for tasks like object detection and localization.

In transformers, translation equivariance is not naturally present. Transformers process input sequences as tokens, where each token attends to all other tokens through self-attention. transformers don’t inherently “understand” spatial shifts in the same way that CNNs do because each token’s positional information is encoded separately, often through positional embeddings.

All my concerns are written in Weaknesses section.
soundness: 2
strengths: * Clear and Detailed Writing: The paper is well-organized, with clear explanations of the rationale behind pixel-based tokenization, the potential benefits, and the authors are upfront about the trade-offs involved. I only suggest a few writing modifications below.

* Thorough Experimental Evaluation: The paper includes a comprehensive set of experiments across multiple tasks—classification, self-supervised learning, and image generation, although adding more baselines will make the evaluation more complete and convincing.
summary: The paper studies the role of locality biases in Vision Transformers (ViTs) by examining whether treating individual pixels as tokens (1x1 patches)—rather than conventional 16×16 patches—is sufficient for computer vision tasks (image classification or generation). 

This pixel-based approach yields 2 benefits:
1. Reduced Vocabulary Size: By treating each pixel as an independent token, the "vocabulary" size (i.e., unique values the model must handle) becomes more manageable.
2. Model Agnosticism to Locality Bias: The pixel-based Transformer performs competitively across various tasks, such as image classification, self-supervised learning, and image generation, suggesting that locality may not be as essential for vision tasks for ViT as previously thought.

The authors evaluate this approach on tasks like classification (CIFAR-100, ImageNet), self-supervised learning, and image generation through diffusion models. They find that the pixel-based Transformer achieves results comparable to patch-based ViTs, though with higher computational costs due to increased sequence lengths.

While the paper is well-written, it falls short in addressing the practical implications of using 1x1 patches beyond merely demonstrating feasibility. (i.e. other than knowing that we can use 1x1 patches, what is the value of using 1x1 in practice?). I am hesitant to recommend this paper for NeurIPS.
weaknesses: * The empirical results in Table 3 for Case Study #1 show only marginal performance improvements when using 1×1 tokens on datasets like ImageNet and Oxford-102-Flower. The evaluation would be more convincing if results were presented for all four size variants (ViT-T/S/B/L) and included a 16×16 baseline for comparison.
* In Case Study #2, there is a lack of detailed information on the setup and evaluation. It’s unclear what new value or insights this setup offers beyond those in Case Study #1. This section appears to have been completed in a rush.
* In Table 5, there is no clear advantage in using 1×1 tokens over 2×2 tokens. Results might benefit from additional baselines, such as 4×4 or 8×8 tokens.
* In Figure 3, qualitative examples for the 1×1 approach are shown without comparable examples from baseline methods. Including baseline results would provide a more effective comparison.
* While the authors are upfront about the inefficiency of using 1x1 patches, I am unsure about the practical value of using 1x1 patches (though vocab size benefit was mentioned). The overall takeaway for me is: “We can use 1x1 rather than 2x2 and we still get good performance” but the answer why this finding matters is not clearly explained.

Meta-review: This paper explores whether inductive bias towards locality, a hallmark of convolutional networks and Vision Transformers (ViTs), is truly necessary for computer vision tasks. The study replaces patch-based tokenization (e.g., 16x16 patches) with a "pixels-as-tokens" approach, where each individual pixel is treated as a token. The results demonstrate that this locality-free design performs competitively across various tasks, including supervised learning, self-supervised learning, and image generation. However, this approach is computationally intensive, owing to the increased sequence length.

Strengths:
* Challenging status of quo: The work challenges a fundamental belief in computer vision about the importance of locality and provides a new perspective for Transformer-based vision models.
* Thorough Experiments: Extensive evaluations across tasks and datasets add credibility to the findings.
* Clarity: The paper is well-written, with clear explanations of the motivation, experimental setup, and results.

Weaknesses:
* Practical Limitations: The approach is computationally inefficient due to the quadratic growth of tokens, making it less practical for large-scale vision tasks.
* Expected Findings: Some reviewers felt the results were not surprising, given prior knowledge that reducing inductive bias often leads to competitive results in deep learning.
* Late Introduced Claims: Some claims, such as comparisons between patchification and advanced tokenization, were introduced late in the rebuttal process and lacked sufficient substantiation.

Despite concerns about practicality and broader significance, this paper provides a meaningful contribution to the field by challenging established assumptions about locality in vision models. Its findings, though not entirely unexpected, are supported by robust experiments and could inspire further innovations in designing Transformers for vision tasks and beyond.

Decision: Accept (Poster)