Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Authors: Feng Wang
Yaodong Yu
Wei Shao
Yuyin Zhou
Alan Yuille
Cihang Xie
Venue: ICML 2025 poster (2025)
Keywords: Scaling law
Patchification
Primary area: applications->computer_vision

Abstract: Since the introduction of Vision Transformer (ViT), patchification has long been regarded as a common image pre-processing approach for plain visual architectures. By compressing the spatial size of images, this approach can effectively shorten the token sequence and reduce the computational cost of ViT-like plain architectures. In this work, we aim to thoroughly examine the information loss caused by this patchification-based compressive encoding paradigm and how it affects visual understanding. We conduct extensive patch size scaling experiments and excitedly observe an intriguing scaling law in patchification: the models can consistently benefit from decreased patch sizes and attain improved predictive performance, until it reaches the minimum patch size of 1*1, i.e., pixel tokenization. This conclusion is broadly applicable across different vision tasks, various input scales, and diverse architectures such as ViT and the recent Mamba models. Moreover, as a by-product, we discover that with smaller patches, task-specific decoder heads become less critical for dense prediction. In the experiments, we successfully scale up the visual sequence to an exceptional length of 50,176 tokens, achieving a competitive test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We hope this study can provide insights and theoretical foundations for future works of building non-compressive vision models.

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Feng Wang' Yaodong Yu? 2 Wei Shao
Yuyin Zhou Alan Yuille' Cihang Xie
Abstract
series of visual foundation models learned with this plain architecture achieving highly competitive performance and establishing effective connections to other modalities such as natural language (Dosovitskiy et al., 2021; Caron et al., 2021; Radford et al., 2021; Yu et al., 2022a; Rombach et al., 2022; Kirillov et al., 2023; Liu et al., 2023). A key insight behind the ViT-like architectures lies in a compressive en- coding paradigm: instead of directly processing raw pixels that introduces significant complexity, these architectures leverage a patchification layer to compress images into spa- tially smaller feature maps, making the representation space of an image roughly equivalent to that of a medium-length text consisting of a few hundred tokens.
Since the introduction of Vision Transformer (VIT), patchification has long been regarded as a de facto image tokenization approach for plain visual architectures. By compressing the spa- tial size of images, this approach can effectively shorten the token sequence and reduce the com- putational cost of ViT-like plain architectures. In this work, we aim thoroughly examine the in- formation loss caused by this patchification-baat based compressive encoding paradigm and how it af- fects visual understanding. We conduct exten- sive patch size scaling experiments and excitedly observe an intriguing scaling law in patchifica- tion: the models can consistently benefit from decreased patch sizes and attain improved predic- tive performance, until it reaches the minimum patch size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable across different vision tasks, various input scales, and diverse ar- chitectures such as ViT and the recent Mamba models. Moreover, as by-product, we discover that with smaller patches, task-specific decoder heads become less critical for dense prediction. In the experiments, we successfully scale up the vi- sual sequence to an exceptional length of 50, 176 tokens, achieving a competitive test accuracy of 84.6% with a base-sized model on the ImageNet- Ik benchmark. We hope this study can provide insights and theoretical foundations for future works of building non-compressive vision models. Code is available at https: / wtchbb.ccm wangf 4
However, we argue that this operation often incurs irre- versible information loss to visual inputs. For example, intu- itively, we believe the information contained in x resolution image is generally much richer than that in text consisting of 196 words; however they have nearly the same size of representation space under a VIT encoder with patch size 16x 16 (we suppose the vision and language encoders share the same embedding dimension). The difference in information content between visual and textual data can also be directly reflected in their storage requirements: storing an uncompressed 24-bit, 224x224 resolution image requires approximately 147KB, whereas storing 196-word text only needs about 1.1.. Empirically, if we manually reduce the compression rate, for example, by changing the patch size of DeiT-Base from 16x 16 8x8, we can observe a signif- icant accuracy improvement from 81.8% to 83.5% on the ImageNet-Ik classification benchmark (Deng al., 2009).
Nonetheless, since the computation of self-attention scales quadratically with sequence length, ViT architectures are sensitive to the patch size. At the time when ViT was first introduced in late 2020, it needed to ensure that its computa- tional cost was comparable to that of the CNN counterparts; and given the computational capacity at that time, the mod- els had to be computationally manageable in terms of mem- ory consumption and training time, especially when trained with the medium- resolution, medium-scall ImageNet (Deng et al., 2009) and beyond (Sun et al., 2017). As a result, the architectural design of VIT had to compromise with a com- pressive encoding paradigm achieved through patchification.
1. Introduction
In the past few years, we have witnessed the great success of Vision Transformers (ViTs) in representation learning, with
'Johns Hopkins University 'UC Berkeley 'University of Florida *UC Santa Cruz. Correspondence to: Cihang Xie <cixie@uese.cduu
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
The success of this design, patchification with typical 16x 16 16-pieel kernel, has led to its widespread adoption as

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
2.2177
0,987
1.765
0.93
0914
0696
Test loss
0.892
Test loss
0307
Test loss
0,022
0.34
0,37
0.64
0.221
0.1
0.67
16
16
16
Patch size
Patch size
Patch
(a) DeiT-B, 64x64 Input, CLS
(b) Adventurer-B, 128x 128 Input, CLS
(c) Adventurer-BB B, 224x224 Input, CLS
33g
0.238
0.252
0
335
Test loss (bbox)
0.223
Test loss (mask)
0238
Test loss
0
0. 327
0,009
022
0 522
0.177
0.203
0.317
0.86
0102
12
Patch size
Patch size
Patch size
(d) ADE20K Semantic Segmentation
(e) COCO Object Detection
(I) COCO Instance Segmentation
168
0.968
009
0924
0.867
0.794
Test loss
a556
Test loss
0.22
Test loss
0.225
8002
07102
0000
0.792
0.695
32
16
16
Patch size
Patch size
Patch size
(g) DeiT-BB 128x 128 Input, CLS
(h) Adventurer-L, 128x 128, CLS
(i) Adventurer-T, 224x224, CLS
Figure 1. Patchification Scaling Laws. We observe smooth and consistent decrease loss across different vision tasks, input resolutions, and model architectures when reducing the patch size. The performance gains remain considerably significant even when scaling down the patch size to 1xL I. In sub-figures, both and y axes are log scale. CLS denotes ImageNet- Ik classification.
a default component in various subsequent architectures, even including those non-attention models such as Con- vNeXt (Liu al., 2022) and Vision Mamba (Zhu et al., 2024), while the impact of information loss posed by this compressive encoding paradigm has not been well studied.
lock the compressed information by reducing the patch size. We highlight that through patchification, there significant room for scaling up the model's 's computation, and new scaling law may emerge during this process.
Thanks to the rapid advancements in hardware, efficient attention mechanisms (Dao et al., 2022; Kwon et al., 2023), as well as linear-complexity structures (Katharopoulos et al., 2020; Peng et al., 2023; Gu Dao, 2023), we can now ex- tensively validate the impact of patchification a standard input size (e.g.. 224 224 for ImageNet) with manageable computing resources. We conduct series of straightfor- ward scaling experiments on patchification, gradually reduc- ing the model's patch size from the typical 16x 16 down to 1x to lower the compression rate and observe how performance changes. We employ both ViT and Adven- turer (Wang et al., 2024c), a Mamba-based (Dao Gu, 2024) cinear-complexity architecture, to make our conclu-
In this work, we aim to thoroughly examine how compres- sive encoding affects visual representations and whether patch size can be a new scaling dimension for modern visual architectures. While the concept of Scaling Laws (Kaplan al., 2020) has been broadly testified in natural language processing, leading to great prosperity of Large Language Models over the past few years (Touvron et al., 2023; Team et al., 2023; Achiam et al., 2023), the scaling-up of vision models faces practical issues in the dimensions of both pa- rameter size and input size (detailed Section 4.3). Here, we aim revisit the scaling potential of vision models from new perspective spatial compression, attempting to un-
2

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
sions generalizable and experiments affordable in compu- tation. To our surprise, this simple scaling study delivers three intriguing discoveries:
The landscape of visual backbones underwent another round of significant transformation with the introduction of ViTs (Dosovitskiy et al., 2021) in late 2020, where a novel plain architecture was proposed that treats images akin to language sequences. This model utilizes a simple patchification layer to convert images into sequences of to- kens, which are then processed using mechanisms adapted from language models. This approach opened new avenues in handling visual data without the inductive biases inher- ent in CNNs, demonstrating competitive performance on several benchmarks. The success of ViT's have spurred rapid development and innovations in data-efficient training strategies (Touvron et al., 202 2022a;b), seff-uupereiee learning techniques (Caron et al., 2021; Chen et al., 2021b; Bao et al., 2022: He et al., 2022; Ren et al., 2024b), vision- language understanding (Radford et al., 2021; Jia al., 2021; Liu et al., 2023; Alayrac et al., 2022: Yu et al., 2022a; Wang et al., 2024a), and hierarchical architecture designs (Liu et al.. 2021; Chen et al., 2021a; Yuan et al., 2021; Yu et al., 2022b).
First, as shown in Figure 1, we excitedly observe new scal- ing law for patchification in vision models. Similar to the scaling laws discovered in early studies (Kaplan et al., 2020) on language- where continuously increasing model parame- ters reliably leads consistent performance gains-we we have identified new scaling dimension for vision models, which is reflected in the observation that as the compression rate (i.e., patch size) decreases, the model's 's test loss smoothly declines, reaching its limit at single-pixe patch sizes that es- sentially form a compeessreee encoding paradigm. This conclusion broadly holds true for various vision tasks, di- verse input scales, and different visual architectures.
Second, confirm that visual encoding can be performed in very long token sequence, while patchification is not a requisite for building effective vision models, but rather a compromise to memory and computation overhead when resource is limited. The information lost in the compression of the patchification layer is actually crucial for the model's prediction: on the standard 224 x 224-resolution ImageNet- Ik classification benchmark, we remove the patchification operation and form a super-long visual sequence consist- ing of 50,176 tokens, by which we boost the model's ' test accuracy from 82.6% to a remarkable result of 84.6%.
Inspired by the patchification design of transformers, there have been many CNN-based (Liu et al., 2022) and State Space Model (Kalman, 1960; Gu et al., 2022; 2021) based architectures (Zhu et al., 2024; Wang et al., 2024b;c) follow- ing the same paradigm. Notably, the Mamba (Gu & Dao, 2023; Dao & Gu, 2024) token mixer, due to its advantage of linear complexity, has recently been widely used to ex- plore vision tasks and has achieved competitive results (Zhu et al., 2024; Wang et al., 2024b;c; Ren et al., 2024a; Liu et al., 2024; Yang et al., 2024; Hatamizadeh & Kautz, 2024; Huang et al., 2024; Li et al., 2024; Ren et al., 2024c; Lieber et al., 2024; Wei & Chellappa, 2025). Among them, the Adventurer (Wang et al., 2024c) architecture, which sig- nificantly simplifies the overall model, has demonstrated superior speed compared to the Transformer. In this paper, we employ it as one of the primary experimental models.
Finally, we observe a compelling phenomenon in semantic segmentation: as we transit from patch- based tokenization to pixel-level modeling, the traditional necessity for de- coder head-long considered a default component since the inception of deep network architectures-can be eliminated without compromising performance. This architectural sim- plification is potentially profound, suggesting the possibility of developing decoder-free dense prediction models and il- luminating the path toward universal, encoder-only visual architecture capable of learning from every pixel.
Visual architecture scaling. Scaling laws was initially studied in natural language processing (Kaplan et al., 2020). In vision, a similar concept has guided the community to scale up foundational models in both parameter size and data volume. For example, in the age of CNNs, Efficient- Nets (Tan & Le, 2019; 2021) have proposed to scale-up the models in depth, width, and resolution. These advance- ments were then integrated into ResNets, leading to nearly Billion-level parameter CNNs (Xie et al., 2020; Kolesnikov et al., 2020; Huang et al., 2019; Bello et al., 2021; Wightman et al., 2021). Scaling the parameter count of Vision Trans- formers has also shown a great success in modern visual understanding benchmarks and has exhibited state-of-the-at results (Touvron et al., 2021b; Zhou et al., 2021; Zhai et al., 2022: Dehghaai et al., 2023). More recently, Nguyen et al. introduce Pixel Transformers, with standard patchification grids scaled down to pixels, showcasing promising scaling
2. Related Work
Generic visual backbones. The development of visual backbones has fundamentally shaped the field of computer vision. Initially dominated by Convolutional Neural Net- works (CNNs), these architectures have evolved to gain increasing capabilities for visual representation learning. Pioneering works such LeNet (LeCun et al., 1998) and AlexNet (Krizhevsky et al., 2012) have proven the signif- icant effectiveness of convolutional architectures in large- scale image classification tasks. Following these founda- tional models, the architecture has been refined with the innovations in model depth (Simonyan & Zisserman, 2015), residual connection (He al., 2016; Huang et al., 2017), and efficient neural architecture search (Tan & Le, 2019).
3

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
results for low-resolution (e.g., 32x32) input images.
as decoder head. Note that we use the same patch size for classification pretraining and downstream finetuning to ensure consistency in the scaling property.
3. Method
For semantic segmentation. in addition to evaluating the standard encoder-decoder structure, we also explore a decoder-free approach to observe the emerging properties of patchification scaling. Specifically, instead of using a deep UperNet (Xiao et al., 2018) as the default segmenta- tion head, we employ simple linear layer to project the dense features extracted by the backbone into the category dimension for training the semantic segmentation task.
3.1. Problem Formulation
This work aims to investigate the impact of spatial com- pression on the representation capability of modern visual architectures by scaling the downsampling rate of the patchi- fication operation. The primary experiments are conducted on ViT-like plain architectures, with their definition as fol- lows: The image encoder F g3xwxh + R con- sists of patchification layer at the beginning, positional embeddings, and a number of cascade token mixers and channel mixers. The patchification layer divides the input image x E wxx into non-overlapping patches of size px x p. flattening and projects them into a ID token sequence x' E' RLLDD The following mixer layers extract deep visual features while keeping the sequence length L and the feature dimensionality D unchanged-whie means the patchifica- tion layer makes the only spatial compression throughout the whole visual encoder.
This modification is based on the following prior assump- tion: in dense prediction tasks like semantic segmentation, the decoder head serves two main functions. The first is addressing the issue where the backbone's 's high downsam- pling rate results in feature granularity that is insufficient for pixel-level predictions-tppicee as atrous convolution and multi-scale feature fusion (Chen et al., 2017; 2018). The second function is enhancing the model's learning capacity by introducing additional train- able parameters. Under this assumption, we believe that if the backbone's compression rate is already very low, the decoder's 's benefits would be limited to the second aspect. Therefore, task-specific decoder head designs become less critical, and training a general high-fidelity backbone alone would be sufficient to handle various vision tasks.
mitigated by designs such
To eliminate the influence of different mixer types on the results of patchification scaling, we conduct the main exper- iments using two visual encoders: the standard ViT (Doso- vitskiy al., 2021) and Adventurer (Wang et al., 2024c). Due to the significant memory and computation challenges posed by the quadratic complexity of self-attention, ViT is only used for context lengths within 4,096 in this work. For longer sequence tasks, we employ Adventurer, a re- cent Mamba- (Gu & Dao, 2023; Dao & Gu, 2024) efficient architecture that excels in modeling long range de- pendencies with linear complexity. Adventurer shares the same plain framework VIT, with spatial compression only presents in the initial patchification layer, while the key dif- ference is that Adventurer leverages the recent Mamba (Dao & Gu, 2024) module as its token mixer, which has a lin- ear complexity relative to sequence length and allows us to perform pixel tokenization for even the standard 224x224 resolution inputs within reasonable computational resources (e.g., 256 A100 GPUs). Remarkably, in our experiments, we form a super-long visual sequence of 50,176 tokens for ImageNet inputs by scaling down the patch size to I 1x
4. Experiments
4.1. Experimental Setup
The experiments are conducted on the standard ImageNet- Ik (Deng et al., 2009) classification, ADE20k (Zhou et al., 2019) semantic segmentation, and COCO (Caesar et al., 2018) object detection and instance segmentation bench- marks. For ViTs, we follow the data-efficient strategy of DeiT (Touvron et al., 2021a) to train the model for 300 epochs by an AdamW (Loshchilov & Hutter, 2019) opti- mizer with 1024 batch size, 0.001 learning rate and 0.05 weight decay. For Adventurer, basically refer their optimized multi-stage training recipe to improve efficiency and obtain competitive results. The details of the training strategy can be found in Appendix. In semantic segmenta- tion, we follow the prior practice of DeiT and Adventurer to finetune the classification models with an AdamW opti- mizer, 5e-5 learning rate, 0.01 weight decay, a total batch size of 16 for 160k iterations. We train object detection and instance segmentation with Adam optimizer, le-4 learning rate and 0.05 weight decay for 12 epochs.
3.2. Technical Details
We conduct patchification scaling experiments on image classification, semantic segmentation, object detection and instance segmentation tasks. Following the standard design of ViTs (Dosovitskiy et al., 2021) and Adventurer, we ex- tract holistic visual features by learnable [CLS] token for classification. For object detection and instance segmenta- tion, we load backbones pretrained with classification and employ Cascade Mask R-CNN (Cai & Vasconcelos, 2019)
4.2. Main Results
As shown in Figure 1, we first evaluate the model's patchifi- cation scaling performance using test loss as unified metric
4

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
patch tokenization
pixel tokenization
Model
Input size
y=16
p=8
p=4
p=2
p=l
seq. length
DeiT-Base (Touvron et al., 2021a)
64
68.2
76.9
80.1
80.8
81.3
4,096
DeiT-Base (Touvron et al., 2021a)
128 x 128
78.1
81.0
82.3
82.9
Adventurer-BB (Wang al., 2024c)
64x64
69.2
77.2
80.0
80.5
80.9
4,096
Adventurer- Base (Wang et al., 2024c)
128 x 128
79.0
81.5
81.8
82.2
82.4
16,384
Adventurer-Base (Wang al., 2024c)
224
82.6
83.9
84.3
84.5
84.6
50,176
Table 1. Detailed NetgeNet classification results. As patch size (denoted as p) decreases, the test accuracy (%) on ImageNet-1k (Deng al., 2009) consistently improves and reaches the best performance with pixel tokenization. We highlight that we successfully scale up the visual token sequence to an unprecedented length of 50,176, with a competitive 84,6 test accuracy obtained by a basc-sized model.
across different input sizes, tasks, and parameter scales. We observe an interesting phenomenon that the model's pre- dictive performance consistently improves as the patch size decreases. This observation effectively highlights the nega- tive impact of the existing compressive encoding approach in visual models and supports our initial hypothesis: patchi- fication is not a necessary component for visual encoders; its primary role is to improve computational efficiency at the cost of partial information loss. Although this efficiency gain is significant for Transformer models with quadratic complexity, our findings suggest that when the comput- ing resource allows-ane and indeed, computational power has evolved rapidly over years-We we should reconsider the tradi- tional compressive encoding approach and begin embracing the notion of "a pixel is worth token" that stands for non-compressive representation learning paradigm.
Model
Decoder
Params
Patch size
mloU
UperNet
17M
16x 16
41.3
None
12M
16x 16
40.0
Adventurer-T
None
12M
8x8
41.6
None
13M
4x4
42.1
None
13M
2x2
42.5
UperNet
112M
16x16
45.7
None
99M
16x16
44.0
Adventurer-B B
None
99M
8x8
45.5
None
100M
4x4
46.3
None
100M
2x2
46.8
Table 2. ADE20k semantic segmentation. We focus on decoder- free structures and observe the mloU score improves smoothly when patch size shrinks. We highlight the results that reach the limits hardware capabilities in blue best results bolded.
We also observe that reducing the patch size not only im- proves performance in dense prediction tasks like semantic naturally
be effectively performed from very long contexts.
segmentation and instance segmentation- favor fine feature granularities and for which smaller patch size is direct solution-but also benefits holistic tasks like image classification, which inherently do not require fine- grained representations. This result indicates that the pri- mary benefit of reducing the patch size comes from unlock- ing the visual information that is previously compressed by patchification. This information, often considered insignifi- cant level features in the past, actually considerably critical for visual understanding.
47
w/o decoder
+0,1
w/ decoder
+0.3
+0.7
mloU (%)
46
+1.7
45
44
ImageNet classification results elaborated in Table 1. As shown, in terms of test accuracy, the models also ex- perience a smooth and consistent performance improve- ment with patch size decreasing. Notably, with the help of Adventurer's linear time complexity and efficient memory consumption, we successfully scale up the visual token se- quence length of 50, 176 in the ImageNet classification task. To knowledge, this is the first time that modern visual architectures have extended the input sequence to such a length and processed it directly without partitioning. It not only achieves a highly competitive 84.6% test accu- racy with a base-sized model ( (100M parameters), but more importantly, it demonstrates that visual understanding can
16
8
2
Patch Size
Figure 2. Decoder's impact on semantic segmentation. We train semantic segmentation model with the same backbone but differ- ent decoder heads: an UperNet with 13M parameters and simple linear layer with 0.2M parameters. We observe that as patch size decreases, the impact the decoder head diminishes.
ADE20k semantic segmentation results summarized in Table 2. As shown, we observe the same scaling behav- ior in this dense prediction task, with its test loss smoothly decreasing (see Figure 1) and mloU score consistently im- proving as patch size shrinks. It is worth noting that even though we eliminate the task-specific decoder head in this
5

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Model
Patch
AP"
AP
7P
APm
AP
AP7s
Adventurer-B Adventurer-T
32x32
44.7
63.3
48.6
38.4
60.4
41.4
84.5
16
46.5
65.2
50.4
40.3
62.2
43.5
8x8
48.0
66.7
51.8
41.7
63.6
45.0
ImageNet Top-I Accuracy
4x4
48.5
67.1
52.3
42.2
64.1
45,4
84.0
Training collapsed with more parameters
2x2
48.7
67.3
52.4
42.4
64.3
45.7
8
64x64
44.1
62.8
48.0
38.3
60.1
41.8
32x32
46.4
65.0
50.3
40.6
62.5
43.1
83.5
Huge (759M)
16x 16
48.4
67.2
52.4
42.0
64.8
45.0
Large (346M)
8x8
49.5
67.9
53.3
42.9
65.5
46.1
4x4
50.3
68.5
54.0
43.4
66.0
46.6
83.0
Patch size scaling
Base (99M)
Parameter scaling
Table COCO object detection and instance segmentation. Similar to classification and semantic segmentation results, these tasks exhibit consistently enhanced performance as patch size decreases. We highlight the results that reach the limits of hardware capabilities in blue and best results bolded.
82.5
1x
4x
16x
56x
150x
FLOPs
(a) Scaling from Adventurer-Basel/ 224x224 input.
83.0
ImageNet Top-l Accuracy (%)
experiment, the encoder-only models-whetiee the 13M- 82.0
Training collapsed with more parameter's
parameter tiny-sized model or the 100M-parameter base- sized model-can still produce competitive results when the encoding compression rate becomes sufficiently low.
p-8
81.0
Huge (631M)
80.0
Large (304M)
Figure presents a direct comparison on the impact of de- coders in semantic segmentation, where we load the same pretrained backbone (Adventurer-Basee and finetune sepa- rately with UperNet (Xiao al., 2018) and simple linear projection layer. As shown, with high spatial compres- sion rate such as 16x, the model can easily benefit from decoder head; however, as the patch size decreases and the encoder itself can produce sufficiently fine-grained features, the functionality of decoders starts to be marginalized.
79.0
Patch size scaling
Base (86M)
Parameter scaling
78.0
1x
20x
130x
FLOPs
(b) Scaling form ViT-Base/16, 128x 128 input.
Figure 3. Patch size scaling vs. parameter scaling, Given an Adventurer- Base with 224x224- inputs, we scale up the model along two dimensions respectively. The model struggles to achieve further accuracy improvements beyond 7700 param- eters, whereas scaling down the patch size continues to show a consistent upward trend in performance.
Interestingly, this experiment validates our hypothesis pre- sented in Section 3.2, demonstrating that the core compo- nent of developing dense prediction models lies in reducing the spatial compression rate, while the help that decoder heads can provide is very limited. This insight further sug- gests that with non-compressive encoders, it becomes fea- sible to build a visual foundation model that could provide pixel-level representations and effectively supports various downstream tasks without requiring significant efforts to adapt to their specific objectives. In this work, we keep focusing the exploration of patchification scaling and leave the development of foundation models for fu- ture research. We hope that our findings here can provide a solid theoretical foundation for such endeavors.
Notably, we have conducted patch size scaling experiments across four tasks: object classification, semantic segmen- tation, object detection, and instance segmentation. These experiments span a variety of input resolutions (from 66 to a short side of 800), different training objectives, and dif- ferent token mixer types iself-attentin and Mamba (Gu & Dao, 2023)). Despite these variations, a consistent and gen- eralizable conclusion emerges: Reducing patch size reliably guarantees performance gains.
COCO object detection and instance segmentation tasks also showcase similar effect of patchification scaling. As summarized in Table 3, both tasks achieve their best perfor- mance when the patch size reaches the hardware's 's compu- tational limits (2x2). Compared to the high compression baselines, both Adventurer- Tiny and Base models demon- strate significant precision improvements, such as 48.7% 7% vs. 44.7% for Adventurer- Tiny and 50.3% vs. 44.1 1 for Base.
4.3. Ablation Studies
Patchification scaling vs. parameter scaling, In Figure 3a, we compare the impact of scaling down the patch size ver- sus scaling up the parameter count on the performance of Adventurer. As shown, for fixed patch size and input
6

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
size, increasing the parameter count within a certain range (e.g., up 760M parameters) yields significant performance gains. However, further scaling beyond this point does not necessarily lead to additional benefits. In fact, overcom- ing the parameter scaling bottleneck in vision models is both technically challenging and costly. It often requires investing in higher-quality training data (Zhai et al., 2022; Radford et al., 2021), incorporating self-supervised learning approaches (Caron et al., 2021; He et al., 2022), and mak- ing extensive hyperparameter tuning efforts (Touvron et al., 2022b).
ImageNet Top- Accuracy (%)
82.9
9.4M
82.5
2.3M
37.7M
151.0M
82.1
ImageNet average imput size
81.7
Parameters for patchification
813
0.6M
In contrast, patch size scaling not only exhibits a better cnmputatioo-accuracy tradeoff and achieves higher perfor- mance limits than parameter scaling, but it also offers a simpler and more straightforward learning process: when training with different patch sizes, there is no need to modify training strategies or datasets, and all experiments can be done in single run using the same set of hyperparameters.
112
224
448
896
1792
Input Size
Figure 4. Input size scaling with fixed sequence length. We fix the ratio of mmage_size/patch. size for ImageNet classification. As shown, when the input size is scaled beyond its original resolutions (e.g., typically 460 for ImageNet), further interpolating the input images does not yield additional accuracy gains. Instead, it leads to a rapid increase in patchification parameters, resulting in training instability that ultimately harms performance.
and scale up the input
The potential of patchification scaling is even more evident in ViT. With the same input scale, reducing the patch size yields greater performance improvements for ViT compared to the linear-complexity Adventurer. Additionally, in terms of FLOPs, ViT has more room for scaling, as its computa- tion grows quadratically with sequence length. As shown in Figure 3b, due to this quadratic complexity, ViT expe- riences a larger increase in FLOPSs than Adventurer when scaling down the patch size, leading to a similar accuracy growth over FLOPs as that of parameter scaling (e.g., VYT- Base/8 vs. ViT-Large//6). However, when investing higher FLOPs, parameter scaling falls significant short and may easily collapse with higher parameter counts.
put size while proportionally increasing the patch size- the model undergoes a rapid growth in parameter count in the patchification layer, which may easily result in reduced model efficiency and stability during scaling. We showcase this issue in Figure 4, where it is observed that resizing in- puts beyond their original resolutions does not provide addi- tional information gains. Instead, the over-parameetrrizaton of the patchification layer leads to a considerable perfor- mance degradation. As comparison, the direct scaling of patch size can effectively avoid the paratttrttttinnnnooonnn issue, making it a flexible and practical scaling dimension for modern visual architectures.
Limitations of input size scaling, Compared to scal- ing down the patch size at a fixed input size, another merhod-directly scaling up the input size-can achieve a similar effect of reducing the compression rate and extend- ing the token sequence. However, we contend that changing the input size is not flexible and applicable approach for effective scaling, as its upper bound is easily constrained by the original resolution of the image. For example, in the standard ImageNet classification benchmark, images are resized to 224x224 for both training and evaluation stages (Touvron et al., 2021a; Liu et al., 2022; Wang et al., 2024c). This input size has actually compressed the visual information, as the average ImageNet image size is approxi- mately 490 pixels. Within this range, scaling the input size is generally more effective than scaling the patch size. For example, Adventurer-B/16 with a 448x448 input out- performs Adventurer-B -/8 with a 224x224 input by 0.4%, despite having similar parameter counts and FLOPs.
Scaling in both dimensions. In Table 4, we provide more ImageNet classification results with Adventurer models, where we scale them in both model size (parameter count) and patch size. As shown, the two scaling dimensions work synergistically when offering performance gains, with the highest accuracies consistently achieved by either the largest models or the smallest patch sizes. As analyzed earlier, the function of patch size scaling lies in reducing the spatial compression rate and thereby enabling the extraction of richer information from the data itself. Intuitively, this ef- fect does not conflict with scaling up the model size, where performance gains mainly stem from enhanced fitting capa- bilities provided by increased parameters.
The results in this experiment suggest that, given sufficient computing resources, we can easily transfer past advance- ments in parameter scaling to patchification scaling. In other words, patchification scaling can serve as a complement to model size scaling- with the current data volume, we have
However, beyond this input size, further increasing the input dimensions provides diminishing returns in performance gains. If we fix the sequence sength-scaling up the in-
7

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Patch size
Memory (per image)
GPU hours
Model size
Patch
Length
16x
16
8x8
4x4
2x2
1x1
DeiT-Base
Adv-Base
with 128x 128 resolution inputs:
16
196
62MB
0.36
0.45
Tiny
72.6
78.5
80.4
80.6
80.7
784
252MB
1.86
1.76
3,136
1,024MBB
9.79
6.86
Small
77.6
80.5
80.9
81.2
81.4
12,544
4,057MB
80.06
27.45
Base
79.0
81.5
81.8
82.2
82.4
50,176
16,1
967.99
115.08
Large
79.8
82.2
82.6
82.9
83.1
with 224x 224 resolution imput:
Table 6. Computational overhead for training a DeiT-Base and Adventurer Baatturer-Baeeeaa at 224x224 resolution inputs and different patch sizes. Memory usage is calculated based on the per- per-image con- sumption in ViT. GPU hours (for each ImageNet epoch) are esti- mated on a single A100 GPU. The models are trained at Float16 precision with FlashAttention (Dao et al., 2022) applied in VIT. The detailed evaluation protocol can be found in Appendix.
Tiny 78.2
80.9
81.3
81.7
81.9
Small 81.8
83.0
83.5
83.7
83.8
Base
82.6
83.9
84.3
84.5
84.6
Table 4. Scaling both patch and model sizes. The gains from patch size scaling and model size scaling are not conflicting; com- bining both can lead to further performance improvements. The numbers denote ImageNet accuracy (%) with Adventurer models. We associate the results with different shades for clear observation.
that extends the input sequence interpolating on existing to- kens. Specifically, in this comparison, we retain the original large patch size (16x 16) but perform spatial interpolation on the tokens produced by the patch embedding, by which we extend the input sequence without introducing any new information. As shown in Table 5, this approach does not bring substantial improvements to the model's performance (see column "By extending"). In contrast to the significant gains achieved through patchification scaling (e.g., 4.2% ac- curacy on ImageNet), this ablation study effectively demon- strates that the benefits of our approach primarily stem from unlocking the visual information compressed by large patch sizes, enabling the model to focus more detailed visual features, while simply scaling the sequence length itself has only a minimal impact on performance.
Model
Length
By extending
By scaling
64
78.1
78.1
DeiT-Base,
256
78.2 (+0.1)
81.0 (+2.9)
128x 128 input
1,024
78.2 (+0.1)
82.3 (44.2)
196
82.6
82.6
Adventurer-Ban Base,
784
82.7 (+0.1)
83.9 (+1.3)
224x224 input
3,136
82.8 (+0.2)
84.3 (+1.7)
12,544
82.8 (+0.2)
84.5 (+1.9)
Table 5. Ablation of sequence length. Extending the sequence length alone does not yield significant improvements (column "by extending"),, whereas reducing patch size and lowering informa- tion compression rate is the primary source of performance gains (column "by scaling"). Performance is measured by ImageNet- k accuracy (%), with longest sequences highlighted in blue.
4.4. Discussions
already observed the limitations of parameter scaling in vi- sion models (Zhai et al., 2022: Dehghani et al., 2023), while it is promising to see more future breakthroughs in visual encoding with the help of this new scaling dimension.
We summarize the computational requirements involved in the patchification scaling experiments in Table 6. As shown, the super-long visual token sequences associated with small patch sizes impose a significant hardware overhead on ViT architectures. This overhead was indeed a major challenge around five years ago, when V100 GPUs with 16/32GB memory remained the mainstream hardware for Al training. However, with rapid advancements in hardware develop- ment, efficient parallel computing mechanisms, as well as low-complexity visual architectures, the idea of "learning from pixels" has become increasingly feasible.
Impact of sequence length. Intuitively, scaling down the patch size has two direct effects. First, smaller patch sizes allow the model to receive richer, more fine-grained input information, which can greatly benefit its inference abilities. Second, reducing the patch size directly extends the token sequence, and for token mixers like self-attention or Mamba, longer sequences inherently expand the model's representa- tional space, enhancing its capabilities in feature processing. Both factors can potentially have a significant impact. We seek to demonstrate that the performance improvement from reducing the patch size primarily arises from the informa- tion gain due to a lower compression rate, rather than from the enhanced representational capacity associated with an extended sequence length.
In the experiments, we have demonstrated many key ben- efits of patchification scaling, such as direct performance improvements, reduced dependence on decoders, and the ability to overcome many limitations of parameter scaling and input size scaling. These emerging properties suggest that, when computational resources allow, we should gradu- ally reduce or even abandon the spatial compression mecha- nisms in vision encoders, fully exploiting all the information inherently provided by the data. We hope this paper can pro-
We conduct a direct ablation study: in contrast to our patch size scaling approach, we set up an additional experiment
8

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
vide insights and inspiration for transition from the current patch-based compressive encoding paradigm pixel-based non-compressive visual foundation models.
learning. NeurIPS, 2022.
Bao, H., Dong, L.,, and Wei, F. BETT: BERT pre-training of image transformers. In ICLR, 2022.
5. Conclusion
Bello, I., Fedus, W., Du, X., Cubuk, E. D., Srinivas, A.,
Lin, T.-Y., Shlens, J., and Zoph, B. Revisiting resnets: Improved training and scaling strategies. NewrIPS, 2021.
In this work, we conduct extensive studies in reducing the spatial compression rate patchification layers and discover new scaling dimension for visual encoding, which we term Patchification Scaling Laws. The new scaling laws sug- gest that, with more computational resources invested, lever- aging smaller patch sizes consistently leads to improved predictive performance. This conclusion is broadly applica- ble across various vision tasks, different input resolutions, and diverse model architectures. As a by-product, we also identify an interesting emerging property of patchification scaling: when the encoder patch size becomes sufficiently small, the benefits provided by task-specific decoder heads diminish significantly. We hope the discoveries in this paper can provide solid theoretical foundation for the future pixel learning paradigm and the development of non non-compressivy visual foundation models.
Caesar, H., Uijlings, J., and Ferrari, V. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
Cai, Z. and Vasconcelos, N. Cascade r-cnn: High quality object detection and instance segmentation. IEEE trans- actions pattern analysis and machine intelligence, 2019.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Chen, C.-F. R., Fan, Q. and Panda, R. Crossvit: Cross- attention multi-scale vision transformer for image classi- fication. In ICCV, 2021a.
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and
Acknowledgments
Yuille, A. L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE TPAMI, 2017.
This work is supported by ONR: N00014-21-1-2690 and
National Eye Institute (NEI) with Award ID: RO1EY037193.
We would like to thank TPU Research Cloud (TRC) pro- gram and Google Cloud Research Credits program for par- tially supporting our computing needs.
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F,, and Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
Impact Statement
Chen, X., Xie, S.,, and He, K. An empirical study of training self-supervised vision transformers. In ICCV, 2021b.
This paper presents work whose goal is to advance the field of Machine Learning. Our experiments involved approx- imately 50,000 A100 GPU hours, which is considered a modest level of resource consumption compared to large- scale vision or language model research. While there are many potential societal consequences of our work, none are significant enough to warrant specific highlighting in this context. We believe the ethical impacts and societal impli- cations are well-aligned with the advancement of machine learning technology.
Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In ICML, 2024.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficien exact attention with io-awareness. NeurIPS, 2022.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers to 22 billion parameters. In ICML, 2023.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874. 2023.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, In ICLR, 2021.
Alayrac, J.-B., Donahue, J., Lue, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few-shot

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00772, 2023.
Kolesnikov, A., Beyer, L.,, Zhai, X., Puigcerver, J., Yung,
J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and Ré, C. Combining recurrent, convolutional, and continuous-timg models with linear state space layers. In NeurIPS, 2021.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012.
Kwon, W., Li, Z., Zhuang. S., Sheng, Y., Zheng, L., Yu, H., Gonzalez. J., Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sympo- sium on Operating Systems Principles. 2023.
Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022.
Hatamizadeh, A. and Kautz, Mambavision: A hybrid mamba-traassoorer vision backbone. arXiv preprint arXiv:2407.08008, 2024.
LeCun, Y., Bottou, L.,, Bengio, Y., and Haffner, Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 1998.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Li, K., Li, X., Wang, Y., He. Y., Wang, Y., Wang. L., and Qiao, Y. Videomamba: State space model for efficient video understanding. arXiv preprint 2024.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I. Safahi, E., Meirom, S., Belinkov, Y., Shalev- Shwartz, S., et al. Jamba: A hybrid transformer-mambe language model. arXiv preprint arXiv:2403. 2024.
Huang, G., Liu, Z., Van Der Maaten, L.., and Weinberger, K. Q. Densely connected convolutional networks. In CVPR, 2017.
Huang. T., Pei, X.. You, S., Wang, F., Qian, C., and Xu, C. Localmamba: Visual state space model with windowed selective scan. arXiv preprint MeXiv:2403.09338 2024.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304.0844 2023.
Liu, Y.. Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. Vmamba: Visual state space model. arXiv preprint arXiv: 24Xv:2401..0116. 2024.
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V.. Wu, Y., et al. Gpipe: Efficient training giant neural networks using pipeline parallelism. NeurlPS, 2019.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z.,, Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Jia, C., Yang, Y., Xia, Y., Chen, Y-T.,, Parekh, Z., Pham, H., Le, Q., Sung, Y--H., Li, 7.,, and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML,
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In CVPR, 2022.
Loshchilov, and Hutter, F. Decoupled weight decay regu- larization. In JCLR, 2019.
Kalman, R. E. A new approach to linear filtering and pre- diction problems. 1960.
Nguyen, D.-K., Assran, M., Jain, U., Oswald, M. R., Snoek, C. G., and Chen, X. An image is worth more than 16x16 patches: Exploring transformers on individual pixels. arXiv preprint arXiv:2406.0944 09415, 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint ar wxXiv2201.08360 2020.
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung. M., Grella, M., GV, K. K., et al. Rwkv: Reinventing mnns for the transformer era. arXiv preprint arXiv:2305.13048 2023.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, 2020.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-.., et al. Segment anything. arXiv preprint arXiv:2304. 026444 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. JCML, 2021.
10

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Ren, S., Li, X., Tu, H., Wang, F,, Shu, F,, Zhang, L.,, Mei, J., Yang, L., Wang, P., Wang, H., et al. Autoregres- sive pretraining with mamba in vision. arXiv preprint arXiv:2406.07537, 2024a.
Wang, E, Wang, J., Ren, S., Wei, G., Mei, J., Shao, W., Zhou, Y., Yuille, A., and Xie, C. Mamba-r: Vision mamba also needs registers. arXiv preprint arXiv:2405. 148588 2024b.
Ren, S., Wang, Z., Zhu, H., Xiao, J., Yuille, A., and Xie, C. Rejuvenating i-gpt for scalable visual representation learning. In ICML, 2024b.
Wang, F., Yang, T., Yu, Y., Ren, S., Wei, G., Wang, A., Shao, W., Zhou, Y., Yuille, A., and Xie, C. Causal image mod- eling for efficient visual understanding. arXiv preprint arXiv:2410.077 10.09999 2024c.
Ren, S., Yu, Y., Ruiz, N., Wang, F., Yuille, A., and Xie, C. M-var: Decoupled scale-wise autoregressive model- ing for high-quality image generation. arXiv preprint arXiv:2411. 10433, 2024c.
Wei, G. and Chellappa, R. Vit-linearizer: Distilling quadratic knowledge into linear-time vision models. arXiv preprint arXiv:2504.00033 2025.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Wightman, R., Touvron, H., and Jégou, H. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2 10.00476, 2021.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Sun, C., Shrivastava, A., Singh, S., and Gupta, A. Revisiting unreasonable effecttivnness of data in deep learning era. In ICCV, 2017.
Xie, Q., Luong, M.-T., Hovy, E., and Le, Q. V. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
Tan, M. and Le, Q. Efficientnet: for convolutional neural networks. In ICML, 2019.
Rethinking model scaling
Yang, C.,, Chen, 7.,, Espinosa, M., Fricsson, I.., Wang, Z.,, Liu, J., and Crowley, E. Plainmamba: Improving non- hierarchical mamba in visual recognition. arXiv preprint arXiv: 2403xx2403..1 17695, 2024.
Tan, M. and Le, Q. Efficientnetv2: Smaller models and faster training. In ICML, 2021.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11880 2023.
Yu, J., Wang, Z., Vasudevan, V., Yeung. L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models. arXiv preprint ar 20111220.0000.. 2022a.
Touvron, H., Cord, M., Douze, M., Massa, F,, Sablayrolles, A., and Jégou, H. Training data-efficient image trans- formers & distillation through attention. In /CML, 2021a.
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. Metaformer actually what you need for vision. In CVPR, 2022b.
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., and Jégou, H. Going deeper with image transformers. In ICCV, 2021b.
Yuan, L., Chen, Y., Wang. T., Yu, W., Shi, Y., Jiang, Z.-H., Tay, F. E., Feng, J., and Yan, S. Tokens-to-token vit: Training vision transformers from scratch imagenet. In ICCV, 2021.
Touvron, H., Cord, M., El-Nouby, A., Verbeek, J., and Jegou, H. Three things everyone should know about vision transformers. ECCV, 2022a.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In CVPR, 2022.
Touvron, H., Cord, M., and Jégou, H. Deit ii:: Revenge of the vit. In ECCV, 2022b.
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-.,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023.
Zhou, D.,. Kang, B., Jin, X., Yang. L., Lian, X., Jiang. Z., Hou, Q., and Feng, Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 11886 2021.
Wang, F., Mei, J., and Yuille, A. Sclip: Rethinking self- attention for dense vision- lsion-Language inference. In ECCV, 2024a.
Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., and Wang, X. Vision mamba: Efficient visual representation learning with bidirectional state space model. In ICMI., 2024.

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Appendix
A. More Technical Details
The detailed configuration of the models used in this paper are claborated in Table For ViTs, We basically follow the configurations in the DeiT series models (Touvron et al., 2021a; 2022b), but change the default patch size of DeiT-Huge to 16x 16, the same as the other DeiT models. For Adventurer, we scale up the model a huge size following the same rule of DeiT; set its embedding dimension to 1,280, keeping its original MLP ratio and employ 32 blocks in total.
Model
Embedding dimension
MLP dimension
Blocks
Parameters
DeiT-Tiny (Touvron et al., 2021a)
192
768
12
5M
DeiT-Small (Touvron et al., 2021a)
384
1,536
12
22M
DeiT-Base (Touvron et al., 2021a)
768
3,072
12
86M
DeiT-Large (Touvron al., 2022h)
1,024
4,096
24
304M
DeiT--Huge (Touvron et al., 2022b)
1,280
5,120
32
631M
Adventurer- (Wang et al., 2024c)
256
640
12
12M
Saventurer-Small (Wang et al., 2024c)
512
1,280
12
44M
Adventurer- Base (Wang et al., 2024c)
768
1,920
12
99M
Adventurer- Large (Wang et al., 2024c)
1,024
2,560
24
346M
Adventurer- Huge (Wang et al., 2024c)
1,280
3,200
32
759M
Table 7. Model configurations. All models have 16x 16 patch size by default.
Protocols of estimating memory and GPU hours. In Table present estimation of the GPU memory and training hours required for DeiT and Adventurer. Here give more details of how they evaluated. We calculate the memory consumption by each image. That means, the reported numbers have excluded the memory used for storing the model, optimizer, and other hyper-parameters. The actual memory demand increases linearly with batch size. To evaluate the GPU hours required for training, we set a total batch size of 1,024 and use the minimum number of nodes necessary for training (depends on the total memory demand). Each node is equipped with 8 A A100/80GB GPUs. The estimated training hours are then multiplied by the total number of GPUs used to ensure that the reported numbers are normalized.
Config
Tiny/Small/Bas
Large/Huge
optimizer
AdamW
base learning rate
5c-4
2c-4
weight decay
0.05
0.3
epochs
300
200
optimizer betas
0.9. 0.999
0.9, 0.95
batch size
1024
4096
warmup epochs
5
20
stochastic depth (drop path)
0.1
0.2
layer-wis, decay
label smoothing
random erasing
Rand Augmentation
repeated augmentation
ThrrenAumeenttionn
Table Recipe of the pretraining stage, for 64x64 or 128x 128 pixel inputs.
Training recipes. this work, we train DeiT-Tiny, Small, and Base with the official repository (Touvron et al., 2021a) and recipe. For Leig-Large and Huge, there is not training configuration in the original DeiT paper so we follow the supervised training pipeline reported in (He al., 2022). Note that the Pixel Transformer (Nguyen et al., 2024) which conduct pixel tokenization experiments with low resolution images (28 x 28) employs the same training recipe.
For Adventurer, we mostly follow its original multi-stage strategy (Wang et al., 2024c) to train our models. Specifically, for 64 resolution inputs, we simply perform the pretraining stage (shown in Table 8) for 300 epochs for all model sizes. For 128 x 128 resolution inputs, we additionally perform a finetuning stage (shown in Table 9) for enhanced results. For the standard 224x 224 resolution inputs, we follow the practice of Mamba-Reg (Wang al., 2024b) and Adventurer (Wang
12

---

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
al., 2024c) load the pretrained model x28x 128, performing intermediate training stage (shown in Table 10) for 100 epochs and then finetuning stage for 20 epochs. We highlight that this multi-stage training strategy is highly efficient for our experiments as we can fully exploit the models pretrained lower resolutions. For example, we only need train the x24-224-im models for 120 epochs since we can load the weights pretrained at 128x 128 resolution inputs.
Notably, for both DeiT' and Adventurer, there is no need to adjust training recipes for different patch sizes, which we consider be one of the flexible and practical advantages of patchification scaling.
Config
Small/Base
Large
optimizer
AdamW
base learning rate
le-5
2e-5
weight decay
0.1
0.1
epochs
20
50
optimizer betas batch size
0.9, 0.999
0.9, 0.95
512
512
warmup epochs
5
5
stochastic depth (drop path)
0.4 (S), 0.6 (B)
0.6
layer-wise Ir decay
x
0.95
label smoothing
0.1
random erasing
X
Rand Augmentation
rand-m9-mmttd0---mg
repeated augmentation
X
Three Augrentngttnttooonn
Table 9.
Recipe of the finetuning stage, for 128x 128
224x224 pixel inputs.
Config
Small/Base
Large
optimizer
AdamW
base learning rate weight decay epochs
5e-4
8e-4
0.05
0.3
100
optimizer betas batch size
0.9, 0.999
0.9, 0.95
1024
4096
warmup epochs
5
20
stochastic depth (drop path)
0.2 (S), 0.4 (B)
0.4
layer-wise Ir decay
x
0.9
label smoothing
random erasing
Rand Augmentation
X
repeated augmentation
Turrenuugmnttatioon
Table 10. Recipe the intermediate training stage, 224x224 pixel inputs.
13


Reviews:
[{"review_id": "2EZQepOh4T", "reviewer": "Reviewer_DwN7", "summary": "The paper addressed the scalability issue of patch sizes in vision transformer (ViT), which is a widely-adopted backbone in vision-related task. In past research a moderate parameter of patch size is used by default when ViT is chosen as the backbone. This study empirically investigate the effect of varying patch size (e.g., reducing it down to even 1*1) and find a scaling-law like rule in a variety of computer vision tasks (recognition, detection, segmentation).\n\n## update after rebuttal\n\nThanks for the response and additional experiments to clarify my concern on SwinTransformer. I have also checked the comments from other reviewers and stick to my original recommendation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s constructive feedback and detailed suggestions. The detailed response to your questions/concerns are presented below:\n\n**Q1: *Additional experiments and discussion about sparse and spatially local attention networks.***\n\n**A1:** Thank you for your constructive suggestion. We followed your feedback and conducted additional experiments on Swin Transformer, with the summarized results provided below. Specifically, we use Swin-T as the base model, which has an initial patch size of 4×4 and hierarchically downsamples the feature map by a final factor of 8 in both width and height during intermediate stages. We scaled it up by reducing the initial patch size to 2×2 and 1×1, while keeping the default downsampling rates unchanged. This scaling strategy closely mirrors the approach we used when handling standard ViT models. As shown in the table, our conclusion also holds for Swin Transformer—smaller patch sizes consistently lead to lower test loss and improved accuracy. In the revision, we will include larger local-attention models such as Swin-S and Swin-B to further support our findings.\n\n| Initial patch size | Inference time | Test loss | Accuracy |\n|--------------------|----------:|----------:|---------:|\n| 4x4                |       1x  |     0.806 |    81.3  |\n| 2x2                |     3.9x  |     0.734 |    82.0  |\n| 1x1                |    15.1x  |     0.697 |    82.5  |\n\n**Q2: *Limitations of input size scaling.***\n\n**A2**: We appreciate your insightful comments on this matter. We would like to explain the motivation behind this set of experiments: in fact, input size scaling and patch size scaling are largely equivalent—they both proportionally change the feature processing granularity and the sequence length of the model. Through these experiments, our goal is simply to show that patch size is a more suitable choice for a standardized direction of scaling, since it does not affect input storage, and input size scaling does not necessarily offer performance advantages over patch size scaling beyond a certain range.\n\nSpecifically, we know that increasing the input size within a certain range tends to improve model performance. For example, on ImageNet, using the same model such as DeiT-Base/Patch16, an input resolution of 384×384 achieves noticeably better accuracy compared to 224×224 (e.g., 83.1 vs. 81.8). We believe this improvement basically comes from two sources: 1)  the direct benefit of increased computation from scaling up the input; and 2) a higher input resolution reduces the distortion caused by resizing images during preprocessing. To ablate these two effects, we keep a fixed ratio between input size and patch size, which helps maintain a roughly constant computational cost and figure out the impact of the second term. Then our experiment demonstrates that this benefit tends to vanish once the input resolution exceeds the average original image size in the dataset.\n\nWe will carefully rephrase the description and analysis of this experiment in the revised version based on your suggestions and comments.", "claims_and_evidence": "All are fine except for possibly problematic setting in the section \"limitations of input size scaling\" and Figure 4. The comparisons may not be on some fair basis.", "methods_and_evaluation_criteria": "The evaluations are conducted on several most popular benchmarks (e.g., ImageNet, COCO etc.) and with standard metrics (e.g., average precision for object detection and instance segmentation). I have not concern about the evaluations.", "theoretical_claims": "There is no theoretic proof or claim. This is a work fully based on empirical evaluations on data.", "experimental_designs_or_analyses": "In fact most of pages in the paper were devoted to experiments, including both the reported performance scores on the chosen benchmarks and a series of ablation studies to reveal the effect of key factors (e.g., scaling of patch or parameter). I carefully check the experimental settings. They seem following previous practice, with sufficient details presented.", "relation_to_broader_scientific_literature": "The work reveals the importance of choosing proper patch size in using ViT. Given the popularity of ViT in a large number of domains (computer vision, medical image analysis, weather forecasting, earthquake prediction etc.), the insight reported here may be valuable in improving many models nowadays used in these domains.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Overall I regard this is a good work with clear insight and experimental designs. The key idea (varying patch size to investigate whether and kind of scaling law holds) seem empirically validated by a series of experimental results in this work. As far as I know such a study is still missing in the literature, thus I currently lean to recommend to accept this work.\n\nHowever, there is still some space for improving the work. One key difference between computer vision and other general tasks lies in the sparsity of attentions in ViT. One example is the swinTransformer as proposed by Microsoft research. The work reveals the scalability of increasing visual token-based sequences. It is not clearly discussed whether the claims still hold when it is combined with sparse and spatially local attentions. I would suggest the authors to include additional experiments and discussion. \n\nThe section \"limitations of input size scaling\" is not reasonable. There are some key technical details missing in the main text, particularly the specific way of increasing the number of parameters in patchification. Fixing the ratio of image-size / patch-size makes patches from different images not granularity-equal. According to my experiences, this will complicated the training of the model since it has to tackle more complex input during generalization. All above make the claim not fully convincing.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "Please read me comments on the weakness of this work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "There is no ethical issue found in the submission."}}, {"review_id": "gTCqbpdL9g", "reviewer": "Reviewer_vQ14", "summary": "The paper evaluates the performance (test loss) of vision models (standard ViT and Adventurer) against different patch sizes. The paper's findings are that with reduced patch size, the performance of the networks increases and in the extreme case of 1x1 patch size, the segmentation tasks do not need a decoder head. The paper evaluates the patchification scaling on the standard ImageNet-\n1k classification, ADE20k semantic segmentation, COCO object detection and instance segmentation benchmarks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and feedback during the review stage. We‘d like to respectfully point out that your review contains several factual misunderstandings of our paper. These points were clearly presented in the main text, and we'd like to first clarify them below:\n\n1. ***Absence of 1x1 patch size for ViT:*** This is factually incorrect. We do implement pixel patchification in ViTs, albeit at lower input resolutions. The results are clearly shown in the first row of Tab.1.\n\n2. ***“The overall method… showing results on vision and timeseries tasks.”*** Our work is solely focused on vision tasks, and there is no mention or evaluation on time series data anywhere in the paper.\n\n3. ***“without ever indulging into the GLOPs, test and training time compute”*** This is completely inaccurate. We provide thorough analysis of FLOPs (Fig.3), memory usage and training time (Tab.6), and discuss computational costs of patch scaling in multiple places (e.g., Lines 324–329, 436l–408r).\n\n4. ***\"no ablation that discusses the patchification vs. embedding dimension size trade-off.\"*** We compared patch size and parameter size scaling in the first experiment of the ablation study in Section 4.3. The embedding dimension is highly equivalent to the parameter size. E.g., in the standard scaling of ViT models from ViT-T to ViT-B, the only change lies in the embedding dimension; from ViT-B to larger models such as ViT-L, it still involves increased embedding dim while depth is also changed.\n\n**Q1: *Compare to 32 patch size***\n\nOur study focuses on exploring scaling laws of patchification, which means we need to examine both scaling up and scaling down. We aim to collect a wide-range performance curve to demonstrate that the gains are smooth and consistent across different patch sizes. The 32 patch size results are reported to provide a more comprehensive view; we did not hide the results for the mainstream 16 patch size—they are shown in the same table and scaling from it leads to a +2.2 box AP.\n\n**Q2: *Generalization capability***\n\nWe demonstrated that patch scaling consistently leads to improved predictive performance, which is generalizable for different models, tasks, and input resolutions. Below we include more results of generalization capability against out-of-domain data, where our conclusions hold true for ImageNet variant test samples as well.\n\n|Patch|IN-v2|IN-R|IN-A|IN-S|\n|-|-|-|-|-|\n|16|71.7|87.5|31.4|36.8|\n|8|72.9|87.9|36.6|38.3|\n|1|74.3|88.8|41.6|39.5|\n\n**Q3: *Patchify in NLP***\n\nWord-level tokenization has indeed been prominent in NLP. However, we believe this is only a phase-based conclusion but no one has ever proven that it is the only correct approach. In contrast, recent studies(e.g. Byte Latent Transformer) have shown that byte-level LLMs can outperform word-level models and offer better scaling potential. Thus, whether patchify is necessary remains an open question that deserves further investigation. Moreover, directly transferring conclusions from NLP to CV is not always appropriate. Images and text differ significantly in terms of information density; a pixel cannot be naively equated to a character in text. These differences call for careful study, and our work aims to contribute to this ongoing exploration.\n\n**Q4: *Quantify the information loss from the information theory perspective.***\n\nThis paper is not a theory-oriented study, but rather an empirical investigation into scaling laws. We believe both empirical and theoretical perspectives are equally important, and our experimental results can serve as a solid foundation for future theoretical work. In fact, our evaluations already offer explanations from an information point of view, in which compression can be assessed via entropy differences. Fig.1 explicitly shows how cross-entropy loss changes during patch scaling, which is actually measuring the KL divergence between ground-truth distribution and encoded distribution. This typically serves as a direct proxy for compression in information theories.\n\n**Q5: *Results marginally better; underperform SOTA models***\n\nOur focus and contribution is figuring out how patch size affects the performance of vision models, rather than chasing SOTA on open-ended architectures. Whether a performance gain is considered marginal is subjective and varies by context. For fixed architecture, improving accuracy from 82.6 to 84.6 is a significant achievement. It is widely acknowledged that hierarchical models (e.g., Swin, SpatialMamba) are more capable of fitting ImageNet-scale datasets compared to plain models, but plain models have wider applications in multimodal tasks. We focus on patch scaling within plain architectures so the performance gap with SOTA models is expected. In our response to R#DwN7 we also show that patchification scaling laws hold true for Swin, so we believe that future follow-up work applying patch size scaling to SOTA architectures could reasonably expect further performance improvements.", "claims_and_evidence": "1. The paper examines how compressive encoding affects visual representations and whether patch size can be a new scaling dimension for modern visual architectures.\n    It provides evidence in the form of empirical results obtained by changing the patch size down to 1x1 (pixel level).\n2. The paper claims tokenization/patchification is another dimension for scaling law. The evidence is provided by reducing the patch and getting marginal gains.", "methods_and_evaluation_criteria": "This study should have been done very carefully and meticulously because there are many aspects and variables that need to be carefully analyzed and ablated. \nThe overall method simply involves getting two networks and reducing the patch size down to 1x1 (not for pure transformers) and then showing results on vision and timeseries tasks. \nThe issue is that the method is counter to common wisdom of balancing the compute and gains trade off. It does not make sense to increase compute just because more hardware is available without getting real gains from it. The method showcases its results on metrics without ever indulging into the GLOPs, test and training time compute, throughput and other aspects that come with reducing the patch size of the input. \n\nThe paper raises the slogan of “a pixel is worth a token” (line 233) but does not practice it in ViT because the “a pixel is worth a token” requires a lot of resources.\n\nLine 312-313L: The paper provides misleading sense  of great results but the comparison is unfair since the numbers are compared to 32x32 patch size instead of the mainstream/common 16x16 or 14x14 patch size. The wosre results on uncommon 32x32 patch size makes it looks like the results are good (which they arent)", "theoretical_claims": "This paper actually could ground itself from a mathematical point of view and framework when talking about compression and information contained in the embedding vectors of a given size. The lack of such framework and relying entirely on empirical results is not much convincing.", "experimental_designs_or_analyses": "The experimental design and ablations in the paper are either flawed or entirely absent. Please refer to the other sections for the stated problems.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "It's surprising that \"On the relationship between self-attention and convolutional layers\" by JB Cordonnier et. al., which was the basis of ViTs and started with the patch size of 2x2 (which was later changed to 16x16 in ViT paper) has not been discussed at all. \nComrpession and compression related topics (e.g. information theory) are entirely missing form the paper.", "other_strengths_and_weaknesses": "Weaknesses:\n1. The overall idea of scaling is to increase the generalization capability of the models. It is not supported by the evidence shown in the paper that patchification leads to better generalization. Moreover, smaller patchification leads to more computation without having significant benefits. \n2. Analogous patchification in NLP would be the character-level patchification which has been shown not to be optimal. It is not clear why and why not similar will be true for vision models from this work. \n3. There are claims and arguments (discussed above) that are not supported by any experiments or ablation studies. \n4. There is no effort to introduce or quantify the information loss lost by compressive encoding from the information theory perspective.\n5. There is no ablation that discusses the patchification vs. embedding dimension size trade-off. \n6. The results themselves are only marginally better and yet underperforming to SOTA methods with a greater patch size (Spatial Mamba for example has better results).", "other_comments_or_suggestions": "Missing footer. Not using the ICML template or the template is altered. \n\nLine 250-257: Sounds like the paper is claiming the credit for the method \"Adventurer\" only for increasing the input token size by reducing the patch size. Increasing the input size shouldn't be posed as an achievement.  \n\nLine 351R: The same condition does not hold for the paper's own method.", "questions_for_authors": "Line 15-16R (Right paragraph): \"We  argue  that  this  operation  often  incurs irreversible information loss to visual inputs.\" There is no reference, ablation or supporting evidence provided for this argument. \n\nLine 105: Why exactly computation needs to be scaled up? \n\nLine 114-115: Single pixel patch (or even few pixels patch) expanded to embedding dimension is not even compressive paradigm anymore but \"expansive\" regime so to say. There is no discussion on compression ratio. \n\nLine 324R: \"patch size scaling not only exhibits a better computation-accuracy tradeoff\": Where is the trade-off provided?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UOeZzMwTcx", "reviewer": "Reviewer_KY13", "summary": "The authors perform a study regarding the size of patches used in modern vision transformers or state space models. Utilizing the Adventurer state space model, the authors are able to experiment with resulting sequence lengths of up to 50,176 tokens.\nThe work arrives at the conclusion that there exists a scaling law that holds until each input pixel is represented by a dedicated patch/token. Smaller patches naturally align well with dense prediction tasks like semantic segmentation, where the need for a decoder diminishes with the use of smaller patches.\n\n## update after rebuttal\n\nI thank the authors for their efforts in answering my questions and providing additional experiments. Still, I am of the opinion that the introduction would benefit from references to other works, that already experimented and observed the benefit of smaller patches.\nFurthermore, the authors explain the improvement by a reduction of the information loss in the patchification step. Which is very reasonable and certainly true for smaller models. The insightful experiment presented in Figure 4. shows that the scaling potential is bounded by the original image resolution. I think a similar experiment with respect to the hidden dimension of the model would be of high relevance.  Thus, I still think that the paper would greatly benefit from an experiment that investigates the scaling potential in a regime where no compression is needed to create patches.  \n\nThe experiments provided during the rebuttal support the information loss argument, as the smaller patch versions consistently show a much smaller reconstruction loss. Yet on a second glance, the results are not totally plausible, as they show no impact of the bottleneck dimension of the autoencoder whatsoever.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive review, insightful feedback, and recognition of the empirical contribution of this work. The detailed response is summarized below.\n\n**Q1: *No technical or theoretical contribution.***\n\nThanks for the comment. As you noted, this is indeed an experiment-driven study, and our focus is on delivering empirical results to demonstrate a new scaling law. We respectfully ask the reviewers to take into consideration that all studies on scaling laws are inherently grounded in empirical observations. For large vision or language models, it is extremely difficult—if not impossible—to theoretically derive precise upper or lower bounds on their fitting capacity. Therefore, empirical evidence remains the most practical and informative approach for uncovering such trends.\n\nIf we consider the initial scaling law proposed by Kaplan et al. (2020) as a theoretical foundation, this work will be a meaningful extension of their theories in vision. Specifically, in the initial study, they show that in language modeling tasks, when the model size (parameter count $M$) and the amount of training data (token count $N$) increase, the perplexity (or loss) on the validation set tends to follow a near power-law relationship:\n\n$ L(N,M)\\approx\\alpha N^{-\\beta}+\\gamma M^{-\\delta}+\\text{(smaller interaction terms)}$\n\nwhere $L$ represents the validation loss (e.g., cross-entropy), and \n$\\alpha,\\beta,\\gamma,\\delta$ are constants fitted from large-scale experiments. In this work, we not only reproduced similar scaling trends on vision tasks, but also validated the second component of the scaling law (the token count) in the vision domain. Specifically, we show that increasing token count in vision can be achieved by reducing patch size rather than solely increasing dataset size, which we believe is an important theoretical extension of the original scaling law foundation. We sincerely appreciate your suggestion on this point and will include the corresponding discussion in the revised version.\n\n**Q2: *The benefit of smaller patches has been known and has been exploited for quite some time***\n\nThanks. We agree that many existing studies have observed that using smaller patch sizes within a certain range can improve prediction performance. However, prior to our work, these observations remained scattered and lacked a unified theoretical or empirical framework. In contrast, our study elevates patchification scaling from isolated findings to a law-level conclusion.\n\nWe believe this distinction is substantial—not only in terms of the conclusions themselves, but also in their implications for guiding future progress in the community. For example, before the NLP scaling laws were formally introduced by Kaplan et al., practitioners had already noticed from experience that \"larger models and more data tend to yield lower loss.\" Yet such heuristic insights were insufficient to offer reliable guidance or theoretical grounding for large-scale language model development.\n\nThe formulation of scaling laws provided the community with a much clearer signal: the scaling curve had no evident upper bound, or at least we were far from reaching it. This encouraged researchers to confidently invest significant resources in scaling up language models, ultimately contributing to the breakthrough success of models like the GPT series.\n\nIn the same spirit, we would like to highlight the contribution of our work. We aim to offer a long-term, principled perspective on vision model development. The value of patchification scaling laws lies in showing that patch size is a reliable scaling dimension—even at today’s typical input scales, shrinking the patch size down to 1×1 continues to yield noticeable gains. This suggests that for those seeking to push the limits of model performance, reducing patch size remains a viable and promising direction.\n\n**Q3: *Concerning the argument that the scaling law can be attributed to the information loss in the patch creation step . Why should this hold in the case where the hidden dimensions are large enough to simply stack the corresponding pixels?***\n\nWhen the hidden dimension exceeds the total number of pixels within a patch, the patch embedding may theoretically support a lossless projection. However, in practice, the high-dimensional hidden features are optimized to represent the holistic semantic of the whole patch, rather than preserving the pixel-level representations. This is because the token mixing process (e.g., self-attention) is performed on patch-level features. That says, the different segments within a feature vector undergo the same operation in token mixing. Moreover, the hidden dimension cannot be flexibly scaled up since it quadratically increases parameter count for all linear projection layers and can easily lead to training collapse (e.g., also observed in Dehghani, et al., 2023).\n\n**Q4: *Typo fig 3b.***\n\nThanks. We will fix it in the revision.", "claims_and_evidence": "The work claims that there exists a scaling law when it comes to the size of patch tokens used in transformer or state space models for image processing. The authors provide empirical results that clearly support the claim.", "methods_and_evaluation_criteria": "Supervised training on ImageNet is a standard evaluation that allows for a comparison to a wide range of models. Semantic segmentation on ADE20k is common as well. The authors clearly state that they focus on decoder free evaluation as it aligns with the benefits of smaller patches.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes. Experiments follow recipes proposed in the respective publications (e.g. DeiT and Adventurer). Adaptations are listed in the Appendix.", "relation_to_broader_scientific_literature": "The contribution of this paper adds another example to the set of scaling laws that have shown empirically that an exponential increase in resources leads to a linear performance improvement.", "essential_references_not_discussed": "Not essential, but there are quite few works in self-supervised learning that already utilize smaller patches to improve performance. And especially for smaller models that is a widely used practice. Already the 14x14 ViT Huge model in the ViT paper suggests that the potential benefit of smaller patches has been known for quite some time.\nE.g. [1-3], especially in 3 the authors already work with small 4x4 patches to achieve an advantage in the resulting per-parameter comparison to other models\n\n[1] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n[2] Zhou, Jinghao, et al. \"Image BERT Pre-training with Online Tokenizer.\" International Conference on Learning Representations.\n[3] Assran, Mahmoud, et al. \"Masked siamese networks for label-efficient learning.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.", "other_strengths_and_weaknesses": "Strengths:\n- the work provides empirical evidence that smaller and smaller patches steadily improve the performance of visual backbones and follow a scaling law.\n- both for transformer and state space models. \n- the experimental setup is well documented. \n- Additional studies that one patch per pixel is the optimal size and that more patches than pixels shows no benefits.\n\nWeakness:\n- no technical or theoretical contribution \n- the benefit of smaller patches has been known and has been exploited for quite some time", "other_comments_or_suggestions": "typo fig 3b. form - from", "questions_for_authors": "Concerning the argument that the scaling law can be attributed to the information loss in the patch creation step . Why should this hold in the case where the hidden dimensions are large enough to simply stack the corresponding pixels?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None."}}, {"review_id": "WoDLGuHm4f", "reviewer": "Reviewer_UjKW", "summary": "This paper aims to explore how the performance of vision transformers changes when the patch sizes are scaled up. The main contribution is the scaling law, which has not been presented in previous work, that as the patch sizes get smaller, the classification and segmentation performance can be lifted. The authors also show that the benefit of scaling up patch number is larger than scaling up the model size. This is really an interesting phenomenon.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their careful evaluation and thoughtful comments. Detailed responses can be found below.\n\n**Q1: *Experiments with other architectures (e.g., Swin Transformer, LV-ViT)***\n\nThank you for your insightful feedback. We would like to highlight that our conclusion also holds for pyramid networks such as Swin Transformer (see results in the response of Q1 for Reviewer DwN7) and stronger ViT baselines such as LV-ViT (see table below). To complete this experiment within a limited time frame, here we use 112×112 input images and employ LV-ViT-S as the backbone. Under this stronger baseline, we observe a performance trend similar to that in the standard ViTs, which demonstrates the architecture-wise robustness of patch size scaling. We will include the corresponding discussion in the revised version.\n\n| Patch | 16x16 | 8x8 | 4x4 | 2x2 |\n|-------|-------|-----|-----|-----|\n| Acc.  | 78.0  | 81.1| 82.5| 83.2|\n\n**Q2: *Comparison with early convolutions to the plain ViT model.***\n\nThank you for your suggestion of comparing with early convolutions. We summarize the results of patchification scaling with early convolution in the table below. As shown, when the patch size is at the standard 16×16 level, applying early convolution brings a noticeable performance gain. However, this benefit diminishes rapidly as the patch size decreases, and becomes negligible at 8×8. In simple terms, the basic idea behind early convolution is to decompose a single 16×16 kernel size and 16×16 stride convolution layer into a stack of smaller kernels (e.g., 3×3) with smaller strides (e.g., 1×1 or 2×2). This approach effectively addresses the training instability caused by the abrupt spatial resolution drop at the patchification layer in standard ViTs. However, as the patch size decreases, this issue becomes much less pronounced. We are encouraged to see that our paper shares some core insights with this line of research—namely, that large-kernel patchification in standard ViTs can limit model expressivity. The difference is that we take a more direct approach to reduce the spatial downsampling effect in patchification. When extended to the extreme case of pixel tokenization, this issue is already fundamentally resolved. We will include these results and discussions in the revised version.\n\n| Patch size | Original accuracy | Early conv. accuracy |\n|------------|-------------------|----------------------|\n| 16x16      | 82.6              | 83.1                 |\n| 8x8        | 83.9              | 83.9                 |\n\n**Q3: *Could these two types of scaling methods (patch size and parameter size) benefit from each other?***\n\nThank you for your insightful question. Yes, they do benefit each other and we have related results in Table 4 and discussions in line 364-376r. Table 4 shows a consistent upward trend from the top-left (small model and large patch size) to the bottom-right (large model and small patch size) corner. Additionally, as observed from both Table 1 and Table 4, reducing the patch size and increasing the input resolution can both have a positive impact, demonstrating the good potential of jointly employing these scaling dimensions.\n\n**Q4: *Application to Mamba-like models.***\n\nThank you for your feedback. We evaluate the patch size scaling performance with ViT and Adventurer models, where for Adventurer, we are actually using its Mamba-based setup so the application to Mamba models is already included. We will clarify this point in the revision.\n\n**Q5: *I was supposed that a novel method is presented to solve the computation overhead when using smaller patches. If such a method is proposed, I think the overall quality of this paper could be further improved.***\n\nWe appreciate your constructive comment on this point! In fact, we first identified a solution to address the computational challenges brought by small patch sizes before conducting the patchification scaling study: we chose to carry out the main experiments using the Adventurer model, whose computational cost scales linearly—as opposed to ViT’s quadratic scaling—with respect to sequence length. This linear complexity fundamentally resolves the computation bottleneck, which enables us to perform pixel tokenization experiments using modest computational resources (256 A100 GPUs). We also discuss the advantages of this linear architecture in Table 6: in the most demanding training setup with the longest sequences, Adventurer achieves an 8.4× speedup compared to ViT with FlashAttention. This substantial efficiency gain is what made our large-scale pixel tokenization experiments practically feasible.", "claims_and_evidence": "The authors have made extensive experiments to support the claims.", "methods_and_evaluation_criteria": "The criteria is widely-used.", "theoretical_claims": "These is no theoretical claims. Not applicable.", "experimental_designs_or_analyses": "The experimental designs are thoroughout. The authors conduct a series of experiments to demonstrate the claims and the designs are also meaningful.\n\n- One concern about the experimental design is that the authors only use two baseline models, i.e., DeiT and Adventurer. I would like to see whether the scaling law still exists when stronger baselines are used, i.e., [A] and [B].\n\n- In addition, the experimental designs are all based on the ViT-like plain architectures. This type of architectures has been widely used in vision tasks, but the pyramid architecture is also important. For example, Swin Transformer is such a type of architecture adopting a pyramid network. Have the authors conducted experiments on this type of architectures?\n\n[A] All tokens matter: Token labeling for training better vision transformers. NeurIPS, 2021.\n[B] Early convolutions help transformers see better. NeurIPS, 2021.", "relation_to_broader_scientific_literature": "The scaling law presented in this paper has not been discussed in previous work to my knowledge.", "essential_references_not_discussed": "See the 'Experimental Designs Or Analyses' section.", "other_strengths_and_weaknesses": "Strengths:\n\n- The presentation of this paper is good. The authors clearly explain the motivation of this paper and the method of this paper is easy to follow.\n- Though the method does not present an interesting method, the phenomenon observed by the authors are interesting.\n\nWeakness:\n\n- The paper [B] has shown that adding early convolutions to the plain ViT model can largely improve the model performance. However, the authors did not compare with this type of works, which aim to replace the patchification methods. It would be good to add some comparisons about this.\n\n- In Fig. 3, it is good to see the comparison between patch size scaling and parameter scaling. However, it is straightforward to come up with a new question: Could these two types of scaling methods benefit from each other? In other words, could the performance be further improved if fine-grained patchification is used when taking higher-resolution images as inputs?\n\n- The authors claim that the proposed approach is also applicable to Mamba-like models. However, it seems that there are no experimental results supporting this.\n\n[B] Early convolutions help transformers see better. NeurIPS, 2021.", "other_comments_or_suggestions": "Actually, when I read the introduction section of this paper, I was supposed that a novel method is presented to solve the computation overhead when using smaller patches. If such a method is proposed, I think the overall quality of this paper could  be further improved.", "questions_for_authors": "Not applicable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews, with the two of reviewers (DwN7, UjKW) recommending acceptance, and two reviewers (vQ14, KY13) suggesting rejection. The AC has thoroughly examined the submitted materials, including the paper itself, the reviewers’ concerns and the authors’ responses, and has carefully considered all factors in the decision-making process. 

To the AC, the main concern is on the limited utility of the patch size scaling law, as the paper shows (in Fig. 4) further scaling the input size beyond original image size is no longer helpful; therefore compared to model size scaling, there is a potential hard limit in this scaling axis, which compromises the promise to further scale. Nevertheless, the work has done a solid exploration along this direction, already adopting an efficient architecture to mitigate the computational concern, and the overall value is deemed outweighing the concerns which are mostly addressed by the responses. 

Therefore recommending acceptance. Please incorporate promised changes in the final version.
Decision: Accept (poster)