Faster Vision Transformers with Adaptive Patches
Authors: Rohan Choudhury
JungEun Kim
Jinhyung Park
Eunho Yang
Laszlo A. Jeni
Kris Kitani
Venue: ICLR 2026 Poster (2026)
Keywords: efficient vision
vision transformers
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We accelerate vision transformers by adaptively allocating different patch sizes within the same image, reducing the number of input tokens.

Abstract: Vision Transformers (ViTs) partition input images into uniformly sized patches regardless of their content, resulting in long input sequence lengths for high-resolution images. We present Adaptive Patch Transformers (APT), which addresses this by using multiple different patch sizes within the same image. APT reduces the total number of input tokens by allocating larger patch sizes in more homogeneous areas and smaller patches in more complex ones.
APT achieves a drastic speedup in ViT inference and training, increasing throughput by 40\% on ViT-L and 50\% on ViT-H while maintaining downstream performance.
It can be applied to a previously fine-tuned ViT and converges in as little as 1 epoch.
It also significantly reduces training and inference time without loss of performance in high-resolution dense visual tasks, achieving up to 30\% faster training and inference in visual QA, object detection, and semantic segmentation.
We will release all code and trained models.

Published as a conference paper at ICLR 2026
ACCELERATING VISION TRANSFORMERS WITH ADAPTIVE PATCH SIZES
Rohan Choudhury!- JungEun Kim
Jinhyung Park'
Eunho Yang² László A. Jenilt Kris M. Kitani
*Camegie Mellon University 2KAIST
*General Roboties
ABSTRACT
Vision Transformers (ViTs) partition input images into uniformly sized patches regardless of their content, resulting long input sequence lengths for high- resolution images. We present Adaptive Patch Transformers (APT), which ad- dresses this by using multiple different patch sizes within the same image. APT reduces the total number of input tokens by allocating larger patch sizes in more homogeneous areas and smaller patches in more complex ones. APT achieves a drastic speedup in VIT inference and training, increasing throughput by 40% ViT-L and 50% on ViT-H while maintaining downstream performance. It can be applied a previously fine-tuned VIT and converges as little as I epoch. It also significantly reduces training and inference time without loss of performance in high-resolution dense visual tasks, achieving up 30% faster training and infer- ence in visual QA, object detection, and semantic segmentation. Our project page is available this link.
INTRODUCTION
Vision Transformers (ViTs) (Dosovitskiy et al., 2020) have become the dominant paradigm for vi- sual recognition, but their scalability is limited by the quadratic cost of self-attention with respect to sequence length. Since inputs are divided into fixed-size patches, image resolution directly de- termines sequence length: higher resolution images yield disproportionately long token sequences despite much higher redundancy.
Many prior works have proposed solutions to this issue, typically by merging a fixed proportion of similar tokens (Bolya et al., 2022) pruning uninformative ones with auxiliary predictors (Rao et al., 2021; Yin et al., 2022). While these reduce theoretical FI.OPs, they face two drawbacks. Firstly, fixed reduction ratio is mismatched to image complexity: merging only half the tokens in pure white image is insufficient, while merging half the tokens in a busy cityscape is harmful. Secondly, pruning during the forward pass introduces padding and irregular shapes, often negating speedups in practice (Dehghani et al., 2021). In contrast to vision transformers, language models rely on adaptive tokenizers such as Byte-Pair Encoding (Sennrich al., 2016) and SentencePiece Kudo & Richardson (2018), which flexibly assign tokens of varying lengths depending on subword frequency. This reduces input sequence size while improving performance, suggesting that variable- granularity tokenization can be more efficient than fixed-size splits.
Our key insight is that a similar idea can be applied to vision transformers. As illustrated in Figure 1, VITs use the same amount of computation a uniform green background on the complex patches on the head of the bird, despite the significant difference in visual complexity. We introduce the Adaptive Patch Transformer (APT), which addresses this mismatch by varying patch sizes within single image. Regions that are smooth and redundant can be represented with large patches, while regions rich in detail allocated smaller patches. This content-aware patchification preserves important information where it matters while reducing redundancy elsewhere. To do this, APT computes entropy at multiple scales and assigns larger patch sizes to regions with the lowest entropy,
"quual contribution
I Equal advising.

---

Published as a conference paper at ICLR 2026
Original Image
Standard Vision Transformer
Adaptive Patch Transfomerr (Ours)
Figure I: Adaptive Patch Sizing, We present APT, Adaptive Patch Transformers, which signif- icantly accelerate vision transformer training and inference by patchifying images based on their content. Complex regions receive more, smaller tokens, while simpler, homogeneous regions re- ceive fewer.
resulting in significantly fewer input tokens. We then down-sample the larger patches and combine their patch embeddings with the information from the original large patch using zero-initialized ML.,, allowing APT to converge without harming the network.
APT speeds up ViT inference and training by almost 40%, with even larger boosts for higher reso- lution images and larger models. When initialized from a self-supervised or large-scale pretrained checkpoint, APT reaches the same performance the original ViT after fine-tuning. If applied di- rectly to an already fine-tuned ImageNet checkpoint, APT incurs only a small accuracy drop without additional training. With our zero-initialized MLP, this gap can be closed in as little as a single epoch of finettning. We also find that unlike most prior token reduction works, APT can successfully ac- celerate vision transformers on a wide range of image understanding tasks, such as visual question answering, object detection, and semantic segmentation, while matching the baseline performance.
In summary, we (1) introduce the Adaptive Patch Transformer (APT), which accelerates Vision Transformers by up to 40% through content-aware patch sizes, with larger gains at higher resolutions and model scales; (2) show that APT preserves the accuracy of standard pretrained models across resolutions and scales; and (3) demonstrate that APT extends beyond ImageNet, performing well on dense prediction and vision- laniuaee tasks.
2 RELATED WORK
Vision Transformers and Patchification. Vision Transformers (ViT') (Dosovitskiy et al., 2020) are currently the de facto standard architecture for computer vision backbones (Xu et al., 2022; Kirillov et al., 2023; Peebles & Xie, 2022). In contrast to language models, which typically use subword tokenizers (Sennrich et al., 2016; Kudo & Richardson, 2018) with varying numbers of bits per token, ViTs patchify images into equally sized patches, each becoming a token. This can re- sult in an enormous number of tokens, especially at high resolution. Transformer-hasee generative models (Peebles & Xie, 2022; Esser et al., 2020) use visual tokenizers, typically using a variational auto-encoder (Kingma al., 2013; Van Den Oord al., 2017). project images into a compressed latent space, reducing the input size significantly. Some recent works explore adaptive visual to- kenizers (Yan al., 2024; Duggal al., 2024), which dynamically allocate more tokens more complex visual inputs, but do not meaningfully speed up training or generation. As a result, image understanding tasks are typically limited to lower resolution.
Reducing VIT Tokens. Accelerating ViTs by removing tokens is a rich area of research. Methods such as pruning (Yu Xiang, 2023; Yang et al., 2023; Zheng et al., 2022), compressed representa- tions (Wu et al., 2018; Park & Johnson, 2023), or quantization (Liu et al., 2021b; Li et al., 2022c; Moon et al., 2024) remove redundancies or compactly encode parameters, reducing inference time and memory usage. Alternative attention mechanisms, such as linearized (Katharopoulos et al., 2020: Lu et al., 2021) or local window attention (Liu et al., 2021a; Wei et al., 2023; Chen et al., 2023b), improve efficiency by limiting token interactions. More related to work methods that exploit the inherent redundancy of images (Meng et al., 2022; Yin et al., 2022; Kong et al., 2022; Rao et al., 2021) and videos (Choudhury et al., 2025; Ding et al., 2023; Wu et al., 2023) by
2

---

Published as a conference paper at ICLR 2026
pruning uninformative tokens. While these works content-aware, most require learning which tokens are unhelpful, negating any training speedup and preventing inference on batch sizes greater than 1. Several methods instead merge tokens based on similarity (Bolya et al., 2022; Bolya & Hoff- man, 2023; Liang et al., 2022b; Shang et al., 2024; Cao et al., 2023; Liang et al., 2022a; Tran et al., 2024; Kallini at., 2024; Lee & Hong, 2024; Wu al., 2023), which accelerates training. However, merging methods typically combine a constant number of tokens for each input, which can be sub- optimal for inputs with varying complexities. APT strikes balance between these two lines of work by providing significant acceleration to training and inference while maintaining content-awarenee
Adaptive Patch Sizing for Efficient ViTs. Our work is not the first to propose using multiple patch sizes for faster ViTs. Early attempts in this direction (Chen et al., 2021; Beyer et al., 2023; Wang et al., 2024; 2021; Zhou & Zhu, 2023; Hu et al., 2024) train models that are capable of using different patch sizes, but still require single patch size for each image. Closer to APT are works that allow for varying patch sizes within a single image (An et al., 2024; Ronen et al., 2023; Chen et al., 2023a; Bai et al., 2024). However, CF-ViT (Chen et al., 2023a) and Quadformer (Ronen et al., 2023) rely on a fixed number of patches, neglecting the variability of semantic information across images, which can lead to suboptimal performance. MG- VIT (Zhang et al., 2023b) also supports two patch scales, but relies on attention scores to decide patch sizes, preventing use of efficient attention kernels, and also requires training from scratch, which is significantly more expensive than APT.
Closest to our work is MS-ViT (Havtorn et al., 2023), which, like DynamicViT (Rao et al., 2021) learns gating network determine patch sizes and defines separate patch embedding networks for each size. However, it requires significant fine-tuning on pre-trained networks and does not speed up training. APT resolves this issue while demonstrating dramatically larger speedups at higher resolutions and on larger models.
3 METHOD
Our goal is to achieve a significant wall-clock speedup during both training and inference by using different-sized patches in different regions of the image. We first describe how we allocate different patch sizes within an image (Section 3.1) and then how we process different-sized regions into the same embedding space (Section 3.2). We then explain how we efficiently handle different input sizes and how we can adapt APT to work on dense visual prediction tasks like object detection.
3.1 DECIDING PATCH SIZES
Consider vision transformer that takes H W C image input. The standard ViT partitions the image into a set of X P patches. A linear layer E is applied to each patch to convert it into token, of size dembed, resulting in a sequence of N (HW/p?) tokens.
In contrast, our goal is to decide patch size based on the image content, instead of using a constant number of patches. Concretely, we define fixed number of patch scales S, where the set of patches consists of P Pi U P2 U Ps. with each patch in P. having size 2'p x 2'p. For example, if = 3. 16, we are trying to find a smaller set of 16 16, 32 32 and 64 64 patches while maximizing 'information' conveyed. For simplicity, we also impose the constraint that all patches follow a quadtree-like structure following regular grid.
We use entropy H as a measure of patch's 's compressibility, given by:
H(P) -
Σ
Pi logz Pt.
(1)
where pt the probability of pixel intensity i. Since patches contain discrete pixel values, we approximate this by binning pixel intensities and computing entropy from the resulting distribution. Entropy quantifies the unpredictability and thus information content of a patch, making it a useful predictor of compressibility- lower entropy indicates higher redundancy. A large patch with low entropy should therefore be efficiently representable by a -imubee--imenioonal vector. We discuss alternative measures further in the Appendix.
We obtain the patchification of the image hierarchically, as illustrated in Figure 3. We first divide the image into patches the coarsest scale 25p x 2 and compute their entropies. We then retain
3

---

Published as a conference paper at ICLR 2026
1. Split input image into multi scale putches und calculate entrupy
2. Assign paich sizes according the entropy
Patch Embedding
Transformer
Resize ZeroMLP
3. Project different- sized tokens into single embeedding space
Pass tokens into transformer to prodace output encoding
Figure 2: APT overview. APT works by measuring the entropy at multiple scales and assigning large patch sizes to low entropy patches. All patches are projected to the same size token embedding, and the reduced size input sequence passed to the transformer.
3
2
Patch Embedding
2
4
5
Resize
3
4
6
6
Split
5
2
Conv
4
ZeroMLP
6
Figure 3: Embedding Different Patch Sizes. The smallest size patches are projected with the patch embedding. Larger patches are both split into their sub-patches and resized; the sub-patches embedded, aggregated with a convolution layer. These are combined with the resized embedding with zero-initialized MLP (Zhang et al., 2023a).
all such patches with entropy below a fixed threshold TÜ which is tunable hyperparameter for each level. We repeat this process until reach the smallest possible patch size p p, which assign all remaining patches.
3.2 PATCH AGGREGATION
After dividing the image into different patch sizes, we need convert these patches into embeddings with dimension dembed: in standard ViTs, this is done with single linear layer E. Prior work on vision transformers with varying patch sizes either resize every patch to x p (Ronen et al., 2023), resize E for each size (Beyer al., 2023), train S separate patch embedding layers & for each possible patch size (Havtorn et al., 2023), adding overhead. Resizing allows reasonable performance with training but can be improved upon uses strictly less information than if we applied & to the higher-resolution
patches.
We combine these strategies, shown in Figure 3. but retain copies larger original patches. For a given patch P, of size 2'p x 2'p, we define its constituent p sub-patches the (Pz). Each sub-patch Pj is embedded using the standard
We resize patches to uniform size p
4

---

Published as a conference paper at ICLR 2026
embedding layer E. The final embedding for patch P, is then computed as:
E(P)) = -erooMLP (Conv2 PP) c.)) P.)c)))))))))) +
+ ((Resize
(2)
where Conv2d indicates applying a convolutional downsampling layer i times, aggregating em- beddings from sub-patches back to size p x p. The ZeroMLP, a single linear layer initialized with zero weights inspired by ControlNet (Zhang et al., 2023a), allows the model to gradually incorpo- rate high-resolution details without initially degrading performance, facilitating faster convergence during fine-tuning, In particular, this enables APT to be applied to any pre-trained ViT and matches the performance of the initial model with a single epoch of accelcrated fine-tuning.
3.3 DYNAMIC INPUT SIZFS
Since APT is content-aware, the number of tokens for each image can vary widely. However, in contrast to token pruning works (Rao al., 2021; Liang et al., 2022b), we do not reduce the size of the input at each layer, but before running the model. While most vision works use a fixed resolution, our setting is closer to that of language modeling, and in vision to RLT (Choudhury et al., 2025) and NaVIT (Dehghani et al., 2023), where the number of tokens varies, but is predictably dictated by the input data. We follow these methods and employ sequence packing. For a batch of input images with sequence lengths NN, NB), we concatenate the tokens into a single sequence with length E.= N₂ and construct block- diigonal mask that ensures tokens only attend to tokens from the same example. This is natively implemented in commonly available attention backends such as FlashAttention (Dao al., 2022; Dao, 2024) or xFrmers (Lefaudeux et al., 2022), and adds no overhead to the network itself as the mask does not change. After running the network, we split the resulting sequence into its constituent subsequences and either extract the class token or compute a pooled representation for each subsequence.
Positional Encodings. To handle the positional encodings for the new variable-length sequences, we use positional encoding interpolation, introduced in NaVIT (Dehghani et al., 2023). Each smallest- size patch grid of size x W. is assigned an initial positional encoding, whether learned, sinusoidal or RoPE, where p is the base patch size. For larger patch sizes, we obtain the corresponding posi- H x sp sp grid, whose positional encodings are computed by sampling from the original H W. encoding map. For exam- plc, patches of size 2p use encodings interpolated to an 2p 2p and so on. This approach naturally generalizes positional information across different scales while maintaining spatial consistency.
tional encodings through interpolation: patches of size sp (where 1)
p H. W. grid, patches of size 4p use 4p
H. W x 4p*
x
Adaptation to Downstream Tasks. Standard methods for dense visual tasks like object detection semantic segmentation often rely on feature map that has the same aspect ratio the image. This required for methods that rely on transposed convolutions to upsample an input feature map for per-pixel predictions (Li et al., 2022a). In contrast, APT produces a different number of tokens per image, which cannot be simply reshaped into a rectangular feature map. To handle this, rely on the assumption that the tokens representing larger patches encode simpler features and simply repeat them 2 times, as in (Havtorn et al., 2023; Bolya & Hoffman, 2023). This yields fully differentiable feature map that can be upsampled by transposed convolutions and seamlessly applied to downstream tasks. Furthermore, tasks requiring high-resolution dense prediction, such as object detection, often rely on window attention (L.iu et al., 2021a; Yuan et al., 2021; Fang al., 2024), where the image is subdivided into multiple window regions to localize attention and increase efficiency. APT can still be applied even with window-attention. To do this, we divide the image into windows that multiples of the largest patch-size, and apply our patch assignment strategy as before. Now, each window contains variable numbers of tokens rather than constant number, and attention is applied within each window. As before. this can be straightforwardly implemented using sequence packing and attention masks with light overhead.
5

---

Published as a conference paper at ICLR 2026
Model
Res/Patch
Acc +
Img/s 1
GFLOPS
WC Time 4
Speedup 1
MAE
384/16
1151
49.4
11.6h
Random
384/16
83.4
1401
21.5
8.8h
+32%
Resizing
384/16
83.9
1390
21.5
9,0h
+29%
APT (Ours)
384/16
84.2
1390
21.9
9.0h
+29%
MAE
336/14
86.1
395
174.7
15.9h
Random
336/14
550
76.2
9.6h
+66%
Resizing
336/14
527
76.2
9,9h
+61%
APT (Ours)
336/14
86.1
527
76.8
9.9h
+61% %
VIT-L MAR
448/14
86.4
190
645
31.4h
Random
448/14
85.8
314
267
16.2h
+94%
Resizing
448/14
86.0
302
267
16.9h
+86%
APT (Ours)
448/14
86.3
302
268
16.9h
+86%
Table Full Fine-Tuning on ImageNet. APT significantly reduces the wall-clock time to fine-ture tin-tune pre-trate rreitra backbone ImageNet with no degradation in accuracy. We the MAE (He al., 2) training recipe for all cases. Note that ViT-B is trained for 2x more epochs than ViT-L.
Model
Res/Patch
Acc
GFLOPS
Img/s
Speedup
Res/Patch
Acc 1
GFL.OPS
Img/s
Speedup
VIT-B
221/16
85.1
16.9
384/16
86.1
49.4
1151
Random
224/16
83.7
12.5
3751
+13%
384/16
85.0
21.5
1401
+22%
Resizing
224/16
84.6
12.5
3540
+7%
384/16
85.7
21.5
1390
+21%
APT-B (Ours)
224/16
85.1
12.7
3540
+7%
384/16
86.1
21.9
1390
+21%
VIT-1.
224/14
87.9
59.7
883
336/11
88
174.7
395
Random
224/14
44.3
+19%
336/14
87.3
76.2
550
+39%
Resizing
224/14
87.4
44.3
993
+12%
336/14 114
87.9
76.2
527
+335%
APT-L (Ours)
224/14
44.5
+12%
336/14
88.1
527
+33%
VIT-H
224/14
88.3
162.0
441
336/14
88.5
363.7
175
Random
221/14
87.4
92,1
568
+29%
336/11
87.0
158.3
272
+55%
Resizing
92.1
336/14
88.0
263
+50%
APT-H (Ours)
224/14
88.3
92.3
542
+2356
336/14
88.4
158.9
263
+50%
Table 2: 1-epoch Fine-Tuning on ImageNet. APT consistently achieves large speedups while matching sometimes exceeding the original network's performance after fine-tuning for 1 more epoch. Compared to only random masking or only resizing, APT offers the best tradeoff between speed and accuracy.
EXPERIMENTS
4.1 BASELINES
We categorize token merging approaches into two groups: input-level and layer-level. Input-level merging reduces tokens directly from image patches before entering the model, which is the category our method belongs to. In contrast, layer-level merging performs reduction within the network dur- ing feature propagation. We adopt input-level merging as main baseline for fairest comparison, but compare to layer-level methods as well.
Input-level Merging Baselines. We use three main baselines: random-masking, resizing-only (He et al., 2021; ;; Li et al., 2023) and the original optimized Vanilla implementation from t timm. Resizing refers to only resizing the larger patches the base patch size; This is stronger version of Quad- former (Ronen et al., 2023), which also resized the larger patches, but used a constant, nonadaptive number of patches per image; instead we use a threshold so that the sequence length adaptive. Random is stronger version of FL.IP (Li et al., 2023); we compute the token reduction obtained from APT and set as the random patch dropping rate.
Layer-level Merging Baselines. We benchmark against four baselines: EVIT (Liang et al., 2022a),
ToMe (Bolya et al., 2022), PPT (Wu et al., 2023) and DTEM (Lee & Hong, 2024). These baselines perform token merging across the ViT layers, removing constant number of tokens regardless of image content. However, they share a key limitation: none of them are natively compatible with FlashAttention, which makes them even slower than the Vanilla ViT equipped with FlashAttention. To provide a fairer and stronger comparison, we re-implement 'advanced" versions these base- lines with FlashAttention with the exception of PPT, which relies attention scores and is thus

---

Published as a conference paper at ICLR 2026
88.0
88.6
07.5
88.2
Accuracy (%)
Vanilla
Accuracy (9)
06.0
Vanills
Ours
Ours
Evit
Evit
(Adv)
Evit (Adv)
07.6
TMMe
ToMe
87.4
(Adv)
DTEM
DTEM
87.2
DTEM
PPT
PPT
85,5
87.0
6
10'
10a
Throughput (Imgls)
Throughput (Img/s)
(a) Trade-off comparison ViT-L/14, 221 x 224.
(b) Trade-off comparison on ViT-H/14, 336 X 336.
Figure 4: Accuracy vs. Throughput under different compute budgets. Comparison between APT and layer-level merging methods on ViT-L and ViT-H. For fairer evaluation, also include their re-implementee Advanced (Adv) versions with FlashAttention, shown with a dashed line. APT consistently outperforms the baselines in both throughput and accuracy across all compute budgets.
incompatible. For methods that use weighted attention, we disable it to enable FlashAttention. We provide more details in Appendix A.
4.2 IMAGE CLASSIFICATION
Full Fine-Tuning. We provide the results of fine-tuning a vision transformer with APT on two different model scales and resolutions in Table 1. Here, 'full" fine-tuning refers to training from pre-trained self-supervised ViT backhone, rather than an already fine-tuned classification network. In all experiments, we use the official MAE (He et al., 2021) pre-trained checkpoint, interpolated to match the target resolution. At lower resolution, APT provides a 100 speedup over the base- line with 141 token reduction. However, the speedup increases dramatically for higher reso- lutions-oo on 336x336 the speedup doubles. APT also reduces training time more at larger model scales, likely due to the fact that the attention computation dominates much more of the training time per iteration. In all cases, APT matches the baseline while using the exact same training recipe-it it standard patching.
can be considered absolute improvement
Short Fine-Tunme Tuning. We next present results from training with APT for epoch starting from checkpoint already fine-tuned on ImageNet (Deng et al., 2009), rather than pre-trained self- supervised checkpoint. Compared to other methods like Dynamic (Rao et al., 2021) or MS-VIT (Havtorn et al., 2023) which require 50 more epochs of fine-tuning learn scoring function, thanks to our use of a vero-initialized layer, APT models make high-quality predictions from initial- ization. With no training, APT only resizes the larger patches to the base size, which stronger version of Quadformer (Ronen et al., 2023). However, we observe that just one epoch is sufficient to "heal" the degradation from the new patchification scheme and match the original performance of the model, shown in Table 2.
We provide a comparison with representative baselines in Figure 4. We compare throughput and ImageNet accuracy to our short fine-tuning results on ViT-L/14 with resolution of 224 and ViT- H/14 with a resolution of 336. APT consistently outperforms baselines, including their original versions well as our improved reproductions using FlashAttention. The results confirm that input- level merging is inherently more efficient and reliable than layer-level merging.
4.3 VQA AND DENSE VISUAL TASKS
Vision transformers are used for a wide range of tasks beyond image classification; we evaluate how APT affects downstream performance in vision-language understanding tasks as well as dense prediction. Building on our fine-tuning experiment for image classification, we start with fully fine- tuned model, and fine-tun for 5% of the total iterations used in each model's fine-tuning scheme. For Visual QA, we fine-tune tune only the newly introduced ZeroCony and MLP modules. while for detection and segmentation, we fine-tune the entire model.
7

---

Published as a conference paper at ICLR 2026
Model
Img's
Speedup
VQA"
GQA
SQA'
VQA"
POPE
MME
MMB
MMB
MMV
LLaVA-1.5-)
3.70
78.5
61.0
67.8
58.2
86.9
1510.1
64.6
58.1
30.7
Random
4.58
+24%
76.9
60.9
67.2
54.1
86.1
1460.5
62.7
57.6
30.5
Resizing
4.51
+22%
77.5
61.1
66.8
56.5
86.6
1473,8
63.2
58.1
30.2
APT (Ours)
4.51
+22%
77.9
61.4
67.5
56.9
86.4
1474.0
63.8
58.2
30.8
LLaVA-1,5-138
2.22
80.0
63.2
72.7
61.2
87.1
1530.6
68.5
63.4
35.4
Random
2.79
+26%
78.0
60.7
72.0
55.7
86.5
1484.0
64.7
60.8
32.0
Resizing
2.72
+23%
78.9
72.0
86.8
1496.9
33,9
APT (Ours)
2.72
+23%
79.4
63.0
72.4
59.5
87.2
1511.2
66.5
63.7
34.7
Table 3: Transfer to VQA. APT enables significant throughput increase while matching exceed- ing performance to the baseline.
Model
Img/s
Speedup
mAP
AP50
Model
Res
Img/s
Speedup
aAcc
mloU
EVA-02-B
1536
3.86
58.93
77.85
EVA-02-L
512
4,40
86.67
59.77
Resizing
1536
4.41
+14%
58.43
77.22
Resizing
512
4.87
+11%
86.09
58.81
APT (Ours)
1536
4.41
+14%
58.79
77.65
APT (Ours)
512
4.87
+11%
86.68
59.70
EVA-02-L
1536
1.62
62.28
80. 80
EVA-02-L
640
2.55
86.83
60.05
Resizing
1536
2.17
+30%
61.75
80,27
Resizing
640
2.83
+11%
86.06
58.83
APT (Ours)
1536
2.17
+30%
62.07
80.64
APT (Ours)
640
2.83
+11%
86.82
60.01
Table 4: Transfer to Object Detection. APT Table 5: can be scaled to high-resolution dense image APT can handle pixel-level fine-grained tasks tasks supporting window attention.
Transfer to Semantic Segmentation.
without compromising visual acuity.
Visual QA. We first apply APT to the vision backbone of LLaVA (Liu et al., 2023; 2024a). LLaVA vision language model (VLM) that combines vision transformer backbone with a language backbone via projection layer. In the original paper (Liu et al., 2023), the vision encoder was completely frozen, and only the projection layer was updated. APT matches the original model per- formance while reducing image tokens and increasing throughput by 23%. Note that APT provides no speedup to the language component, but by reducing the number of visual tokens, accelerates both the vision backbone and cross attention layers. We find that APT exceeds the original perfor- mance of the LLaVA model on a range of vision-language benchmarks (Goyal al., 2017; Hudson & Manning, 2019; Lu et al., 2022; Singh et al., 2019; Schwenk et al., 2022: Fu et al., 2024; Liu et al., 2024b; Yu et al., 2023).
Object Detection. One might expect APT to degrade performance for tasks that require pixel-level understanding, such as object detection. To investigate this, we trained an object detector using the EVA-02 (Fang et al., 2024) backbone with window attention, with ViTDet (Li et al., 2022b) style detection head. We conduct experiments on the COCO (Lin et al., 2014) dataset at 1536 x 1536 resolution. APT is able reduce impressive 30% of input tokens, drastically speeding up training and inference, while matching the final performance on mAP and AP50. Furthermore, these results demonstrate that APT remains effective under window attention beyond naive full attention, broadening the scope of its application.
Semantic Segmentation. We conduct another experiment on semantic segmentation, which re- quires fine-grained understanding of object boundaries. We again use the protocol of EVA-02 (Fang backbone with UperNet (Xiao al., 2018) segmentation model on top. When tested on ADE20K (Zhou et al., 2019; 2017), ,), APT attains baseline performance while reduc- ing 28-322% the input tokens depending on image resolution, thereby substantially accelerating inference. APT's success at semantic segmentation is particularly encouraging, since it implies that it reduces compute while not sacrificing visual acuity at the pixel level.
al., 2024), using
4.4 ABLATIONS
We ablate components of APT to evaluate their effect speed and accuracy.
Measuring APT overhead. Next, we measure the computational overhead introduced by APT. Re- arranging the input patches and using masks does not have computational cost, and given that GPUs are highly optimized for constant input shapes, understanding the cost of adding APT is im- portant. The results of this analysis are in Table 6, showing that with sequence reduction, APT is
8

---

Published as a conference paper at ICLR 2026
(mraee dio 334, Thrmen 5.75)
0.24
2.45
3040
Sequannee Raccing
2000
0.034
Patch Embeddiin iZeroConwi
Taarffrmer
300
25
Time (milliseconds)
Figure 5: Sequence Length Distribution. The distribution of sequence lengths is concentrated near the maximum, and slowly tails off, stop- ping at around 30% of the maximum value.
Figure 6: Runtime Breakdown. Although the standard VIT has faster patch embedding step, APT's transformer cost is less from the reduced token length, leading significant overall speedup.
Res/Patch
Base (Img/s)
APT, ==
Model
w/o training
w/ training
ViT-B 224/16
3310
3090
Base
88.15
88.15
VTT-B 384/16
1151
1030
Residual
87.40
87.52
ViT-L 224/16
883
811
NonZero
87.50
87.81
336/14
395
360
Zero (Ours)
87.98
88.13
VIT-H 224/14
441
418
VIT-H 336/14
190
180
Table Ablating Zero-initialization. Using zero-initialized connection works the best for training APT networks to properly incorporate higher resolution details from the original image, while preserving good-quality predictions before training.
Table 6: APT overhead with no reduction. With no token reduction, APT incurs nontriv- ial overhead. However, token reduction gives 20%+ speedups relative the standard imple- mentation, more than covering the discrepancy.
about 10% slower. Next, Figure Figure 6 measures the speed of the ZeroConv and sequence packing operations. When setting 11, the ZeroConv operation is much faster, since no additional com- putation is incurred. However, when 5.75, it takes about 10% of the end-to-end model latency. The resulting speedup from removing extra tokens more than makes up for this, though, yielding a net 33% speedup. We also perform the entropy computation the CPU dataloader, parallelizing across multiple cores and overlapping it with the GPÚ model computation. This yields no additional overhead.
Zero-initialization. Finally, we ablate the use of our zero-initialized connection for incorporating rngher-reoouuton details in larger patches. In Table 7, compare with simple residual connection, a non-zero initialized connection, and resizing. We find that initializing to zero offers the best off- the-shelf accuracy as well as the strongest performance after epoch of training. This matches the finding of ControlNet (Zhang et al., 2023a), which showed that zero-initialization works well for adding new capabilities without adding harmful noise the original model.
Thresholds. The main tunable parameter in APT is the entropy threshold, which can differ per scale and controls how compressible region must be in order be retained. For most tasks, use 7y2 = 5.75, 4 4.0, which works well out-of-the-bae box. We found that object detection required lower threshold of 2, likely due to the task's reliance on precise edge localization. Despite the smaller threshold, we still observe large speedups in object detection, due to the extremely high resolution inputs. The speed-accuracy trade-off resulting from threshold adjustment shown in Figure 4, and additional analysis and visualizations are provided in Appendix C. We observe that as the threshold is increased, the accuracy begins to slowly decrease, and eventually the accuracy drops significantly more and more useful information is blurred in the input.

---

Published as a conference paper at ICLR 2026
Visualized Examples. APT consistently places large patches on more homogenous re- gions and smaller patches on more complex ones. We use conservative thresholds to limit informa- tion loss. Images are best viewed zoomed More visualizations are Appendix
Figure 7:
4.5 EXAMPLE VISUALIZATIONS
We provide qualitative visualizations of the patchification produced by APT with 3 patch scales in Figure As desired, APT consistently assigned larger patches to more homogeneous regions of the image. Dark backgrounds, blue sky, and blurry backdrops (64x64) patches, while smaller regions that are still simple are given the second largest x32x32). Regions with more detail high frequency receive smaller patches: people's faces objects in focus are allocated the most. Each image has different number of patches, depending on its inherent complexity the cityscape in the bottom right has significantly more than the simpler cartoon image in the top left. By design, the patches produced by APT are agnostic to downstream goals and unaware of what a might desire from an image. If the top right image along with the question "What color is the background?", APT would still assign extremely coarse patches the pink wall due to its textural simplicity. Additional visualizations are provided in Appendix C.
all covered by the largest
input VLM
5 CONCLUSION
We presented Adaptive Patch Transformer (APT), method to accelerate ViTs that uses larger patches in simpler areas and smaller patches in more complex ones. It significantly improves train- ing and inference speeds, especially for larger models and higher resolutions. APT can be applied to any pretrained ViT backbone and converges in epoch or less, enabling users to quickly train their models to be faster on wide range of vision tasks. Our results suggest that APT will benefit the broader vision community by reducing the compute budget required to train state-of-the-a models.
Limitations. Although APT provides significant speedups, it still relies hand-crafted heuristic to determine patch sizes, which may not always align with downstream users' preferences and could empirically- tuned threshold hyperparameter,
likely be improved. Additionally, APT relies which can add friction to adoption on downstream tasks. Finally, while APT works for image understanding tasks, currently does support image generation, which operates with extremely high-resolution images and large models, making it an ideal application for our work. Future work will be required overcome these limitations, and hope that APT can inspire further research on efficient ViTs.
ACKNOWLEDGMENTS
RC is supported by the NSF Graduate Research Fellowship (GRFP). JK is supported by an IIT grant from the Korean government (MSIT) under the Al Excellence Global Innovative Leader Edu- cation Program (RS-2022-00113911)
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Xiaoqi An, Lin Zhao, Chen Gong, Nannan Wang, Di Wang, and Jian Yang. Sharpose: Sparse high- resolution representation for human pose estimation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 691-699, 2024.
Zhuhua Bai, Weiqing Li, Guolin Yang, Fantong Meng, Renke Kang, and Zhigang Dong. A coarse- to-fine framework for point voxel transformer. In 2024 27th International Conference Com- puter Supported Cooperative Work in Design (CSCWD), pp. 205-211. IEEE, 2024.
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Korblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flex- ivit: One model for all patch sizes. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 14496-14506, 2023.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4599-4603, 2023.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXivv2210.09461, 2022.
Qingqing Cao, Bhargavi Paranjape, and Hannaneh Hajishirzi. Pumer: Pruning and merging tokens for efficient vision language models. arXiv preprint arXiv:2305.17533 2023.
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international con- ference on computer vision, pp. 357-366, 2021.
Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji. Cf-vit: general coarse-to-fine method for vision transformer. Proceedings of the AAAI con- ference artificial intelligence, volume 37, pp. 7042-7052, 2023a.
Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. Sparsevit: Revis- iting activation sparsity for efficient higl-resolution vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2061-2070, 2023b.
Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Ninumma, Kris Kitani, and László Jeni. Don't look twice: Faster video transformers with run-length tokenization. Advances in Neural Informa- fion Processing Systems,
2025.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Fltstonttntion: Fast and memory-efficient exact attention with aO-awarensss. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency mis- 2021.
nomer. arXiv preprint
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, et Patch pack: Navit, vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36:2252-222 2274, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
11

---

Published as a conference paper at ICLR 2026
Shuangrui Ding, Peisen Zhao, Xiaopeng Zhang, Rui Qian, Hongkai Xiong, and Qi Tian. Prune spatio-temporal tokens by semantic-aware temporal accumulation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 16945-16956, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Shivam Duggal, Phillip Isola, Antonio Torralba, and William T Freeman. Adaptive length image tokenization via recurrent allocation. arXiv preprint arXiv:2411.02399 2024.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing. 149:105171, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL https: /ttp:///axvvvvvo .rrg/ abs/ 2306. 13394.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Jakob Drachmann Havtorn, Amélie Royer, Tijmen Blankevoort, and Babak Ehteshami Bejnordi. Msvit: Dynamic mixed-scale tokenization for vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 838-848, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377, 2021.
Youbing Hu, Yun Cheng, Anqi Lu, Zhiqiang Cao, Dawei Wei, Jie Liu, and Zhijun Li. Lf-vit: Reducing spatial redundancy vision transformer for efficient image recognition. Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 2274-2284, 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Julie Kallini, Shikhar Murty, Christopher D Manning, Christopher Potts, and Róbert Csordás. Mrt5: Dynamic token merging for efficient byte-level language models. arXiv preprint arXiv:2410.20771, 2024.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165. PMLR, 2020.
Diederik Kingma, Max Welling, al. Auto-encoding variational bayes, 2013.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 4015-4026, 2023.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In European conference on computer vision, pp. 620-640. Springer, 2022.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing:
12

---

Published as a conference paper at ICLR 2026
System Demonsttrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Com- putational Linguistics. doi: 10.18655/v1/DD1--2022 URL D18-2012/.
org//
Dong Hoon Lee and Seunghoon Hong. Learning to merge tokens via decoupled embedding for efficient vision transformers. Advances in Neural Information Processing Systems, 37:54079- 54104, 2024.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang. Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov, xformers: A modular and hackable trans- former modelling library. https:/ giggtithhh..e..e http://gghbbb..cooaaabb.aaaacaaa ham///faboooreearcchh.. wfoomerrs..roooooooooooomm 2022.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. In European conference on computer vision, pp. 280-296. Springer, 2022a.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. In European conference on computer vision, pp. 280-296. Springer, 2022b.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image image pre-training via masking, In CVPR, 2023.
Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Ac- curate and fully quantized low-bit vision transformer. Advances neural information processing 2022c.
systems,
Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800, 2022a.
Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07880, 2022b.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6- 6-12, 2014, pro- ceedings, part v 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-349166 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 16296-26300 2024a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? In European conference computer vision, pp.
Springer, 2024b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021a.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quanti- zation for vision transformer. Advances in Neural Information Processing Systems, 34:28092-
28103, 2021b.
Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu. Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: Softmax- -ree transformer with linear complexity. Advances Neural Information Processing Systems, 34:21297-21309, 2021.
13

---

Published as a conference paper at ICLR 2026
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems. 35:2507-2521, 2022.
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser- Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 12309-12318, 2022.
Jaehyeon Moon, Dohyung Kim, Junyong Cheon, and Bumsub Ham. Instance-aware group quanti- zation for vision transformers. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 16132-16141, 2024.
Jeongsoo Park and Justin Johnson. Rgb no more: Minimally-decoded jpeg vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09744, 2022.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 13937-13999, 2021.
Tomer Ronen, Omer Levy, and Avram Golbert. Vision transformers with mixed-resolution tokeniza- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4613-4622, 2023.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, 1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16 1162.
URL https: / /aaaacaannollooooooeo wwwphhhlllg...oor/egeeeoe org /Plg//P1e 6 1162/.
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint wxXiv:2403115388 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Chau Tran, Duy MH Nguyen, Manh-Duy Nguyen, TrungTin Nguyen, Ngan Le, Pengtao Xie, Daniel Sonntag, James Y Zou, Binh Nguyen, and Mathias Niepert. Accelerating transformers with spectrum-preserving token merging. Advances in Neural Information Processing Systems, 37: 30772-30810, 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Advances in neural information processing systems, 34:1 1960-11973, 2021.
Yunke Wang, Bo Du, Wenyuan Wang, and Chang Xu. Multi-tailed vision transformer for efficient inference. Neural Networks, 174:106235, 2024.
Cong Wei. Brendan Duke, Ruowei Jiang, Parham Aarabi, Graham W Taylor, and Florian Shkurti. Sparsifiner: Learning sparse instance-dependent attention for efficient vision transformers. In
14

---

Published as a conference paper at ICLR 2026
Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 12680-22689 2023.
Ross Wightman. Pytorch image models, 2019. URL http::://ggthhbb.cmm/////
srrorch-image-monn
Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R Manmatha, Alexander J Smola, and Philipp Krähenbühl. Compressed video action recognition. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 6026-6035, 2018.
Xinjian Wu, Fanhu Zeng, Xiudong Wang, and Xinghao Chen. Ppt: Token pruning and pooling for efficient vision transformers. arXiv preprint 2rXiv:2310.013: 2023.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pp- 418-434, 2018.
Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer base- lines for human pose estimation. Advances in neural information processing systems, 35:38571- - 38584, 2022.
Wilson Yan, Matei Zaharia, Volodymyr Mnih, Pieter Abbeel, Aleksandra Faust, and Hao I Liu. Elas- tictok: Adaptive tokenization for image and video. arXiv preprint arXlv:2410.08368, 2024.
Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian- aware saliency. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18547-18557, 2023.
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov, A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10809-10818, 2022.
Lu Yu and Wei Xiang, X- X-pruner: explainable pruning for vision transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 24355 14355-24363, 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 230.022088.02400 2023.
Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang- Hrformer: High-resolutinn transformer for dense prediction. arXiv preprint 1xXiv:2100009) 2021.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023a.
Yu Zhang, Yepeng Liu, Duoqian Miao, Qi Zhang, Yiwei Shi, and Liang Hu. Mg-vit: multi- granularity method for compact and efficient vision transformers. Advances in Neural Information Processing Systems, 36:69328 69347, 2023b.
Chuanyang Zheng, Kai Zhang, Zhi Yang, Wenming Tan, Jun Xiao, Ye Ren, Shiliang Pu, et al. Savit: Structure-aware vision transformer pruning via collaborative optimization. Advances in Neural Information Processing Systems, 35:9010-9000 9023, 2022.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 13001- 13008, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2017.
15

---

Published as a conference paper at ICLR 2026
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321. -20022-21,,7 2019.
Qiqi Zhou and Yichen Zhu. Make long image short: Adaptive token length for vision transformers. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 69 85- Springer, 2023.
16

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
Hardware Setup. All ImageNet experiments were conducted on a node of 8x NVIDIA A A100s, and the experiments on object detection, segmentation, and visual QA were conducted with 8xN- VIDIA RTX A6000, The inference-time results were computed on a single GPU, along with the throughput and FLOPS analysis. We used a single node for all work on this paper.
Layer-level Merging Baselines. We used the official repositories for EVIT (Liang et al., 2022a), ToMe (Bolya et al., 2022), and DTEM (Lee & Hong, 2024). Since implementations and experi- ments for ViT-L and ViT-H were not provided, we extended the code to include these two model configurations. In addition to adding the ViT-L and ViT-H variants, all experimental settings (train- ing schedule, enhancements, optimizers, input resolutions, and other hyperparameters) were kept identical to the original baselines to ensure a fair comparison.
Image Classification. We implemented image classification models using the timm library (Wightman, 2019), leveraging its pretrained checkpoints. The ImageNet-1K IK dataset (Deng et al., 2009) was used for training and evaluation, following prior work (Havtorn al., 2023; Ronen al., 2023). For the full fine-tuning experiment, we follow the exact MAE training recipe (He et al., 2021), training VIT-B for 100 epochs and VIT-L for 50, We use a base learning rate of 1.5e-3 and use standard augmentations, namely RandAug (Cubuk et al., 2020), Random Erasing (Zhong et al., 2020), random flipping, and cropping. All training was done with 8 GPUs and used batch size 1024. We set layer decay to 0.75 during long fine-tuning, For short fine-tuning, we train the network for epoch with layer decay set to 0.99, and learning rate set to 1e-6, and disable augmentations.
Visual QA. For our Visual Question Answering (VQA) experiments, we utilized the official VLa (Liu et al., 2024a) implementation and its pretrained checkpoints. Unlike the origi- nal approach, which collects data and fine-tunes the entire dataset for one epoch, we fine-tuned only 5% of the dataset, as we initialized from an already fine-tuned checkpoint. To adapt to this setting, we reduced the learning rate by a factor of 10 while following all other fine-tuning procedures rec- ommended by LLaVA. The base image resolution was set to 336 with patch size of 14, as specified in LLaVA's default configuration. A threshold of 5.75 was applied to determine a patch size of 28.
Object Detection. For object detection, used the official implementation of EVA-02 (Fang et al., 2024) along with its pretrained checkpoints, which utilize a window attention mechanism. Fine-tuning was conducted following the recommended procedures outlined in EVA-02. Consistent with our previous experiments, we fine-tuned for 5% of the total iterations while reducing the learn- ing rate by factor of 10. Following EVA-02's settings, the image resolution was 1536 with a patch size of 16. Patch sizes of 128, 64, and 32 were determined based on threshold values of 0.3, 2, and 2, respectively.
Semantic Segmentation. We also utilized the official EVA-02 implementation along with its pre- trained checkpoints for semantic segmentation. The ADE20K dataset (Zhou et al., 2019; 2017) was used for training and evaluation. Fine-tuning followed the recommended procedures outlined in EVA-02. In alignment with our previous experiments, we fine-tuned for 5% of the total itera- tions while reducing the learning rate by factor of 10. According EVA-02's settings, the image resolution was either 512 or 640, with a patch size of 16. A threshold of 5.75 was applied.
Advanced baselines with Attentoon. Many adaptive-token baselines including EViT (Liang et al., 2022a), ToMe (Bolya et al., 2022), and DTEM (Lee & Hong, 2024) incorporate token- weighted attention mechanisms on top of standard scaled dot-product attention. While this weight- ing is central to their design, it also prevents the use of FlashAttention, forcing these models to rely on the much slower unfused vanilla attention kernel. As a result, the basclines run substan- tially slower than a standard VIT equipped with FlashAttention, creating unfair comparison in throughput-oriente
evaluations. To address this, we re-implement "advanced" versions of these baselines using the unified operator in Listing 1. During inference, we disable the weighting oper- ation, allowing the model to employ FlashAttention and achieve speeds comparable to modern ViT implementations. This design ensures that all baselines benefit from FlashAttention when possible, providing more equitable and technically up-to-date comparison across methods.
17

---

Published as a conference paper at ICLR 2026
Listing 1: Advanced baselines with FlashAttention. Baselines originally rely on vanilla scaled dot-product attention combined with token-weighted attention, which prevents the use of FlashAt- tention and significantly slows inference. Our implementation enables fairer comparison by sup- porting FlashAttention.
class AdvancedAttenn ion (
def (sel,, x, mode="wot "wigh size=Nene): 6**
Args:
x: Tensor of shape (B, M, C)
mode:
"flash attn": use scaled dot poooupprruutt..oo attbettttttniion "weighted_ attn": proportion attention with token weights "size'
"vanilla_attn"" standard scaled dot-product attention
10
size: Tensor of shape (B, with per-token weights.
***
13
B, N, x. shape
14
outl, out2 self.qkv (x)
15
ękv = outol reshape (B, N, 3, self.num_heads, self. haffhheadddmm) permute 0, 3, 1, 4)
16
e, k, v gkv. unbind (0)
17
S, = self.
self.k
18
G k, q. float k. flfloat(), v.float
20
with corch.cuda.ad .mt..uuuuaaa..autccar hww.ttccsstcyyyyttortr...ccoeeee (dtype-torch. fleat32, enabled-True)
FlashAttention without weighting
22
if mode == "flash_attn",
_ = q, k, v,
attn_mask=Nene,
26
www.ootttssell..tt....o is_causal=False
27
30
Original implementation of baselines
31
elif mode = "weighted attn" and size is not None:
q self.scale
attn
k. (- -1)
_attn attn terch.max (attn, dim=-1, keepdim-True) [0]
35
attt _attn.exp_() sizel: None, None, :] type (torch.
float32)
attn = _attn / Latt. sum (dim==1, keepdim=True) attn self.attn, attn drop
38
_ attn
39
40
Standard scaled dot-product attention elif mode = "vanilla_attn";
q q self.scale
43
attn q 8 k.traaspose (-2, -1)
44
attn = attn attm.softmax (dim=-ammddmm=111
attn = self.attn drop (attn) attn
48
else:
49
raise ValueError (f"Unknwwn attention mode: (mode)
x. www.tpppeecc...... dt.d)y
52
53
X x.transpose (1, 2) reshape (B, N, C) self.proj (x)
54
self.prof drop (x)
57
return x
18

---

Published as a conference paper at ICLR 2026
Captacian Ubsarrple
28-
0.873
Accurooc (50
07
26
0.872
85
04
Throughput (Imglsec, log
0.863
Threshold (r)
55
% Takens
Figure 8: Threshold Effect. Increasing the threshold increases throughput significantly, but after approximately
Figure 9: Analyzing Scorers. We compare the accuracy on ViT-L/336 for different scor- ers, controlling for the fraction of retained kens. We find that the the entropy (standard) scorer performs best high reductions, but that all three are relatively similar.
6.0, the accuracy be- gins to severely drop off, and is not 'fixable" with fine-tuning.
B RUNTIME DETAILS
All ImageNet experiments were conducted on node of 8x NVIDIA A100s, and the experiments on object detection, segmentation, and visual QA were conducted with 8xNVIDIA RTX A6000. The inference-tee time results were computed on single GPU, along with the throughput and FLOPS analysis. We used a single node for all work on this paper.
C ADDITIONAL RESULTS
We provide additional visualizations illustrate how APT (Adaptive Patch Token) prunes tokens and to analyze the qualitative effects of varying the difference threshold т, augmentation and scorers. All visualizations were conducted using images resolution of 336 336 and patch size of 14 x 14.
Threshold Analysis. The main tunable parameter in APT is the entropy threshold, which can differ per scale and controls how compressible region must be in order to be retained. Lower values indicate higher sensitivity, and for the vast majority of experiments this paper, we used Ti 5.75, 72 4.0. In Figure 14, measuring ImageNet accuracy. We observe that for threshold values larger than 6.0, accuracy drops significantly, while throughput continues to increase. We find that 5.75 offers good trade- off between acceleration and maintaining quality and hypothesize that this is close to the 'true' threshold for compressibility; beyond this point, coarse-scale patches result in information loss. Figure 10 shows diverse set of sample images and how our method prunes tokens with relatively lower amounts of information (e.g., background regions uniform color patches). We fix T2 4.0 and change from 4.5 to 7. Observing various categories of images, one can see that patches containing high-free frequery details or salient object features are consistently preserved. In contrast, less critical regions-suck as large uniform areas-are pruned. This visualization confirms that the model potentially increases efficiency by ignoring parts of the image that contribute less to the downstream task.
vary Tj for 3 model scales with resolution 336 and patch size
Scorer Analysis. Figure/ 11 qualitatively contrasts the results an entropy-based scorer with two alternative scores. The entropy-based scorer measures how diverse complex the distribution of vixel-values within patch is. If a patch has pixels with wide range of intensities or colors, scores higher and is more likely be retained. This approach naturally favors regions with
19

---

Published as a conference paper at ICLR 2026
Image
r 4.5
T 5.0
5.5
6.0
T 6.5
T= 7.0
Threshold visualization. We can see that patches containing high-frequency details or salient object features are consistently preserved under various thresholds. We used v = 5.5 for most of the experiments. Zoom in for the best view.
Figure 10:
intricate textures, multiple color transitions, or high levels of detail. In comparison, the Laplacian- based scorer uses a second-derivative operator (or second-order difference) detect edges or sharp transitions. Specifically, it looks at how abruptly the pixel intensity changes within a patch. As Laplacian
a result, there is a strong boundary or a sharp difference in color or brightness, score becomes high, signaling that the patch likely contains important edge information and should be preserved. Finally, we tested an upsampling-
scorer, which downsamples the image by factor of 2" for each scale index then upsamples back the original resolution. It then compares the average mean squared difference for each patch. This scorer performs similarly to the Laplacian scorer, but can be little less sensitive smaller details.
We also measured the accuracy of using each scorer, controlling for the fraction of reduced tokens, the results of which shown in Figure 9. Although they perform similarly, the entropy scorer works better at higher token reductions. At higher token reductions, the Laplacian and upsampling- based scorers tend to remove more information that critical the model, which results in slightly worse performance. However, the differences are quite small and in practice we expect all three could be used interchangeably.
Augmentation Analysis. We compare how APT operates under different data augmentation tech- niques in Figure 12, since these apply transforms images that make them 'less natural'. In partic-
20

---

Published as a conference paper at ICLR 2026
Original Image
Standard
Entropy
Laplacian
Upsample
Scorer visualization. The entropy, Laplacian and upsampling scorers follow generally the same patterns with minor variations. The entropy scorer uses larger patches on regions with very few differing colors, while the upsampling and Laplacian scorers consistently use small patches on high-texture regions.
Figure 11:
ular, random erasing removes parts of the image, causing the overall information to be reduced from the outset. As result, the total number of retained tokens also decreases because many regions lose their distinguishing features. This phenomenon implies that the speed-up up gain could be higher during training or tinettnnng-- when augmentations are applied repeatedly than during inference.
Qualitative Results. APT generalizes effectively to downstream visual tasks that require spatial precision, including object detection and semantic segmentation. As illustrated Figure 13 and Figure 14, APT reliably allocates larger patches to uniform background regions while preserving fine-grained structures with smaller patches around object boundaries and textured areas. Each results support accurate bounding box regression and maintain the pixel-level fidelity necessary for segmentation, demonstrating that APT can deliver significant computational savings without compromising spatial detail task performance.
21

---

Published as a conference paper at ICLR 2026
Original Image
Augmented Image
Original Image
Augmented Image
Figure 12:
Augmentation visualization. We observe that augmentations generally lead to fewer tokens. In particular, Random Erasing (Zhong et al., 2020), leads to regions that can be tokenized with the large patch sizes, significantly increasing throughput compared to inference time.
22

---

Published as a conference paper at ICLR 2026
Vanilla
APT
CARLO NTE ALOO
AREENTE
Vanilla
ATTTARLO AE- ARLO W
APT
Vanilla
APT
Vanilla
APT
Figure 13: Object Detection Examples. First and third columns show the adaptive patch lay- outs produced by APT, where larger patches correspond to more homogeneous regions and smaller patches capture high-frequency object details. Second and fourth columns show the corresponding object detection outputs, demonstrating that APT preserves essential features for accurate bounding box prediction despite reducing the number of tokens. Images are best viewed zoomed in.
23

---

Published as a conference paper at ICLR 2026
Vanilla
APT
Vanilla
APT
Vanilla
APT
Vanilla
APT
Figure 14: Semantic Segmentation Examples. Left and third columns visualize the adaptive patch assignments generated by APT, illustrating how fine-grain me--grine regions (e.g., object boundaries) re- ceive smaller patches. Right and fourth columns display the resulting segmentation maps, showing that pixel-level details are preserved sufficiently for dense prediction tasks, even under token reduc- tion. Images are best viewed zoomed in.
24


Reviews:
[{"review_id": "NoaeJHhcgn", "reviewer": "Reviewer_sF7q", "summary": "The manuscript proposes Adaptive Patch Transformers (APT) considering using multiple different patch sizes within the same image processed by a Vision Transformer (ViT). Larger patch sizes are allocated in more homogeneous areas while smaller patches are allocated to more complex ones. The proposed approach accelerates the ViT with 40%.\n\nEntropy is used as a measure of a patch’s compressibility with lower entropy indicates higher redundancy.\n\nPatch aggregation is also employed aggregating embeddings from sub-patches back to size \n\nToken merging approaches is done at input-level. Input-level merging reduces tokens directly from image patches before entering the model. The method is also compared to Layer-level token merging.", "questions": "How is the adaptive patch sizes work with other transformer models, such as for example the Swin transformer.\nZ. Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV 2021\n\nCan other methods be used of detecting complex regions for using smaller patches?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* A more efficient and adaptive transformer model is proposed, where the adaptation refers to the fact that complex information is processed in more detail with smaller patches. Meanwhile less complex regions are processed with larger patches.\n* Extensive experimental results are provided.", "weaknesses": "* Lack of theoretical analysis\n* Lack of computational analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GPydZbsXcS", "reviewer": "Reviewer_sSUC", "summary": "This paper proposes Adaptive Patch Transformers (APT) , which accelerates Vision Transformers (ViTs) by replacing uniform patch splitting with content-aware, multi-granularity patching   based on entropy calculation . The method utilizes a Resize + ZeroMLP mechanism to fuse features from different scales into a unified embedding space, significantly reducing the input token count . The key contributions include achieving drastic throughput speedup (up to 50% on ViT-H) while maintaining accuracy across classification and dense prediction tasks, and ensuring   fast, stable adaptation to fine-tuned models via its zero-initialized fusion layer.", "questions": "Refer to weakness. If these concerns are well addressed, I will raise the rating to a positive one.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Strong Experimental Validation: The paper features a comprehensive set of ablation studies   across various tasks, effectively demonstrating the efficacy of the proposed mechanisms.\n2. Significant Efficiency and Generalization: APT delivers substantial throughput improvements (up to 50% on ViT-H) on large models, exhibits fast convergence (1 epoch fine-tuning), and shows   robust generalization   across classification and dense prediction benchmarks.\n3. Clarity and Presentation: The paper is well-structured, and the figures are high-quality.", "weaknesses": "1. Missing Similar Methods Comparison: The evaluation is incomplete because it fails to include a direct comparison against methods addressing the same task, such as MG-ViT[1] and PPT[2]. A detailed analysis of the methodological and empirical differences among APT, MG-ViT and PPT would substantially strengthen the paper.\n\n2. Hyperparameter Dependency: The performance is sensitive to the entropy threshold , which appears to be a manually tuned hyperparameter . This dependency might complicate achieving optimal efficiency across different downstream tasks, as the definition of \"salient information\" can vary significantly between tasks.\n\n3. In object detection tasks, does the use of entropy to determine patch size risk ignoring subtle object boundaries? Entropy measures pixel intensity distribution diversity, which may not perfectly align with semantically critical edges, especially when compared to gradient-based measures.\n\n4. For higher resolution images, which naturally result in a larger number of base patches, could the authors explore further patch fusion/aggregation operations after the initial adaptive patching step, particularly when several adjacent low-entropy patches exhibit similar entropy scores\n\n5. When patch size change, should entropy threshold change? That means one size patch correspond to one entropy threshold, and different size patch correspond to different entropy threshold.\n\n[1]Zhang Y, Liu Y, Miao D, et al. MG-ViT: a multi-granularity method for compact and efficient vision transformers[J]. Advances in Neural Information Processing Systems, 2023, 36: 69328-69347.\n\n[2]Wu, Xinjian, et al. \"Ppt: Token pruning and pooling for efficient vision transformers.\" arXiv preprint arXiv:2310.01812 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "upUGvbsCTO", "reviewer": "Reviewer_u6bT", "summary": "This paper introduces an adaptive patching (tokenization) method for ViTs. The main idea is to measure the entropy (how variable the pixel values are) of local regions of images and use finer (smaller) patches to tokenize the regions with higher variation. The patches of different sizes are resized (and split) to the same size for embedding. It reduces the number of tokens compared to the ViT using the same-sized patches for the whole image. This method is adapted to different ViTs and used in different vision tasks such as ImageNet classification, VQA, and object detection. The method generally shows better accuracy-effeciency trade-off than the original ViTs and some efficient ViTs.", "questions": "- For dynamic input size (seciton 3.3), you concatenate the tokens of a batch of images into a single sequence and use block attention. Why not padding the sequence of each image into the same length?\n- In the Input-level Merging Baselines, what do you mean by saying Resizing represents a stronger version of Quadformer? It seems that Resizing is a variant of APT by removing the zero initialized layer, and is not really related to Quadformer. Similar question for \"Random\". Clearer explantion is needed.\n- For the dense prediction tasks (section 4.3), you mentioned training only the newly added component. Does it mean only training the conv layers and ZeroMLP as in Fig 3? For example, for LLaVa, the language model and the projection layer are frozen. Is that correct?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well written overall; the adaptive patching idea (larger patches for low-entropy regions, smaller for high-entropy regions) is intuitive and easy to follow. The entropy formulation and hierarchical quadtree patchification are clearly described, with alternatives noted for the appendix.\n- The zero-initialized MLP lets the model incorporate high-res details without hurting initialization, enabling quick convergence from existing ViTs.\n- The proposed method APT plugs into several ViT backbones and tasks, including classification, VQA, detection, and segmentation. It also works with window attention (e.g., EVA/ViTDet).\n- APT reported 40–50% throughput gains on large models/resolutions while matching accuracy, and also some speedups on dense tasks, achieving better accuracy-effeciency trade-off than several well-known efficient ViTs such as EViT and ToMe.\n- The paper re-implements layer-level merging baselines with FlashAttention for a fairer comparison (and shows APT outperforms across compute budgets).", "weaknesses": "- Table 3 shows +22–26% throughput on LLaVA-1.5 (7B/13B), with some metrics slightly down (e.g., VQAv2 −0.6 for 13B) and others on par or up; overall the Pareto looks close but not strictly better across all benchmarks. Table 4 shows +14–30% throughput on detection with essentially unchanged mAP/AP50. This is positive, but the improvements are less decisive than in classification and would benefit from a Pareto plot analogous to Fig. 4 for these tasks.\n- Currently APT uses entropy to measure the variation of pixels in image regions. It would be beneficial to add ablations for different measures, such as standard deviation of the pixels and local frequency (e.g., DCT-band energy).\n- The writing of the experimental setup can be clearer, specifically on the difference between Full Fine-Tuning and Short Fine-Tuning (Section 4.2). Sometime it could be a bit confusing as to what is the pre-trained MAE; is it one only trained with masked autoencoding or one with both masked autoencoding and classification training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sKo1BrsHDP", "reviewer": "Reviewer_W17G", "summary": "The paper proposes the Adaptive Patch Transformer (APT): it adaptively employs multiple patch sizes within the same image based on content—using larger patches for flat/redundant regions and smaller patches for detail-rich areas, thereby reducing input tokens and boosting throughput. Core Implementation: Multi-scale histogram entropy (Eq.(1)) serves as compressibility metric, while hierarchical thresholds (ω) determine whether to “stop” at a layer or continue subdivision. For large patches, dual-path information is simultaneously utilized: “Embedding resized to base dimensions + sub-patch embeddings aggregated via Conv,” fused through a zero-initialized MLP. Inference/training side: “Sequence packing + block-diagonal mask” accommodates variable sequence lengths, accelerated by FlashAttention. Experiments show: Full ImageNet and 1-epoch fine-tuning achieve 20%–90% throughput gains while maintaining accuracy, with accelerated performance on LLaVA's VQA, COCO detection, and ADE20K segmentation. Authors also report: Without token reduction, APT incurs non-zero overhead; zero-initialized connections deliver the most stable “plug-and-play” fine-tuning convergence.", "questions": "1. How are the hierarchical thresholds (ωᵢ) determined for different datasets or tasks? Are they manually tuned or selected automatically?\n\n2. How is the histogram entropy computed — what bin settings and value ranges are used? Have other texture or frequency-based criteria been compared?\n\n3. What is the preprocessing overhead of multi-scale entropy computation and token packing on CPU and GPU? Are these times included in the reported runtime measurements?\n\n4. Since token counts vary across samples, what is the distribution of sequence lengths, and how does this variation affect throughput and latency?\n\n5. Can the object detection results be further analyzed by object size (small, medium, large) to better understand performance on fine-grained details?\n\n6. Does the “repeat 2^{2i}” reconstruction step cause aliasing or checkerboard artifacts? Has any boundary or contour accuracy been evaluated to confirm visual quality?\n\n7. How are positional embeddings handled for variable patch sizes and packed sequences? Is there any scale-aware interpolation or adjustment applied?\n\n8. What is the parameter and memory overhead introduced by the zero-initialized MLP fusion layer, and how stable is it during longer fine-tuning?\n\n9. For baselines adapted to support FlashAttention, can the authors provide a detailed list of implementation changes and verify that all models were tested under identical conditions?\n\n10. Does the aggressive merging of smooth regions lead to errors when fine-grained or background details are required for prediction? Has any failure case analysis been performed?\n\n**If you address my concerns, I will consider raising my score.**", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Comparison tables are provided across four task categories—classification, VQA, detection, and segmentation—clearly demonstrating how variable tokens are transformed into regular feature maps for dense prediction tasks.\n\n2. Key methodological formulas and structural diagrams are presented, along with component ablation studies (zero initialization vs. non-zero/residual; system overhead without compression) to facilitate reproducibility and pinpoint sources of improvement.", "weaknesses": "1. The adaptive patch size relies on a hierarchical entropy threshold (Eq. 1, §3.1), which is fixed and manually tuned for each scale. The paper gives no data-driven method to set these thresholds, and poor choices can cause information loss or accuracy drops.\n\n2. To use FlashAttention, some baselines were re-implemented with modifications like disabling weighted attention (§4.1). These changes may alter results, so runtime and throughput comparisons might not be fair without full implementation details or code release.\n\n3. The reconstruction method (§3.3) repeats large-patch features to form dense grids, which may create block artifacts and hurt small-object accuracy. The paper reports only overall mAP/mIoU and shows no failure examples, leaving fine-detail loss unverified.\n\n4. Table 6 shows that APT is slower than the baseline when no compression is applied, suggesting preprocessing overhead from entropy computation and token packing. The paper omits CPU/GPU timing breakdowns, and speed gains are smaller than FLOPs reductions, implying memory or pipeline bottlenecks.\n\n5. The method mainly benefits high-resolution, large-model settings. Key parameters (thresholds ω, binning strategy, search range) are missing, and code is unreleased, making reproduction and fair comparison difficult.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)