Thicker and Quicker: The Jumbo Token for Fast Plain Vision Transformers
Authors: Anthony Fuller
Yousef Yassin
Daniel Kyrollos
Evan Shelhamer
James R Green
Venue: ICLR 2026 Poster (2026)
Keywords: efficient deep learning
computer vision
vision transformers
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We add a new wider "Jumbo" token to ViTs to improve accuracy and efficiency by adding model capacity in just the right way while preserving generality.

Abstract: ViTs are general and accurate, and address many tasks, but ViTs are slow, and are not always practical when efficiency is key. Existing methods for faster ViTs design hybrid non-ViT architectures, losing generality, or shrink their tokens, sacrificing accuracy. Many non-ViT architectures are both fast and accurate. Yet, without significant modifications, they cannot do what ViTs can: process other input shapes, pre-train by SOTA self-supervised learning, reduce computation by dropping tokens, and more. We make ViTs faster by reducing patch token width while increasing global token width by adding a new Jumbo token. Our wider Jumbo token is processed by its own wider FFN to increase model capacity. Yet our Jumbo FFN is efficient: it processes a single token, for speed, and its parameters are shared across all layers, for memory. Crucially, our Jumbo is attention-only and non-hierarchical, like a plain ViT, so it is simple, scalable, flexible, and compatible with ViT methods new and old. Jumbo improves over ViT baselines with Registers from Nano to Large scales while maintaining speed/throughput on ImageNet-1K (0.1-13%). Jumbo also improves segmentation (1.9-3.1% on ADE20K), MAE pre-training (4.9% linear probing on ImageNet-1K), test-time adaptation (5.2% on ImageNet-C), and time series modeling. Our Jumbo models even achieve better speed-accuracy trade-offs than specialized non-ViT compute-efficient models, while maintaining plain-ViT compatibility for practicality. Code and weights are available: https://github.com/antofuller/jumbo

THICKER AND QUICKER: A JUMBO TOKEN FOR FAST PLAIN VISION TRANSFORMERS
Anthony Fuller!.3 Yousef Yassin', Daniel G. Kyrollos', Evan Shelhamer 12,3* Carleton University', University of British Columbia?, Vector Institute equal advising
,James R. Green
ABSTRACT
ViTs general and accurate, and address many tasks, but ViTs slow, and are not always practical when efficiency is key. Existing methods for faster ViTs design hybrid non-ViT architectures, losing generality, sacrificing accuracy. Many non-ViT architectures are both fast and accurate. Yet, without significant modifications, they cannot do what VITs can: process other input shapes, pre-train by SOTA self-supervised learning, reduce computation by dropping tokens, and more. We make ViTs faster by reducing patch token width while increasing global token width by adding a new Jumbo token. Our wider Jumbo token is processed by its own wider FFN to increase model capacity. Yet our Jumbo FFN is efficient: it processes single token, for speed, and its parameters are shared across all layers, for memory. Crucially, our Jumbo atlention-only and non non-hierarchical, like plain ViT. it is simple, scalable, flexible, and compatible with ViT methods new and old. Jumbo improves over ViT baselines with Registers from Nano to Large scales while maintaining speed/throughppu ImageNet-IK
shrink their tokens,
Jumbo also improves segmentation (11.9-3.0 3.1% on ADE20K), MAE pre-training (14.9% linear probing ImageNet-1K] test-time adaptation 5.5% on ImageNet-C), and time series modeling, Our Jumbo models even achieve better speed-accuracy trade-offss than specialized non-ViT compute- efficient models, while maintaining plain-Vil compatibility for practicality, Code and weights are available: hattps://githie com/ www.////noollee//iniioo..
INTRODUCTION: ARCHITECTURE, ACCURACY, AND EFFICIENCY
For most model sizes, the vision transformer (VIT; Dosovitskiy et al. (02T is the go-to architecture in computer vision viston-powering foundation models like DINOv2 (Qquab et al.] 2024 language- aligned models like CLIP (Radford et al.) (0211), segmentation models like SAM (Kirillvv et al.)
2023 3D vision models like DUST3R (Wang et al. 20244, and diffusion models like DiT (Peebles) & Xie 2022). These are all "plain" ViTs, which are crucially attention-only and non-hiirarchiall.
At the smallest scales-offering the highest speedss/hrrughhus-ppha ViTs competitive with highly specialized architectures (Yun & Ro 2024). We attribute the worse accuracy-speed of plain ViTs their width (number of channels). Existing work scales width equally across all tokens and layers so higher speed requires lower width: ViT-Base (768) + ViT-Small (384) + ViT-Tiny (192).
We scale width differently across tokens and equally across layers. Our architecture adds Jumbo token, which replaces the conventional CLS token, that is Jx wider than the patch tokens, with its own wider feed-forward network (FFN), effectively and efficiently boost model capacity. For self-attention, the Jumbo token is split into Jx as many tokens/heads, but the Jumbo FFN is only applied to the one (merged) token reduce time and shared across layers to reduce memory. Jumbo keeps the defining traits of plain WTT-aatention-only and non-bierarchiaal-so Jumbo applies anywhere a plain ViT does but at higher speed.
The simplicity of ViTs is due to their attention-only and non-hierarchical architecture. Multiple uses of ViTs rely on this architectural "interface" for their computation and function. For instance, this interface enables efficient sparse computation through masking/token dropping. Random token dropping enables efficient training (Liu et al. 2023 Dehghani et al.) 2024 [erry et al. 2024) and learned token dropping enables efficient deployment (Bolya et al. 2023 Fuller et al. 2025 Several

---

(a) ImageNet- Val
(b) 21K
47
ViT+Jumbo (ours)
Jumbo (ours)
Top-1 Ace. (%)
80
Top-1 Acc. (%)
46
Bixx
45
75
70
Veg+Registere
MobileNet SHVI
44
Vegiiegsss
1.9x faster
43
65
Efficient
42
60
3
10
20
30
40
3
4
6
8
Throughput (K images/s)
Throughput (K images/s)
(c) ImageNet-v2
Attention- only
Non-
75
Legend
Method
hierarchical
Top-1 Acc. (%)
70
+umbo fours)
EfficientViT
SHViT
65
BiX
MobileNerV4
ViT+Registers
60
MobileNer SHVI
BiXT
JiT+J (ours)
55
ViT+Reg
Our Jumbo is attention-only and lon-hierarchical, which provides out-of-the-box support for SOTA self-supervised learning algorithms, multimodal data, and non-2D data -whle being much simpler than existing efficient architectures.
50
3
10
20
30
40
Throughput (K images/s)
Figure 1: Plain ViTs are in red and others are in blue ViT+Jumbo outperforms SOTA compute- efficient architectures while maintaining the advantages of plain ViTs. ViT+Jumbo outperforms ViT+Registers on ImageNet- IK and the more challenging ImageNet-21KK dataset. Throughput is measured on an RTX 4090 GPU using Py PyTocch 2.6.0, torch. comp e, and 512 batch size.
SOTA self-supervised learning (SSL) algorithms require token dropping for learning He et al. 2022
Garrido et al.] 2024 Wei al., 2025 Venkataramanan et al. 2025). This same interface enables
flexible processing of different input shapes, like time series (Nie et al. 2023) or video (Arnab et al.
2021p. Moreover, many extensions and applications- object detection and segmentation heads
Fang et al. 2023 Zhang et al. 2022) test-time adaptation algorithms (Niu et al. 2023 are
designed for this plain ViT interface. Architectures that maintain ViT compatibility inherit all this.
Our experiments show that Jumbo improves speed-accuracy performance across tasks, datasets, and modalities. O Image classification: Jumbo outperforms ViTs with Registers (Darcet et al.) 2024 by 0.1-13% ImageNet-1K K and 1 3.1% ImageNet-21 while maintaining throughpuf and achieves the pareto frontier vs. compute-efficee efficcien architectures. ? Segmentation: Jumbo gains 1.9-3.1% on ADE20K (Zhou al. 2017 using a standard segmentation head. O Self-Supervised Learning (SSL): Jumbo improves MAE (He et al. (2022) pre-training measured with linear probing by 4.9% ImageNet-IK at ViT-Base scale- this ViT-Base+Jumbo ties the ViT-Large baseline, with 2.3x fewer parameters, 3.5x fewer FLOPs, and 3.1 x higher throughput. O Test-time adaptation (TTA): Jumbo is more accurate and more robust with 5.2% improvement on ImageNet-C C using SOTA adaptation method for transformers (SAR (Niu et al. 2023)). Time series: Jumbo generalizes beyond vision to rank first across 20 time series benchmarks vs. transformer baselines.
Jumbo is such an efficient ViT-compatible architecture that it outperforms highly specialized existing architectures on ImageNet- 1K (Fig. [ This is notable because such compute-efficient architectures
(Chen et al. 2023 Howard et al. 2017) sacrifice generality and compatibility with other techniques and applications. Even efficient architectures based on ViTs include convolutions, hierarchy, and batch normalization (Yun & Ro 2024 Vasu al. 2023b Cai al. 2023) that make them incompatible out of the box with SSL by MAE, TTA by SAR, time series, VIT heads, etc. Jumbo delivers compute efficiency while maintaining plain- VIT compatibility.
2

---

2 BACKGROUND AND RELATED WORK: GENERALISTS AND SPECIALISTS
2.1 VISION TRANSFORMERS, SIMPLE, FLEXIBLE, BUT NOT YET FAST
Jumbo extends ViTs. A VIT splits an image into a patch grid, R xxxXxC -
where C is the number of channels, Y / X are the image height width, / /N2 are the grid height / width, and py / Pa are the patch height / width in pixels (equal to Y Ny x)), Next, they flatten the grid into sequence and flatten the patches into vectors, C.N,NN.XE XP,xxxxxxxxxCX + RNxx,i,, where N is the number of patches (equal to N2 Na), and Dpia is the number of pixel values per patch (equal to Py Pa -C). Next, they apply learnable linear projection form patch embeddings, RNxD2 gD1 TR. ND. where D is the token width, also known the embedding dimension. Next, they add position embeddings to patch embeddings. These operations produce patch tokens x" E 12 x that represent local information typicalll 16 x 16 px square. Crucially for us, ViTs prepend a learnable CLS token x to the sequence of patch tokens, x
x 1/ox E IR (1)+) where llo denotes concatenation along the t (sequence) dimension. Finally, the input x is processed by a plain transformer and the CLS token, having attended to all other tokens, can serve as the global representation of the image. ViT sizes vary w.r.t. depth and width. ViT-Large has 24 layers, others have 12 layers, while the widths vary {96, 128, 192, 384, 768, 1024]. corresponding to names (Pico, Nano, Tiny, Small, Base, Large}. Narrower ViTs require less computation and are thus faster.
standard image size of 224 x 224 px and a standard patch size of 16 x 16 px result in 196 local tokens. A single CLS token-desipee designed to aggregate global information for classification - 1 1/197th of a model's 's representational capacity to global information (and this fraction decreases with larger images and/or smaller patches). This allocation is imbalanced, and may not be optimal. Recent work finds evidence to support this intuition and proposes a fix: register tokens.
Registers. Darcet et al. (2024) find that ViTs learn to repurpose some patch tokens to behave like additional CLS tokens by collecting global information and discarding patch-specific local information. The same work proposes fix: prepend extra learnable tokens called registers
xeg E R R2 where R is the number of registers to the input sequence, = x | Reg 1x' E
RR +N++)))) Registers improve accuracy (by ~0.4% on ImageNet- IK (Russakovsky et al. 2015
ViT-Base) and reduce attention map artifacts/noise by provisioning more global capacity.
Registers are elegant, simple, and keep the plain ViT interface. In theory, registers can benefit any plain, non-causal transformer. These advantages account for registers' significant and immediate
impact including in applications beyond images (Dong et al.] [024 Vaquero et al.] [2024] Leigh et al.
2024 Messaoud et al. 2025 Hu et al. 2024 Tiimonier al. [024]] Omranpour al. 2024). Our
Jumbo is inspired by VeT+Registers: see Fig. for their relationship and key differences.
2.2
COMPUTE-EFFICIES ARCHITECTURES FAST, BUT NOT SIMPLE NOR FLEXIBLE
The Jumbo architecture is accurate and compute-effiiient so we highlight 3 architectures and use them as baselines for high-speed ViTs. EfficientViT' (Cai et al. 2023) and @ SHViT YYun & Ro 2024) improve the efficiency of ViT's by incorporating efficient attention, pooling, and convolutional layers. 4 MobileNetV4 (Qin al. 2025)) improves the efficiency of CNNs by leveraging many strategies (and different strategies for different model sizes). These baselines represent the SOTA in computational efficiency; please refer Appendix A.2 for descriptions of these model architectures.
Beyond these, there is a rich literature on compute-efficient vision architectures. For example, several efficient -bNed architectures exist (Howard (2017] [Sandler al., 2018 Howwrd al. 2019
Han et al. 2020 Tan et al.] 2019 Vasu et al. (023a); however, these are surpassed by MobileNetVA
(Qin et al.
2025). Since the invention of the VIT, there have been many compute-efficient "ViT'" that
incorporate e efficiencies inspired by CNN-based approaches (Vasu et al. 2023b Mehta & Rastegari
2021 2022 Li et al. 2023) Pan et al., 2002 Chen et al. 2022; Li et al. 2022). SHViT (Yun & Ro
2024 has recently surpassed these architectures. Despite their impact and ingenuity, none of these hybrid architectures meets the definition of a plain ViT, which is attention-only and -in-hiirarchical; they thus lose many advantages of ViTs that wish to keep. On the other hand, BiXT (Hiller et al.) 2024] models are an efficient extension of the Perceiver architecture (Jaegle et al. 2021 y that keeps the attention-only and non-hierarchical properties of ViTs, which are natural comparison to Jumbo.
3

---

ViT+Jumbo (ours)
RiT+Regsstess Darcet et al.
2024
Jumbo FFN
FFN
FFN
FFN
FFN
FFN
FFN
FFN
FFN
FFN
FFN
FFN
Layers
Llayers
concat
MHSA
MHSA
split
Jumbo
CLS
Regi
Reg
global
(image patches)
global
tokens (image patches)
Figure 2: (Lefft) Our ViT+Jumbo method creates wide global token that gets split into several tokens, with width equal to the patch width, prior to multi-header self-attention (MHSA). After attention, the split Jumbo token is reassembled via concatenation, and is then processed by its own FFN. Patches are processed by their own, shared FFN. (Right) ViT+Regssters creates register tokens all equal to the patch width and all tokens are processed by a shared FFN. ViT+Jumbo enhances global processing as the (split) global tokens can interact via an expressive FFN, plus attention.
METHOD: JUMBO TOKEN FOR A COMPUTE--FFICIENT PLAIN VIT
3.1 DESIGN MOTIVATION AND INTUITION
Capacity and Cost. Although Jumbo adds wider token and FFN, the cost is minimal. The key insight is that a single wide token affords much greater width and more processing with- out slower speed. As shown Fig. the main drivers of computational cost (FLOPs per layer) are sequence length and patch width, D. The FLOP contribution from our Jumbo token is comparatively negligible. Since our architec- ture shares Jumbo FFN parameters across all ViT layers, its memory costs are also minimal.
FLOPs Per Layer (x100
VIT
ViT-hase, D=768
ViT+Jumbo
0.5
VIT-small, D=3884
ViT-tiny. D=192
ViT-nano, D=128
200
400
600
800
1,000
Non-hierarchical and attention-only. Jumbo preserves the oon-hierarchial shape of ViTs (also known columnar isotropic shape). By foregoing convolutions, spatial information only moves through attention. These two properties have several ad advantages that we now discuss.
Patches (sequence length)
The cost of layers is largely determined by the number patches and their width D. The cost of our Jumbo token (J=6) is negligible.
Figure 3:
Token Dropping Masking, Although convolutions are capable of processing a sparse subset of patches via sparse compute kernels, these kernels can be complex, challenging to use, and require updating when new hardware arrives. Furthermore, sparse convolutional kernels will never be as efficient as simply indexing from sequence i.... how transformers drop tokens. As comparison, ConvNeXt V2 Woo al. 2203) reports a 1.3x speedup using a 60% masking ratio with the Minkowski Engine v0.5.4 (Choy et al. 2019). Conversely, MAE (He et al. [002] report 2.8 - 4.1x speedups using 75% masking ratio with plain ViTs. Efficient token dropping is required for
SOTA SSL algorithms (Assran al. 2023 Fu al.) 22244: Garrido et al.) 20244 Wei et al.) 2025
Venkataramanan et al. Qquab et al. 2024). Token dropping also speeds up supervised training
(Dehghani et al.) 2024).
We demonstrate Jumbo's token dropping ability in subsections 4.2 and 4.4
Other Data Modalities and Shapes. These properties explain the input flexibility transformers, which Jumbo keeps. For example, users need only adjust tokenization strategies for 1D time series, 3D point clouds, or multimodal data. We show a 1D time series application of Jumbo in subsection 4.6 and preliminary application to vision-language data in subsection 4.7
4

---

Plain ViT's Ecosystem. These two properties nonnhierarchical and attention- only-maintain support for methods invented for the plain ViT. For example, segmentation and object detection heads
(Fng et al. 2023 Liu et al. [025 Zhang et al. 2022p, which expect ViT's unpooled feature map;
lest-time adaptation methods (Niu et al. 2023). designed for the LayerNorm (Ba et al.) 2016) not
BatchNorm (loffe & Szegedy 2015 and attention improvements, such Flash Attention (Dao et al.)
2022 which can speed up self-attention by >5x. Jumbo supports these innovations out of the box.
Crucially, none of the compute-efficient architectures in subsection 2.2 immediately benefit from these advances, support other data modalities or token dropping, or integrate with the ViT ecosystem.
Two hypotheses. Jumbo asymmetrically increases the model capacity. Thus, we expect increasing gains due to Jumbo with decreasing patch token width. @ We expect increasing gains due to Jumbo with increasing task output dimensionality. We explore both of these hypotheses using experiments with ViTs of different widths and datasets of different complexities.
3.2 DESIGN SPECIFICS FOR TOKEN- ASYMMETRY
Exactly like the original ViT, Jumbo computes patch embeddings. xP E NxD Unlike the original ViT, our method creates Jumbo token that is J times wider than the patch width D, Jumbo E R.D.D Architecturally identical transformer layers then process these inputs.
Before self-attention, the Jumbo token is split into J tokens, | dumbo RIIJ.D + RJDD, where I denotes splitting into J segments along the 1st (feature) dimension. Next, the split Jumbo token is concatenated with patch embeddings along the sequence dimension, x x x lox E R (DD)) This sequence sent through a plain multi-headed self-attention layer. Afterward, the Jumbo token extracted from the sequence by splitting along the sequence dimension, 12 x (N++)xD (IR (R Ig. [B-VxD), XD), where the first element contains the (still split) Jumbo token and the second element contains the patch representations. Finally, the Jumbo token reassembled through concatenation along the channel dimension, x = -11 x JxD - R IxX--, These two splits and two concatenations add negligible runtime overhead.
After self- attentntnnion, the Jumbo token is processed by its own FFN that does not share parameters with the patch FFN. Fig. indicates this by coloring the Jumbo and patch FFNs differently. After processing by all layers, we project the Jumbo token to C' class logits, JDD - RC
Layer sharing. We share our Jumbo FFN parameters across all layers to reduce memory use (through fewer model parameters). All other model parameters are not shared across layers, as usual. Sharing also acts as regularization. Empirically, we find sharing keeps (and sometimes increases) Jumbo's accuracy gains compared with not layer sharing-whin while effectively controlling memory use. Sharing the FFN layer is thus the default in our Jumbo architecture.
4 EXPERIMENTS: ACCURACY, COMPUTE EFFICIENCY, AND GENERALITY
For all experiments, we measure throughput on an RTX 4090 GPU using PyTorch 2.6.0, torch. compile, and 512 batch size.
4.1 IMAGENET-IK K EXPERIMENTS WITH COMPUTE-EFTICIEN BASELINES
Setup. We perform controlled experiments to evaluate Jumbo. Specifically, we train models from scratch on ImageNet-Il IK (Russakovsky et al. 2013) at 128x px for 400 epochs, then for 20 epochs at x24 224 px. We leverage distillation to improve convergence, which is a common strategy.
We train each model architecture twice, once for each learning rate (1e-3, 3e-3) (Touvron et al. 2022: Yun & Ro 2024 using 1024 batch size with the AdamW optimizer (Loshchilov 2017). We report the results of the best learning rate for each model architecture. Please see Appendix A. A.4.1e & A.5 for hyperparameters and complete results, respectively.
Baselines. We choose the high-speed models for each family: ViT+Registers NNano, Tiny, Small,
Base) (Dareet et al.
2024 BiXT has 1 size (Tiny), O EfficientViT (BO, B1) (Cai et al. 2023). O Cony-Medhum, Hybrid- We compare these architectures with high-speed ViT+Jumbo variants
SHVIT S2, S3
Yun & Ro 2024), and ① MobileNetV4
Medium) Qin et al.
5

---

ImageNet-Real
ImageNet-HR
ImageNet-R
90
Top- Acc. (%)
1 11%
+ 15%
85
35
+ +11%
80
30
75
75
25
70
70
20
10
20
30
40
10
20
30
40
10
20
30
40
Throughput (K images/s)
Throughput (K images/s)
Throughput (K images/s)
Legend: ViT+Jumbo (ours), -eg+Regsss @BXX,, MobileeeeV4, SHViT, EfficientViT
Figure 4: ViT+Jumbo achieves the Pareto frontier and is much simpler than specialized compute- efficient architectures. Results are plotted for each model's best learning rate. Throughput measured on an RTX 4090 GPU using PyTorch 2.6.0, torch. compi le, and a 512 batch size.
(Pico, Nano, Tiny, Small, Base]. Darcet ct al. (2224) show ViT+Registers with R=16 performs best, which we confirm in the appendix Table 7 and in these experiments. ViT+Jumbo robust to the choice of J; we use J=3 for Base and J=6 otherwise.
Test Sets. We test all models the three most common ImageNet- 1K test sets: ImageNet- Val (Russakovsky et al.] 2015). ImageNet-Rest (Beyer et al.) 2020). ) and ImageNet-v2 (Recht et al. 2019). To further evaluate generalization also test all models on ImageNet-HR (Fuller al. 2024), for its image diversity and high-quality annotations, and ImageNet-R (Hendrycks et al. 2021 for its dittr--distiibution images.
Results. As illustrated in Fig 4 Jumbo achieves the Pareto frontier on ImageNet-IK Crucially, Jumbo achieves these results while preserving the many advantages and simplicity of plain ViTs. Even matching the specialized compute-efficient architectures makes a strong case for ViT+Jumbo.
ViT+Jumbo outperforms +eg+seiss by 13% at the nano scale and 4% the tiny scale, where such gains are significant. This confirms our first hypothesis that Jumbo's 's gains should increase as we decrease the patch width, i.c., from Small (384) to Tiny (192) to Nano (128) (Figs. la
ViT+Jumbo is clear choice if a researcher or practitioner requires high speed and out-of-the-bue ViT compatibility for SSL algorithms or multimodal processing. ViT+Registers is not as accurate at high speed, while the specialized compute-efficient architectures do not support most SOTA SSL algorithms or flexible processing across modalities. Remote sensing (Rolf et al.) 2024 and autonomous driving (Muhammad et al. 2020) are two of many applications where this combination of speed, SSL. support, and multimodal processing particularly valuable.
4.2 IMAGENET-2IK EXPERIMENTS WITH VIT COMPARISONS
ImageNet-IK is a subset of the more challenging, original ImageNet (Deng al. 2009), now referred to as Iea We use common variant comprising 10,450 classes that includes processing to make a more accessible benchmark [Ridnik et al.) [02T). This dataset provides more than 10x the number of classes and samples ImageNet-IK, making well suited to test our second hypothesis- that is, gains due to Jumbo should increase with increasing task-output dimensionality.
Setup. We train models from scratch on ImageNet-21K.. Since training models on ImageNet-21K is expensive, we leverage a token dropping strategy to reduce costs. Specifically, we start training with 90% token drop rate and linearly decrease this value to 10%; this halves the total number of tokens processed. Dehghani et al. (2024) demonstrate the effectiveness of this strategy, i.c., leveraging "masking" with plain supervised training. Plain ViTs support masking with minimal code changes.

---

We train each model architecture once, for 50 epochs using 3e-3 learning rate and 1024 batch size with the AdamW optimizer [Loshchilov] 2017) (see the Appendix [A.4. for other hyperparameters).
Baselines. We choose ViT+Registers (Small, Base) to compare with our ViT+Jumbo (Small, Base) sizes. This a more narrow but valid comparison, as the plain VIT is the vision community's preferred architecture at these scales. For ViT+Regssters, use R=16; for ViT+Jumbo, use J=6 for the Small model and J=3 for the Base model.
Results. ViT+Jumbo outperforms ViT+Registers by 3.1% and 1.2% ViT-Small and ViT- Base scales, respectively (see Fig. [b.. Gains due to Jumbo increase when scaling from ImageNet- to ImageNet-21 K for given model size, e.g., ViT-Small gains increase from 0.8% (Fig. Ta to 3.1 1.. Thus, these findings confirm our second hypothesis that gains due to Jumbo should increase with increasing output dimensionality. Furthermore, for given accuracy Jumbo is 1.9x faster (Fig. lbp.
4.3 SEMANTIC SEGMENTATION EXPERIMENTS
Setup. We fine-tune on ADE20K Zhou et al, 2017) for 100 epochs with a batch size of 16 and sweep learning rates (le-5, 2c-5, 4e-5, 8c-5] J for each model size [ViT-Base, ViT-Small, ViT-Tiny and global strategies [Jumbo, Registers). dataset. We initialize model parameters from our ImageNet-IK supervised training experiments. We attach a Segmenter head (Strudel et al. 2021), which a commonly used head for ViTs.
use 512x512 px images, which is standard for this
Results. Our Jumbo achieves mloU of 44.4% / 39.1% 1% 35.5%. while Registers achieves mloU of 42.5% / 37.0% 42.4% for ViT-Base Small / Tiny sizes. This shows Jumbo supports segmentation since it non-hierarchical-aaa and gains when using standard plain- ViT segmentation head.
4.4 MASKED AUTOENCODING EXPERIMENTS
Setup. We pre-train Jumbo ViT-Base and ViT-Large models using masked autoencoding (MAE: (He et al. 2022)) using the default settings. This tests Jumbo's 's ability in a standard SSL framework. This also tests Jumbo's scalability to larger models (up to ViT-Large) and longer training schedules (up to 1600 epochs on ImageNet-1K: These experiments are expensive, so we leverage free TPU resources, perform no hyperparameter tuning, and compare against plain VIT results obtained with the same MAE implementation. After pre-training, we linear probe on ImageNet-IK to obtain accuracies.
Table 1: MAE Pretraining. Jumbo significantly outperforms standard ViT. Jumbo scales to these large MAE models and their long training sched- ules (1600 epochs for ViT-Base, 800 epochs for ViT-Large). After pretrain- ing, we linearly probe to compute top- 1 accuracy ImageNet-IK.
Speed K imgs/s
Params Memory FLOPs
Top- I Acc. %
Architecture
M
GB
G
ViT-Base
3.1
86.6
3,3
16.5
68.1 1
ViT-Base+Jumbee
3.1
130.7
3.9
16.9
73.0
ViT-Large
304.4
5.0
59.7
73.0
ViT- ViT-Lo
1.0
382.2
5.2
59.9
74.0
Results. Our ViT-Base+Jumbo MAE outperforms the baseline by 4.9% on ImageNet-IK. ViT- Base+Jumbo ties the ViT-Large MAE, while Jumbo is 3x faster with only 0.43x the parameters. This shows Jumbo can be applied SSL by MAE to improve performance without further modification. The role of masking in the MAE suggests that the wider Jumbo token stores more global information. For this MAE, Jumbo a more efficient way scale model parameters than the wider VIT.
4.5 ROBUSTNESS AND TEST-TIME ADAPTATION EXPERIMENTS
Setup. We measure robustness to corruption with and without adaptation. We follow SAR Niu et al. exactly, swapping in ViT-S models with Registers
2023
Jumbo from Sec. 4.1 and measure
robustness to 15 corruptions at the highest severity from ImageNet-C (Hendrycks & Dietterich 2019).
Results. Jumbo is both more accurate than Registers (+0.8% IN-Val) and more robust than Registers on corrupted data (+3.6% on IN-C). Test-time adaptation by SAR further increases the robustness gain to +5.2%. In principle test-time adaptation apply any architecture, but in
7

---

Table 2: Test-Time Adaptation (TTA). Jumbo improves plain- robustness without TTA (avg. 13.6%) and with TTA (avg. 15.2%) on ImageNet-C. We follow SAR (Niu et al. (203) and test across 15 shifts at the highest severity. Jumbo directly compatible with SOTA methods designed for ViTs, for instant use without tuning, unlike
architectures (MobileNet, SHViT, ...).
Method
Gauss. Shot Impul. Defoc. Glass Motion Zoom Snow Frost Fog Brit, Contr. Flastic Pixel JPEG Avg.
Registers
13.6
14.2
13,0
29.3
20.3
34.9
28.7
49.0
50.4
56.6 73.5
47.8
29.0
45.6
56.1
37.5
Jumbo
25.9
26.6
25.8
31.2
21.4
33.3
30.6
53.1
51.9
57.3 75.2
49.4
30.5
47.5
57.1
41.1
Registens+SAR
38.9
39.2
41.5
48.2
48.7
56.5
32.1
62.0
59.4
68.9
76.1
61.5
59.1
65.5
66.1
54.9
Jumbo+SAR
45.2
49.6
51.2
53.3
53.2
61.1
44,5
66.2
58.5
71.7
77.5
67.1
65.2
69.0
69.0
60.1
practice methods specialize. SOTA methods such as SAR are designed for the plain ViT LayerNorm, and not the BatchNorm of SHViT, MobileNerV4, and Efficient so ViT compatibility is a plus.
4.6 TIME SERIES EXPERIMENTS
Jumbo can easily process different input shapes (beyond images) because it maintains the plain transformer interface. We apply Jumbo to time series inputs. PatchTST (Nie et al. [023) is a SOTA
patch-based transformer for time series that we extend with registers (etishTTTRRegistess) or Jumbo (PatchTST+Jumbo)
10 univariate time series datasets from the UCR archive (Dau et al. 2018), and @ 10 multivariate time series datasets from the UEA archive (Bagnall et al.
Setup. We train models from scratch
2018: both of which commonly used benchmarks (Zervas et al. 2021: Grover et al. 2024] et al. 2024b. For each dataset and model, we perform a hyperparameter sweep from the Cartesian product of learning rate (3e-3, 1e-3, 3e-4, le-4), and dropout (0.0, 0.1, 0.2). More details are in
Appendix A.2. We report the best run and the average of all 12 runs per experiment in the appendix
Tables 12
To summarize these results, we compute the rank between models and then average the ranks over the 10 univariate and 10 multivariate datasets.
13
Baselines. We compare PatchTST with our PatchTST+Jumbo method and our PatchTST+Registee baseline. We experiment with 8 and 42 patches per sequence for all three models. Jumbo and registers are both simple to adopt for PatchTST because they remain plain transformers.
Table 3: Time series rankings using PatchTST (Nie al. 2023) with Registers or Jumbo (lower is better and the best bold). We rank over 10 univariate and 10 multivariate datasets. "Best" is the best run of our 12-run hyperparame- ter sweep and "Avg" the average over the sweep. Jumbo achieves the best ranking in all experiments. We use two patch sizes: 8/42 (results are formatted likewise).
PatehTST PatchTST +Registers +Jumbo
PatchTST
Best 2.0/1.9
2.5/2.1
1.5/1.7
Univar.
Avg 2.9/2.3
2.1/2.4
1.0/1.3
Best 2.1/2.0
2.1/1.9
1.6/1.7
Multivar.
Avg 2.7/2.6
2.0/2.4
1.3/1.0
Results. PatchTST outperforms strong PatchTST and PatchTSTT-Reissers baselines (Tab. 3 Jumbo gains the most with fewer patches and when considering overall results across hyperparameters. These results establish that Jumbo can improve non-causal transformers beyond ViTs.
4.7 PRELIMINARY LANGUAGE EXPERIMENTS
We perform proof-of-concept experiments on a vision-language task (image-caption retrieval) and language-only task (masked modeling) to show that Jumbo may help natural language processing (NLP) tasks. These experiments are not meant to achieve SOTA results- we leave comprehensive NLP experiments for future work.
Image-caption retrieval. We align our Jumbo and Register models (ViT-Base size) to language embeddings by fine-tuming to predict image-caption embeddings given images. We pick the Qwen3 embedding model (Zhang al.] (2025) and fine-tune ViTs for 500K steps with 128 batch size on the CC3M dataset (Changpinyo et al. 202T). For evaluation, we embed all captions in CC3M's validation set with Qwen3 and embed all corresponding images with our fine-tuned ViTs.. Jumbo
8

---

image embeddings retrieve the correct caption embedding 34% of the time in the top 10, whereas Register image embeddings score 30%.
Masked language modeling (MLM). We train Jumbo and Register models (Base size) from scratch using BERT's MLM objective and hyperparameters (Devlin et al. 2019). We pick the dataset (Raffel et al. 2020 and train for 500K steps with 128 batch size. Jumbo achieves validation perplexity of 4.8, while Registers achieves 4.9 (lower is better). MLM perplexity correlates with performance on downstream tasks.
4.8 ABLATIONS
Table 4: Jumbo's shared FFN
Speed
Params Memory Fl FLOPs Top-1 Acc.
increases accuracy and is mem-
Architecture
K imgs/s f
M
GB
G
%
ory efficient. Our Jumbo FFN can be enlarged (J=10) for even
Jumbo (Fig. lb
7.9
88
2.6
4.6
44.61
Jumbo without layer sharing Jumbo without Jumbo FFN
7.7
557
4.1
4.6
44,95
higher performance, at relatively low cost. We report top-1 accuracy on ImageNet-2IK.
8.4
46
2.2
43.64
Jumbo LoRA, rank=8
7.7
89
44,94
Jumbo 6-+10
6.9
180
45.62
Setup. We follow our ImageNet-21K training recipe and ablate Jumbo's design at ViT-Small scale to better understand the contributions of the architecture and its design choices.
Results. (Tab. Not sharing the Jumbo FFN across layers slightly improves accuracy this scale. However, we can fully recover from the drop with sharing by adapting the Jumbo FFN parameters with LoRAs (Hu al. 2022 still share the Jumbo FFN across layers but apply layer-specific LoRAs to specialize efficienlly. LoRAs recover accuracy at negligible cost in speed and memory. Jumbo without Jumbo FFNs performs well enough 2% better than Reg+Regsss but worse than Jumbo: the main difference between this ablation and ViT+Registers is that it concatenates all global tokens as input the classifier (rather than discarding registers). Yet, our best ViT-Small includes the Jumbo FFN: with J=10 its shared FFN achieves 45.6% top-1 accuracy. This Jumbo model beats ViT Small+Registers by 4.1% and matches ViT- ReT-Basss (0.1% difference) with higher speed (2.4x faster) and less memory.
4.9 ANALYSIS: HOw TO SCALE EFFICIENCY AND CAPACITY
Speed K imgs/s
Memory FLOPS Top-1
Table 5: ViT-Base+Jumbo matches a symmetrically wider ViT+Registers with equal params; yet our Jumbo is 1.7x faster. Jumbo also outperforms other ways of adding global capacity, e.g.. ① uses an FFN for patches, and separate FFN for CLS+Reg. tokens, e uses an FFN for patches+CLS, and separate FFN for Reg. tokens. We re- port I accuracy on ImageNet-21K.
Architecture
GB +
G
t
ViT-Base models Jumbo
3.1
152
4.1
16.5
47.0
Reg. Darcet et al.
2024
2.9
94
3.5
18.2
45.7
Registers 1): 768 1024
1.8
163
4.5
32.4
47.1
Alt. O: CLS+Reg FFN
2.8
151
4.1
18.2
46.0
Alt. @:
Reg. FFN
2.8
151
4.1
18.2
46.4
ViT-Small models
Jumbo
7.9
88
2.6
4.6
44.6
Reg.
Darcet 2024
8.0
26
2.3
4.6
41.5
Alt.
CLS+Reg FFN
7.7
40
2.3
4.6
41.5
Alt.
Reg. FFN
7.7
40
2.3
4.6
42.1
Is Jumbo more accurate just because it has more parameters? No. We take ViT- B+Registers and increase its width 768-+1022 to equalize the number of parameters with our ViT-Basc+Jumbo (Tab. 5 rows & 3). These models differ in accuracy by 0.1%, yet Jumbo is more efficient with 1.7x the throughput, 0.5x the FLOPs, and 0.9x the memory. Our novel asymmetric- design of the Jumbo token and FEN is crucial to its better efficiency.
Alternate Veg+Register designs. We experiment with two more architectures investigate the role of adding separate FFNs for different types of tokens (Tab, 5. Alternative O has an FFN for all patch tokens with separate FFN for the CI.S and registers. Alternative has FFN for all patch tokens

---

and the CLS token with separate FFN for the registers. Neither model gains much: the asymmetric token width of Jumbo explains its success, and not the addition of more parameters alone.
Attention maps of split Jumbo tokens
Attention maps of register tokens
Split Jumbo norms
Patch norms
Register norms
Patch norms
10f
Seeely
10"1
Jensty
10-1
10-2
0
10
10
10
morm
Lz 22mor
Ag norm
momm
Figure 5: Jumbo (left two subfigures) eliminates high-norm, outlier tokens in our measurements. According to Darcet et al. 22024) outlier tokens cause attention-may artifacts, and their presence can be reduced by adding registers (right two subfigures). By inspection, Jumbo also learns artifact-free attention maps, and split Jumbo tokens seem to specialize.
Does Jumbo also reduce high-norm tokens? Registers reduce high-norm, outlier tokens that cause attention map artifacts (Darcet et al. (2024). We Jumbo does the same. The Jum+o models train are in fact more effective at reducing outlier tokens than Ree+Regsssess (Fig. 5 We also show attention maps in the Appendix A.7 where we again see similar effect.
DISCUSSION: EFFICIENCY, GENERALITY, AND CAPACITY
Limitations and Future Work. In this work, we conduct only proof-of-concepp experiments using Jumbo in vision language (e.g., CLIP (Radford et al. (021bb) and language-on applications (e.g.. BERT (Devlin et al. 2019), which is non-causal and could benefit from Jumbo theory). We save further exploring these applications for future work.
Conclusion. Jumbo highly efficient, simple, and general: our Jumbo ViTss achieve SOTA accuracy- speed trade-offs by targeted increase in the global computation and parameter capacity of any plain VIT. We show that upgrading a plain ViT with Jumbo improves accuracy at the same speed maintains accuracy faster speeds for supervised image classification, image segmentation, self-supervised learning, time series modeling, and test-time adaptation. Jumbo is the first attention- only and hinhaierarcliaal architecture to outperform specialized compute-efficiet architectures like Efficient VIT (Cai et al, (023). To do so Jumbo increases width asymmetrically, across tokens, and not across layers (in contrast to existing hierarchical models). While increasing model capacity can increase accuracy, it is critical to add capacity in the right places to achieve high efficiency and maintain model flexibility as we show with Jumbo.
ACKNOWLEDGEMENTS
AF is primarily supported by an NSERC PGS-D scholarship. ES is supported by Canada CIFAR AI Chair. YY is primarily supported by Ontario Graduate Scholarship (OGS) and Vector Scholarship AI. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We thank the Google TPU Research Cloud (TRC) for providing the TPUs on which we conducted the MAE experiments. We thank Eric Tzeng and Max Argus for their helpful review and feedback the exposition.
10

---

REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. ViVIT: A Video Vision Transformer In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021. doi: 10. 1wL.L0/CCVA49222222..0000 206666
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619-15629, 2023,
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL org/ abs 1 1607666700 06450
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul
Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. URL
https: //aa:////xxiii arxiv. org/ abs 1811 00075
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.077555 2020.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: good teacher patient and consistent. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. (0925-10934) 2022.
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Confer- ence Learning Representations, 2023. URL https: httss:///ppeeeveeeee
wwwpnnrwwwwwwwweeeeeeeeeee http:/neefffooooooeeoo net/forum?id
JrozRaRw 7Eu
Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17302-17313, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568, 2021.
Jierun Chen, Shiu-hong Kao, Hao He, Weipeng Zhuo Song Wen, Chul-Ho Lee, and S-H Gary Chan. Run, don't walk: chasing higher flops for faster neural networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12021 2023.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5270 2270-5279, 2022.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3075-3084, 2019.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.0993 2018.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 35:1 35:16344 16359, 2022.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL https: htttp::///penreviee .ne/ http://ommmmmdmddmmm..aa
Hoang Anh Dau, Anthony J. Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn J. Keogh. The UCR time series archive. arXiv preprint arXiv: 181000775 2018.
11

---

Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch pack: Navit, vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Yhin Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head architecture for small language models, 2024. URL
org/ aws//41131 13676
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL htttpss//oppeereiitit net gut/foo htte/forum?idd ii- cNTTY
Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang. Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures. arXiv preprint arXiv:2403.02530 2024.
Yuxin Fang. Shusheng Yang, Shijie Wang, Yixiao Ge, Ying Shan, and Xinggang Wang. Unleashing vanilla vision transformer with masked image modeling for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6244 6253, 2023.
Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, Xudong Wang, Adam Yala, Trevor Darrell, Alexei A. Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders. arXiv preprint 1XXi:22401.14391, 2024.
Anthony Fuller, Daniel Kyrollos, Yousef Yassin, and James R Green. Lookhere: Vision transformers with directed attention generalize and extrapolate. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httpsttt: /wtp//ppevvvve.ee/ee ntt.
idd-7DDGG220
Anthony Fuller, Yousef Yassin, Junfeng Wen, Daniel G. Kyrollos, Tarek Ibrahim, James R. Green, and Evan Shelhamer. Lookwhere? efficient visual recognition by learning where to look and what to see from self-supervision, 2025. URL htttts:ss//aaii....e/p/a
abs///a///// 2505. 18051
Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun, Learning and leveraging world models in visual representation learning. arXiv preprins arXiv:2403.00504, 2024.
Shivam Grover, Amin Jalali, and Ali Etemad. Segment, shuffle, and stitch: A simple layer for improv- ing time-series representations. In The Thirty-eighih Annual Conference on Neural Information Processing Systems, 2024. URL https: / wwwpeenwwwiiee.eee//rrreree Eorum wwt.:orrwmmimmmmmmmmmmmmmmmm...m Lcg
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1580-1589, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
12

---

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendryeks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint (etb:::6000041133 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021.
Markus Hiller, Krista A. Ehinger, and Tom Drummond. Perceiving longer sequences with bi- directional cross-attention transformers. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 94097-94129, 2024.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019.
Andrew G Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applica- tions. arXiv preprini arXiv: / 770404461 2017.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017. URL
/704 04861
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International
Conference on Learning Representations, 2022. URL httpss////pnneee.e..eeeeeeeeeeee
[d=nZeVKeeFYrF)
Yang Hu, Xiao Wang, Lirong Wu, Huatian Zhang, Stan Z Li. Sheng Wang, and Tianlong Chen. Fm-ts: Flow matching for time series generation. arXiv preprint wxXiv:2411.07500 2024.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by 03167
reducing internal covariate shift, 2015. URL
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651 -664. PMLR, 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304..00 02240006433 2023.
Muan-May Le, Ling Luo, Uwe Aickelin, and Minh-Tuan Tran. Shapeformer: Shapelet transformer for multivariate time series classification. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp. 1484-1494 New York, NY, USA, 2024. Association for Computing Machinery. doi: 10. 10.1145/3635528.6671666 21114//33367522286812222
URL https
Matthew Leigh, Samuel Klein, François Charton, Tobias Golling, Lukas Heinrich, Michael Kagan, Inês Ochoa, and Margarita Osadchy. Is tokenization needed for masked particle modelling? arXiv preprint arXiv:2409. 12589, 2024.
Vincent Leroy, Jerome Revaud, Thomas Lucas, and Philippe Weinzaepfel. Win-win: Training high-resolution vision transformers from two windows. In The Twelfth International Confer- ence on Learning Representations, 2024. URL ltttp:://ppeneviiee net/ wtr.morrrmiic N2334ybMJJ
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 15:12944-12949 2022.
13

---

Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for mobilenet size and speed. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. (6889-16900, 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 38-55. Springer, 2025.
Yue Liu, Christos Matsoukas, Fredrik Strand, Hossein Azizpour, and Kevin Smith. Patchdropout: Economizing vision transformers using patch dropout. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 3953-3962, 2023.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19976-11866 2022.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Sachin Mehta and Mohammad Rastegari. Mobilevit: light,weight, general-purposs and mobile- friendly vision transformer. arXiv preprint arXiv:2110.02178, 2021.
Sachin Mehta and Mohammad Rastegari. Separable self-attention for mobile vision transformers. arXiv preprint arXiv:2206.026680 2022.
Kaouther Messaoud, Matthieu Cord, and Alexandre Alahi. Towards generalizable trajectory prediction using dual-level representation learning and adaptive prompting. arXiv preprint 55X4v22501.04815, 2025.
Khan Muhammad, Amin Ullah, Jaime Lloret, Javier Del Ser, and Victor Hugo C de Albuquerque. Deep learning for safe autonomous driving: Current challenges and future directions. IEEE Transactions Intelligent Transportation Systems, 22(7):4316 2020.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
Shuaicheng NIu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In The Eleventh International Conference on Learning Representations, 2023. URL htllhttps://ooo httttp:://opezevveew net/ forum?
id=g2YzaF75Tj
Soroush Omranpour, Guillaume Rabusseau, and Reihaneh Rabbany. Higher order transform- ers: Enhancing stock movement prediction on multimodal time-series data. arXiv preprint arXiv:2412.10544 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features with- out supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL Featured Certification.
Misra,
httpss://ppnneeieey net forum? wow.mmiddaaassa iiw 6zFt
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light- cnns on mobile devices with vision transformers. In European Conference on Computer Vision, pp. 294-311. Springer, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 12209922 2022.
14

---

Danfeng Qin, Chas Leichner, Manolis Delakis, Marco Fornoni, Shixin Luo, Fan Yang, Weijun Wang, Colby Banbury. Chengxi Ye, Berkin Akin, et al. Mobilenetv4: Universal models for the mobile ecosystem. In European Conference on Computer Vision, pp. 78- 16-9.. Springer, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021a. URL https
//arxiv. org/ wttp:///1....00000
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021b.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1-6, 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL TRL httpps::/oopeneeveeee.eeeeeeeees http://wwwweeeeeeeeeeoooooimaaaa
Vc26o1
Esther Rolf, Konstantin Klemmer, Caleb Robinson, and Hannah Kerner. Position: Mission critical satellite data is distinct modality in machine learning. In Proceedings of the 41st International Conference on Machine Learning, pp. 42691-42706, 2024.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252. 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520 2018.
Andreas Peter Steiner, Alexander Kolesnikov. Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL
htttp:://epenteveew.nt n wwteee//ouuuneeeeeeeeeeeeeeeeeee forummid=44nPar
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2262-72722 2021.
Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2820-2828, 2019.
Hugo Thimonier, José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, and Bich-Liën
Doan. T-jepa: Augmentation-free self-supervised learning for tabular data. arXiv preprint arXiv:2410.05016 2024.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European conference on computer vision, pp. 516-533. Springer, 2022.
Gabriel Tseng, Anthony Fuller, Marlena Reil, Henry Herzog, Patrick Beukema, Favyen Bastani, James R Green, Evan Shelhamer, Hannah Kerner, and David Rolnick. Galileo: Learning global & local features of many remote sensing modalities. In Forry-second International Conference on Machine Learning, 2025. http.htts///// ptttps://oopereviiee net wtt.://oooom/l//-g/000533333333
15

---

Lorenzo Vaquero, Yihong Xu, Xavier Alameda- Pineda, Victor M Brea, and Manuel Mucientes. Lost and found: Overcoming detector failures in online multi-object tracking. In European Conference Computer Viston, pp.
Springer, 2024.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An improved one millisecond mobile backbone. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 7907-7917, 2023a.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Fastvit: fast hybrid vision transformer using structural reparameterization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023b.
Shashanka Venkataramanan, Valentinos Pariza, Mohammadreza Salehi, Lukas Knobel, Spyros Gidaris, Elias Ramzi, Andrei Bursuc, and Yuki M. Asano. Franca: Nested matryoshka clustering for scalable visual representation learning, 2025. URL httts:://araa:: wwwp:iii.rgg/abooo org/abs/250
14137
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovski,, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 20697-20709, June 2024.
Yibing Wei, Abhinav Gupta, and Pedro Morgado. Towards latent masked image modeling for self-supervised visual representation learning. In European Conference on Computer Vision, pp. 1-17. Springer, 2025.
Ross Wightman. Pytorch image models. pytorch- ima-e----e----- models 2019.
https://github.co com/ htt.:/wig///a/aamnmnmmmmmmmmm
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16133-16142, 2023.
Seokju Yun and Youngmin Ro. Shvit: Single-head vision transformer with memory efficient maero design. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5756-5767, 2024.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &; Data Mining, KDD '21, pp- 2114-2124 2021.
Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Semantic segmentation with plain vision transformers. Advances in Neural Information Processing Systems, 35:4971-443 4982, 2022.
Yanzhao Zhang. Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embed- ding: Advancing text embedding and reranking through foundation models, 2025. URL
httpps://axiv... org/ abss /.5..////.6...... 05176
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprini 401.4:2401.04417. 2024.
16

---

A APPENDIX
A.1 IMPACT STATEMENT
This work presents new designs and empirical results for deep network architectures for more accurate and computationally efficient modeling applied to visual recognition and time series processing. This general topic does not have more specific societal consequences aside from those inherited, good or bad, from the adoption of machine learning.
A.2 COMPUTE-EFRICIEN ARCHITECTURE DESCRIPTIONS
EfficientViT Cai et al. 2023 is hierarchical architecture with four stages and one head. Stages 1 and 2 consist MBConv layers (Sandler et al.) 2018). Stages 3 and 4 consist of MBConv sublayers and their novel EfficientViT sublayer, consisting of an efficient attention module and an FFN+DWCony module (Howard) 2017). Their attention module creates queries, keys, and values of three scales via three DWConvs, and then each set of queries, keys, and values undergoes efficient linear attention. Finally, the head receives outputs from Stages 2, 3, and 4, and applies a final MBConv. EfficientViT variants differ in stage depths and widths, as well head width.
@ SHViT YYun & Ro 2024) is a hierarchical architecture with three stages. Stage consists of a # sublayer and an FFN sublayer. Stages 2 and incorporate their novel single- headed self-attention (SHSA) sublayer between the stage 1 sublayers. SHSA consists of performing single-headed self-attention on fraction of dimensions (1/4.67 ratio); the other dimensions pass straight through, further reducing cost. Both FFN and SHSA sublayers also replace linear layers with DWConv. SHViT variants differ in stage depths and widths.
MobileNetV4 (Qin et al. 2025) variants use their FusedIB, ExtraDW. and Mobile MQA (multi- query attention) modules along with MBConv, ConvNext-Likk (Liu et al. (022). and FFN modules. Variants differ in stage depths and widths, the number of stages, and stage architectures built with combination of the listed modules.
A.3 DISCUSSION OF SEQUENCE BASED ARCHITECTURES
Both Vim Zhu et al. 2024 and V-RWKV (Duan et al. 2024) are not ViTs nor attention-only: thus, they require further customization process other shapes. Plain ViTs simply require defining position embeddings to accommodate new shapes, which is trivial (e.g., one line of code).
Plain ViTs (including with Registers Jumbo) are flexible input shapes:
(1) ViTs can drop/mask tokens to process any subset of patches simply via indexing, which is not possible for other architectures. For example, SHViT, MobileNerV4, and VificientViT all use convolutions and pooling. Mamba-based architectures, such as Vim, require defining a patch-order, e.g., left-to-right. Dropping patches in such a sequence model incorrectly assigns their positions, unlike transformers. Out of the box, V-RWKV's "Q-Shift" does not support token dropping since convolutional, but the authors of V-RWKV modify it to enable MAE pre-training. Token dropping/masken is used in all SOTA SSL algorithms, can speed up training and inference (e.g.. through adaptive computation). and is thus a desirable trait of model architectures.
(2) ViTs can process non- non-2D data, for example, remote sensing inputs or combinations of different shapes. SOTA remote sensing foundation models process inputs of shape: height x width spectral groups x time (Tseng et al. (025). Applying Mamba- based architectures to this input would require defining complex (and unnatural) scans over each axis. Applying V-RWKV to this input would require re-designing the Q-Shift operation. Plain ViTs require defining position embeddings, which trivial and explains their dominance in remote sensing and other applications. Plain ViTs can also process inputs composed of different shapes, e.g., a joint iaguage-vision encoder, via position embeddings. SHVIT, MobileNetV4, EfficientViT, Vim, and V-RWKV would all require significant customization jointly encode 1D (language) and 2D (vision) inputs.
17

---

A.4 EXPERIMENTAL DETAILS
A.4.1 IMAGENET-IK AND -21K HYPERPARAMETERS
We pick these recipes based on findings in the itteratre-ssch as (Tovrron et al. 2022). (Fuller et al. 2024 (Beyer al. 2022), (Dehghani et al. 2024), and (Steiner al. 2022 and past experience indicating that these recipes would result in strong models.
ImageNet-IK training recipe: 128 x 128 px images, 400 epochs, 1024 batch
size, PyTorch's AdamW optimizer with 0.05 weight decay, 1.0 clip grad norm, deit3 b-it3-base pat-ppatchhh 6-224. fbb 1eto--nn2kk-f-iinn ft-nnkk teacher [Touvron
al. 2022 given
224 x 224 px images using (Wightman 2019 's implementation, KL divergence loss between student and teacher logits (Beyer et al. 2022). linear learning rate warmup for 10% of steps to fle-3, 3e-3) and cooldown using cosine decay schedule to le-5, mixup 0.8, cutmix a 1, and 3-Augment data augmentation (Touvron et al., 2022). Then we continue training at 224 x 224 px images, 20 epochs, 512 batch size, PyTorch's s AdamW optimizer with 0.1 weight decay, 1.0 clip grad norm, chi 5- 22 4. fb- in22k-ft-ininn teacher (Touvron et al., 2022) given 224 x 224 px images using (Wightman 2019 's implementation, KL divergence loss between student and teacher logits (Beyer et al.) 2022). linear learning rate warmup for 25% of steps to 5e-5 and cooldown using cosine decay schedule 1e-5, mixup a 0.8, cutmix a = 1, and AutoAugment
-aadd-m9-mstd0---inn 1"" data augmentation (Cubuk et al. 2018 following DEIT III's (Touvron) high-res finetuning recipe.
al.
2022
ImageNet-21K training recipe: 224 x 224 px images, 50 epochs, 1024 batch size, PyTorch's AdamW optimizer with 0.02 weight decay, 1.0 clip grad norm, cross-entropy loss, linear learning rate warmup for 10% of steps to 3e-3 and cooldown using a cosine decay schedule to 1c-5, mixup = 0.8, cutmix = 0, and 3-Augment data augmentation (Touvron et al.) (022). To speed up training, we also employ a token dropping strategy starting at 90%, linearly decreasing to 10%.
A.4.2 TIME SERIES EXPERIMENTS
We adopt the PatchTST NNe et al. 2023 architecture for our time series experiments. PatchTST is a patch-based transformer architecture for time series processing. The method splits univariate time series into patches processed as they are in ViTs for classification, aside from position encoding (2D) vs. ID). For multivariate series, each channel is processed independently using the shared transformer backbone, with the final-layer CLS tokens from each channel concatenated before classification. We extend this shared backbone with registers FRtchhSSTRRegitteee and Jumbo PrachTST+JJumbn
We closely follow the PatchTST training recipe for our experiments, making minor adjustments based prior experience enhance performance. This method remains competitive with recent transformer- benchmarks for time series classification (Zerveas et al.] 2021 Grover et al.)
2024 Le al.] 2024). Apart from variations in time series length, all experiments the same hyperparameters and methodology.
PatchTST Hyperparameters: The model comprises encoder layers, each with 16 attention heads and a token width of D = 128. The transformer FFN includes two linear layers with a GELU activation (Hendrycks Gimpel 2016); the first expands the hidden dimension 256, while the second projects it back to 128. For MitchTST+Jumbo we use J = 4. For calculated match FLOPs.
R is
Time Series training recipe: We perform a hyperparameter sweep over the Cartesian product of learning rates (3e-3, le-3, 3e-4, 1e-4) and dropout rates (0.0, 0.1, 0.2). Each configuration uses either 8 or 42 equally sized patches of maximum possible length, with end-padding applied as needed. The stride length is set to half the patch length. Unless stated otherwise, all experiments follow the same setup: 100 epochs, 256 batch size, PyTorch's AdamW optimizer with a 0.02 weight decay, cross-entropy loss, and a linear learning rate warmup for the first 10% of steps, followed by cooldown using cosine decay to 1e-8. For large datasets, reduce the number of epochs to ensure efficient processing within a reasonable time frame; specifically, we train datasets (Sleep, Tiselac, FaceDetection) ) for 20 epochs.
18

---

Each dataset from the UEA and UCR archives includes a prescribed validation set. We create new 50/50 test/validation split from each of these original validation sets, selecting the best run based on validation performance. All reported results are from the test set.
The 20 datasets were selected in decreasing order of their number of training examples; datasets with either (i) fewer than 42 total timesteps (ii) significant data preparation issues were excluded.
19

---

A.5 DETAILED IMAGENET- IK RESULTS
Table All final results obtained on 224 x 224 px images (%).
Throughput 224
ImageNet-We Val ImageNet-Real
ImageNet-v2 ImageNet-R
ImageNet-HIR
Architecture
Size
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
D=96, J=6
43.7K
69.1
88.5
76.6
92.3
56.2
79.2
21.9
35.0
78.6
92.8
D=128, J=6
31.3K
74.6
91.9
94.9
61.6
83.6
28.3
43.0
83.4
94.9
ViT+Jumbo
D=192, J=6
20.4K
78.4
94.0
84.6
96.2
66.1
86.6
31.7
46.7
87.0
96.7
D=384, J=6
7.6K
82.6
96.5
87.8
97.9
71.8
90.8
38.6
54.9
90.7
98.2
D=768, J=3
3.1K
84.9
97.5
89.3
98.5
75.7
93.0
45.0
62.4
92.8
99.0
D=128, R=16
25.5K
61.0
84.1
68.9
88.9
49.1
74.1
18.7
32.4
69.7
88.9
VrT+Registers
D=192, R=16
16.7K
74.5
92.3
82.3
95.4
62.5
84.7
28.2
43.5
83.7
95.6
D=384, R=16
6.6K
81.9
96.0
97.7
71.4
90.2
38.0
53.8
90.0
98.0
D=768, R=16
2.9K
84.3
97.2
89.0
98.3
74.8
92.5
43.8
60.7
92.0
98.9
conv-small
33.7K
65.6
86.2
73.3
90.7
52.5
75.5
22.3
37.5
75.7
91.3
MobileNetV4
conv-medium
11.2K
74.9
92.6
82.4
95.5
63.1
84.8
29.4
45.9
84.7
95.8
hybrid-medium
8.8K
78.1
94.3
85.0
96.7
67.0
87.5
33.0
49.4
87.2
96.8
S1
42.1K
67.9
88.2
75.7
92.2
54.7
78.2
23.1
38.0
77.6
92.7
SHVIT
S2
34.5K
71.0
90.0
78.6
93.6
58.4
80.5
25.6
41.1
80.9
93.9
S3
23.7K
74.3
92.0
81.6
95.0
61.8
83.5
28.3
43.9
84.0
95.4
BO
25.2K
66.3
86.5
73.9
90.7
53.6
76.3
22.2
36.7
75.8
91.0
Efficient VIT
B1
9.8K
76.9
93.5
96.2
64.5
85.9
31.3
47.2
85.8
96.4
Table 7: Reg+Reesss results, obtained on 128 x 128 px images (%).
Patch
Num.
Learning
Throughput ImageNet-Val Val ImageNet-Real, ImageNet-v2 ImageNet-R ImageNet-HR
Width Registers
Rate
imgs/s
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5 Top-1 Top-5 Top-1
Top-5
16
3e 3
53.6
78.5
60.8
83.6
42.4
67.8
15.9
28.6
61.9
83,4
128
107.0K
16
le-3
51.9
76.8
59.0
81.8
40.8
65.4
13.7
24.9
60.5
82.6
8
3e-3
65.7K
68.5
88.8
76.1
92.6
55.7
78.8
24.9
39.4
77.8
92.2
192
16
3e
68.8
88.9
76.6
92.6
55.9
79.4
24.8
38.9
78.5
92.5
59.9K
16
le 3
66.1
87.2
74.0
91.2
54.2
22.9
36.2
75.4
91.6
3e
24.6K
84.3
96.2
65.8
86.3
48.6
86.8
96.5
384
16
3e-3
78.1
94.0
84.5
96.3
66.1
86.6
33.3
48.6
86.6
96.5
21.8K
16
1c-3
78.2
94.1
84.5
96.3
66.4
86.6
33.5
48.4
87.2
96.6
Table 8: ViT+Jumbo results, obtained on 128 x 128 px images (%).
Patch Learning Throughput
ImageNet- Val ImageNet-Rec Real
ImageNet-v2 ImageNet-R R ImageNet- HR Top-5
Width
Rate
imgs/s
Top-1
Top-5
Top-1
Top-5
Top-1
Top-1
Top-5 Top-1
Top-5
3e
62.4
83.7
69.6
88.1
49.3
72.9
20.1
33.5
71.4
89.1
96
136.0K
le-3
60.8
82.9
68.0
87.3
48.3
71.8
18.9
31.6
70.3
87.6
3e-3
67.7
87.5
75.0
77.6
24.1
37.5
92.1
128
103.1K
le-3
68.4
87.9
75.6
91.6
55.2
78.0
23.8
37.6
77.0
92.1
3e-3
73.3
91.2
94.1
82.1
28.0
42.2
82.7
94.6
192
57.3K
le-3
73.5
91.3
80.3
94.1
60.5
81.8
27.8
42.2
82.8
94.4
3e
79.3
94.4
85.3
96.5
67.3
87.0
34.3
49.5
88.3
96.8
384
20.4K
1c-3
79.3
94.5
85.1
96.6
66.7
86.7
33.4
48.4
87.7
96.6
20

---

Table 9: MobileNetV4 results, obtained on 128 x 128 px images (%).
Learning
Throughput imgs/s
ImageNet-Val Val ImageNet-Real.
ImageNet-v2 ImageNet-R ImageNet-HR
Size
Rate
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1 Top-5
Top-1
Top-5
3c-3
62.1
83.6
69.2
88.1
49.1
72.8
20.4
34.3
71.8
89.4
conv-small
142.7K
1e-3
60.0
82.0
67.2
86.7
47.6
71.4
18.9
32.2
69.8
87.7
3e 3
73.3
91.5
80.5
94.6
60.6
82.9
27.7
42.8
83,2
95.3
conv-medium
53.8K
le 3
72.2
90.7
79.4
94.0
59.5
81.7
27.0
42.0
82.0
94.6
3c-3
74.9
92.4
81.8
95.3
62.4
84.0
29.5
44,8
84,4
95.5
hybrid-medium
43.5K
le-3
75.2
92.5
82.0
95.3
63.0
84.5
29.1
44.5
84.2
95.4
Table 10: SHViT results, obtained on 128 x 128 px images (%).
Learning
Throughput ImageNet- Val
ImageNet-Redt Real
ImageNet-v2 ImageNet-R ImageNet-HR
Size
Rate
imgs/s
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1 Top-5
Top-1
Top-5
3e 3
63.5
84.9
70.9
89.1
50.8
74.5
22.2
35.7
73.7
90.0
S1
81.0K
le
63.5
85.1
71.0
89.3
50.9
74.4
21.3
34.7
72.9
90.5
3e-3
66.6
87.0
73.9
90.8
54.0
76.6
23,9
38.0
76.1
91.7
S1
76.1K
le-3
66.7
87.0
73.8
90.8
53.7
76.8
24.0
37.8
76.7
92.0
3e
70.5
89.8
77.7
93.1
58.1
80.4
26.6
41.0
80.4
93.8
S3
73.8K
le-3
71.2
90.0
78.3
93.3
58.6
80.7
26.7
40.7
80.7
93.9
Table 11: VificientVTT results, obtained on 128 x 128 px images (%).
Learning
Throughput ImageNet- Val ImageNet-Real
ImageNet-v2 ImageNet-R R ImageNet-HR
Size
Rate
imgs/s
Top-1
Top-5
Top-1
Top-5
Top-1 Top-5 Top-1 Top-5 Top-1
Top-5
3e-3
59.5
81.9
66.8
86.7
46.8
70.3
18.6
32.0
69.3
87.6
B0
98.6K
60.8
82.6
87.2
48.3
71.6
19.3
32.6
70.4
87.7
3e
71.8
90.7
79.2
94.0
59.7
81.8
27.4
42.2
81.5
94.4
B
38.7K
le-3
72.8
91.0
79.8
94.2
60.4
81.9
27.1
42.3
82.5
94.8
21

---

A.6 DETAILED TIMESERIES RESULTS
Table 12: Univariate time series classification results (%) "Best" refers to the best run of our 12-run hyperparameter sweep and "Avg" refers to the average the sweep.
PatchTST/8
PatchTST/8 +Jumbo
PatchTST/42
PatchTST/42
PatchTST/8
PatchTST/42
+Registers
+Registers
+Jumbo
Best 70.9
70.7
73.3
70.5
70.6
70.3
Sleep
Avg
67.5
67.7
68.3
67.2
67.1
67.6
Best
83.7
85.6
InsectSound
Avg
76.7
76.0
78.7
78.7
78.7
79.7
Best
92.2
90.9
92.2
95.2
95.0
95.1
FruitFlics
Avg
88.4
88.4
89,4
93.1
92.9
93.9
Best
97.0
96.1
RightWhaleCalls
Avg 92.8
93.5
94.2
94.0
94.8
95.1
Best
98.0
97.7
98.1
99.6
99.8
99.8
FaultDetectionA
Avg 94.6
95.2
97.2
99.2
99.2
99.5
Best
89.0
88.8
90.1
92.4
92.4
92.5
ElectricDevices
Avg
83.2
84.0
85.2
88.1
Best 80.9
81.2
82.0
81.2
80.6
82.2
Crop
Avg 69.4
70.9
72.0
68.7
68.3
68.7
Best 98.8
97.3
97.7
97.7
96.5
96.5
FordB
Avg 95.4
96.0
96.6
95.7
94,9
94.8
Best
97.3
98.0
97.7
97.1
97.1
97.5
FordA
Avg 96.1
96.6
97.2
95.5
95.7
96.0
Best 92.1
91.5
91.0
90.4
91.0
93.1
MelbournePsdestrran
Avg 81.8
82.8
83.9
83.5
83.8
84.9
Table 13: Multivariate time series classification results (%). "Best" refers the best of our 12-run hyperparameter sweep and "Avg" refers to the average over the sweep.
PatchTST/8 +Registers
PatchTST/88 +Jumbo
PatchTST/42
PatchTST/422
PatchTST/8
PatchTST/42
+Registers
+Jumbo
Best 96.6
96.9
97.2
96.4
96.4
96.7
Tisclac
Avg
86.9
87.8
90.1
84.7
85.0
87.9
Best
96.0
96.0
96.5
98.0
97.6
97.6
WalkinggSitingStandiin
Avg
91.7
89.8
93.5
93.9
93.9
94.5
Best
99.9
99.7
99.9
99.7
99.9
99.9
Spoken.ArabicDigits
Avg 99.5
99,6
99.6
99.6
99.5
99.7
Best 87.8
88.1
87.5
86.8
86.6
84,8
FaceDetection
Avg 78.9
80.9
80.0
77.4
77.4
78.8
Best
57.1
$9.1
57.6
58.9
PhonemeSpectra
Avg 38.2
38.7
46.5
42,9
44.5
47.7
Best 78.7
79.5
79.9
74,6
75.7
79.9
L.SST
Avg 69.3
69,4
71.2
61.4
61.4
67.3
Best 92.7
88.5
87.5
94.8
99.0
94.8
UWaveGestureLLbrray
Avg 76.8
79.9
83.4
81.3
85.4
Best
99.0
98.0
99.6
98.7
99.3
98.4
Character/rajeectriiss
Avg
93.6
94.2
96.6
96.6
95.2
97.0
AppaiPaaemee/yyyyoorinnnss
Best
72.3
77.7
81.1
89.5
88.5
89.5
Avg
72.7
75.8
77.0
81.2
82.1
83.7
Best
87.5
83.3
77.1
79.2
66.7
87.5
MotorImagery
Avg 84.9
83.3
83.2
73.6
74.0
81.9
22

---

A.7 ATTENTION MAPS
(a) Attention maps of the Jumbo token split into 6 (b) Attention maps of the CL. and the first five register smaller global tokens. Like VrT+Registers, VuT+Jumbo tokens. learns relatively artifact-free attention maps (as com- pared with the attention maps (Darcet et al. 2024)).
23


Reviews:
[{"review_id": "26BzgclHLx", "reviewer": "Reviewer_oXPN", "summary": "This paper proposes adding a jumbo token to ViTs, demonstrating good efficiency compared to ViT register baselines. The paper is well-written and the idea is intuitive.", "questions": "1. Can the authors report the model parameters in Tab. 5 for the Alt.1 and Alt. 2 model variants?\n2. Why did the authors perform ablations on different scales of ViTs in Tab. 5? Would the conclusions change for Alt. 1 and Alt. 2 if using ViT-Base models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written.\n2. The authors perform systematic and high-quality ablation studies.", "weaknesses": "1. Thanks for the authors' discussions in Sec. 4.7, which show that the additional parameters indeed improve the accuracy. The advantage of Jumbo token is the efficiency, which limits the upper bound of this method.\n\n2. The current use case seems limited. I am curious to see its potential in DiT for C2I or T2I image generation. \n\n3. As discussed in the limitation section, does this method have potential in visual-language understanding or language models?\n\n4. Can the authors explain and analyze why Jumbo reduces the high norms of tokens more effectively than Register as shown in Fig. 5?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DCbMXSSeQw", "reviewer": "Reviewer_rHVB", "summary": "The paper proposes ViT+Jumbo which is a attention-only and non-hierarchical design that increases global capacity by replacing the CLS token with a single Jumbo token. To increase the model capacity, the Jumbo token is wider than patch tokens and has its own FFN whose parameters are shared across layers. The Jumbo token is split before MHSA and re-concatenated after attention. Only the Jumbo token passes through the Jumbo-FFN, keeping overhead small. The authors provide extensive experiments results to verfr their modification design on ImageNet-1K/21K, MAE pretraining, test-time adaptation, and time-series classification. Experimental results suggest that ViT+Jumbo improves speed-accuracy trade-offs versus ViT baselines and competes with specialized efficient backbones while preserving ViT compatibility.", "questions": "1. Why \"While many non-ViT architectures are both fast and accurate, they cannot flexibly process other input shapes\" in the abstract? Non-hierarchical non-ViT models such as Vim and V-RWKV are also sequence models. How do they differ from ViTs in terms of input shape?\n2. How do register-based methods compare to Jumbo in terms of performance, given similar model capacities?\n3. Why does the Jumbo method work more effectively on smaller models? Can it be understood that the benefits are more pronounced with larger model sizes?\n4. Please refer to the section on Weaknesses for other questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Good practicality: The method remains attention-only and non-hierarchical, then the Jumbo token simply widens global processing while keeping the ViT interface intact. This preserves applicability of token dropping, SSL, and ViT heads. The design requires only two splits and concats plus a shared FFN with minimizing implementation burden while adding capacity where it matters.\n2. Increasing capacity while maintaining throughput is promising: On ImageNet-1K the method attains or approaches the frontier across speeds and outperforming register-base models especially at small sizes. This shows effectiveness under constrained regimes.\n3. This paper tests the proposed scheme from different experimental perspectives and tasks (supervised classification, SSL, TTA, and time-series classification), and provides a wealth of ablation experimental results and a relatively detailed explanation of the effects of hyperparameter tuning, which is commendable.\n4. Some visual comparison results are provided, especially the visualization of the norm range, which can be compared according to the assumption of its most relevant baseline (i.e., the Register method).", "weaknesses": "1. Fairness and measurement choices may bias comparisons: ImageNet-1K training uses DeiT-III distillation and a specific two-stage recipe. But it may unclear that all baselines were tuned equivalently under this recipe. Without such distillation training, could the proposed schedule works well to coverage? \n2. No experiments for downstream tasks like the common compared detection and segmentation which are important for verifying the robustness of vision encoders. So downstream dense tasks remain unverified as vision processing is the main field of the work. Besides, how to utilize Jumbo tokens at different layers in different downstream tasks is also worth exploring.\n3. In the MAE self-supervised learning experiment, the linear probing setting was used to illustrate that performance cannot fully reflect the model's capabilities. Full fine-tuning is masking SSL, which is a more common choice, including MAE. It is better to add such results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PU2R5EgX2U", "reviewer": "Reviewer_j2Lh", "summary": "This paper introduces Jumbo, a simple yet effective extension of Vision Transformers (ViTs) that replaces the conventional single CLS token with a wider one. The “Jumbo” token is divided before self-attention and then reassembled afterward, enabling more powerful global representation learning at minimal additional cost. The approach preserves the elegance and simplicity of standard ViTs while integrating seamlessly with techniques such as token dropping and multimodal learning. Extensive experiments demonstrate consistent performance gains across various benchmarks, including ImageNet-1K and ImageNet-21K.", "questions": "The proposed approach can potentially have a big increase in the cost of efficiency with a little performance gain.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The Jumbo design can be seamlessly integrated into existing ViT architectures without requiring substantial structural modifications. It consistently outperforms both ViT+Registers and several optimized models, with particularly strong gains observed in smaller model settings. The paper provides extensive empirical evidence, including comprehensive comparisons, ablation studies, and diverse use-case evaluations.", "weaknesses": "Compared with ViT+Registers, the primary difference lies in using a single large token instead of multiple smaller ones—a design choice that may appear incremental rather than fundamentally innovative. Although Jumbo achieves notable performance gains, it comes at the cost of increased computation and a larger number of parameters, which may limit its practicality in real-world applications. The contributions are incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)