SPoT: Subpixel Placement of Tokens in Vision Transformers
Authors: Martine Hjelkrem-Tan
Marius Aasan
Gabriel Y. Arteaga
Adín Ramírez Rivera
Venue: Accepted by TMLR ()

Abstract: Vision Transformers naturally accommodate sparsity, yet standard tokenization methods confine features to discrete patch grids. This constraint prevents models from fully exploiting sparse regimes, forcing awkward compromises. We propose Subpixel Placement of Tokens (SPoT), a novel tokenization strategy that positions tokens continuously within images, effectively sidestepping grid-based limitations.  With our proposed oracle-guided search, we uncover substantial performance gains achievable with ideal subpixel token positioning, drastically reducing the number of tokens necessary for accurate predictions during inference. SPoT provides a new direction for flexible, efficient, and interpretable ViT architectures, redefining sparsity as a strategic advantage rather than an imposed limitation.

Published in Transactions on Machine Learning Research (3/2026)
SPoT: Subpixel Placement of Tokens in Vision Transformers
Martine Hjelkrem-Taae Tan", Marius Aasan", Gabriel Y. Arteaga', and Adín Ramírez Rivera
Department of Informatics, University of Oslo
(natan, mariuans, gabrieya, adi.r)
Reviewed on OpenReview: httpp://gppaarrreee...e///mo/tts
Abstract
Vision Transformers naturally accommodate sparsity, yet standard tokenization methods confine features to discrete patch grids. This constraint prevents models from fully exploiting sparse regimes, forcing awkward compromises. We propose Subpixel Placement of Tokens (SPoT), a novel tokenization strategy that positions tokens continuously within images, effectively sidestepping grid-based limitations. With our proposed oracle-guided search, we uncover substantial performance gains achievable with ideal subpixel token positioning, drastically reducing the number tokens necessary for accurate predictions during inference. SPoT provides new direction for flexible, efficient, and interpretable ViT architectures, redefining sparsity strategic advantage rather than imposed limitation.
Tokens
Representations
Tokens
Representations
Dense
Encoder
Classifier
Sparse
Encoder
Classifier
Figure 1: (Left) standard ViT splits the image into a fixed grid of non-overlapping patches. (Right) With SPoT,
contimuuulyy sampled set of subpixel-precise patches are extracted.
1 Introduction
Sparsity the fine art of doing more with less is an attractive prospect in systems design and modeling, As models grow ever larger, sparse features alleviates the computational demands a model to provide lower latency, lower memory overhcad, and higher throughput-all important propertics for real-time applications. Incidentally, sparse selection of features offers inherent interpretability and transparency for increasingly complex models (Tibshirani, 1996). Clever adaptations of the Vision Transformer (VIT) (Dosovitskiy et al., 2021) architecture have shown that this family of models et al., 2023; Chen et al., 2023; Rao et al., 2021; Yin et al., 2022; Bolya et al., 2023), accelerating inference by selectively processing a reduced subset of the input. Sparsification has even indirectly inspired entirely new paradigms for efficient unsupervised training in the form of masked image modeling (MIM) (He et al., 2022; Zhou et al., 2022; Oquab et al., 2024).
handle sparse inputs remarkably well (Liu
Code available httpa:/ //tg://ggttttttt con/dsb- ifi/SPoT

---

Published in Transactions on Machine Learning Research (3/2026)
(b)
B
B
C
C
C C
Figure 2: Grids cannot align over all key features. (a) A 5 5 patch grid (gray) with three optimal region placements for sparse feature selection. The green patch is well aligned (A), yellow straddles two cells (B), and red lies corner (C) and leaks into four cells. Translating the grid only swaps which peak is misaligned always bad. (b) Our subpixel tokenizer drops fixed-size windows (green squares) directly on each peak, eliminating the alignment trade-off while still allowing conventional grid tokens when they are well aligned.
patch is
However, in carefully studying the fine print of Dosovitskiy et al.'s (2021) work, one notes an insistence on aligning features with an underlying grid, mirroring the structure inherited by its language counter- part (Vaswani et al., 2017; Devlin et al., 2019) where inputs are naturally represented as sequences of discrete tokens. This discretization might seem natural-after all, are not pixels fundamentally discrete? Our hypothesis that this adherence turns sparsity into an awkward dance; forcing the selection of entire tiles, even if the true optimal feature set hides in-between rigid lines. Like eating soup with fork: possible, but decidedly inefficient and frustrating.
We propose simple remedy with Subpizel Placement of Tokens (SPoT). By allowing patches to occupy continuous subpixel positions instead of constraining leatures to a discrete grid, we expand our modeling toolkit to include gradient based search and sampling for discovering optimal sparse feature sets. Figure succinctly illustrates the core idca, while the example in Figure 2 shows the limitations of discrete grid-basod approach in redueing tokens while preserving predictive quality. Our contributions include:
We propose SPoT, a novel tokenization framework placing features at continuous subpixel positions, significantly enhancing the robustness and efficiency of ViTs.
We introduce Oracle-guided Neighborhood search (SPoT-ON), an analysis tool to empirically quantify optimal subpixel pasitions, showing that carefully selected sparse placements can outperform dense grids with only 12.5% of the original tokens. SPoT-ON provides an empirical upper bound on performance that can be gained by only changing what the model sees, and we show that optimal regions discovered with one model improve performance in another.
We syslematically investigate spalial priors for subpizel token placement, and find that dense regimes prefer coverage, while sparse regimes benefit from center bias and saliency-driven priors.
2 Sparse Visual Bag-of -fo-df-Words Words and ViTs
At first glance it might seem like ViT is designed to process image globally via partitioning an image into patches. However, there is nothing in the transformer architecture that requires a discrete partition. Because self-attention is permutation-invariant. a ViT encoder effectively treats its input tokens as an unordered multiset; a visual bag-of-words, analogous to BERT (Devlin et al., 2019). This observation suggests we need not restrict tokens to a grid, leading to our formulation of sparse feature selection for ViIs.
We denote a VIT encoder as go: Ω + Rd, where I is dataset of source images, and Ω space of positions from which to sample image features. For example, with standard tokenization Pgrid is fixed,
2

---

Published in Transactions on Machine Learning Research (3/2026)
discrete set of non-overlapping square patches tiling the image with fixed window size grid of pixels. The sparse feature selection (SFS) problem can then be formulated as
min Es-pp [Cigo (I, s).)) s.t. S S 2 |S|
[
In other words, for each image we are looking for a probability distribution Ps over subsets of Ω that minimizes loss function L We note that for the discrete non-overlapping case of Agrid, there is an implicit assumption that sampling of S is done without replacement, since sampling the same feature more than once is unlikely to improve model performance. Three specific issues arise from the ViT sparse sampling problem:
1. Interdependence: Transformers process tokens as a set. This means that the marginal distribution of one token is dependent on the inclusion of other tokens. Furthermore, the optimal distribution Ps for a given image may vary depending on the choice of number of tokens.
2. Combinatorial scarch: The discrcte nature of Pgrid means that selecting a subset of tokens is combina- torial knapsack problem. This makes search difficult and gradient methods intractable, particularly since cardinality-constrranedd subset selection is NP-hard (Nemhauser & Wolsey, 1978).
3. Misalignment:
By quantizing patches a fixed grid, key patterns for discriminating an image could be missed in SFS. Concretely, if the grid imposed by agrid is misaligned with key features in the image, SFS could be challenging, central shape or texture may be spread over multiple patches, making subset selection more challenging.
These issues hinder efficient optimization of SFS under standard tokenization.
3 Subpixel Placement of Tokens: SPoT
We propose a more flexible tokenization scheme to tackle SFS problems in ViIs, Instead of considering Q as a fixed discrete partition, we instead imagine Daubpix = [0. H 1] x [0, W-WW1 1] as a continuous space of subpixel positions from which to select features within H W image. Put simply, we parametrize subset of positions S = {si,-... 8m) a set of points of interest from which to extract features from within an image. By sampling tokens from continuous subpixel positions, our tokenizer directly addresses the intrinsic misalignment issue imposed by traditional grid-based based methods, as illustrated in Figure 2. To tackle the combinatorial search problem, we use a bilinear interpolation function and window size k such that each subpixcl position s; = (h, w) provides an extracted feature
(s; = 1 1
h w :
(2)
This allows us to formulate SFS as continuous, probabilistic optimization problem rather than an intractable discrete subset-sclection. The key insight is that our novel tokenizer allows us to (1) investigate placing different priors on Ps; and (2) use gradient based optimization to search for S by way of gradients through Ig. Since we select q to be bilinear, its partial derivatives w.r.t. 8 exist everywhere except at pixel boundaries. so gradients propagate cleanly back to the placements {mi We note that subpixel tokens do not impose any constraint on non-overlapping patches.
Since Pgrid S Dsubpix, patch tokenization is just special case of our tokenization method. This means that models can be evaluated with the exact same features a standard patch-based VIT by letting S = Qgrid- Our tokenizer extends existing work showing that more generalized tokenizers can be constructed to be mod- ularly commensurable to standard ViT models, and adopt their kernelized positional embedding (Aasan et al., 2024).
3.1 Spatial Priors
By allowing subpixel freedom in token placement, we lose the implicit spatial prior that discrete grids naturally encode. Hence the shift to continnous domain raises a dilemma: in sparse regimes, what should
Typically standard cross-entropy used for
3

---

Published in Transactions on Machine Learning Research (3/2026)
Uniform
Gauss
Inotropic
Ceater
Salient
Figure 3: Different sampling priors which can be employed with SPoT. The Sobol prior (not figured) produces uniform quasirandom placements with explicit constraints on coverage.
guide choice of token positions? An appropriately selected prior enables efficient sparse representations while preserving performance, whereas an ill-suited one may lead to substantial degradation. We compare several spatial priors, each encoding different assumptions about feature importance and spatial distribution, illustrated Figure 3:
Uniform: randomly samples locations with spatial bias, assuming all regions are equally important.
Gaussian: randomly samples locations with central bias, which encodes a prior belief that subjects are typically centered in images.
Sobol: provides quasirandom sampling aimed uniform coverage while reducing overlap (Sobol, 1967).
Isotropic: deterministically distributes tokens evenly in a subpixel grid, emphasizing coverage.
Center: deterministically distributes tokens evenly with slight central-bias, commonly seen in classifica- lion datasets (Szabó & Horváth, 2022).
Salient: encodes object-centric bias by placing tokens based on regions identified as visually salient from pretrained saliency model (Lüddecke & Ecker, 2022).
Morcover, we highlight that prior could be learned to suit particular problem. However, we note that interdependency requires
complex parametric family than univariate spatial distribution. We therefore leave this step future work, and focus the proposal of the continuous positions of tokens and its evaluation.
3.2 Oracle Neighborhoods: SPoT-ON
In addition to invostigating spatial priors, we also look to directly explore differentiable optimization for token placement. To probe for ideal choices of S = (s1...., we optimize a constrained version of the SFS problem given by equation directly for each image. We freeze the encoder 9o, and directly apply gradient search per image I to optimize
arg min [C(go(I,S).u) s.t. S C Quupiixs [S] m
for a set number of lokens m with initial positions 5 ~ Pa sampled from a chosen prior pø- This provides an Oracle Neighborhood (ON) adjustments of the initial placements for SPoT. SPoT-ON reveals ideal locations for classifying each image, which allows us to ascertain the existence of an optimal set of positions S for cach image, and estimate upper bound performance gain from effective token sampling. We specify that SPoT-ON incurs higher computational cost for classification, and is not intended as a practical solution for inference. Rather, it acts as a tool for analyzing the nature of sparse ViTss, demonstrating the potential of learnable token positions.
4 Experiments: Case Studies
We examine SPoT by adapting two standard VIT models (Steiner al., 2022), trained on ImageNet-21k and ImageNet-1k (CI.S-N21K, CLS-INIK) and self-suervised Masked Autoencoder (MAF-INIk) (He et al.,

---

Published in Transactions on Machine Learning Research (3/2026)
Table Oracle accuracy of grid-constrained and off- grid patch representations in extreme sparse setting with 12.5% of tokens. The grid-based configuration mimics the discrete patch selection of standard ViTs. The off- grid configuration permits subpixel placement in contin- nous space. Results demonstrate that allowing contin- wous positioning enhances representational quality un- der sparse token regimes. We report the performance of initially sampled points (SPoT) and oracle optimized placements (SPoT-ON).
Table Accuracy (%) for different spatial initialization priors in extreme sparse setting with 25 tokens. We show SPoT performance (Acc@1) and potential increase performance obtained under oracle optimization using
SPoT-ON (Oracle A).
Prior k-NN (%)
Acc@1 (%)
Oracle A
SPoT CLS-IN21K
Uniform
45.23+0.10
44.05
34.22
Gaussian
45.27:10.10
45.22
122.83
Sobol
46.48+0.20
43.67
1 355.83
Method Ir
Steps Acc@1 (%) Oracle
Isotropic 48.19
46.85
144.85
SPoT
61.7
Center 52.18
52.45
131.07
Grid
Salient
56.83+0.26
55.71
131.90
SPoT-ON 3e-3
5
66.2
SPoT MAE-INIK
SPoT-ON
le-2
10
74.0
Uniform
19.72:10.13
56.71
13141
Subpixel
Gaussian
49.49++0.33
57.58
1 29966
SPoT-ON
3e-3
5
90.2
Sobol
53.54++0.39
60.62
1 29940
SPoT-ON
1e-2
10
90.9
Isotropic 54.56
61.72
1 299.21
Center 55.61
62.83
166.70
Salient 60.80++0.08
66.13
166.59
2022). All models utilize the ViT-B/16 architecture (Dosovitskly et al., 2021). Supervised models initialize weights from TIMM (Wightman, 2019), while MAE uses official pre-trained weights. To integrate our subpixel tokenizer, each model undergoes 50-epoch retrofitting step on ImageNet-13 1k (Deng al., 2009), after which the MAF-based model is further fine-funed for classification original protocol (He et al., 2022). Additional details are given in Scction 5.2. We explore the effectivencss and properties of our approach through four targeted case studies addressing grid limitations, object-centric priors, oracle guidance preferences, and transferability of guided placements.
100 epochs, aligning with its
4.1 Are Grids Inherent Limitation of ViTs?
In our first case study, we investigate whether moving away from fixed-grid token representations toward subpixel placements in continuous space enhances representational quality. Traditional grid-basod represen- lations restrict tokens to fixed intervals, which often require multiple patches to cover important features adequately, as illustrated in Figure 2. Subpixel placement, the other hand, allows tokens to align precisely with these features, potentially enabling more efficient representations with sparse set of tokens.
To investigate grid versus off-grid representations, we design an experiment using SPoT-ON to directly compare contimuou subpixel placement with discrete, grid-based positioning, all under a fixed token budget of 12.5% of the standard 196 ViT-B/16 architectures. For the discrete setting, the learned subpixel positions were mapped to their nearest locations on standard 14 token grid, mimicking a conventional ViT configuration. We consider two optimization configurations: one with a learning rate of 1 over optimization steps, and another with higher lcarning rate of x 10 over 10 steps.
The results in Table clearly demonstrate the advantage of subpixel placement, which achieves at least 16.9 percentage point improvement in accuracy over the grid-constrained method. Interestingly, increasing both the learning rate and the number of optimization steps allows the grid-based approach to discover more effective token positions. Nevertheless, the constraints of discrete, gasdd positioning hinders per- formance, even under more aggressive optimization. The consistent performance gains highlights significant benefits of continuous, subpixel token placement in resource constrained settings.
Finding 1: Off-grid token placement enables greater ficxibility and yields substantially better perfor- mance than grid-based approaches under sparse loken seltings.
5

---

Published in Transactions on Machine Learning Research (3/2026)
4.2 Do Object-Centric Priors Improve Predictions?
To investigate spatial priors and how they interact with oracle supervision the sparse setting, we initialize our adaptive sampler using the spatial priors introduced Section 3.1. Each initialization defines the starting coordinates 5 of loken placements, which are subsequently refined by SPoT-ON to minimize classification loss with a learning rate of x 10-3 over 5 optimization steps.
Table 2 reports the resulting downstream accuracy for both supervised and self-supervised backbones in the sparse setting with a token budgel of 25 tokens (12.5% of original). We observe that sampling from saliency heatmaps yields the highest performance both out-of-the -ho--tte-bbxx and after oracle supervision. This aligns with common intuition, object-centric features are more relevant the classification task. The center grid prior also shows higher performance, very likely due to the center bias ImageNet images. Finally, grid-based initializations (e.g.; regular grid, center grid, and Sobol) consistently lead to higher accuracy than the uniform random and Gaussia-stochassic priors, as these result in overlapping placements that are less efficient than structured alternatives.
Comparing these results to Table 5, we see that the benefit of object centric sampling disappears for higher token budgets. Instead, the best performing prior the regular grid, which ensures broad spatial coverage rather than concentrating solely on the object. This reveals surprising inductive llas-coverage is more critical than object-centrictyy for classification under high token budgets. We hypothesize that this is because the information provided by object-foeused features quickly saturates, and with higher token budgets the model benefits from the broader context provided by even coverage of the image.
Finding 2: Object-centric priors yield higher performance in sparse regimes. In dense regimes, even and structured coverage provides better performance.
4.3 Does Oracle Guidance Prefer Salient Regions?
Intuitively, object-centric priors should help classifier, but do they actually steer our oracle-guided tok- enizer? We investigate whether oracle gradient search in SPoT-ON moves tokens towards pixels with higher class saliency.
We design an experiment as follows; starting with a isotropic prior, we optimize trajectories s (0)) , (9) via oracle gradient search following equation Our goal is to measure the shift of token placements toward higher-saliency regions between the initial position (()) and the final position s(4), Given saliency mask M, we compute score for placements by
k2 My (s; k):
score(s) = R22
Σ 1=1
(4)
The relative saliency gain for cach trajectory is given by
RSG score(s(*) - (0))
(5)
score(s(0))
Table shows the result of averaging relative saliency gains over ImageNetlk, showing that there is slight gain in saliency scores for each of our three models. However, the results are not significant enough to claim that saliency alone guides the placements during the oracle search.
We illustrate thirty examples with different trajectories in Fig. 4, which sheds more light on the behavior of SPoT-ON. While trajectories are drawn to discriminative regions, such as spots on a ladybug (left) or hands of a clock (center right), other placements seem more arbitrary, and even loop back on themselves. The oracle often positions tokens close to to-rather than on on-the object; providing context that self-attention exploit. Hence, interdependency rather than saliency alone, drives the final placements.

---

Published in Transactions on Machine Learning Research (3/2026)
Figure 4: Illustration of oracle placements with 25 tokens with SPoT-ON. By optimizing oracle-neighborhoood search equation 3 all the way through the model, the oracle discovers optimal placement points, yielding an accuracy of 90.9% ImageNet with only 12.5% of the tokens. Trajectories colored starting with dark purple for initial points, with endpoints colored bright yellow.
Finding 3: While oracle gradient search yields slight bias toward higher-salicncy pixels, the results are not highly significant. The results suggest that token interdependency as opposed to pure object saliency - is a predominant factor for optimal placements.

---

Published in Transactions on Machine Learning Research (3/2026)
Table Quantitative analysis on object-seeking behavior with SPoT-ON. each region, we compute the average saliency in a 16 16 window centered each point. We compare the saliency scores of initial placements compared to oracle placements, compute the relative saliency gain (RSG) over ImageNet1k.
RSG (%)
Backbone 25 Tok.
49 Tok.
100 Tok.
196 Tok.
CLS-IN2IK
1..34
0.33
1.447
40.14
CLS-INIK
10.77
10.77
1.775
4.553
MAE-INIK 10.49
1.554
1..51
J. 0.0.8
Table 4: Transfer properties of SPoT-ON positions between models, the sparse setting with 12.5% token budget. We optimize oracle positions using gradient optimization for a source model, initialized with the isotropic prior. We then evaluate the discovered points independently trained target model. Each model a significant increase in performance, even from points derived from an independent model.
Acc@1 (%) (25 Tokens)
Source
Target
Original
Transfer
CLS-INIK CLS-IN2IK
46.85
54.06
7.21
MAE-INIK + CLS-IN2IK
46.85
56.71
1.886
CLS-IN21K + CLS-INIk
33.52
37.91
4.39
MAF-INIK CLS-INIK
33.52
39.19
1.667
CLS-IN2IK + MAE-INIK
61.72
68.50
1.778
CLS-INIK MAE-INIK
61.72
67.81
1.009
4.4 Do Oracle Guided Placements Transfer?
If discovered token placements with oracle guidance captures structure rather than model-specifie quirks, placement learned by one model should benefit another. To test this, investigate transferability of token placements. Given two independently trained models, 9B, independently optimize two feature sets; SA. SB, respectively, following equation 3. Both feature sets are initialized with the same isotropic spatial prior Sg- Then, for target labels y. we compute the difference in accuracy from the initial placements So to the optimized placements with the alternate model, i.e,
A Elga (1, So) v) Elga(I, y):
(6)
with symmetrical computations for model gg on S The results can be found in Table 4. Our experiments show that placements discovered with one model transfers to yield improvement in performance in different, independently trained model.
Finding 4: Discovered positions via SPoT-ON generalize between models; set of placements optimized via model will improve results with another independently trained model in sparse regimes.
5 Extended Experimental Results
We present the performance of SPoT under varying sparsity configurations and compare against. baseline models, including the supervised backbones from TIMM (Wightman, 2019) and the officially fine-tuned MAE model (He et al., 2022). For clarity, all baselines are denoted ViT-B/16 in Table 5. To evaluate the baselines under sparsity constraints, apply PatchDropout (Liu et al., 2023), which randomly drops input patches during inference.
The results in Table 5 reveal several noteworthy observations. First, the self-supervised MAE model (He et al., 2022) consistently outperforms its supervised counterparts under sparse configurations. This advan- tage is likely due to its pre-training objective, which inherently involves reconstructing inputs from partial observations, thereby fostering robustness patch dropout. Second, we observe that under full-token condi- tions, SPoT achieves marginally higher performance than both the supervised and self-supervised baselines.
8

---

Published in Transactions on Machine Learning Research (3/2026)
Table 5: Classification top-1 and kNN accuracies for supervised and and self-sup models using different token priors. We find that center-bias in spatial priors is beneficial in sparse regimes, while coverage becomes more important as token budgets increase.
25 Tokens
49 Tokens
Tokens
Tokens
Model
Prior
Oracle
Acc@1
kNN
Acc@1
kNN
Acc@1
kNN
Acc@1
kNN
CLS-IN21K
VIT-B/16
Patch Grid
24.72
27.86
56.29
57.19
78.75
78.77
85.11
83.96
SPoT-B/16
Uniform
44.05
45.23
67.77
66.38
79.64
78.03
83.76
81.85
SPoT-B/16
Gaussian
45.22
45.27
68.64
66.96
79.75
77.74
83.45
81.48
SPoT-B/16
Sobol
43.67
46.48
69.02
68.60
81.63
79.35
84.66
82.62
SPoT-B/16
Tsotropic
46.85
48.19
70.61
70.29
82.20
80.73
85.15
83.42
SPoT-B/16
Center
52.45
52.18
69.22
68.16
80.84
78.56
84.01
82.23
SPoT-B/16
Salient
55.71
56,65
72.89
72.38
79.91
80.56
84.56
82.59
SPoT-ON-B/16
Isotropic
81.70
70.65
94.28
88.58
95.97
92.92
96.12
93.52
CLS-IN1k
VIT-B/16
Patch Grid
9.24
12.05
41.05
44.38
71.22
71.41
79.14
77.64
SPoT-B/16
Uniform
29.87
33.88
60.64
60.84
74.44
73.18
79.38
77.36
SPoT-B/16
Gaussian
29.27
33.07
60.47
60.23
74.37
72.82
79.02
77.00
SPoT-B/16
Sobol
30.67
35.23
64.42
63.88
76.45
75.18
79.96
78.17
SPoT-B/16
Isotropic
33.52
37.84
66.18
66.25
77.58
76.29
80.61
79.04
SPoT-B/16
Center
39.91
42.47
63.04
62.65
75.41
73.63
79.32
77.71
SPoT-B/16
Salient
39.83
43.72
66.32
66.00
74.36
75.25
79.54
78.03
SPoT-ON-B/16
Isotropic
73.99
74.42
94.21
90.11
95.79
93.61
96.04
93.97
MAE-IN1k
VIT-B/16
Patch Grid
55.43
48.85
70.69
67.15
79.53
78.41
83.60
82.07
SPoT-B/16
Uniform
56.71
49.72
73.22
65.85
80.53
74.76
82.78
78.21
SPoT-B/16
Gaussian
57.58
49.49
72.51
65.59
80.31
74.52
82.55
77.90
SPoT-B/16
Sobal
60.62
53.54
75.71
68.71
82.19
76.24
83.51
79.09
SPoT-B/16
Tsotropic
61.72
54.56
76.84
70.02
82.76
77.24
83.89
79.53
SPoT-B/16
Center
62.83
55.61
74.63
67.31
81.06
75.20
82.97
78.54
SPoT-B/16
Salient
66.13
60.80
77.10
72.24
81.46
77.25
81.64
79.13
SPoT-ON-B/16
Isotropic
90.93
79.73
94.87
87.87
96.09
90.76
96.24
91.28
Table 6: Analysis on harmful spatial priors and adversarial oracles in sparse regimes. background prior samples from inverse saliency maps; the boundary prior samples with image edge bias. Ascent shows accuracy under worst- case token placements, discovered via SPoT--O, Label obfuscation optimizes placements for randomized labels. Each is compared with baseline SPoT performance using the isotropic prior- the performance drop is shown to the right of each score.
Acc@1 (%) (25 Tokens)
Backbone Backgrd.
Boundary
Ascent
Lab.Obf.
CLS-IN21k
40.80
10.68
13.75133.09
1.17:
CLS-INIK
20.06; 13.46
4.15 29.37
5.90 $27.82
2.68 33.52
MAE-INIK
31.89, 29.83
10.83 50.89
16.54) 45.18
1.111 60.61
This suggests that even in dense settings, additional gains can be realized by leveraging the flexibility in- troduced by subpixel representations. Third, as the level of sparsity increases, SPoT consistently surpasses all baselines, regardless of spatial prior. Notably, performance further improved with priors that pro- mote spatial coverage, compared stochastic uniform sampling, which demonstrates the importance of an appropriate token placement scheme under sparsity constraints.
In Figure 5 show image throughput versus accuracy, comparing SPoT with the baselines across varying sparsity levels. As sparsity increases, throughput improves significantly, albeit with an associated tradc-off in accuracy. Notably, SPoT achieves the most. favorable trade-off, maintaining substantially more of the full-model accuracy while enabling higher throughput than competing approaches. Further, we observe only slight variation throughput between the models at each sparsity level, indicating that SPoT incurs very minimal computational overhead compared baselines. We also include our oraclo-guided variant SPoT-ON in the figure, which illustrates ceiling achievable performance when placements ideally sampled.

---

Published in Transactions on Machine Learning Research (3/2026)
25
50
100
196 Tokens
CLS-IN2IK - CLS-INIK - MAE-INIk - SPoT 4.. SPoT-ON
100
Ceiling
98
Accuracy (%)
90
Gap
80
60
10
15
20
25
30
35
Throughput (k img/s)
Figure 5: We show ImageNetlk accuracy vs. throughput with models at four sparsity levels. The ceiling denotes performance unlikely to be achieved given the intrinsic label noise in ImageNet (Beyer et al., 2020). The gap highlights the margin between SPoT with optimal configuration and SPoT-ON, illustrating possible performance gain through better token placement.
5.1 Robustness and Sensitivity Analysis
To thoroughly validate the semantic relevance of optimized subpixel token placements, we conduct tar- geted robustness analyses. Specifically, we evaluate performance under intentionally adversarial condi- tions-including inverse priors favoring irrelevant regions such backgrounds image boundaries, and gradient ascent adversarially maximizing loss randomized label assignments. Our results in Table 6 demonstrate substantial performance degradation in all these adversarial scenarios, strongly suggesting that our token placement mechanism indeed leverages meaningful semantic cues rather than trivial spatial corre- lations or easily exploitable priors.
Importantly, the gradient-ascent oracle still receives the correct labels, so its sharp accuracy collapse shows that semantically aligned token positions are indispensable- enough if the tokens pushed onto irrelevant regions. In contrast, the near-chance performance under label- obfuscation demoustrates that the model does not easily adapt to arbitrary image label pairings, confirming that our selector grounds predictions in genuine object evidence rather than flexible location- shortcuts. Background sampling leads to reduced performance; however, we posit that the token set still captures some object edges, providing the model with useful information. This hypothesis reinforced by the significantly larger performance drop observed when sampling with a strong bias loward image boundaries. Adversarial optimization shows similarly extreme drop, indicating that good token placements are not trivial.
keeping the right supervision is not
5.2 Retrofitting and Finetuning Details
Table 7 details the training configurations for the three backbone variants: CLS-INIK, CL.S-IN11, and MAE-INIK. By retrofitting, mean a fine-tuning protocol where the goal to adapt models pretrained parameters I to proposed subpixel tokenization scheme, i.c.
min (L s.t. S = {Sk ~ (Ssuubiii)) k k= 1,,,,,,, i.i.d. for each image T.
Explicitly, we let. Σ,~z denote the expectation over a dataset of images 1 ~ I of the loss function L of the model outputs go and the labels y. The positions S consists of m i.i.d. uniformly distributed subpixel positions for each image I, such that each position s; ~ ((Qsubpix)
The official TIMM (Wightman, 2019) model card names are listed below each corresponding subtable for reference. All protocols use isotropic sampling and for 50 epochs 224 x 224 images on the ImageNet-
10

---

Published in Transactions on Machine Learning Research (3/2026)
Table Training protocols for retrofitting. We use training protocols from He et al. (2022) in MAE finetuning.
(a) CLS-INIK Retrofitting
(b) CLS-IN211 Retrofitting
(c) MAF-INIK Retrofitting
config
value
config
value
config
value
sampler
isotropic
sampler
isotropic
sampler
isotropic
batch size
2048
batch size
2048
batch size
4096
epochs
50
epochs
50
epochs
50
dataset
ImageNetlk
dataset
ImageNetIk
datasct
ImageNet1k
img.size
224 x 224
iig-size
224 224
img.size
224 224
loss fn.
CE (0.1 smooth.)
loss fn.
CE (0.1 smooth.)
loss fn.
MSE
optimizer
AdamW
optimizer
AdamW
optimizer
AdamW
momentum
0.99
momentum
0.9, 0.99
momentum
0.9, 0.95
Ir.sche.
cos.decay (5 w.u.)
Ir.ched.
cos.decay (5 w.u.)
Ir.sched.
cos.decay (5 w.n.)
6e-5
Ir dropout path
6e-5
Ir dropout path
3e-3
dropout path
0.1
opt.
1e-8
opt.
1e-8
opt. rrc
1e-8
augment
rrc randaug(15, .5)
augment
mrc randaug(15, .5)
augment
mixup a
0.8
mixup a
0.8
mixup alpha
0.8
cutmix a 1lrd
1.0
cutmix a
1.0
cutmix 1.0
0.65
1lrd
0.65
lrrd
0.75
vSt. wtt_baoe.gaschhit.o twngrae2_tn2ii.ff..oo
1 dalaset (Deng et al., 2009). Layer-wise learning rate decay (LLRD) is employed with slightly more aggressive parameter in the MAE retrofitting, while the MAE finetuning follows the original protocol outlined by He et al. (2022), with minor exceptions..
Evaluation Protocol. Our evaluation protocol closely follows existing works (Oquab et al., 2024; He et al., 2022; Zhou et al., 2022). We use bicubic interpolation and a crop ratio of 0.875 our evaluations. All models are trained with standard ImageNet normalization, noting that the TIMM basclines (Wightman, 2019) adopt the convention of using& flat normalization of jRGB ORGB (0.5, 0.5, 0.5). Our kNN evaluation protocol was adapted from Caron et al.'s (2021) work.
6 Related Work
Leveraging sparsity to reduce computational overhead is well-ostablished research direction. Previous work introduced sparsity through masking during pre-training, in self-supervised (He al., 2022) and language- supervised contexts (Li et al., 2023). Liu et al. (2023) applied sparsity at the fine-tuning stage by initially upsampling images and subsequently randomly dropping patches, thus enhancing efficiency and reducing computational complexity. Distinct from maaiing-centtivn sparsity approaches, our work induces sparsity during inference by retrofitting ViTs with subpixel tokenizer, significantly improving throughput. Another line of research explores inference- time sparsity via selective pruming to either discard (Chen et al., 2023; Rao et al., 2021; Yin et al., 2022) merge (Bolya et al., 2023) tokens (ToMe) based different heuristies. In contrast, our approach achieves sparsity by sampling rather than selectively pruning tokens during inference.
We include comparison with ToMe (Bolya et al., 2023) using their VIT-B/16 models in Table 8. ToMe begins with full loken budget and progressively reduces the number of tokens by merging them across transformer layers. In contrast, SPoT performs token reduction at the tokenization stage by sampling a smaller set of tokens and maintains constant token count throughout the network. For this comparison, we report results using the perforperg SPoT configurations and evaluate performance when sampling 100 tokens, and select ToMe configurations that achieve the highest throughput speed-up, these most closely match the SPoT performance. SPoT achieves higher throughput improvements while incurring smaller drop in accuracy.
Recently, other works have explored non-grid based tokenization. One interesting line of research looks to leverage subobject tokenization, which extracts fine-grained segmentations as opposed to square
2Cosine warmup in Ir-scheuuler, starting from 10 with peak learning rate of 10-3, and gradient clipping set 3.
11

---

Published in Transactions on Machine Learning Research (3/2026)
Table Comparing SPoT with 100 token budget against ToMe. We show throughput improvement (Speed-up) and drop in accuracy (A Acc@1) relative to the full token budget baseline. We use ToMe's officially reported results.
marks that token reduction was applied at the finetuning stage. Otherwise, models apply reduction during inference.
Model
Speed-up
A Ace@1
CLS-IN21K
ToMe
1.95x
-4.20
SPoT
3.31x
-2.55
MAE-INIK
ToMe
1.94x
-4.87
ToMe!
1.95x
-1.71
SPoT
3.31x
-1.13
Table Comparing ElasticVTT to SPoT in sparse regimes.
Acc@1 (%) / Number of Tokens
Model
39
59
78
98
118
137
157
176
196
EIasticVTT
67.17
72.65
75.47
77.47
78.18
79.73
80.81
81.31
82.04
SPOT-MAE-INIK
71.81
77.86
80.47
81.89
82.55
83.05
83.34
83.52
83.85
patches (Aasan al., 2024; Chen et al., 2025). Other works apply learnable clustering into the transformer architecture via cross-allention operators (Fan et al., 2024; van Steenkiste et al., 2024), while Nguyen et al. (2025) proposed to tokenize each individual pixel. Deformable patches was first proposed in relation to object detection (Xia et al., 2022), but was further extended to general purpose modeling in ElasticViT (Pardyl et al., 2025), which proposed elastic windows as local uugmentations standard classification tasks. These are defined as stochastic patch perturbations scale, position, and erasure via patch dropout. Put simply, ViT relaxes the traditional patch grid of ViTs by randomly shifting, rescaling, and dropping patches during training.
ViT differs from SPoT in key aspects. First, we relax the discrete grid assumption not by perturbing existing patch positions, but rather by directly sampling arbitrary continuous-valued points within the image. Conversely, ElasticViT uses discrete pixel positions, and does not adapt a continuous subpixel approach. Second, we do not explicitly train our model to handle sparse inputs; instead, our method's inherent robustness to sparse token configurations naturally arises from training on continuously sampled points. Nevertheless, comparing our method to Elastic is insightful, as their approach explicitly trained to handle sisiimuous-vllled positions and sparse token scenarios. Table 9 compares SPoT with ElasticViTT officially reported results, demonstrating that SPoT consistently outperforms ViT across all evaluated sparse configurations.
7 Conclusion
We proposed SPoT for extracting features at contimuus subpixel positions, and used oracle-guided gradient search to probe the nature of optimal token placements and ideal spatial sampling priors. Our case studies showed that the flexibility of contimuuss off-grid placements improves performance out-of-the-box, especially in sparse token budget settings. SPoT-ON provided an estimate of best-case performance from optimal token placement. Although placements are guided via an oracle, these optimal features exist independently of how they were discovered, revcaling a performance gap that better informed priors could help bridge. While we focused on analyzing the elfects of subpixel tokenization under varying sparsity configurations with different spatial priors, the development of learnable spatial priors is next step towards narrowing the oracle performance gap.
Concretely, since the oracle derives placements from image-dependent statistics, we foresee that a light weight "policy network" can be trained as a structured regressor over dense local cues to predict good token place- ments a single forward pass. A feasible instantiation would involve a lightweight CNN or MIP operating at patch-gric grid resolution, taking low-level or carly-stage features as input to produce a dense importance map to obtain token placements, e.g. via rudgel selection. Potential learning signals include
12

---

Published in Transactions on Machine Learning Research (3/2026)
distillation from SPoT-ON, self-supervised alignment, or joint end-to-end training with the backbone. We emphasize that introducing policy network is orthogonal to the core contribution of SPoT. The current work intentionally focuses on analytic, interpretable placement to isolate the effect of token geometry.
Our study with SPoT on different spatial priors focused on classification on ImageNet, but optimal placements may vary for different datasets and downstream tasks. Beyond image-level classification, we foresee that SPoT can be useful in tasks that require more explicit spatial reasoning such as object localization and detection, where spatial priors could be adapted to emphasize fine-grained multi-scale positional structure, potentially improving localization accuracy. Similarly, for video understanding tasks like action recognition and temporal localization, SPoT can be extended to incorporate spatiotemporal priors that jointly encode spatial layout and temporal continuity, encouraging consistent feature aligument across frames. Overall, we expect the design and placement of spatial priors to be desk-depen,,nt, and exploring SPoT in temporal prediction settings is a promising direction for future work.
By enabling continuous token positioning, SPoT facilitates gradient-based optimization of token placement, which can be advantageous in resource-constrainee environments where sparsification is beneficial. Although we limit our scope to employ an oracle to determine optimal token placements, exploring oracle-independent strategies represents a compelling direction for future research. Specifically, integrating efficient saliency- driven objectives or heuristics during inference could potentially enhance throughput efficiency while main- taining competitive performance compared to models utilizing a full token budget. Further improvements may also be seen by allowing the model to adjust the patch window size dynamically during training, More- over, while the scope of this work is towards modeling in sparse regimes, our results indicate that continuous subpixel token placements provide a novel research direction for ViTs a more general level.
Acknowledgments
This work was funded by RCN (the Research Council of Norway) through Visual Intelligence, Centre for Research-based Innovation (309439). We acknowledge Sigma2 (Project NN8104K) for access to the LUMT supercomputer, owned by the EuroHPPC Joint Undertaking, hosted by CSC (Finland) and the LUMI con- sortium through Sigma2, Norway.
References
Marius Aasan, Odd Kolbjornsen, Anne Schistad Solberg, and Adin Ramírez Rivera. A Spitting Image: Modu- lar Superpixel Tokenization Vision Transformers, In European Conf. Comput. Vis. Wksps, (ECCVW), 2024. doi:10.
11. 3, 12
Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with ImageNet? CoRR, abs/2006.07159, 2020.
10
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token Merging: Your VIT but Faster. In Inter. Conf. Learn. Represent. (ICLR), 2023. har.5///aXii.221000000 1, 11
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Superviee Vision Transformers. In IEEE Inter. Conf. Comput. Vis. (ICCV), pp. 9630-9640. IEEE, 2021. 1et.10.119/ICCV/ 2019922.002..00 .0095101111. 11
Delong Chen, Samuel Cahyawijaya, Jianfeng Liu, Bаoyuan Wang, and Pascale Fung. Subobject-level Image Tok- enization. In Inter. Conf. Mach. Learn. (ICML), 2025. do 48550/ar5Xii 2402.....0..333 12
Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. SparseVIT: Revisiting Activation Sparsity for Efficient High-resolution Vision Transformer. In IEEE/CVF Inter. Conf. Comput. Vis. Pallern Recog. (CVPR), pp. 2061-200100 2070, 2023. doi:10. 1we.111999/9R222 CVPR52729 (T1.:/2229.2222.00000 202..00222. 1, 11
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNel: A Large-Scale Hierarchi- cal Image Database. In IEEE/CVF Inter. Conf. Comput. Vis. Pallern Recog. (CVPR), pp. 248 288-255, 2009. (C0..1100///VP.20000....
5,
13

---

Published in Transactions on Machine Learning Research (3/2026)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Conf. North Amer. Ch. Assoc. Comput. Läng. (NAACL), pp. 4171-4186. Association for Computational Linguistics, June 2019.
18653//1/N19914225
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesuikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16 x 16 Words: Transformers for Image Recognition at Scale. In Inter. Conf. Learn. Represent. (ICLR)) 2021. 185:00 1053500/rXiv.21000 10000....119.... 11929. 1, 2,
Ke Fan, Zechen Bai, Tianjun Xiao, Tong He, Max Horn, Yanwei Fu, Francesco Locatello, and Zheng Zhang. Adaptive Slot Attention: Object Discovery with Dynamic Slot Number. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern /C/////V33.222224444422 Recog. (CVPR), pp. 23062-23071, 2024. doi:10.1109/ 00t10..119/CVPRE2222
12
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders are Scalable Vision Learners. In IEEE/CVP Inter. Conf. Comput. Vis. Pattern Recog, (CVPR), pp. 16000-16009, June 2022. doi:10.1109) 11t.1...1//..111.. B1L.0/////R25S..22222 2022.01152 1, 4, 5, 8, 11, 16
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling Language-Imag Pre- Training via Masking. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog, (CVPR), pp. 23390-23400 2023, doi: 10w.000000////CP22..... 200.52272..02.022210 11
Yue Liu, Christos Matsoukas, Fredrik Strand, Hossein Azizpour, and Kevin Smith. PatchDropout: Economizing Vision Transformers Using Patch Dropout. In IEEE Wint. Conf. Appl. Comput. Vis. (WACV), pp. 3953-3962 2023. doi:10.1109/ 1oi110.119/WAACV0 WAC39/WWCC66888.. 2023.00391. 1, 8, 11
Timo Lüddecke and Alexander S. Ecker. Image Segmentation Using Text and Image Prompts. In JEEE/CVE /CVF Infer.
Conf. Compul. Vis. Pallern Recog, (CVPR), pp. 7076-7086, 2022. doi: 10:10.11000 1ae.1001119//V/ (0000//CVR52668.2222222 202..0009....00055. 4
George L. Nemhauser and Laurence A. Wolsey, Best algorithms for approximating the maximum of a submodular set function. Mathematics of Operations Research, 3(3):177 188, 1978. dbt:10.1287mmo0r333.. 3
Duy-Kien Nguyen, Mahmoud Assran, Unnat Jain, Martin R. Oswald, Cees GM Snoek, and Xinlei Chen. An Image is worth More than 16x16 Patches: Exploring Transformers on Individual Pixels. In Inter. Conf. Learn. Represent. (ICLR), 2025. doi:10.44550/ 11111008550///xiivv..22 http:/a..........9.111 12
Maxime Oquab, Timothóe Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniee, Vasil Khalidov, Pierre For- nandez, Daniel HAZIZA, Francisco Massa, Alazeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang-Wer Lı, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Syn- naeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. Trans. Mach. Learn. Res., 2024. ISSN 2835-8856. Noi:10.48550// Xav 230..0793.. 1, 11
Adam Pardyl, Grzegorz Kurzejamski, Jan Olszewski, Tomasz Trzciáski, and Bartosz Zielifskii. Beyond Grids; Ex- ploring Elastic Input Sampling for Vision Transformers. In IEEE Wint. Conf. Appl. Comput. Vis. (WACV), pp. 8536-8545 2025. doi: 100.0.111////MM 2022660041.02250008222 12
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Adv. Neural Inf. Process. Sys. (NeurIPS), volume 34, pp. 13937-13949 Curran Associates, Inc., 2021. doi:10.483500 (0000000000//032221 /ttp///.....1..2233.3
1, 11
LM Sobol. On the Distribution of Points in a Cube and the Approximate Evaluation of Integrals. USSR Computational Mathematics and Mathematioal Physics, 7(4):86-112, 1967. ISSN 0041-5553. doi: 1et1100116/0000 (00060001-555366666000000 (90)666666094001 44-9. 4
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to Train Your VIT? Data, Augmentation, and Regularization in Vision Transformers. Trans. Mach. Learn Res., 2022. ISSN 2835-8856. 2ob10.48550/arrii 10270.. 4
Gergely Szabó and András Horváth. Mitigating the Bias of Centered Objects in Common Datasets. In IEEE Inter. Conf. Pattern Recog. (ICPR), pp. 1786 1792, 2022. 16f:00.119//CCPR6630 http://CRR33311222222222
4
Robert Tibshirani. Regression Shrinkage and Selection via the Lasso. J. R. Stat. Soc., B: Stat. Methodol, 58(1): 267-288, 1996. ISSN 00359246. 1111100111112517--1111111111 wtw.1666...99...0000000.000 1
14

---

Published in Transactions on Machine Learning Research (3/2026)
Sjoerd Steenkiste, Daniel Zoran, Yi Yang, Yulia Rubanova, Rishabh Kabra, Carl Doersch, Dilara Gokay, Etienne Pot, Klaus Greff, Drew Hudson, et al. Moving Off-the-Grid: Seene-Grounded Video Representations. In Adv. Neural Inf. Process. Sys. (NeurIPS), volume 37, pp. 124319 124346, 2024. doi:10.48550/ 48550/arXiv......
12
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In Adv. Neural Inf. Process. Sys. (NeurIPS), volume 30, 2017. (oi:10048550/arXii 110.1.07.00....
Ross Wightman. PyTorch Image Models, 2019. URL httpa:///ithhb...... http:///ggggaaccc.ooo///atta/a/p -ag.://aaaapppooorrraae-emmaaaaa 5, 8, 10, 11
Zhmofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision Transformer with Deformable Attention. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 1791 4803, 2022.
doi:10.1119 100.01119///VP52265600 2022.00115. 12
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-VII: Adaptive Tokens for Efficient Vision Transformer. In IEEE/CVF Inter. Conf. Comput. Vis. Pottern Hecog. (CVPR), pp. 10809 10818, 2022. 1att1..119/CV/PRSSE 201226688.022.00055. 1,
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT Pre- training with Online Tokenizer. In Inter. Conf. Learn. Represent. (ICLR), 2022. Noi:10.48530) 48550/arXiv /aroir///iiviaa 211.0222... 1, 11
15

---

Published in Transactions on Machine Learning Research (3/2026)
A Retrofitting and Finetuning Terminology
We distinguish between retrofitting and fine-tuning, which serve different purposes:
Retrofitting adapts a pretrained Vision Transformer to a new Lokenizer while preserving the original train- ing objective (supervised or self-supervised). All model parameters are learnable, but we apply layer-wise lcarning-rate decay where carly layers incur a higher learning rate than later layers. Retrofitting is per- formed for a limited number of epochs align the model with the new tokenization scheme while preserving high-level representations.
Fine-tuning adapts pretrained encoder to downstream task. In our work, this corresponds to ImageNet classification with linear head attached. Both the encoder and head are trained jointly using the task- specific objective. We fine-tune the pretrained MAE following standard fine-tuning protocols from the original paper (He et al., 2022).
B Retrofitting ablations
In Figure B.1 we show training and validation accuracies during the retrofitting stage for the CLS-IN2IK model. We observe that the encoder initially aligns quickly with our tokenizer during the first epochs, and that the validation accuracy plateaus during the last 10 epochs. Basod on this result, we retrofit the CLS-INIK and MAE-INIk models for 50 epochs well.
85
Accuracy (%)
80
75
Train
70
Val
15
20
35
40
Epoch
Figure B.1: Accuracy per training epoch during retrofitting for our CLS-IN21K model.
C Computational overhead of SPoT
SPoT samples tokens by using bilinear interpolation over a grid defined by the window size k. Per point in the grid, bilinear interpolation computes a 4-term weighted sum, requiring 4 multiplications and additions (7 FLOPs; or 4 FLOPs under an FMA-counting convention) for cach image channel. Thus, the interpolation cost 7C FLOPs. Computing the interpolation weights incurs an additional constant cost of FLOPs per point. Since this operation is performed for T 122 points, the total added cost is T 122 (7C + 8) FLOPs. For the standard & 16 window size, C = 3 channels, and T = 25 tokens, this is 185600 FLOPs added to the tokenization step. We count one multiplication and one addition/subtree /subtractiooo as one FLOP each; non-arithmetic operations (e.g., indexing, clamping, and rounding) are excluded.
More generally, the additional cost sub-pixel sampling has O(T -C) complexity. In contrast, self-attention and MLP blocks incur O C)) and O(T C2) complexity, respectively. Therefore, the introduced overhead is asymptotically negligible.
To complement the results in Fig. 5, we show the exact throughput values for SPoT and the patch-based ViTs in Tab. C.1. Despite introducing sub-pixel sampling and interpolation, SPoT closely matches the throughput
3 Validation accuracy is somewhat lower than what shown in Table duc different validation augmentation schemes. Results Table follow the standard of resizing 256 and center cropping to 224, while the accuracies reported during training in Figure B.1 resized with RandomResi LandomnesizedCrapoooo zedCrop (224, scale=(1.0) (1.0,1.0))
closely align with training geometry.
16

---

Published in Transactions on Machine Learning Research (3/2026)
of a standard ViT across all token counts, indicating that the additional operations incur minimal overhead in practice.
Table C.I: Throughput (k imgs/s) for standard ViTs and SPoT for different token budgets.
Model
25 tok
50 tok
100 tok
196 tok
VIT
34.81
20.41
10.77
3.29
SPoT
34.78
20.39
10.75
3.25
D Additional Qualitative Results
We provide additional examples of oracle trajectorics salient image regions for different priors in Fig. D.1. Similar to Fig. 4, they show that the optimized placements do not significantly align with object saliency.
To complement the results in Sec. 5.1, we also provide visualizations of adversarial oracle trajectories with gradient ascent Figure D.2. This shows that subpixel sampling can also result in reduced performance if adversarial placements are chosen. Similar to the oracle descent case, these placements do not immediately appear good or bad to the human eye, but result a drastic decrease in performance. In worst-case scenarios, probabilistic priors may accidentally sample token placements that lead reduced performance.
We note that if the number of oracle iterations is increased to 20 more, substantial amount of the token placements move outside the image boundary.
17

---

Published in Transactions on Machine Learning Research (3/2026)
Uniform
13
Gaussian
21
08
13
10
Sobol
Isotropic
P
Center
Salient
Figure D.1: Illustration of oracle placements over salient regions for various priors with 25 tokens using SPoT-ON. Salient image regions are highlighted in white while the background has dark mask. Trajectories are colored from dark purple (start) to bright yellow (end).
18

---

Published in Transactions on Machine Learning Research (3/2026)
Figure D.2: Illustration of adversarial oracle placements with 25 tokens using SPoT-ON with gradient ascent. Trajectories are colored starting with dark purple for initial points, with endpoints colored red.
19


Reviews:
[{"review_id": "eYbLC5rs5n", "reviewer": "Reviewer_uzzG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper challenges the conventional reliance on discrete, grid-aligned tokenization in Vision Transformers (ViTs). The authors argue that forcing tokens into a fixed grid is sub-optimal for sparse feature selection, as critical image patterns often straddle grid lines. To resolve this, they introduce Subpixel Placement of Tokens (SPOT), a framework that allows tokens to occupy continuous coordinates within the image space. By leveraging a bilinear interpolation function, the token positions become differentiable, enabling gradient-based optimization. The authors propose SPOT-ON, an oracle-guided search that identifies optimal token locations per image. Furthermore, the paper investigates various spatial priors, demonstrating that object-centric biases significantly enhance performance in sparse regimes, and that optimized placements are transferable across independently trained models.\n\nKey Strengths\n\n1. The work elegantly re-frames the ViT as a \"visual bag-of-words\" by decoupling tokenization from the rigid pixel grid, providing a more flexible inductive bias for sparse modeling.\n\n2. The empirical evidence shows that SPOT-ON can achieve 90.9% accuracy on ImageNet-1k with only 12.5% of tokens (25 tokens), revealing a massive performance \"ceiling\" that current sparse methods fail to reach.\n\n3. In high-sparsity settings, the method outperforms grid-constrained approaches by over 16.9, highlighting the inherent advantage of subpixel alignment.\n\n4. The finding that token placements optimized by one model can improve the performance of another model suggests that SPOT-ON captures fundamental semantic structures rather than model-specific noise.\n\nKey Weaknesses\n\n1. The primary performance gains stem from SPOT-ON, which requires iterative gradient-based optimization during inference. This makes the most powerful version of the proposed method computationally prohibitive for real-time applications.\n\n2. There is a substantial disparity between \"out-of-the-box\" spatial priors and the oracle-guided results. The paper lacks a practical, lightweight mechanism (e.g., a predictor network) to bridge this gap without iterative optimization.\n\n3. The method requires a 50-epoch \"retrofitting\" phase on ImageNet-1k to adapt pre-trained weights to the subpixel tokenizer. While shorter than pre-training, it remains a non-trivial cost for large-scale models.\n\n4. The scope is limited to image classification. It remains unproven whether subpixel flexibility offers similar advantages for dense prediction tasks like object detection or segmentation, where spatial precision is traditionally handled by grid-based heuristics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide multi-faceted evidence, including direct comparisons between grid and subpixel optimization in identical settings. The robustness tests using adversarial oracles and transferability experiments further confirm that the model leverages genuine semantic cues.", "audience_yn": "Yes", "audience_explanation": "This work addresses fundamental questions regarding ViT architecture and efficiency. Researchers focused on efficient AI, Transformer inductive biases, and spatial representation will find the \"visual bag-of-words\" perspective highly valuable.", "requested_changes": "1. Provide a detailed quantitative analysis comparing the FLOPs/latency added by the bilinear interpolation step in SPOT vs. standard linear patch projection.\n\n2. The authors mention learnable priors as future work. Please expand this into a concrete discussion on the feasibility of training a lightweight \"policy network\" to predict SPOT-ON placements in a single forward pass.\n\n3. Include an ablation study showing how performance scales with the number of retrofitting epochs (e.g., results after 5, 10, and 25 epochs) to identify the minimum adaptation required.\n\n4. Supplement the qualitative results with examples where SPOT-ON fails to improve performance, helping the audience understand the limits of subpixel sampling.", "broader_impact_concerns": "The paper focuses on technical efficiency and architectural improvements for AI reliability. There are no significant ethical or social impact concerns beyond the standard biases inherited from large-scale datasets like ImageNet.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qtPCyNCQIW", "reviewer": "Reviewer_31UR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission proposes a method called SPoT to identify the optimal subpixel patch combination for the label \"y\" in the pre-trained DiT model. It could serve as a tool to analyze or investigate which key patches we can select in an overlapped way during model inference to achieve better accuracy. \n\nHowever, the leakage of ground-truth labels leads to the abnormal high accuracy (over 30% improvement on imagenet-1k eval.) and unfair comparison. \nThus, these experimental results can not be used to demonstrate the claim that the proposed SPoT method could surpass and replace the classical image tokenization way.", "claims_and_evidence_yn": "No", "claims_explanation": "Authors challenge the widely-used fixed-grid patchify method with three limitations and claim that their proposed flexible SPoT method addresses these limitations and achieves better accuracy. \n\nHowever, the methodology design and experiment cannot support their claim due to the following cues:\n\n1. Data leakage and unfair comparison. The label information is used to optimize a better combination, causing the ground truth label 'y' to be leaked, as shown in equation (3), which could definitely get a higher accuracy. This is expected, as the optimization direction during patch selection aligns with the classification goal; lower logit loss is associated with higher accuracy.\n2. SPoT is applied on the pre-trained ViT model and cannot be used in model training from scratch in such way.", "audience_yn": "Yes", "audience_explanation": "This work could serve as an analysis tool for the ViT model, like an extension of the gradient-based saliency map at the token level of the ViT-based model, providing a way to identify the flexible patch combinations that most contribute to classification output.", "requested_changes": "I strongly encourage the authors to majorly modify and reorganize the paper to give a clear position for the proposed method. The current submission quite confuses readers about what the paper is exactly doing.\n\nThe current version of the submission shows that the authors post a large motivation, but only propose a simple tool method to analyze the pre-trained DiT model with some unsuitable experiment comparisons. For example, \n1) authors challenge the widely-used patchify method with three limitations, but their proposed SPoT-ON can not be used for model training and only achieves higher inference accuracy via the ground truth label leakage way. \n2) authors post many offline sampling priors in Sec. 3.1 &Figure 3 and recognize the necessity of a learnable prior, but leave this step as future work. etc.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HptDuI3E7R", "reviewer": "Reviewer_ujJj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents SPoT, a sub-pixel tokenization framework, where token positions are defined in continuous coordinates rather than discrete grid cells. This formulation enables differentiable optimization of feature locations. The proposed framework has the potential of enhancing the robustness and efficiency of ViTs. To analyze the nature of sparse ViTs and demonstrate the potential of learnable token positions, the paper proposes SPoT-ON, a gradient-based procedure that identifies near-optimal token placements for each image, providing approximate upper bounds on achievable performance in sparse settings. The paper also explores how different priors (including uniform, Gaussian, Sobol, isotropic, center, and saliency-based sampling) affect sparse token performance, revealing that object-centric priors are better in sparse regimes while coverage-based priors are better in dense ones.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s contributions are concisely summarized at the end of the introduction. The authors accurately point out that SPoT-ON is not a practical solution for inference but instead a tool to demonstrate the potential of learnable token positions. The claims made by the paper are supported by empirical studies across several ViT backbones. Results from tables 1-5 show that sub-pixel placement consistently improves performance over all baselines at varying sparsity levels. The paper also includes additional experiment results for robustness and transferability, showing the generalization ability of the learned token placements.", "audience_yn": "Yes", "audience_explanation": "The findings of the paper is highly relevant to the machine learning community, especially researchers working with computer vision and vision transformers. The proposed continuous and differentiable tokenization scheme has the potential of making vision transformers more efficient. The paper also offer insights into sparse vision modeling, helping researchers understand where ViTs optimally look at when constrained by token budgets.", "requested_changes": "- Include an analysis on computational overhead: While throughput is briefly mentioned, the cost of sub-pixel sampling and interpolation is not analyzed in detail (e.g., per-image latency, FLOPs). It would be helpful for the readers to judge the potential of the proposed method by including a computational complexity and runtime comparison with baseline ViTs.\n- The paper claims that SPoT provides a new direction for interpretable ViT architectures, and the visualizations in Figures 1 and 2 also hints that the sub-pixel patches will be more interpretable and more aligned with the salient objects in the frame. However, the visualizations in Figure 4 does not show any alignment between oracle placements and the salient objects in the frame. As a result I think the paper could be better motivated with some different visualizations as figures 1 and 2. I also encourage the authors to provide more visual examples of saliency maps vs. oracle token positions to better support interpretability claims.\n- The authors can discuss the potential applications of SPoT beyond classification (e.g., localization or video understanding).", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)