MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution
Authors: Wenzhuo Liu
Fei Zhu
Shijie Ma
Cheng-Lin Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-Resolution Vision Model
Vision Transformers
Multi-Scale Patch Embedding
Primary area: machine_vision

Abstract: Although Vision Transformers (ViTs) have recently advanced computer vision tasks significantly, an important real-world problem was overlooked: adapting to variable input resolutions. Typically, images are resized to a fixed resolution, such as 224x224, for efficiency during training and inference. However, uniform input size conflicts with real-world scenarios where images naturally vary in resolution. Modifying the preset resolution of a model may severely degrade the performance. In this work, we propose to enhance the model adaptability to resolution variation by optimizing the patch embedding. The proposed method, called Multi-Scale Patch Embedding (MSPE), substitutes the standard patch embedding with multiple variable-sized patch kernels and selects the best parameters for different resolutions, eliminating the need to resize the original image. Our method does not require high-cost training or modifications to other parts, making it easy to apply to most ViT models. Experiments in image classification, segmentation, and detection tasks demonstrate the effectiveness of MSPE, yielding superior performance on low-resolution inputs and performing comparably on high-resolution inputs with existing methods.

MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution
Wenzhuo Liu Fei Zhu², Shijie Ma Cheng-Liin Liu
Schooo of Artificial Intelligence, UCAS
2 State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA *Centre for Artificial Intelligence and Robotics, HKISI-CAS
(1iuvenzhuo2020, zhufei2018, mashijie2021)@ia. ac. en, liucl@nlpr ia ac
Abstract
Although Vision Transformers (ViTs) have recently advanced computer vision tasks significantly, an important real-world problem was overlooked: adapting to variable input resolutions. Typically, images are resized to a fixed resolution, such as 224x224, for efficiency during training and inference. However, uniform input size conflicts with real-world scenarios where images naturally vary in resolution. Modifying the preset resolution a model may severely degrade the performance. In this work, we propose to enhance the model adaptability to resolution variation by optimizing the patch embedding. The proposed method, called Multi-Soan Scale Patch Embedding (MSPE), substitutes the standard patch embedding with multiple variable-sized patch kernels and selects the best parameters for different resolutions, eliminating the need resize the original image. Our method does not require high-cost training or modifications to other parts, making it easy to apply to most ViT models. Experiments in image classification, segmentation, and detection tasks demonstrate the effectiveness of MSPE, yielding superior performance on low-resolution inputs and performing comparably on high-resolution inputs with existing methods. Code is available at https //gi thub. com/ /Sammmallll "SmallPigPeppa. MSPE.
Introduction
Vision Transformers (ViTs) [1, 2, 4] have achieved significant success in various computer vision tasks, becoming viable alternative to traditional convolutional neural networks [5, 6, 7, 8]. VIT divides an image into multiple patches, converts these patches into tokens via the patch embedding layer, and feeds them into the Transformer model [9, 10]. The token representations are usually obtained using a convolutional neural network (CNN). Early CNN architectures like AlexNet [11] were designed for fixed-size images (c.g.. 224x224). For easy comparison, this setting has been maintained by subsequent image recognition models, including ViT [2], For fitting neural network input layer size, ViTs typically resize the input image to fixed resolution and divide into specific number of patches [12, 13, 14]. This practice restricts VIT models to processing single-resolution inputs. However, fixed input sizes conflict with to camera deviccesparameeerr, object size, and distance. This discrepancy can significantly degrade images. Changing the preset size requires retraining, and
scenarios where image resolution varies due
ViT's performance on liffereen-resolutte every resolution necessitates an individual model. Therefore, a natural question arises: Is it possible for a single ViT model to process different resolutions directly?
"Corresponding author.
"The term resolution refers to the width and height of images input into neural networks. In this paper. resolution and image size used interchangeably.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) height width
(b) height = 128
ImageNet-11 Acc@1
80%
80%
60%
70%
ImageNet-1K
40%
60%
Vani la FlexiviT MSPE
Vanilla
20%
50%
FlexiVIT
0%
40%
MSPE
28
56
84 112 140 224 896
56
112 140 224 896 width
test width
Figure 1: MSPE results on ImageNet-IK. We loaded ViT-B model pre-trained on ImageNet-211
from [19] and evaluated: (a) Height equals width, ranging from 28x to 896x896, and (b) Fixed height=128, width ranging from 28 to 896. Vanilla ViT performance drops with size/aspect ratio changes; ViT [15] significantly improves performance, and our method surpasses FIeXiVIT.
A few recent methods have been proposed for this problem: FlexiViT [15] uses novel resizing method allowing flexible patch size and token length in VIT. ResFormer [16] enhances resolution adaptability through multi-resolution training and global-local positional embedding strategy. NaViT [17] leverages example packing [18] to adjust token lengths during training, boosting efficiency and performance. However, these methods have limitations. First, some of them incompatible with existing transformer models, preventing the effective use of pre-trained ViT', and thus lead to high re-training costs when applied. For example, models like NaViT and ResFormer require training the entire model to achieve multi-resolution performance. Second, their multi-resolution performance insufficient. For example, FlexiViT demonstrates only slight performance improvement when the input images are smaller than the preset size.
We intuitively believe that an ideal solution should be compatible with most existing ViT models and perform well at any size and aspect ratio. To this end, we propose a method called Multi-Scale Patch Embedding (MSPE). It replaces the patch embedding layer in standard ViT models without altering the other parts of the model. This design makes MSPE compatible with most ViT models and allows for low-cost, direct application. Specifically, our method uses kernels instead of fixed ones, which can automatically adjust the size and aspect ratio of the kernels based on the input resolution. directly converts images into patch embeddings without modifying the input size or aspect ratio. The results in Figure show that this simple method significantly improves performance, for example, increasing ImageNet-IK accuracy up 47.9% across various resolutions. Our contributions are as follows:
of learnable adaptive convolution
By analyzing resolution adaptability ViT models, we identify the patch embedding layer as the crucial component and provide low-cost solution.
We propose Multi-Scale Patch Embedding (MSPE), which enhances ViT models by sub- stituting the standard patch embedding layer with learnable, adaptive convolution kernels, enabling ViTs to be applied on any input resolution.
Experiments demonstrate that with minimal training (only five epochs), MSPE significantly enhances performance across different resolutions on classification, segmentation, and detection tasks.
2 Preliminaries and Analysis
2.1 Vision Transformer Models
Vision Transformers (ViTs) leverage the capabilities of the transformer model [20, 21J, initially developed for natural language processing, to address vision tasks [22, 23, 24J. ViTs primarily consist of the patch embedding layer and Transformer encoder.
Patch Embedding converts an image 1 where E Rd is a vector in patchification space Z. This transformation is achieved via patch embedding layer go X + Z, parameterized by Specifically, the function go implemented as convolution convo, using kernel wp Xue xd and bias by E Rd, The kernel size (h,, wx) and the stride (hs, w.). existing ViTs, patch embedding methods are categorized into non-overlapping and overlapping types:
R.xaax the input space X' into a sequence of tokens

---

Non-overlapping patch embedding: In standard Vision Transformers (e.g., ViT [2]), the stride of conva matches the kernel size, i.e. h2 h2 and ws Wk. The number of tokens N is calculated by
Overlapping patch embedding: In models like PVT [4] and MViT [25], the stride smaller than the kernel size, i.e. hg h2, W, wy. The number of tokens N is calculated 1... + pl where p is the padding size.
by h hx
+ p
h2
Transformer Encoder adds position encodings pos; the token sequence, modifying to {z]],=] {z; + pos A class token Rets for global semantics is added to the front, i.e., {Felss zi...., Z/ This sequence feeds into the Transformer Encoder Enca parameterized by Φb For classification, only the class token Zets is used to produce a probability distribution, and other tokens {zi,..., 2'x² are discarded.
2.2 Problem Formulation
In real-world scenarios, images have variable resolutions due to variations camera devices, pa- rameters, object scale, and imaging distance. Our task is ensure the model adapts to different resolutions. To this end, we resize the same image to multiple resolutions and optimize the model for these diverse inputs. Here is the formal definition of this task:
Firstly, the process of resizing (e.g. bilinear reize) formally defined as a linear transformation:
resize," (o) = B."" vec(o..
(1)
where o E fxw is any input, the resolution r of (h, w) transforms to r* of (h*, w*) after resizing, using the transformation matrix BT E hRhww*x/w Each channel of resized independently.
For input (x, y) in dataset D, the learning objective is to minimize the loss function l (c.g., cross- entropy loss) across series of resolutions 1M optimizing performance at each resolution from T1 to TM:
min ((m (((nee (B)) Vr: (r
0
(2)
S.L. (a (((En) w)] & ((a.)) (ee=)))=(((Encc((() (())),((((()))))) v)] + c,c > 0,
where (z) and (z) are Transformer encoder and patch embedding layer, respectively. The slack variable allows minor loss increments in the well-trained model (ge (x)).
The central challenge is maintaining acceptable performance across different resolutions from to TM but keeping the performance of original resolution r, which means adjustments of and é must be careful. In this work, we confirm the key role of patch embedding and only optimizing ge:
i E arg min
Vra E (r:)
(3)
0
2.3 Pseudo-inverse Resize
To solve the optimization problems stated in Eq. (2) and (3), an intuitive solution appears to ensure that embedding layer Go produces consistent features on different resolutions. For this purpose, FlexiViT [15] proposed Pl-resize, which adjusts the embedding kernel to ensure uniform outputs:
(wo, bot = θ e arg min Ex-.x (197((") (")) x) ge (z)))² Vrx E (r.)-
(4)
We arg min E [( wo) - (B n. .,, [6
we
When upscaling with r >> the analytic solution for Eq. (4) is Wp B, (B, B denote (Br
Wa:
( x, (ea = x" B B. (B") B we x² = (x, we).
(5)
When downscaling with ra < r. the matrix Br, T Br, is non-invertible. Under the assumption X' = (o0 1), it is proven that (Br:")+ur Wa is the optimal solution. In summary, Pseudo-inverse resize (PI-resize) is defined follows: Pl-resize," (w) = (B, T>* vec(w).
(6)
3

---

(a) Illustration Motivation
1b) Patch Embedding
(c) Class Token
15.0
Vanila FeuMTT
Varila
Dimensien
Ence
chere trrent
spaact
Density
110.0
12.5
Denstty
x
FleatviT
Lils
7.5
patch embeeddinng spacce
Fitts
2.5
0.0
0.5
0.6
0.7 0.8 0.9
0.2
0.4
0.6
0.8
Dimansion
artty
Cosine Similanty
Figure 2: Similarity in patch embeddings does not guarantee optimal performance (a). We confirm this by evaluating the accuracy and cosine similarity of: (b) patch embeddings { from 56x56 and 224x224 images, and (c) class tokens Zels from 56x56 and 224x224 images.
J
2.4 Motivation
However, is this optimization target FlexiViT [15] truly appropriate? We suggest that there are two problems with Equation 4: First, it is a stricter sufficient condition of Eq. (3), but it ignores the impact the encoder Enc (z) and the objective function f. Second, at lower resolutions, i.e. < this goal has analytical solution; Pl-resize is jujtttt an approximation assuming NO 1), resulting in significant performance degradation.
Moreover, we intuitively suspect that similarity in patch embeddings does not ensure the best performance. As Figure 2(a) illustrates, features derived from patch embeddings are transformed into the classification feature space, namely the class token space, through the Transformer encoder Encæ. The gradient directions of these two feature spaces may not align, resulting image features that are close to optimal in patch embedding being far from the optimal class token after encoder processing.
A more effective method is directly adjusting the weights wy of embedding layer using the objective function & (Eq. (3)). To verify this idea and our assumptions, we evaluate a well-trained ViT- B/16 model [2] (pre-trained on ImageNet-21 224x224, 85.10% accuracy). We measure the cosine similarity between patch embeddings [ and the class token Zets at resolutions of 56x56 and 224x224. The results in Figures (b) and (c) show that FiexiViT has higher patch embedding similarity and classification accuracy compared to the vanilla model; however, our method significantly outperforms FiexiViT with even lower patch embedding similarity. These results confirm that our analysis is reasonable.
Method
As discussed in Section 2.4, optimizing patch embedding layers through objective function is simple but effective. Motivated by this, we propose Multi-Scale Patch Embedding (MSPE). divides the resolution domain into different ranges {ra } each using that adjusts size through Pl-resize. MSPE can replace the patch embedding layers most VIT models, including overlapping and non-overlapping types. Our method is illustrated in Figure 3 and presented below.
-eiggt patch embedding layer
3.1 Architecture of MSPE
MSPE only changes the patch embedding layer of the ViT model, making it directly applicable to well-trained ViT model. As demonstrated in Figure 3, we introduce the following architectural modifications.
Multiple patching kernels. The typical patch embedding layer go employs single convolution size with parameters (wo, bo), making it unsuitable for varying image resolutions. To overcome this, MSPE incorporates K convolutions with differenet kernel sizes {g]. .......... g6 ). where each gè is parameterized by (wy, bb), to support broader range of input image sizes.
Adaptive patching kernels. Although K convolutions different kernel sizes improve adaptability, they cannot cover all possible resolutions. Setting a unique size convolution kernel for each reso- lution is unrealistic. In MSPE, the size and ratio of the kernel (un). by) are adjustable rather than fixed. Specifically, for input of any resolution (h, w). the corresponding kernel size (he, wx) is

---

Class Bail Car
Vision Transformer (VIT) with MSPE
Multi-Scale Patch Embedding
MLP Head
Transformer Encuder
add
Patch + Position
2
5
6
Embedding
Extra tosrablle
Kg
[class] ambhedding
Multi
Patch Embedding (MSPE)
mabyy
K =
divide
Figure 3: Illustration of the ViT model [2, 3] with MSPE. MSPE only replaces the patch embedding layer in the vanilla model, making well-traines VIT models to be directly applied to any size and aspect ratio. In our method, the patch embedding layer has several variable-sized kernels. The Transformer encoder is shared and frozen.
(Lh/N], w//N)) Using Eq. (6), we adjust wi to the corresponding size wi E A xon, As shown in Figure 3 (b), it can be directly convolved with the image patch.
Another module requiring careful design feature map size after patch embedding. In ViT [2] and DeiT [3], the position embedding is bilinearly interpolated to match the feature map size, a method known as resample absolute pos embed, and it has proven effective during fine-tuning [3]. In MSPE, we discover that bilinear interpolation of position embedding satisfies our requirements. To simplify the architecture and minimize changes to the vanilla model, we follow this dynamic positional embedding method.
position embedding, which must correspond to
3.2 Learning Objectives
MSPE optimizes the patch embedding layer through the objective function but does not explicitly constrain the embedded tokens { J.M across different resolutions. Specifically, MSPE is based on multi-resolution training (mixed resolution training), similar to methods like ResFormer and NaViT. The training process as follows.
Firstly, the mixed resolution (ri),, divided into subsets {S* i.c. UK_ S JM and is randomly sampled from Sp. The patching kernel weights O₁ are shared within Sy and transformed into the corresponding weights 02 for each r; according Eq. (6), The loss function is defined as:
K
vl Co y) = Σ ((En ( (((Enc()00(z)). v).
where é is the task loss function (e.g.. cross-entropy loss), and À is hyperparameter to prevent performance degradation. We optimize only the patch embedding parameters O during training, setting the learning rate of &f zero. Algorithm in Appendix E.2 details the training procedure of MSPE and PyToch-style implementation.
3.3 Inference on Any Resolution
For input æ with any resolution r", previous models resize the image to fixed resolution r. The inference procedure is:
Eno (B)+))),
(8)
With MSPE, the model can infer directly from the original image without resizing or altering its aspect ratio. The process is:
Fine (go" (m)),
(9)
5

---

Table I: ImageNet-Ie IK Top-1 accuracy across 28x28 trained for 5 epochs, while ResFormer [16] was trained for 200 epochs, all methods based on the same well-trained model.
28 to 448 448 resolutions: Our method only
Resolution
Method
28
42
56
70
84
98
112
126
140
168
224
448
Vanilla [2]
8.52
37.86
54.67
65.94
72.58
76.29
78.75
80.62
82.00
83.66
85.10
53.81
VFT
ResFormer [16]
2.21
19.29
45.41
59.92
69.30
74.34
77.92
77.48
79.61
81.30
83.04
81.06
FlexiViT [15]
14.86
48.52
63.44
70.53
74.47
78.03
80.24
82.28
83.10
84.70
85.10
85.11
MSPP
56.41
71.02
77.94
79.54
81.63
82.51
83.75
83.81
83.94
84,70
85.10
85.11
Vanilla [30]
0.14
2.45
17.93
40.30
57.23
67.01
72.80
76.20
79.62
83.51
85.66
62.29
DeifIII
ResFormer [16]
1.08
20.27
46.52
60.78
70.28
75.28
78.80
78.79
80.48
82.13
83.47
82.53
FlexiViT [15]
3,76
26.44
49.99
62.72
20:47
74.55
77.57
79.22
80.69
83,51
85.66
85.53
MSPE
48.71
65.66
75.37
77.78
80.76
81.68
83.49
83.28
83.75
84,86
85.66
85.53
Vanilla
8.78
28.20 20
47.52
53.54
61.11
68
76.45
78.29
80.21
81.73
83.12
73.48
PVT
FlexiViT [15]
16.64
48.13
60.21
65.39
70.44
73.27
76.70
78.29
80.21
81.73
83.12
81.57
MSPE
34.20
60.76
72.09
75.39
77.39
78.08
80.36
81.12
81.59
82.00
83.12
82.43
height= 192
height = 256
height = 384
85
85
Accuracy
80
Accuracy
Accuracy
80
82
75
Vanilla FlexiVIT
Vanilla FlexiViT
Vanilla FlexiVIT MSPE
70
75
79
MSPE
MSPE
60
192
144
80
70
192
128
76584
384
288
192
96
width
width
width
Figure 4: ImageNet- IK Top-1 accuracy curves, fixed heights at 192, 256, and 384. Results show
MSPE directly applied across varying input ratios and enhancing performance.
where 0* is the patching weight that matches the resolution r*. Details on the computation of l provided in Appendix E. and (10).
Experiments
Datasets. We conduct experiments on 4 benchmark datasets: ImageNet-IK: [26] for classification tasks, ADE20K [27] and Cityscapes [28] for semantic segmentation, and COC02017 [29] for object detection.
Backbone networks. Our method is assessed on three ViT models, including ViT-B [2] and DeiT-B III [30] (non-overlapping patches); PVT v2-B3 [31] (overlapping patches). VIT and DeiT III are pre-trained using the ImageNet-21K and ImageNet-22K datasets.
Implementation details. MSPE is trained using SGD optimizer for five epochs, with a learning rate of 0.001, momentum of 0.9, weight decay of 0.0005, and batch size of 64 per GPU. To validate our model, we implement ViT [2] and other networks [30, 25] via open-sourced timm library for classification, ViTDet [32] via MMDetection [33] for object detection, and SETR [34] via MMSegmentation [35] for segmentation; additional details are available in Appendix B.
4.1 Image Classification
Comparison with FlexiVIT. As shown in Table and Figure 4, ViT models using non-overlapping (like ViT and DeiT III) and overlapping patch embeddings (like PVTv2 and MViTv2) significantly lose accuracy when the input resolution varies. This is consistent with the results from [15, 16, 17], demonstrating that the vanilla VIT models are not adaptable to input resolution changes. This issue primarily arises from their patch embedding layers failing to adjust to varying resolutions. This leads to high-level features shifting after the patch tokens are fed into the Transformer encoder and significantly degrading performance. ViT shows remarkably stable performance at upscaled resolutions (e.g., 448x448) by ensuring consistency patch tokens across different resolutions, out- performing vanilla models. However, Section analyzes, Flexi VIT still struggles with downscaling.

---

Table 2: Comparative results of semantic segmentation ADE20K and Cityscapes, using well- trained SETR Naive [34] as the segmentation model (ViT-L backbone), evaluated by mIOu, mACC, and F1 -core...
128x
256x256
512x512
Method
ADE20K
mIOU
mACC
Fl-score
mIOU
mACC
Fl-score
mIOU
MACC
Fl-score
Vanilla
2.89
4,91
4.64
7.48
10.81
15.88
45.39
55.70
59.71
FlexiViT
31.44
38.86
44.13
42.97
52.48
57.21
45.39
55.70
59.71
MSPE
39.65
49.01
51.37
44.39
53.73
58,90
45,39
55.70
59.71
192x192
384x.884
768x768
Citycaaps
Method
mIOU
mACC
Fl-scoree
mIOU
mACC
F1-score
mIQU
mACC.
F1 F1-score
Vanilla
13.33
19.49
24.62
20.10
27.34
26.84
77.58
84,94
86. 72
FiexiViT
59.78
68.65
72.74
74.60
82.81
84.58
77.58
84.94
86.72
MSPE
65.90
74.84
77.21
75.79
83.40
85.35
77.58
84,94
86.72
Table 3: Comparative results of object detection and instance segmentation on COCO2017, employing well-trained ViTDeT [32] as the detection model (ViT-B backbone), pre-trained on ImageNet-IK KK via
MAE [36].
1024
Method
APb
AP
AP
AP
AP
APP
APm
AP
AP
AP
APm
API
Vanilla
0.52
0.72
0.57
0.56
0.66
0.46
0.69
0.50
0.27
0.49
0.64
Vanilla
0.34
0.50
0.36
0.19
0.37
0.45
0.29
0.47
0.31
0.14
0,31
0.44
512
FlexiViT
0.44
0.63
0.47
0.27
0.47
0.57
0.39
0.61
0.41
0.21
0.42
0.56
MSPE
0.47
0.68
0.49
0.50
0.62
0.42
0.64
0.43
0.22
0.61
Vanilla
0.03
0.05
0.03
0.01
0.04
0.05
0.03
0.05
0.03
0.01
0.04
0.05
255
FlexiViT
0.19
0.31
0.20
0.10
0.23
0.29
0.17
0.29
0.17
0.07
0.19
0.28
MSPE
0.30
0.42
0.29
0.34
0.44
0.27
0.39
0.27
0.12
0.43
Our method significantly boosts accuracy with targeted optimization goals, outperforming FiexiViT across various resolutions and aspect ratios.
Comparison with ResFormer and VTT. ResFormer improves performance by multi-resolution training across 128x 128, 160x 160, and 224x224 resolutions. However, the modified VIT architecture does not suit networks like PVT and MViT that use overlap patch embedding. Moreover, ResFormer trains for 200 epochs, but method requires only five epochs. NaVIT keeps the original aspect ratio and trains with mixed resolutions from 64x64 to 512x512, leveraging a larger JFT pre-training dataset and longer training cycles (up to 920,000 steps). Table 1 and Figure 5 show that these state-of-the-art methods improve the vanilla model remarkably. Compared to ResFormer and NaViT, MSPE achieves superior multi-resolution performance with far fewer training resources, which proves the essential role of optimized patch embedding layers.
4.2 Semantic Segmentation
To validate the effectiveness of MSPE in semantic segmen- tation, we test the SETR [34] model on ADE20K [27] and Cityscapes [28] datasets, with the vanilla model trained at 512x512 and 768x768 resolutions, respectively. Re- sults in Table 2 show that FiexiViT significantly enhances the vanilla model's ' performance in semantic segmenta- tion (e.g., 59.78 vs. 13.33 the mIOU metric). This confirms that adjusting the patch embedding layer effec- tive for pixel-level tasks, demonstrating its critical role in enhancing musti-resolltion robustness. Moreover, MSPE consistently outperforms FiexiViT across various resolu- tions, proving our method is ready for pixel-dense tasks in real-world scenarios.
Resolution
7 160-
Vanilla
n=
MSPE
60%
65%
70%
75%
8d%
85%
ImageNet-1K Accuracy
Figure Comparison of MSPE, Vanilla, and NaViT: only NaViT was pre-trained on the JFT dataset, baseline results come from [17].
4.3 Object Detection
In our experiments on the COCO2017 dataset for object detection and instance segmentation, utilize the ViTDeT [32] model with ViT-B (pre-trained on ImageNet-IK via MAE [36]) and
7

---

InageVet-1K Acci))
90%
impact Training Epochs
image 1KK Acc8p1
90%-
Impact of Mode! Size
DeiT34
InageNett: 1K Accoe]
85%
Impact of
Imagefeet 1KK Accig1
7/7
Ippact of
80%
70%
60%
S
80%
70%
Stat
K-
60%
132
224
56
224
75%
56
132
224
56
112 Resalution
224
Resolution 4
Resalution
Resolution 4
Figure 6: Comparison results: (a) different train- ing epochs; (b) model sizes of S. B, and L.
Figure 7: Comparison results: (a) hyperparameter X; (b) differenet kernel count K.
Impact of Resize Method
ImageNet-1K Acc@1
Table 4:
Parameter and computational cost of the patch embedding layer de in MSPE and Vanilla models, the parameter count of the entire model remains nearly unchanged.
80%
60%
Area Bilinear
Model
Method
Parms(gu)
FLOPs(ge)
Parms(all)
50%
Vanilla
590.59K
86.57M
40%
Pl- Re-sesize
VIT-B
28
56
140 224 252
Vanilla MSPE
10.50K
2.83
45.25M
test resolution 5
PVTv2-B3
29.70K
0.13-2.83
45.27M
Figure Comparison results of different resiz- ing methods in MSPE. Pl-resize shows the best. performance and robustness.
employed Mask R-CNN [37] the detection head. During the evaluation, we replace the ViT's patch embedding layer with MSPE or FlexiViT, keeping the rest of the architecture unchanged ( aligned with classification and segmentation ). shown in Table 3, MSPE significantly improves the multi-resolution performance of well-trained detection models. This result is consistent with those of classification and segmentation tasks, demonstrating the effectiveness our method across different visual tasks.
4.4 Ablation Study and Analysis
Training epochs. Figure 6 (a) presents the performance of MSPE on different training epochs (1, 5). is observed that MSPE significantly enhances performance with only few training epochs. The models trained for and epochs show similar performance, with no significant improvement from additional epochs. Thus, we train MSPE for epochs in our experiments.
Model size. For evaluating the impact of model size on MSPE, we test on ImageNet- 1 across different sizes of DeiT III models (Small (S), Base (B), Large (I.),, all pre-trained on ImageNet-22K. As shown in Figure 6 (b), the results of the larger model DeiT-L and the smaller model DeiT-S align with the main experiment; the larger model yields higher accuracy at different resolutions. This result demonstrates the effectiveness of our method across models of different sizes.
Hyperparameters. We conduct ablation studies of hyperparameters À in Figure 7 (a). When lambda set to O, the learning of patch embedding is too flexible, leading to inadequate alignment with the original parameter space. Lambda values of 1 and lead to similar performances; this hyperparameter is set to in our experiments.
Kernel count K. Figure 7 (b) shows the impact of different kernel quantities on model performance. In MSPE, the resolution r; is divided into K subsets, with each subset sharing the patch embedding layer weights. Therefore, the number of subsets K equals the number of patchify kernels in MSPE. The results indicate that slightly increasing K can improve performance. Specifically, when K = and K = 4, the model performance is nearly identical, and additional kernels provide little improve- ment. This suggests that patch embedding parameters can shared across different resolutions. In method, K is set to 4.
Resizing method. our method, the patch embedding weights are dynamically resized for images with different sizes and ratios, denoted as adaptive kernels in Section 3.1. To evaluate the effect of resizing methods on MSPE, we load an ImageNet-21K pre-trained VIT-B model from [19] and train MSPE using different resizing methods, such as standard linear resizing. As shown in Figure 8, the

---

Table 5: Comparison results of image resizing and MSPE on ImageNet- IK Top-1 accuracy.
Resolution
Method
28
42
56
70
84
98
112
126
140
168
224
448
Deil VT
IMG-resize
52.99
67.51
74.06
76.86
79.16
80.25
80.88
82.04
82.68
83.99
85,10
84.92
MSPE
56.41
71.02
77.94
79.54
81.63
82.51
83.75
83.81
83.94
84,70
85.10
85.11
IMG-resize
39,97
58.40 40
66.60
71.16
74.20
76.37
78.15
79.67
80.39
82.01
83.39
82.46
MSPE
45.01
62.54
71.92
73.72
77.09
78.18
80.29
79.94
80.58
82.14
83.39
83.39
PVT
IMG-resize
32.09
52.12
61.54
67.36
71.31
74.05
76.32
78.14
79.45
80.95
83.12
82.34
MSPE
34.20
60.76
72.09
75.39
77.39
78.08
80.36
81.12
81.59
82.14
83.12
83.08
results indicate that Pl-resize consistently outperforms other common resizing methods, aligning with findings from the FiexiViT [15].
Parameters and computation overhead. As shown in Table 4, we analyze the impact of MSPE on the parameter and computational cost of the vanilla VIT model. MSPE modifies only the patch embedding layer Ia. increasing its parameter count by 2x-3x. However, MSPE provides more flexible approach to calculating patch embeddings, significantly reducing computational costs compared to the original method. The total parameter count of the model remains nearly unchanged because the patch embedding layer is an extremely small component.
Image resizing v.s. MSPE. Table 5 shows the comparison results of image resizing (IMG-resize) and MSPE on ImageNet- IMG-resize adjusts images of different resolutions to a preset resolution (e.g., 224x224) during testing, whereas our method keeps the original image size and aspect ratio unchanged. In real-world scenarios, resizing small-size images to a larger size incurs digitization noise and increased computation costs, and model performance is not optimal at different resolutions. The results indicate that our method comprehensively outperforms IMG-resize, suggesting that model inference on the original images is viable option.
5 Related Work
Most relevant to our work is VieXiViT [15], which proposed to resize the patch embedding weights, enabling flexible patch sizes and token lengths in ViT models. Pix2struct [38] supports variable aspect ratios by a novel positional embedding method, enhancing efficiency and performance in chart and document understanding. ResFormer [16] enables resolution adaptability through multi-resolution training and global-local positional embedding strategy. NaVIT [17] enhances efficiency and performance by example packing [18] to adjust token lengths for different resolutions during training. Compared to these methods, MSPE only changes the patch embedding layer and achieves better performance. This justifies the patch embedding layer's 's key role in adapting ViT models for different resolutions.
In CNN-based models, Mind the Pooling [39] addresses overfitting resolution by introducing SBPooling to replace max-pooling, enabling CNNs to process different resolutions. Learn to Resize [40] uses a learnable resizing layer to replace bilinear interpolation. Another study [41] examined the relationship between training and testing resolutions, showing that training at slightly lower resolutions than testing can improve performance. Networks like Resolution Adaptive Networks (RANet) [42] and Dynamic Resolution Networks (DRNet) [43] use multiple sub-models to choose the appropriate resolution and model based on task difficulty, thus enhaneing model efficiency and resolution adaptability.
6 Conclusion
To make VIT models compatible with images of different sizes and aspect ratios, we propose MSPE to replace the traditional patch embedding layer for accommodating variable image resolutions. MSPE uses multiple variable-sized patch kernels and selects the best parameters for different resolutions. eliminating the need to resize the original image. Extensive experiments demonstrate that MSPE performs well in various visual tasks (image classification, segmentation, and detection). Particularly, MSPE yields superior performance on low-resolution inputs and performs comparably on high- resolution inputs with previous methods. Our method has the potential for application various vision tasks, and can be extended by optimizing the embedding layer and transformer encoder jointly.

---

7 Acknowledgments and Disclosure of Funding
This work has been supported by the National Natural Science Foundation of China (NSFC) grant U20A20223 and the InnoHK program.
References
[1] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[3] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, R,, 2021.
[4] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568-578, 2021.
[5] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. pages 4700 4708, 2017.
[6] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenety2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4510-4520, 2018.
[7] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1492-1500, 2017.
[8] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 1114- PMLR, 2019.
[9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[10] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, al. A survey on vision transformer. IEEE Transactions on Pattern Analysis and Machine Inteligence, 45(1)=87-110, 2022.
[11] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
[12] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-rcsolution vision transformer for dense predict. Advances in Neural Information Processing Systems, 017728177993 2021.
[13] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEF/CVP Conference on Computer Vision and Pattern Recognition, pages 12104- 12113, 2022.
[14] Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z Pan. Multi-scale high-resolution vision transformer for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12094-12103, 2022.
[15] Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14496 14506, 2023.
10

---

[16] Rui Tian, Zuxuan Wu, Qi Dai, Han Hu, Yu Qiao, and Yu-Gang Jiang. Resformer: Scaling vits with multi-resolution training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22721-22731, 2023.
[17] Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36, 2024.
[18] Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027 2021.
[19] Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization vision transformers. Transactions on Machine Learning Research, 2022.
[20] Max Jaderberg. Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in Neural Information Processing Systems, 28, 2015.
[21] Chenyang Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. Inception transformer. Advances in Neural Information Processing Systems, 35:23495-23555 2022.
[22] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6836-6846, 2021.
[23] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809-10818 10818, 2022.
[24] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Syxtems, 35:38571-38584, 2022.
[25] Yanghao Li. Chao Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhoffer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4804-48144 2022.
[26] Jia Deng, Wei Dong, Richard Socher, et al. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[27] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 177330--21.. 2019.
[28] Marius Cordts, Mohamed Omran. Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson. Uwe Franke, Stefan Roth, and Bemt Schiele. The cityscapes dataset for semantic urban understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3213 13- 3223, 2016.
[29] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pages 740-755. Springer, 2014.
[30] Hugo Touvron, Matthicu Cord, and Hervé Jégou. Deit iii: Revenge the vit. In Proceedings of the European Conference on Computer Vision, pages 516-533. Springer, 2022.
[31] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):115 424, 2022.
[32] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Proceedings of the European Conference on Computer Vision, pages 280- 296. Springer, 2022.
11

---

[33] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu. et al. Mmneetection: Open millbb detection toolbox and benchmark. arXiv preprint urXiv:/906.07100 2019.
[34] Sixiao Zheng. Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo. Yabiao Wang. Yanwei Fu, Jianfeng Feng, Tao Xiang. Philip HS Torr, al. Rethinking semantic segmentation from a sequence- sequence-to-sequeee -o-sequence perspective with transfomers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6881 1881-, 2021.
[35]
Openmmlab semantic segmentation toolbox and benchmark. https://gtt https::/github.... www.cooooopoo open- mml wtt.mmmmmaaaamaaeeeeeeaamma ab/mmsogmentation 2020.
MMSegmennation Contributors. MMSegmentation:
[36] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[37] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 2961-2969, 2017.
[38] Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming- Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pages 18893-18912, PMLR, 2023.
[39] Bilal Alsallakh, David Yan, Narine Kokhlikyan, Vivek Miglani, Orion Bhattacharya, Mind the pool: Convolutional neural networks can overfit input size. In International Conference on Learning Representations, 2022.
and Pamela
[40] Hossein Talehi and Peyman Milanfar.
Learning to resize images for computer vision tasks. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 497-506, 2021.
[41] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou, Fixing the train-test resolution discrepancy. Advances in Neural Information Processing Systems, 32, 2019.
[42] Xin Wang, Fisher Yu, Ziwei Liang, Thomas Huang, Larry Shi, Mathhw Liu, Jan Kautz, and Alan Yuille. Resolution adaptive networks for efficient inference. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-8738. 2018.
[43] Jinjin Chen, Xifeng Pan, Yu-Kun Lai, and Chao Tang. Dynamic relu. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 11-20, 2020.
[44] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision, pages 213-229. Springer, 2020.
[45] Bharat Singh and Larry S Davis. An analysis of scale invariance in object detection snip. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3578-3587, 2018.
[46] Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. Autoaugment: learning augmentation strategies from data. pages 13-123, 2019.
[47] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced scarch space. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[48] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13001-13008, 2020.
[49] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Paz. mixup: Beyond empirical risk minimization, In International Conference on Learning Representations, 2018.
[50] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023-6032 2019.
12

---

A Limitation
Our paper does consider the impact of position embedding on model performance. This because existing work [2, 3] has shown that linear interpolation of position embedding can achieve acceptable performance. Our experimental results validate the feasibility of this strategy. Moreover, we aim to utilize our method within most existing VIT models, hence we do not alter the position embedding method. In the future, our method can be extended with the positional encoding strategies proposed in related works, such as ResFormer [16] and Pix2struct [38].
B Training details
B.1 Multi-resolution training
Multi-resolution training has become widely used paradigm [44, 45], randomly altering crop size and aspect ratio during training. This training strategy critical in multi-resolution vision models. In FlexiViT [15] training. the size of image patches and the kernel size of the patch embedding layer are randomly changed from 8x8 to 48x48. In ResFormer [16]. input images are resized to 128 x 128, 160x 160 and 224 x224 and simultancously training. In ViT [17], images retain their original size and aspect ratio and are packed into a single pack for training, denoted as mixed-resolution training. The range of its training resolutions is 64 256. Our method is most similar to FlexiViTy randomly resizing each image batch to different resolutions (from 56 to 256) during training.
B.2 Data augmentation
Classification. Our method leverages timm implementations of VIT [2). DeïT III [30], and PVT v2 [31] models for classification tasks. The training data augmentations include RandomResizedCrop RandomHorizzotallFig and ColorJitter. For Resfomer [16]. follow the official-releasee code. Its training data augmentation includes Auto-Augment [46], RandAugment [47], random crasing [48]. MixUp [49], and CutMix [50].
Semantic segmentation. We use the SETR [34] model implemented by MMSegmentatiin [35]. We adhere data settings in MMSegmentation, including RandomResizeeCrrpp RandomFlip, and Photo- MetrroDisto PhotoMetricDistorttin involves series of transformations: random brightness, random contrast, random saturation, random hue, and converting color between HSV and BGR.
Object detection. We employ the ViTDeT [32] model within the MMDetection [33] for object detection and instance segmentation tasks. We utilize the configurations from MMDetection, which include RaidomResizeddrrp and RandomFlip.
C Compute resources
This paper conducts experiments on machine equipped with two AMD EPYC 7543 32-core processors; each slotted with 32 cores supporting two threads per core. The machine has 496 GB of memory and 8* NVIDIA GeForee RTX 4090 graphics cards. Our method significantly reduces computational resources compared to previous approaches, and all main experiments on ImageNet- completed within 3 hours. In contrast, methods like ResFormer [16] require over 50 hours of training on this machine.
D More experimental results
D.1 Impact of aspect ratio
In Section 4.1, we fixed the height and varied the width, ensuring the height was greater than the width. We also tested the accuracy when fixing the width and varying the height, in which case the width was greater than the height. can seen in Figure that MSPE still achieved significant performance improvements.
Furthermore, comparing the accuracy resolutions (h, w) and (we, h) when > w, we observe that the accuracy of (h, w) resolution was higher than (w, h) resolution. This aligns with the findings
13

---

width 128
width 192
85
80
Accuracy
70
Accuracy
80
75
60
Vanilla
Vanil
70
50
MSPE
FlexiViT
65
FlexiVIT
MSPE
40
601
128
height
height
width = 256
width 384
85
85
Accuracy
80-
Accuracy
82
Vanilla FlexiViT
Vanilla FlexiViT MSPE
79
MSPE
256
192
128
64
76:84
288
192
96
height
height
Figure ImageNet-1K Top-1 accuracy curves, fixed width at 128, 192, 256, and 384.
of NaViT [17], which indicate that larger proportion of images in ImageNet- is h >> This suggests that maintaining the original aspect ratio during inference needs to be considered.
D.2 Results on Higher Resolution
Below experimental results for high-resolution images. We resize ImageNet images (processed to 224x224) up to 4096x4096 and evaluate models with non-overlapping patch embedding (ViT) and overlapping patch embedding (PVT). The results in Table 6 show that MSPE can be effectively applied to high-resolution images, as the shape and size of the patch embedding kernels dynamically adjust based on input resolution and have multiple convolution kernels corresponding to different resolution ranges.
Table 6: Comparison results of higher resolution.
Resolution
Method
224
448
672
896
1120
1792
2240
2688
3360
4032
VIT
Vanilla
85.10
53.04
5.17
0.68
0.33
0.13
0.08
0.08
0.08
0.08
MSPE
85.10
85.11
85.13
85.14
85.14
85.15
85.14
85.14
85.13
85.16
Vanilla
83.12
73.64
71.18
67.04
67.04
64,19
63.73
63.50
64.44
63.26
MSPE
83.12
82.44
83.11
82.76
83.02
82.89
82.93
82.65
82.97
82.58
D.3
Extra training epochs
In main experiment, MSPE is trained for 5 epochs. We increase the number of epochs analyze its performance under extended training. As shown in Table 7, the performance improvement with longer training epochs is minimal. This because MSPE requires learning very small number of parameters, which can reach optimal performance within just few epochs of training. This indicates that our method achieves better performance and requires less training overhead, making highly compatible with well-trained models.
Table 7: Comparison results of extra training epochs.
Resolution
Epochs
28
42
56
70
84
98
112
126
140
168
224
448
28.07
55.79
67.61
73.27
78.32
80.52
82.92
83.21
83.81
84.73
85.10
85.11
54.81
70.68
77.69
79.17
81.49
82.36
83.73
83.65
83.92
84.72
85.10
85.11
56.42
71.01
77.96
79.54
81.64
82.51
83.74
83.80
83.95
84.70
85.10
85.11
57.82
71.42
78.12
79.91
81.68
82.62
83.72
83.83
83.99
84.70
85.10
85.11
20
58.88
71.90
78.22
80.07
81.81
82.79
83.79
83.83
84.02
84.70
85.10
85.11
14

---

Impact of Resize Method
Computation overhead of MSPE
ImageNet- Acc@))
80%
Non-overlap
Overlap
60%
FLOP
2
Pi-Resize
Nearest Bicubic
40%
28
56
84
112
140
224
252
28
56
84
112
140
224
448
test resolution
test resolution
Figure 10: Comparison results of different resiz- Figure 11: Computational Overhead of MSPE at ing methods in MSPE.
Different Resolutions.
D.4 Other resizing methods
We test other resizing methods in MSPE, including nearest and bicubic'. As shown in Figure 10, these traditional resizing methods are not suitable for adjusting the size of neural network parameters, leading to less robust patch embeddings in MSPE training. The experimental results are consistent with Section 4.4, confirming that the combination of Pl-resize and MSPE have significant potential applied to multi-resolution models.
D.5
Computation overhead
We evaluate MSPE's computational overhead across resolutions from 32x32 to 448x448. MSPE dynamically adjusts its parameters based on image resolution. For overlapping patch embedding models like PVT and MVIT, MSPE adjusts the convolutional kernel size and stride. For non- overlapping patch embedding models like ViT. MSPE adjusts the convolutional kernel size. As shown in Figure 11, MSPE's computational overhead changes dynamically, avoiding the unnecessary computational costs associated with resizing images fixed resolution.
E
More details on MSPE
In this section, we provide more details about the implementation of MSPE, including modifications the model structure and the training process, and its application for inference any resolution.
E.1 Inference any resolution
For an input image æ with any resolution r, we calculate its patch embedding parameters 0", composed of u" and b*. We directly compute the patch embedding
the original image as follows: 2 = conv(a,, w=") ) + After that, embedding z is fed into the Transformer encoder Enc(z) usual. In this process, calculating 0* is crucial step. We compute 0* by finding the resolution in the resolution sequence ri that is closest to the original resolution r* and then resizing its kernel match the original resolution. This process formalized follows:
M
w² TT ) vvc(ww)) arg
||r: r'll Vrs E (ra)
(10)
0" (w", bb).
E.2 MSPE structure and training process
As depicted in Section 3,1, MSPE modifies two parts of the ViT model, as shown in the class ViT of Algorithm 1. 1) It adds K patch embedding kernels. 2) During patch embedding, the size and aspect ratio of the kernels can be adjusted based on the input. Here, g and Enc represent the patch embedding layer and the Transformer encoder (z),
Bicubic resizing is implemented by F. interpolate (x, mode "adee"biccbic") in PoToch.
15

---

The implementation of multi-resolltion training in Section 3.1 is as follows: We randomly select K resolutions from a range of resolutions {r:} 1 then compute the corresponding patch embeddings. After that, the model computes the loss as usual.
Algorithm MSPE pseudo-impllementatie
model Vit(...) for batch data: ing,
create K images with randon reselutions
hx.1, hw_2, be_K up. wew.oando. Chndoonchoiie(Cr.1,, x.2, r_M])
ing_1, ing.2, .... ing_K IMG-resize hw,1),, IMG-resize (ing, hx.2), .... IMG-resize (ing, hw.K)
z.1, z.2, z_K cop_conviing,,, gi, hel), cop_conv(ang,,, g2, hw2), .... adp_conv King_K, 8_K, bx_3))
y_1, 7.2, .... y_k Enc(z_1), Enc(x_2),
(nc(x_K)
loss_1, 1oss_2,
loss_K - loss(y_1, label), loss(y_2, label), 1oss(y_K, label)
10 1oss 1ose_1 1ose_2 ... loss_
11
[...] backprop and optimize as usual
13 clasa Moduue))odule):
14
def _._init__ (self, starge, K):
init usual
axge["ing_aize"]]
17 self.g un.
arge)
Broce (*+agge)
19
# add K patching kernela of different aizea.
20
for in range (K):
sacle_i (1 1)/4
g.1 Conv2d eizen scale_i, scale_i)), **args)
24
def sdp_conv(self,,
func, hw):
25
func. furemparem( paran(" v"eight"")
26
b Tunc. panccparann""aaaee (aiaaa""
adaptive patching
PI-resiza(o, hu//selr.N)
eatcclenbedding conv (inage,
30
return patch_anbedding
Notes: Changes existing code highlighted via red background.
16

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The related experimental results are Section 2, and we cite FiexiViT's conclu- sion regarding Pl-resize.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Section 4 and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt S ssisssslicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Section 4 and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The random error in the experiments can be ignored and does not affect the conclusions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///eerrppp.eeeee
http///rris.ccppulliiieecpc ctcc//ublic//ee csCuidel ines?
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "wvbOMXivFY", "reviewer": "Reviewer_tsgU", "summary": "The paper aim to address the challenge of adapting ViTs to variable input resolutions, which is a critical issue often overlooked in real-world applications. The authors propose a new method named Multi-Scale Patch Embedding (MSPE), which enhances the patch embedding layer by incorporating multiple variable-sized patch kernels. This approach allows the model to process images of varying resolutions without resizing, thus maintaining performance across different resolutions.", "questions": "See Weakness.", "limitations": "The authors addressed the limitation in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Problem Identification: This paper clearly identifies the gap in current ViT models, i.e. they are unable to handle variable input resolutions effectively. This is a crucial problem as real-world images often come in various resolutions.\n2. Sound Approach: The proposed MSPE substitutes the standard patch embedding layer with learnable adaptive convolution kernels, which allows the model to adjust to different input resolutions dynamically. \n3. Theoretical Analysis: The paper offers a thorough theoretical analysis of the problem and the proposed solution. This includes a detailed discussion on the patch embedding layer's role and the limitations of existing methods like FlexiViT and ResFormer.\n4. Comprehensive Experiments: The authors provide extensive experimental results demonstrating the effectiveness of MSPE across various tasks, including image classification, segmentation, and detection.", "weaknesses": "1. Technical Contribution: Although the proposed method improves the multi-resolution performance of ViTs, it essentially solves an engineering problem from an engineering perspective. \n2. Scalability Concerns: Although the method is low-cost and compatible with existing ViT models, the paper does not fully address potential scalability issues when applied to very large datasets or extremely high-resolution images.\n3. Ablation Studies: The paper could benefit from more detailed ablation studies to isolate the contributions of different components of the MSPE and better understand the impact of each design choice. \n4. Real-World Applications: While the experiments are comprehensive, the paper could include more real-world application scenarios to demonstrate the practical benefits and robustness of MSPE in diverse settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful comments and suggestions, as they have been helpful in refining and enhancing our work. We have thoroughly reviewed all of your points and have addressed your concerns as outlined below:\n\n> #### **W1: Technical Contribution**\n\nThank you for your kind feedback. In fact, images of different sizes and aspect ratios represent different data distributions, and effectively handling these varying resolutions is critical to addressing data distribution challenges in open-world settings. This is particularly crucial for real-world applications, as image resolutions are diverse, not fixed. However, ViTs lack robustness to different resolutions. As demonstrated in our experiments (Sec. 4.1), a model used for a 224x224 resolution significantly underperforms when processing inputs at 448x448 or 128x128 resolutions.\n\nPrevious works faced two major issues: **(1)** introduced complex modules and strategies that are incompatible with existing ViTs architectures, requiring the entire model to be retrained; **(2)** insufficient adaptability across various sizes and aspect ratios, limiting their performance to multiple resolutions.\nOur method does not require costly training or complex modifications, making it easily applicable to most ViT models. Moreover, it significantly outperforms previous methods. The above discussion will be added in the revised version.\n\n\n\n\n> #### **W2: Scalability Concerns**\n\nWe agree that scalability is crucial for method applicability. In Sec. 4 of our paper, we conducted extensive experiments across multiple scenarios and datasets, including ImageNet-1K, COCO2017, ADE20K, and Cityscapes, as well as on various ViT models like ViT, DeiT III, and PVT. \nTo further address your concerns about scalability, we also performed high-resolution tests on ImageNet-1K. The following experimental results demonstrate that our method performs excellently across large datasets and high-resolution settings. We will add those results to the revised version.\n\n|     | **Method** | **Resolution** |            |            |            |            |            |            |            |            |            |\n| :-: | :--------- | :------------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n|     |            | 224            | 448        | 672        | 896        | 1120       | 1792       | 2240       | 2688       | 3360       | 4032       |\n| ViT | Vanilla    | 85\\.10         | 53\\.04     | 5\\.17      | 0\\.68      | 0\\.33      | 0\\.13      | 0\\.08      | 0\\.08      | 0\\.08      | 0\\.08      |\n|     | MSPE       | **85\\.10**     | **85\\.11** | **85\\.13** | **85\\.14** | **85\\.14** | **85\\.15** | **85\\.14** | **85\\.14** | **85\\.13** | **85\\.16** |\n| PVT | Vanilla    | 83\\.12         | 73\\.64     | 71\\.18     | 67\\.04     | 67\\.04     | 64\\.19     | 63\\.73     | 63\\.50     | 64\\.44     | 63\\.26     |\n|     | MSPE       | **83\\.12**     | **82\\.44** | **83\\.11** | **82\\.76** | **83\\.02** | **82\\.89** | **82\\.93** | **82\\.65** | **82\\.97** | **82\\.58** |\n\n\n\n> #### **W3: Ablation Studies**\n\nThank you for your suggestions. We conducted extensive ablation experiments in Sec. 4.4 and Appendix D, including training epochs, model size, hyperparameters, kernel count (K), and resizing methods. The results are displayed in Figures 6, 7, 8, 10, and Table 6. \nThe experiments show that our method is robust to hyperparameters and that increasing training epochs and kernel counts boosts performance. We achieve good performance and basic convergence with 5 training epochs and 4 kernels.\nWe also evaluated different resizing methods, including nearest, bilinear, and bicubic, with PI-resize proving the most effective, as shown in Figures 6 and 8. We hope these detailed results address your concerns about ablation studies. We would revise the ablation study part (Sec. 4.4) to make it clearer.\n\n\n\n\n> #### **W4: Real-World Applications**\n\nWe completely agree with this. Adapting to different resolutions is a key challenge for ViT models in real-world applications. Our method was evaluated on multiple real-world datasets such as Cityscapes, ADE20K, ImageNet-1K, and COCO2017, including tasks like classification, detection, and segmentation.\nThe City dataset collects street scenes from various cities; ADE20K includes scenes from more than 150 settings; ImageNet-1K and COCO2017 are extensive, covering numerous real-world environments.\n\nTables 5 and 6 in our paper show that MSPE significantly improves across various resolutions and task settings. We hope these experiment results confirm the substantial potential for real-world application.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DSHwLBMph9", "reviewer": "Reviewer_RdsV", "summary": "This paper proposes to substitute the standard patch embedding with multiple variable-sized patch kernels. This eliminates the need to resize the original image. Extensive experiment results are shown to demonstrate the benefits.", "questions": "The highest resolution shown is 448, 2x the training resolution 224. A typical cell phone camera has a resolution around 2K. Could the proposed method bridge this 10x gap?", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The problem is well defined and the proposed method is sound. Convincing results are shown to support the claims.", "weaknesses": "NA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable feedback and suggestions. We greatly appreciate your input and completing the necessary experiments. Here is our detailed response to your questions:\n\n>**Q: Experimental Results on More Than 10x Resolution**\n\nOur method can be directly applied to high-resolution images because the shape and size of the patch embedding kernels dynamically adjust according to the input resolution and have multiple convolution kernels corresponding to different resolution ranges. Below are our experimental results for high-resolution images. We tested models with non-overlapping patch embedding (ViT) and overlapping patch embedding (PVT). We will include those results in the revised version.\n\n|     | **Method** | **Resolution** |            |            |            |            |            |            |            |            |            |\n| :-: | :--------- | :------------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n|     |            | 224            | 448        | 672        | 896        | 1120       | 1792       | 2240       | 2688       | 3360       | 4032       |\n| ViT | Vanilla    | 85\\.10         | 53\\.04     | 5\\.17      | 0\\.68      | 0\\.33      | 0\\.13      | 0\\.08      | 0\\.08      | 0\\.08      | 0\\.08      |\n|     | MSPE       | **85\\.10**     | **85\\.11** | **85\\.13** | **85\\.14** | **85\\.14** | **85\\.15** | **85\\.14** | **85\\.14** | **85\\.13** | **85\\.16** |\n| PVT | Vanilla    | 83\\.12         | 73\\.64     | 71\\.18     | 67\\.04     | 67\\.04     | 64\\.19     | 63\\.73     | 63\\.50     | 64\\.44     | 63\\.26     |\n|     | MSPE       | **83\\.12**     | **82\\.44** | **83\\.11** | **82\\.76** | **83\\.02** | **82\\.89** | **82\\.93** | **82\\.65** | **82\\.97** | **82\\.58** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tnAK6pUbfv", "reviewer": "Reviewer_gLBy", "summary": "The paper introduces Multi-Scale Patch Embedding (MSPE), a novel approach to enhance Vision Transformers (ViTs) by allowing them to adapt to variable input resolutions without resizing images to a fixed resolution. MSPE uses multiple variable-sized patch kernels and selects the optimal parameters for different resolutions, maintaining performance across tasks likes image classification, segmentation, and detection.", "questions": "1. Is selecting the hyperparameter λ during the training process crucial for the model's performance? Can different values of λ yield good results?\n\n2. Can you elaborate on the differences between MSPE with FlexiViT?\n\n3. Is there a rationale behind selecting the hyperparameter K and its corresponding shape in MSPE, or was it arbitrarily chosen?\n\n4. Did the authors attempt to apply the model to uncommon resolutions (e.g., 200x150) to test its generalization performance?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. MSPE replaces the standard patch embedding layer in Vision Transformers (ViTs) with multiple learnable adaptive convolution kernels, enabling the effective handling of different input resolutions.\n\n2. MSPE demonstrates improved performance across varying resolutions in classification, segmentation, and detection tasks with minimal additional training.", "weaknesses": "1. This paper does not experiment with other more effective positional encoding strategies. The paper mentions that linear interpolation of positional embedding has achieved acceptable performance in existing work. \n\n2. The paper analyses the importance of patch embedding but lacks corresponding experimental evidence comparing its importance to other components like encoder and positional encoding. Additionally, the components may show different impacts for different tasks; it does not analyze how the importance of these components may vary across different tasks. \n\n3. The training efficiency of MSPE is highlighted, yet the potential long-term benefits of additional training epochs are not fully explored. It would be beneficial to see more detailed comparisons over extended training periods to understand better the model’s convergence behavior and potential overfitting issues.\n\n4. Minor Error: In 2.1 Vision Transformer Models, the shape of image `x` is defined as `h*w*c`, but the kernel shape obtained is `h_k * w_k * d`. Shouldn't the kernel shape be `h_k * w_k * c * d`?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have carefully considered your questions and would like to address them as below:\n> #### **W1: Experiments of Positional Encoding Strategies**\n\nThanks for your comment. MSPE employs the vanilla position encoding approach, a learnable $(N, DIM)$ vector where $N = \\left\\lfloor \\frac{h}{h_k} \\right\\rfloor \\times \\left\\lfloor \\frac{w}{w_k} \\right\\rfloor$ For non-standard input resolutions, we resize the $(N, DIM)$ vector to $(N_{new}, DIM)$ using linear interpolation, commonly used in ViT models as **_resample absolute position embedding_**. \n\nExperiments in Sec. 4 demonstrate that, using the vanilla position encoding strategy, MSPE significantly outperforms other methods with complex position encoding strategies, such as ResFormer and NaviT.\nActually, we intend not to use other complex positional encoding methods as they do not align with pre-trained ViT models. As shown in the results below, changing the position encoding significantly reduces the performance of pre-trained models (ViT/B-16), and retraining does not recover the expected performance.\n| **Method**               | **Training Epoch** |        |        |        |        |\n| :----------------------- | :----------------: | :----: | :----: | :----: | :----: |\n|                          | 0                  | 5      | 10     | 20     | 50     |\n| Learned 2D (NaViT)       | 0\\.10              | 80\\.36 | 81\\.29 | 81\\.85 | 82\\.33 |\n| Global-Local (ResFormer) | 0\\.10              | 77\\.97 | 78\\.80 | 79\\.54 | 80\\.42 |\n| Vanilla                  | **85\\.10**         | -      | -      | -      | -      |\n\n> #### **W2: Analysis of Patch Embedding and Its Relative Importance**\n\nThank you for this comment. We focus on improving patch embedding rather than the encoder or positional encoding for two reasons. \n- **First**, as demonstrated by the analysis in W1, altering the positional encoding strategy necessitates retraining the trained ViT models, incurring significant costs. \n- **Second**, modifying the encoder presents similar challenges. Our goal, as detailed in motivation (Sec. 2.4), is to adapt pre-trained ViT models to different resolutions with minimal changes. \n\nModifying patch embedding is easy to implement, compatible with pre-trained ViT models, and significantly improves performance at various resolutions without compromising the original performance.\n\n\n- Does the analysis yield different conclusions for different tasks? \n\nIn Sec. 2.4, we examined the impact of patch embedding on classification tasks by analyzing the class token at various resolutions. In classification, the class token carries global semantic information and serves as the feature for classification. In segmentation and detection tasks, such as SETR and ViTDeT, the class token is similarly utilized as the feature map for segmentation and detection. Thus, the analysis of the class token is consistent in principle across these segmentation and detection models. The experimental results in Tables 5 and 6 support our findings, demonstrating performance improvement in segmentation and detection tasks.\n\n\n> #### **W3: Analysis of Extended Training Epochs**\n\nThanks for your suggestion. We have provided results for **1, 3, 5, 10, and 20** training epochs in Table 5 of Appendix B, which we hope will address your concerns. These results show that MSPE stabilizes after 5 epochs, and 20 epochs result in slight improvements without overfitting issues. One advantage of our method is that it requires only 5 training epochs, because MSPE only modifies the patch embedding layer of the ViT model and has few trainable parameters. Therefore, very few training epochs are needed.\n\n\n> #### **W4: Minor Error**\n\nThank you for highlighting this error. We agree with your suggestion about the kernel shape, and we corrected it in the revised version of the paper.\n\n> #### **Q1, Q3: Analysis of $\\lambda$ and $K$**\n\n- The loss function is defined as:\n$$\\mathcal{L}_{\\theta}(x,y)= \\sum _{i=1}^K  \\ell [(\\text{Enc} _{\\phi} (g _{\\widehat{\\theta} _i} (B _{r}^{r _i}x)), y ] + \\lambda \\cdot \\ell [(\\text{Enc} _{\\phi}(g _{\\theta}(x)), y ] $$\n$\\lambda$ is a hyperparameter to prevent performance degradation. The ablation study in Figure 7(a) presents results for $\\lambda$ values of 0, 1, and 2, indicating:\n**(1)** The parameter is essential for maintaining performance, with $\\lambda=1$ enhancing accuracy by 4.2% over $\\lambda=0$.\n**(2)** The hyperparameter shows limited sensitivity, as increasing from $\\lambda=1$ to $\\lambda=2$ makes little difference.\nThe revised version will include a wider range of $\\lambda$ results.\n\n- $K$ represents the number of convolutional kernels used for patch embedding. It is impractical to use a unique kernel size for each resolution. In MSPE, both the size and ratio of kernels $(w_{\\theta}^i, b_{\\theta}^i)$ are adjustable. The ablation study in Figure 7(b) demonstrates that $K=4$ is adequate, employing kernel sizes of 16x16, 12x12, 8x8, and 4x4.\n\n> #### **Q2: Differences between MSPE and FlexiViT**\n\nFlexiVit optimizes patch embeddings consistently across resolutions through explicit analytical transformations. However, it has two disadvantages: \n- **First**, it imposes a strict sufficient condition for handling varying resolutions, but the similarity in patch embeddings does not ensure optimal performance. \n- **Second**, in the downscaling scenario, there is no analytical solution for this goal.\n\nDifferently, MSPE optimizes patch embeddings by minimizing the objective function across different resolutions, employing multiple dynamically adjustable patch embedding kernels to handle different resolutions effectively. Experimental results demonstrate that MSPE significantly outperforms FlexiVit.\n\n> #### **Q4: Test at Non-Standard Resolutions**\n\nIn Figures 1, 4, and 9 of our paper, we tested many uncommon resolutions, such as 192x144, 80x192, 896x128, etc. Our method can also be applied to resolutions like 200x150.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iWbwG9O5l1", "reviewer": "Reviewer_9L8r", "summary": "1. They analyze the resolution adaptability in ViT models, identify the patch embedding layer as the crucial component, and provide a low-cost solution.\n\n2. They propose Multi-Scale Patch Embedding (MSPE), which enhances ViT models by substituting the standard patch embedding layer with learnable, adaptive convolution kernels, enabling ViTs to be applied on any input resolution.\n\n3. Experiments demonstrate that with minimal training (only five epochs), MSPE significantly enhances performance across different resolutions on classification, segmentation, and detection tasks.", "questions": "It's impressive that MSPE achieves much better results with low-resolution images. Is it possible to also show the performance with higher image resolutions larger than 1000?", "limitations": "No", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. They found two problems in the optimization target function in FlexiViT and they illustrate similarity in patch embeddings does not ensure the best performance.\n\n2. MSPE allows ViTs to handle variable input resolutions directly, significantly improving performance on low-resolution inputs and maintaining high performance on high-resolution inputs.\n\n3. MSPE is easy to apply to existing ViT models as it only modifies the patch embedding layer, avoiding the need for high-cost training or extensive model modifications.\n\n4. MSPE significantly improves performance across different visual tasks, including image classification, segmentation, and detection, especially in comparison to previous methods like FlexiViT and ResFormer.", "weaknesses": "1. This method assumes that the input x follows a normal distribution x∼N(0,1) for finding the optimal solution in downscaling scenarios. However, this assumption may not hold true for all input images, particularly in real-world scenarios where image data often does not follow a normal distribution.\n\n2. Equation (3) isolates the optimization to gθ, but a more holistic approach that includes Encϕ could potentially lead to better alignment of the entire model's parameters, ensuring more coherent and efficient learning across different resolutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing us with your valuable feedback and suggestions. We appreciate your input and have carefully considered your questions. Below, we provide detailed responses to each of them:\n\n> #### **W1: Assuming $\\mathcal{X} \\sim \\mathcal{N}(0, 1)$ in Image Processing**\n\nWe completely agree with this point. FlexiViT aims to equalize patch embeddings, using the assumption $\\mathcal{X} \\sim \\mathcal{N}(0, 1)$ for downscaling solutions.\nHowever, in real-world scenarios, image pixels do not necessarily meet this assumption, leading to poor FlexiViT performance at low resolutions. \n\nIn light of this issue, we introduce learnable patch embedding and propose a more reasonable optimization condition that minimizes loss across different resolutions, which could compensate the limitation of the above assumption. In other words, our method does not rely on  $\\mathcal{X} \\sim \\mathcal{N}(0, 1)$ assumption, and the experimental results in Sec. 4 demonstrate that our method significantly outperforms FlexiViT on low-resolution inputs. We would revise the motivation part (Sec. 2.4) to clarify it.\n\n\n> #### **W2: Exploring Optimization of Encoder**\n\nWe appreciate the reviewer's insightful question. In this study, we aim to apply minimal modifications to various pre-trained ViT models, finding that optimizing only the patch embedding layer achieves excellent performance. Indeed, optimizing the encoder can increase the model's performance but adds computational costs. Furthermore, optimizing the encoder must be handled carefully to avoid degrading existing representations and causing knowledge forgetting. \n\n Optimizing the patch embedding layer does not impact the original model's performance, as MSPE only optimizes added convolutional kernel parameters. Below are the experimental results for optimizing the encoder, showing performance improvements at other resolutions but a decrease at the standard 224x224 resolution. We hope these results address your concerns about encoder optimization. We will add the above discussions in the revised version.\n\n\n| **Method**   | **Resolution** |            |            |            |            |\n| :----------- | :------------: | :--------: | :--------: | :--------: | :--------: |\n|              | 56             | 112        | 168        | 224        | 448        |\n| Vanilla      | 54\\.67         | 78\\.75     | 83\\.66     | 85\\.10     | 53\\.81     |\n| MSPE         | 77\\.94         | 83\\.75     | 84\\.70     | **85\\.10** | **85\\.11** |\n| MPSE+Encoder | **79\\.33**     | **83\\.94** | **84\\.75** | 84\\.73     | 85\\.07     |\n\n\n> #### **Q1: Testing on High-Resolution Images**\n\nThank you for bringing up this important point. Below are our experimental results at higher resolutions, where we tested two types of ViT models, including Non-overlapping patch embedding (ViT) and overlap patch embedding (PVT). We will add the results of high-resolution in the revised version.\n\n|     | **Method** | **Resolution** |            |            |            |            |            |            |            |            |            |\n| :-: | :--------- | :------------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n|     |            | 224            | 448        | 672        | 896        | 1120       | 1792       | 2240       | 2688       | 3360       | 4032       |\n| ViT | Vanilla    | 85\\.10         | 53\\.04     | 5\\.17      | 0\\.68      | 0\\.33      | 0\\.13      | 0\\.08      | 0\\.08      | 0\\.08      | 0\\.08      |\n|     | MSPE       | **85\\.10**     | **85\\.11** | **85\\.13** | **85\\.14** | **85\\.14** | **85\\.15** | **85\\.14** | **85\\.14** | **85\\.13** | **85\\.16** |\n| PVT | Vanilla    | 83\\.12         | 73\\.64     | 71\\.18     | 67\\.04     | 67\\.04     | 64\\.19     | 63\\.73     | 63\\.50     | 64\\.44     | 63\\.26     |\n|     | MSPE       | **83\\.12**     | **82\\.44** | **83\\.11** | **82\\.76** | **83\\.02** | **82\\.89** | **82\\.93** | **82\\.65** | **82\\.97** | **82\\.58** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their detailed and perceptive comments, which have significantly refined and improved this paper.

We are grateful that the reviewers appreciate our paper in various aspects, including its well-defined problem and theoretical analysis [tsgU, RdsV, 9L8r], simple but solid method [tsgU, RdsV, gLBy, 9L8r], and remarkable performance [tsgU, RdsV, gLBy, 9L8r].

In this work, we explore the challenges of applying ViT models at different resolutions and propose MSPE, which improves ViT models by replacing the standard patch embedding layer with multiple learnable, adaptive convolution kernels. This makes ViTs adaptable to any input resolution without requiring high-cost training or extensive model modifications.


***
Reviewers are particularly interested in how our method performs at high resolutions. Below are the results at high resolution. The results demonstrate that MSPE significantly enhances performance at high resolutions, enabling a single ViT model to effectively cover resolutions from 28x28 to 4032x4032, which previous methods could not achieve.

|     | **Method** | **Resolution** |            |            |            |            |            |            |            |            |            |
| :-: | :--------- | :------------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |
|     |            | 224            | 448        | 672        | 896        | 1120       | 1792       | 2240       | 2688       | 3360       | 4032       |
| ViT | Vanilla    | 85\.10         | 53\.04     | 5\.17      | 0\.68      | 0\.33      | 0\.13      | 0\.08      | 0\.08      | 0\.08      | 0\.08      |
|     | MSPE       | **85\.10**     | **85\.11** | **85\.13** | **85\.14** | **85\.14** | **85\.15** | **85\.14** | **85\.14** | **85\.13** | **85\.16** |
| PVT | Vanilla    | 83\.12         | 73\.64     | 71\.18     | 67\.04     | 67\.04     | 64\.19     | 63\.73     | 63\.50     | 64\.44     | 63\.26     |
|     | MSPE       | **83\.12**     | **82\.44** | **83\.11** | **82\.76** | **83\.02** | **82\.89** | **82\.93** | **82\.65** | **82\.97** | **82\.58** |



***
For other specific questions, we provide detailed responses to each reviewer below. We will carefully revise the paper based on all the feedback from the four reviewers. Thank you once again for your valuable suggestions!

Meta-review: The manuscript received all positive ratings. The reviewers appreciated the soundness of the proposed approach, theoretical analysis and comprehensive experiments. Reviewers also raised several issues which were addressed in the rebuttal. Given the reviewers comments, rebuttal and discussion, the recommendation is accept. Authors are encouraged to take into account all the reviewers feedback and rebuttal content when preparing the final version.
Decision: Accept (poster)