Title: MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multi-Resolution Vision Model

Vision Transformers

Multi-Scale Patch Embedding
Abstract: Although Vision Transformers (ViTs) have recently advanced computer vision tasks significantly, an important real-world problem was overlooked: adapting to variable input resolutions. Typically, images are resized to a fixed resolution, such as 224x224, for efficiency during training and inference. However, uniform input size conflicts with real-world scenarios where images naturally vary in resolution. Modifying the preset resolution of a model may severely degrade the performance. In this work, we propose to enhance the model adaptability to resolution variation by optimizing the patch embedding. The proposed method, called Multi-Scale Patch Embedding (MSPE), substitutes the standard patch embedding with multiple variable-sized patch kernels and selects the best parameters for different resolutions, eliminating the need to resize the original image. Our method does not require high-cost training or modifications to other parts, making it easy to apply to most ViT models. Experiments in image classification, segmentation, and detection tasks demonstrate the effectiveness of MSPE, yielding superior performance on low-resolution inputs and performing comparably on high-resolution inputs with existing methods.

MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution

Abstract

Although Vision Transformers (ViTs) have recently advanced computer vision tasks significantly, an important real-world problem was overlooked: adapting to variable input resolutions. Typically, images are resized to a fixed resolution, such as 224x224, for efficiency during training and inference. However, uniform input size conflicts with real-world scenarios where images naturally vary in resolution. Modifying the preset resolution of a model may severely degrade the performance. In this work, we propose to enhance the model adaptability to resolution variation by optimizing the patch embedding. The proposed method, called Multi-Scale Patch Embedding (MSPE), substitutes the standard patch embedding with multiple variable-sized patch kernels and selects the best parameters for different resolutions, eliminating the need to resize the original image. Our method does not require high-cost training or modifications to other parts, making it easy to apply to most ViT models. Experiments in image classification, segmentation, and detection tasks demonstrate the effectiveness of MSPE, yielding superior performance on low-resolution inputs and performing comparably on high-resolution inputs with existing methods. Code is available at https://github.com/SmallPigPeppa/ MSPE .

1 Introduction

Vision Transformers (ViTs) [1, 2, 3, 4] have achieved significant success in various computer vision tasks, becoming a viable alternative to traditional convolutional neural networks [5, 6, 7, 8]. ViT divides an image into multiple patches, converts these patches into tokens via the patch embedding layer, and feeds them into the Transformer model [9, 10]. The token representations are usually obtained using a convolutional neural network (CNN). Early CNN architectures like AlexNet [11] were designed for fixed-size images (e.g., 224x224). For easy comparison, this setting has been maintained by subsequent image recognition models, including ViT [2]. For fitting neural network input layer size, ViTs typically resize the input image to a fixed resolution 2 and divide it into a specific number of patches [12, 13, 14]. This practice restricts ViT models to processing single-resolution inputs. However, fixed input sizes conflict with real-world scenarios where image resolution varies due to camera devices/parameters, object size, and distance. This discrepancy can significantly degrade ViT's performance on different-resolution images. Changing the preset size requires retraining, and every resolution necessitates an individual model. Therefore, a natural question arises: Is it possible for a single ViT model to process different resolutions directly?

A few recent methods have been proposed for this problem: FlexiViT [15] uses a novel resizing method allowing flexible patch size and token length in ViT. ResFormer [16] enhances resolution adaptability through multi-resolution training and a global-local positional embedding strategy. NaViT [17] leverages example packing [18] to adjust token lengths during training, boosting efficiency and performance. However, these methods have two limitations. First , some of them are incompatible with existing transformer models, preventing the effective use of pre-trained ViTs, and thus lead to high re-training costs when applied. For example, models like NaViT and ResFormer require training the entire model to achieve multi-resolution performance. Second , their multi-resolution performance is insufficient. For example, FlexiViT demonstrates only slight performance improvement when the input images are smaller than the preset size.

We intuitively believe that an ideal solution should be compatible with most existing ViT models and perform well at any size and aspect ratio. To this end, we propose a method called M ultiS cale P atch E mbedding (MSPE). It replaces the patch embedding layer in standard ViT models without altering the other parts of the model. This design makes MSPE compatible with most ViT models and allows for low-cost, direct application. Specifically, our method uses a set of learnable adaptive convolution kernels instead of fixed ones, which can automatically adjust the size and aspect ratio of the kernels based on the input resolution. It directly converts images into patch embeddings without modifying the input size or aspect ratio. The results in Figure 1 show that this simple method significantly improves performance, for example, increasing ImageNet-1K accuracy up to 47.9% across various resolutions. Our contributions are as follows:

By analyzing resolution adaptability in ViT models, we identify the patch embedding layer as the crucial component and provide a low-cost solution.

We propose Multi-Scale Patch Embedding (MSPE), which enhances ViT models by substituting the standard patch embedding layer with learnable, adaptive convolution kernels, enabling ViTs to be applied on any input resolution.

Experiments demonstrate that with minimal training (only five epochs), MSPE significantly enhances performance across different resolutions on classification, segmentation, and detection tasks.

2 Preliminaries and Analysis

2.1 Vision Transformer Models

Vision Transformers (ViTs) leverage the capabilities of the transformer model [20, 21], initially developed for natural language processing, to address vision tasks [22, 23, 24]. ViTs primarily consist of the patch embedding layer and Transformer encoder.

Patch Embedding converts an image x ∈ R h × w × c in the input space X into a sequence of tokens { z i } N i =1 , where z i ∈ R d is a vector in patchification space Z . This transformation is achieved via patch embedding layer g θ : X → Z , parameterized by θ . Specifically, the function g θ is implemented as convolution conv θ , using kernel w θ ∈ R h k × w k × d and bias b θ ∈ R d . The kernel size is ( h k , w k ) and the stride is ( h s , w s ) . In existing ViTs, patch embedding methods are categorized into non-overlapping and overlapping types:

Non-overlapping patch embedding: In standard Vision Transformers (e.g., ViT [2]), the stride of conv θ matches the kernel size, i.e. , h s = h k and w s = w k . The number of tokens N is calculated by ⌊ h h k ⌋ × ⌊ w w k ⌋ .

Overlapping patch embedding: In models like PVT [4] and MViT [25], the stride is smaller than the kernel size, i.e. , h s < h k , w s < w k . The number of tokens N is calculated by ⌈ h -h k h s + p ⌉ × ⌈ w -w k w s + p ⌉ , where p is the padding size.

Transformer Encoder adds position encodings pos i to the token sequence, modifying it to { z ′ i } N i =1 = { z i + pos i } N i =1 . A class token z cls for global semantics is added to the front, i.e. , { z cls , z ′ 1 , . . . , z ′ N } . This sequence feeds into the Transformer Encoder Enc ϕ ( z ) , parameterized by ϕ . For classification, only the class token z cls is used to produce a probability distribution, and other tokens { z ′ 1 , ..., z ′ N } are discarded.

2.2 Problem Formulation

In real-world scenarios, images have variable resolutions due to variations in camera devices, parameters, object scale, and imaging distance. Our task is to ensure the model adapts to different resolutions. To this end, we resize the same image to multiple resolutions and optimize the model for these diverse inputs. Here is the formal definition of this task:

Firstly, the process of resizing (e.g. bilinear reize) is formally defined as a linear transformation:

where o ∈ R h × w is any input, the resolution r of ( h, w ) transforms to r ∗ of ( h ∗ , w ∗ ) after resizing, using the transformation matrix B r ∗ r ∈ R h ∗ w ∗× hw . Each channel of x is resized independently.

For input ( x , y ) in dataset D , the learning objective is to minimize the loss function ℓ (e.g., crossentropy loss) across a series of resolutions { r i } i M =1 , optimizing performance at each resolution from r 1 to r M :

where Enc ˆ ϕ ( z ) and g ˆ θ ( x ) are Transformer encoder and patch embedding layer, respectively. The slack variable ϵ allows minor loss increments in the well-trained model Enc ϕ ( g θ ( x )) .

The central challenge is maintaining acceptable performance across different resolutions from r 1 to r M but keeping the performance of original resolution r , which means adjustments of θ and ϕ must be careful. In this work, we confirm the key role of patch embedding and only optimizing g θ :

2.3 Pseudo-inverse Resize

To solve the optimization problems stated in Eq. (2) and (3), an intuitive solution appears to ensure that embedding layer g θ produces consistent features on different resolutions. For this purpose, FlexiViT [15] proposed PI-resize, which adjusts the embedding kernel to ensure uniform outputs:

When upscaling with r i > r , the analytic solution for Eq. (4) is ̂ ω θ = B r i r ( B r i r T B r i r ) -1 ω θ , denote as ( B r i r T ) + ω θ :

When downscaling with r i < r , the matrix B r i r T B r i r is non-invertible. Under the assumption X = N (0 , 1) , it is proven that ( B r i r T ) + ω θ is the optimal solution. In summary, Pseudo-inverse resize (PI-resize) is defined as follows:

Acc: 54.67

Acc: 63.44

Acc: 77.94

patch embedding

space

class token

space

near

Enc

!

far

far

near

$

"

$

#

$

$%&

"

$

$%&

#

$

'()

$

$%&

'()

2.4 Motivation

However, is this optimization target in FlexiViT [15] truly appropriate? We suggest that there are two problems with Equation 4: First , it is a stricter sufficient condition of Eq. (3), but it ignores the impact of the encoder Enc ϕ ( z ) and the objective function ℓ . Second , at lower resolutions, i.e. r i < r , this goal has no analytical solution; PI-resize is just an approximation assuming x ∼ N (0 , 1) , resulting in significant performance degradation.

Moreover, we intuitively suspect that similarity in patch embeddings does not ensure the best performance. As Figure 2(a) illustrates, features derived from patch embeddings are transformed into the classification feature space, namely the class token space, through the Transformer encoder Enc ϕ . The gradient directions of these two feature spaces may not align, resulting in image features that are close to optimal in patch embedding being far from the optimal class token after encoder processing.

A more effective method is directly adjusting the weights w θ of embedding layer using the objective function ℓ (Eq. (3)). To verify this idea and our assumptions, we evaluate a well-trained ViTB/16 model [2] (pre-trained on ImageNet-21K, 224x224, 85.10% accuracy). We measure the cosine similarity between patch embeddings { z i } N i =1 and the class token z cls at resolutions of 56x56 and 224x224. The results in Figures 2 (b) and (c) show that FlexiViT has higher patch embedding similarity and classification accuracy compared to the vanilla model; however, our method significantly outperforms FlexiViT with even lower patch embedding similarity. These results confirm that our analysis is reasonable.

3 Method

As discussed in Section 2.4, optimizing patch embedding layers through objective function is simple but effective. Motivated by this, we propose M ultiS cale P atch E mbedding (MSPE). It divides the resolution domain into different ranges { r i } i M =1 , each using a shared-weight patch embedding layer that adjusts size through PI-resize. MSPE can replace the patch embedding layers in most ViT models, including overlapping and non-overlapping types. Our method is illustrated in Figure 3 and presented below.

3.1 Architecture of MSPE

MSPE only changes the patch embedding layer of the ViT model, making it directly applicable to a well-trained ViT model. As demonstrated in Figure 3, we introduce the following architectural modifications.

Multiple patching kernels. The typical patch embedding layer g θ employs a single convolution size with parameters ( ω θ , b θ ) , making it unsuitable for varying image resolutions. To overcome this, MSPE incorporates K convolutions with differenet kernel sizes { g 1 θ , . . . , g K θ } , where each g i θ is parameterized by ( w i θ , b i θ ) , to support a broader range of input image sizes.

Adaptive patching kernels. Although K convolutions of different kernel sizes improve adaptability, they cannot cover all possible resolutions. Setting a unique size convolution kernel for each resolution is unrealistic. In MSPE, the size and ratio of the kernel ( w i θ , b i θ ) are adjustable rather than fixed. Specifically, for input x of any resolution ( h, w ) , the corresponding kernel size ( h k , w k ) is

Transformer Encoder

Class

Bird

Car

…

MLP

Head

0

1

2

3

4

5

6

7

8

9

M

ulti-

S

cale

P

atch

E

mbedding (MSPE)

*

* Extra learnable

[class]

embedding

Patch + Position

Embedding

Vision Transformer (ViT)  with MSPE

divide

𝑟

!

𝑟

"

𝑟

#

Multi-Scale Patch Embedding

resize

{"

!

"

, … , "

#

"

}

conv

&

$

"

add

' (

$

"

'

$

"

)

"

𝑲 = 3

b

( ⌊ h/N ⌋ , ⌊ w/N ⌋ ) . Using Eq. (6), we adjust w i θ to the corresponding size ̂ w i θ ∈ R h k × w k . As shown in Figure 3 (b), it can be directly convolved with the image patch.

Another module requiring careful design is the position embedding, which must correspond to the feature map size after patch embedding. In ViT [2] and DeiT [3], the position embedding is bilinearly interpolated to match the feature map size, a method known as resample absolute pos embed , and it has proven effective during fine-tuning [3]. In MSPE, we discover that bilinear interpolation of position embedding satisfies our requirements. To simplify the architecture and minimize changes to the vanilla model, we follow this dynamic positional embedding method.

3.2 Learning Objectives

MSPE optimizes the patch embedding layer g θ through the objective function but does not explicitly constrain the embedded tokens { z i } N i =1 across different resolutions. Specifically, MSPE is based on multi-resolution training (mixed resolution training), similar to methods like ResFormer and NaViT. The training process is as follows.

Firstly, the mixed resolution { r i } i M =1 is divided into K subsets { S k } K k =1 , i.e. ⋃ K k =1 S k = { r i } i M =1 , and r k is randomly sampled from S k . The patching kernel weights θ k are shared within S k and transformed into the corresponding weights ̂ θ k for each r i according to Eq. (6), The loss function is defined as:

where ℓ is the task loss function (e.g., cross-entropy loss), and λ is a hyperparameter to prevent performance degradation. We optimize only the patch embedding parameters θ during training, setting the learning rate of ϕ to zero. Algorithm 1 in Appendix E.2 details the training procedure of MSPE and PyTorch-style implementation.

3.3 Inference on Any Resolution

For an input x with any resolution r ∗ , previous models resize the image to a fixed resolution r . The inference procedure is:

With MSPE, the model can infer directly from the original image without resizing or altering its aspect ratio. The process is:

Method

Resolution

28

42

56

70

84

98

112

126

140

168

224

448

ViT

Vanilla [2]

8.52

37.86

54.67

65.94

72.58

76.29

78.75

80.62

82.00

83.66

85.10

53.81

ResFormer [16]

2.21

19.29

45.41

59.92

69.30

74.34

77.92

77.48

79.61

81.30

83.04

81.06

FlexiViT [15]

14.86

48.52

63.44

70.53

74.47

78.03

80.24

82.28

83.10

84.70

85.10

85.11

MSPE

56.41

71.02

77.94

79.54

81.63

82.51

83.75

83.81

83.94

84.70

85.10

85.11

DeiTIII

Vanilla [30]

0.14

2.45

17.93

40.30

57.23

67.01

72.80

76.20

79.62

83.51

85.66

62.29

ResFormer [16]

1.08

20.27

46.52

60.78

70.28

75.28

78.80

78.79

80.48

82.13

83.47

82.53

FlexiViT [15]

3.76

26.44

49.99

62.72

70.47

74.55

77.57

79.22

80.69

83.51

85.66

85.53

MSPE

48.71

65.66

75.37

77.78

80.76

81.68

83.49

83.28

83.75

84.86

85.66

85.53

PVT

Vanilla [4]

8.78

28.20

47.52

53.54

61.11

68.64

76.45

78.29

80.21

81.73

83.12

73.48

FlexiViT [15]

16.64

48.13

60.21

65.39

70.44

73.27

76.70

78.29

80.21

81.73

83.12

81.57

MSPE

34.20

60.76

72.09

75.39

77.39

78.08

80.36

81.12

81.59

82.00

83.12

82.43

192

144

80

width

60

65

70

75

80

85

height

=192

Vanilla

FlexiViT

MSPE

256

192

128

64

width

70

75

80

85

height

=256

Vanilla

FlexiViT

MSPE

384

288

192

96

width

76

79

82

85

height

=384

Vanilla

FlexiViT

MSPE

where θ ∗ is the patching weight that matches the resolution r ∗ . Details on the computation of θ ∗ are provided in Appendix E.1 and Eq.(10).

4 Experiments

Datasets. We conduct experiments on 4 benchmark datasets: ImageNet-1K [26] for classification tasks, ADE20K [27] and Cityscapes [28] for semantic segmentation, and COCO2017 [29] for object detection.

Backbone networks. Our method is assessed on three ViT models, including ViT-B [2] and DeiT-B III [30] (non-overlapping patches); PVT v2-B3 [31] (overlapping patches). ViT and DeiT III are pre-trained using the ImageNet-21K and ImageNet-22K datasets.

Implementation details. MSPE is trained using SGD optimizer for five epochs , with a learning rate of 0.001, momentum of 0.9, weight decay of 0.0005, and batch size of 64 per GPU. To validate our model, we implement ViT [2] and other networks [30, 4, 25] via open-sourced timm library for classification, ViTDet [32] via MMDetection [33] for object detection, and SETR [34] via MMSegmentation [35] for segmentation; additional details are available in Appendix B.

4.1 Image Classification

Comparison with FlexiViT. As shown in Table 1 and Figure 4, ViT models using non-overlapping (like ViT and DeiT III) and overlapping patch embeddings (like PVTv2 and MViTv2) significantly lose accuracy when the input resolution varies. This is consistent with the results from [15, 16, 17], demonstrating that the vanilla ViT models are not adaptable to input resolution changes. This issue primarily arises from their patch embedding layers failing to adjust to varying resolutions. This leads to high-level features shifting after the patch tokens are fed into the Transformer encoder and significantly degrading performance. FlexiViT shows remarkably stable performance at upscaled resolutions (e.g., 448x448) by ensuring consistency of patch tokens across different resolutions, outperforming vanilla models. However, as Section 3 analyzes, FlexiViT still struggles with downscaling.

Our method significantly boosts accuracy with targeted optimization goals, outperforming FlexiViT across various resolutions and aspect ratios.

Comparison with ResFormer and NaViT. ResFormer improves performance by multi-resolution training across 128x128, 160x160, and 224x224 resolutions. However, the modified ViT architecture does not suit networks like PVT and MViT that use overlap patch embedding. Moreover, ResFormer trains for 200 epochs, but our method requires only five epochs. NaViT keeps the original aspect ratio and trains with mixed resolutions from 64x64 to 512x512, leveraging a larger JFT pre-training dataset and longer training cycles (up to 920,000 steps). Table 1 and Figure 5 show that these state-of-the-art methods improve the vanilla model remarkably. Compared to ResFormer and NaViT, MSPE achieves superior multi-resolution performance with far fewer training resources, which proves the essential role of optimized patch embedding layers.

4.2 Semantic Segmentation

To validate the effectiveness of MSPE in semantic segmentation, we test the SETR [34] model on ADE20K [27] and Cityscapes [28] datasets, with the vanilla model trained at 512x512 and 768x768 resolutions, respectively. Results in Table 2 show that FlexiViT significantly enhances the vanilla model's performance in semantic segmentation (e.g., 59.78 vs. 13.33 on the mIOU metric). This confirms that adjusting the patch embedding layer is effective for pixel-level tasks, demonstrating its critical role in enhancing multi-resolution robustness. Moreover, MSPE consistently outperforms FlexiViT across various resolutions, proving our method is ready for pixel-dense tasks in real-world scenarios.

4.3 Object Detection

In our experiments on the COCO2017 dataset for object detection and instance segmentation, we utilize the ViTDeT [32] model with ViT-B (pre-trained on ImageNet-1K via MAE [36]) and

28

56

84

112

140

224

252

test resolution

r

40%

50%

60%

70%

80%

ImageNet-1K Acc@1

Impact of Resize Method

Area

Bilinear

PI-Resize

i

employed Mask R-CNN [37] as the detection head. During the evaluation, we replace the ViT's patch embedding layer with MSPE or FlexiViT, keeping the rest of the architecture unchanged ( aligned with classification and segmentation ). As shown in Table 3, MSPE significantly improves the multi-resolution performance of well-trained detection models. This result is consistent with those of classification and segmentation tasks, demonstrating the effectiveness of our method across different visual tasks.

4.4 Ablation Study and Analysis

Training epochs. Figure 6 (a) presents the performance of MSPE on different training epochs ( 1 , 3 , 5 ). It is observed that MSPE significantly enhances performance with only a few training epochs. The models trained for 3 and 5 epochs show similar performance, with no significant improvement from additional epochs. Thus, we train MSPE for 5 epochs in our experiments.

Model size. For evaluating the impact of model size on MSPE, we test on ImageNet-1K across different sizes of DeiT III models (Small (S), Base (B), Large (L)), all pre-trained on ImageNet-22K. As shown in Figure 6 (b), the results of the larger model DeiT-L and the smaller model DeiT-S align with the main experiment; the larger model yields higher accuracy at different resolutions. This result demonstrates the effectiveness of our method across models of different sizes.

Hyperparameters. We conduct ablation studies of hyperparameters λ in Figure 7 (a). When lambda is set to 0, the learning of patch embedding is too flexible, leading to inadequate alignment with the original parameter space. Lambda values of 1 and 2 lead to similar performances; this hyperparameter is set to 1 in our experiments.

Kernel count K . Figure 7 (b) shows the impact of different kernel quantities on model performance. In MSPE, the resolution r i is divided into K subsets, with each subset sharing the patch embedding layer weights. Therefore, the number of subsets K equals the number of patchify kernels in MSPE. The results indicate that slightly increasing K can improve performance. Specifically, when K = 3 and K = 4 , the model performance is nearly identical, and additional kernels provide little improvement. This suggests that patch embedding parameters can be shared across different resolutions. In our method, K is set to 4.

Resizing method. In our method, the patch embedding weights are dynamically resized for images with different sizes and ratios, denoted as adaptive kernels in Section 3.1. To evaluate the effect of resizing methods on MSPE, we load an ImageNet-21K pre-trained ViT-B model from [19] and train MSPE using different resizing methods, such as standard linear resizing. As shown in Figure 8, the

results indicate that PI-resize consistently outperforms other common resizing methods, aligning with findings from the FlexiViT [15].

Parameters and computation overhead. As shown in Table 4, we analyze the impact of MSPE on the parameter and computational cost of the vanilla VIT model. MSPE modifies only the patch embedding layer g θ , increasing its parameter count by 2x~3x. However, MSPE provides a more flexible approach to calculating patch embeddings, significantly reducing computational costs compared to the original method. The total parameter count of the model remains nearly unchanged because the patch embedding layer is an extremely small component.

Image resizing v.s. MSPE. Table 5 shows the comparison results of image resizing (IMG-resize) and MSPE on ImageNet-1K. IMG-resize adjusts images of different resolutions to a preset resolution (e.g., 224x224) during testing, whereas our method keeps the original image size and aspect ratio unchanged. In real-world scenarios, resizing small-size images to a larger size incurs digitization noise and increased computation costs, and model performance is not optimal at different resolutions. The results indicate that our method comprehensively outperforms IMG-resize, suggesting that model inference on the original images is a viable option.

5 Related Work

Most relevant to our work is FlexiViT [15], which proposed to resize the patch embedding weights, enabling flexible patch sizes and token lengths in ViT models. Pix2struct [38] supports variable aspect ratios by a novel positional embedding method, enhancing efficiency and performance in chart and document understanding. ResFormer [16] enables resolution adaptability through multi-resolution training and a global-local positional embedding strategy. NaViT [17] enhances efficiency and performance by example packing [18] to adjust token lengths for different resolutions during training. Compared to these methods, MSPE only changes the patch embedding layer and achieves better performance. This justifies the patch embedding layer's key role in adapting ViT models for different resolutions.

In CNN-based models, Mind the Pooling [39] addresses overfitting on resolution by introducing SBPooling to replace max-pooling, enabling CNNs to process different resolutions. Learn to Resize [40] uses a learnable resizing layer to replace bilinear interpolation. Another study [41] examined the relationship between training and testing resolutions, showing that training at slightly lower resolutions than testing can improve performance. Networks like Resolution Adaptive Networks (RANet) [42] and Dynamic Resolution Networks (DRNet) [43] use multiple sub-models to choose the appropriate resolution and model based on task difficulty, thus enhancing model efficiency and resolution adaptability.

6 Conclusion

To make ViT models compatible with images of different sizes and aspect ratios, we propose MSPE to replace the traditional patch embedding layer for accommodating variable image resolutions. MSPE uses multiple variable-sized patch kernels and selects the best parameters for different resolutions, eliminating the need to resize the original image. Extensive experiments demonstrate that MSPE performs well in various visual tasks (image classification, segmentation, and detection). Particularly, MSPE yields superior performance on low-resolution inputs and performs comparably on highresolution inputs with previous methods. Our method has the potential for application in various vision tasks, and can be extended by optimizing the embedding layer and transformer encoder jointly.

7 Acknowledgments and Disclosure of Funding

This work has been supported by the National Natural Science Foundation of China (NSFC) grant U20A20223 and the InnoHK program.

References

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , pages 10347-10357. PMLR, 2021.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 568-578, 2021.

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. pages 4700-4708, 2017.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4510-4520, 2018.

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1492-1500, 2017.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning , pages 6105-6114. PMLR, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(1):87-110, 2022.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems , 25, 2012.

Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution vision transformer for dense predict. Advances in Neural Information Processing Systems , 34:7281-7293, 2021.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1210412113, 2022.

Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z Pan. Multi-scale high-resolution vision transformer for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12094-12103, 2022.

Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14496-14506, 2023.

Rui Tian, Zuxuan Wu, Qi Dai, Han Hu, Yu Qiao, and Yu-Gang Jiang. Resformer: Scaling vits with multi-resolution training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22721-22731, 2023.

Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n'pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems , 36, 2024.

Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027 , 2021.

Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research , 2022.

Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in Neural Information Processing Systems , 28, 2015.

Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. Inception transformer. Advances in Neural Information Processing Systems , 35:23495-23509, 2022.

Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Luˇ ci´ c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6836-6846, 2021.

Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10809-10818, 2022.

Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems , 35:38571-38584, 2022.

Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4804-4814, 2022.

Jia Deng, Wei Dong, Richard Socher, et al. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision , 127:302-321, 2019.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, UweFranke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 32133223, 2016.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision , pages 740-755. Springer, 2014.

Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In Proceedings of the European Conference on Computer Vision , pages 516-533. Springer, 2022.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media , 8(3):415-424, 2022.

Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Proceedings of the European Conference on Computer Vision , pages 280-296. Springer, 2022.

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 , 2019.

Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6881-6890, 2021.

MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation , 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2961-2969, 2017.

Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning , pages 18893-18912. PMLR, 2023.

Bilal Alsallakh, David Yan, Narine Kokhlikyan, Vivek Miglani, Orion Reblitz-Richardson, and Pamela Bhattacharya. Mind the pool: Convolutional neural networks can overfit input size. In International Conference on Learning Representations , 2022.

Hossein Talebi and Peyman Milanfar. Learning to resize images for computer vision tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 497-506, 2021.

Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. Advances in Neural Information Processing Systems , 32, 2019.

Xin Wang, Fisher Yu, Ziwei Liang, Thomas Huang, Larry Shi, Matthew Liu, Jan Kautz, and Alan Yuille. Resolution adaptive networks for efficient inference. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8729-8738, 2018.

Jinjin Chen, Xifeng Pan, Yu-Kun Lai, and Chao Tang. Dynamic relu. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11-20, 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision , pages 213-229. Springer, 2020.

Bharat Singh and Larry S Davis. An analysis of scale invariance in object detection snip. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3578-3587, 2018.

Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. pages 113-123, 2019.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 702-703, 2020.

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 13001-13008, 2020.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6023-6032, 2019.

A Limitation

Our paper does not consider the impact of position embedding on model performance. This is because existing work [2, 3] has shown that linear interpolation of position embedding can achieve acceptable performance. Our experimental results validate the feasibility of this strategy. Moreover, we aim to utilize our method within most existing ViT models, hence we do not alter the position embedding method. In the future, our method can be extended with the positional encoding strategies proposed in related works, such as ResFormer [16] and Pix2struct [38].

B Training details

B.1 Multi-resolution training

Multi-resolution training has become a widely used paradigm [44, 45], randomly altering crop size and aspect ratio during training. This training strategy is critical in multi-resolution vision models. In FlexiViT [15] training, the size of image patches and the kernel size of the patch embedding layer are randomly changed from 8x8 to 48x48. In ResFormer [16], input images are resized to 128 × 128, 160 × 160 and 224 × 224 and simultaneously training. In NaViT [17], images retain their original size and aspect ratio and are packed into a single pack for training, denoted as mixed-resolution training. The range of its training resolutions is 64 to 256. Our method is most similar to FlexiViT, randomly resizing each image batch to different resolutions (from 56 to 256) during training.

B.2 Data augmentation

Classification. Our method leverages timm implementations of ViT [2], DeiT III [30], and PVT v2 [31] models for classification tasks. The training data augmentations include RandomResizedCrop, RandomHorizontalFlip, and ColorJitter. For Resfomer [16], we follow the official-released code. Its training data augmentation includes Auto-Augment [46], RandAugment [47], random erasing [48], MixUp [49], and CutMix [50].

Semantic segmentation. We use the SETR [34] model implemented by MMSegmentation [35]. We adhere data settings in MMSegmentation, including RandomResizedCrop, RandomFlip, and PhotoMetricDistortion. PhotoMetricDistortion involves a series of transformations: random brightness, random contrast, random saturation, random hue, and converting color between HSV and BGR.

Object detection. We employ the ViTDeT [32] model within the MMDetection [33] for object detection and instance segmentation tasks. We utilize the configurations from MMDetection, which include RandomResizedCrop and RandomFlip.

C Compute resources

This paper conducts experiments on a machine equipped with two AMD EPYC 7543 32-core processors; each slotted with 32 cores supporting two threads per core. The machine has 496 GB of memory and 8* NVIDIA GeForce RTX 4090 graphics cards. Our method significantly reduces computational resources compared to previous approaches, and all main experiments on ImageNet-1K are completed within 3 hours. In contrast, methods like ResFormer [16] require over 50 hours of training on this machine.

D More experimental results

D.1 Impact of aspect ratio

In Section 4.1, we fixed the height and varied the width, ensuring the height was greater than the width. We also tested the accuracy when fixing the width and varying the height, in which case the width was greater than the height. It can be seen in Figure 9 that MSPE still achieved significant performance improvements.

Furthermore, comparing the accuracy of resolutions ( h, w ) and ( w,h ) when h > w , we observe that the accuracy of ( h, w ) resolution was higher than ( w,h ) resolution. This aligns with the findings

of NaViT [17], which indicate that a larger proportion of images in ImageNet-1K is h > w . This suggests that maintaining the original aspect ratio during inference needs to be considered.

D.2 Results on Higher Resolution

Below are experimental results for high-resolution images. We resize ImageNet images (processed to 224x224) up to 4096x4096 and evaluate models with non-overlapping patch embedding (ViT) and overlapping patch embedding (PVT). The results in Table 6 show that MSPE can be effectively applied to high-resolution images, as the shape and size of the patch embedding kernels dynamically adjust based on input resolution and have multiple convolution kernels corresponding to different resolution ranges.

D.3 Extra training epochs

In our main experiment, MSPE is trained for 5 epochs. We increase the number of epochs to analyze its performance under extended training. As shown in Table 7, the performance improvement with longer training epochs is minimal. This is because MSPE requires learning a very small number of parameters, which can reach optimal performance within just a few epochs of training. This indicates that our method achieves better performance and requires less training overhead, making it highly compatible with well-trained models.

D.4 Other resizing methods

We test other resizing methods in MSPE, including nearest and bicubic 3 . As shown in Figure 10, these traditional resizing methods are not suitable for adjusting the size of neural network parameters, leading to less robust patch embeddings in MSPE training. The experimental results are consistent with Section 4.4, confirming that the combination of PI-resize and MSPE have significant potential applied to multi-resolution models.

D.5 Computation overhead

We evaluate MSPE's computational overhead across resolutions from 32x32 to 448x448. MSPE dynamically adjusts its parameters based on image resolution. For overlapping patch embedding models like PVT and MViT, MSPE adjusts the convolutional kernel size and stride. For nonoverlapping patch embedding models like ViT, MSPE adjusts the convolutional kernel size. As shown in Figure 11, MSPE's computational overhead changes dynamically, avoiding the unnecessary computational costs associated with resizing images to a fixed resolution.

E More details on MSPE

In this section, we provide more details about the implementation of MSPE, including modifications to the model structure and the training process, and its application for inference at any resolution.

E.1 Inference on any resolution

For an input image x with any resolution r , we calculate its patch embedding parameters θ ∗ , composed of w ∗ and b ∗ . We directly compute the patch embedding z on the original image x as follows: z = conv ( x , w ∗ ) + b . After that, embedding z is fed into the Transformer encoder Enc ( z ) as usual. In this process, calculating θ ∗ is a crucial step. We compute θ ∗ by finding the resolution r i in the resolution sequence r i M i =1 that is closest to the original resolution r ∗ and then resizing its kernel to match the original resolution. This process is formalized as follows:

E.2 MSPE structure and training process

As depicted in Section 3.1, MSPE modifies two parts of the ViT model, as shown in the class ViT of Algorithm 1. 1) It adds K patch embedding kernels. 2) During patch embedding, the size and aspect ratio of the kernels can be adjusted based on the input. Here, g and Enc represent the patch embedding layer g θ and the Transformer encoder Enc ϕ ( z ) .

The implementation of multi-resolution training in Section 3.1 is as follows: We randomly select K resolutions from a range of resolutions { r i } i M =1 , then compute the corresponding patch embeddings. After that, the model computes the loss as usual.

Algorithm 1 MSPE pseudo-implementation.

Notes : Changes to existing code highlighted via red background.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The related experimental results are Section 2 , and we cite FlexiViT's conclusion regarding PI-resize.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 4 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 4 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The random error in the experiments can be ignored and does not affect the conclusions.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. They analyze the resolution adaptability in ViT models, identify the patch embedding layer as the crucial component, and provide a low-cost solution.\n\n2. They propose Multi-Scale Patch Embedding (MSPE), which enhances ViT models by substituting the standard patch embedding layer with learnable, adaptive convolution kernels, enabling ViTs to be applied on any input resolution.\n\n3. Experiments demonstrate that with minimal training (only five epochs), MSPE significantly enhances performance across different resolutions on classification, segmentation, and detection tasks.\n\nSTRENGTHS:\n1. They found two problems in the optimization target function in FlexiViT and they illustrate similarity in patch embeddings does not ensure the best performance.\n\n2. MSPE allows ViTs to handle variable input resolutions directly, significantly improving performance on low-resolution inputs and maintaining high performance on high-resolution inputs.\n\n3. MSPE is easy to apply to existing ViT models as it only modifies the patch embedding layer, avoiding the need for high-cost training or extensive model modifications.\n\n4. MSPE significantly improves performance across different visual tasks, including image classification, segmentation, and detection, especially in comparison to previous methods like FlexiViT and ResFormer.\n\nWEAKNESSES:\n1. This method assumes that the input x follows a normal distribution x∼N(0,1) for finding the optimal solution in downscaling scenarios. However, this assumption may not hold true for all input images, particularly in real-world scenarios where image data often does not follow a normal distribution.\n\n2. Equation (3) isolates the optimization to gθ, but a more holistic approach that includes Encϕ could potentially lead to better alignment of the entire model's parameters, ensuring more coherent and efficient learning across different resolutions.\n\nQUESTIONS:\nIt's impressive that MSPE achieves much better results with low-resolution images. Is it possible to also show the performance with higher image resolutions larger than 1000?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Multi-Scale Patch Embedding (MSPE), a novel approach to enhance Vision Transformers (ViTs) by allowing them to adapt to variable input resolutions without resizing images to a fixed resolution. MSPE uses multiple variable-sized patch kernels and selects the optimal parameters for different resolutions, maintaining performance across tasks likes image classification, segmentation, and detection.\n\nSTRENGTHS:\n1. MSPE replaces the standard patch embedding layer in Vision Transformers (ViTs) with multiple learnable adaptive convolution kernels, enabling the effective handling of different input resolutions.\n\n2. MSPE demonstrates improved performance across varying resolutions in classification, segmentation, and detection tasks with minimal additional training.\n\nWEAKNESSES:\n1. This paper does not experiment with other more effective positional encoding strategies. The paper mentions that linear interpolation of positional embedding has achieved acceptable performance in existing work. \n\n2. The paper analyses the importance of patch embedding but lacks corresponding experimental evidence comparing its importance to other components like encoder and positional encoding. Additionally, the components may show different impacts for different tasks; it does not analyze how the importance of these components may vary across different tasks. \n\n3. The training efficiency of MSPE is highlighted, yet the potential long-term benefits of additional training epochs are not fully explored. It would be beneficial to see more detailed comparisons over extended training periods to understand better the model’s convergence behavior and potential overfitting issues.\n\n4. Minor Error: In 2.1 Vision Transformer Models, the shape of image `x` is defined as `h*w*c`, but the kernel shape obtained is `h_k * w_k * d`. Shouldn't the kernel shape be `h_k * w_k * c * d`?\n\nQUESTIONS:\n1. Is selecting the hyperparameter λ during the training process crucial for the model's performance? Can different values of λ yield good results?\n\n2. Can you elaborate on the differences between MSPE with FlexiViT?\n\n3. Is there a rationale behind selecting the hyperparameter K and its corresponding shape in MSPE, or was it arbitrarily chosen?\n\n4. Did the authors attempt to apply the model to uncommon resolutions (e.g., 200x150) to test its generalization performance?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes to substitute the standard patch embedding with multiple variable-sized patch kernels. This eliminates the need to resize the original image. Extensive experiment results are shown to demonstrate the benefits.

STRENGTHS:
The problem is well defined and the proposed method is sound. Convincing results are shown to support the claims.

WEAKNESSES:
NA

QUESTIONS:
The highest resolution shown is 448, 2x the training resolution 224. A typical cell phone camera has a resolution around 2K. Could the proposed method bridge this 10x gap?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aim to address the challenge of adapting ViTs to variable input resolutions, which is a critical issue often overlooked in real-world applications. The authors propose a new method named Multi-Scale Patch Embedding (MSPE), which enhances the patch embedding layer by incorporating multiple variable-sized patch kernels. This approach allows the model to process images of varying resolutions without resizing, thus maintaining performance across different resolutions.\n\nSTRENGTHS:\n1. Problem Identification: This paper clearly identifies the gap in current ViT models, i.e. they are unable to handle variable input resolutions effectively. This is a crucial problem as real-world images often come in various resolutions.\n2. Sound Approach: The proposed MSPE substitutes the standard patch embedding layer with learnable adaptive convolution kernels, which allows the model to adjust to different input resolutions dynamically. \n3. Theoretical Analysis: The paper offers a thorough theoretical analysis of the problem and the proposed solution. This includes a detailed discussion on the patch embedding layer's role and the limitations of existing methods like FlexiViT and ResFormer.\n4. Comprehensive Experiments: The authors provide extensive experimental results demonstrating the effectiveness of MSPE across various tasks, including image classification, segmentation, and detection.\n\nWEAKNESSES:\n1. Technical Contribution: Although the proposed method improves the multi-resolution performance of ViTs, it essentially solves an engineering problem from an engineering perspective. \n2. Scalability Concerns: Although the method is low-cost and compatible with existing ViT models, the paper does not fully address potential scalability issues when applied to very large datasets or extremely high-resolution images.\n3. Ablation Studies: The paper could benefit from more detailed ablation studies to isolate the contributions of different components of the MSPE and better understand the impact of each design choice. \n4. Real-World Applications: While the experiments are comprehensive, the paper could include more real-world application scenarios to demonstrate the practical benefits and robustness of MSPE in diverse settings.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nThe authors addressed the limitation in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}