Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models
Authors: Shuxia Lin
Miaosen Zhang
Ruiming Chen
Xu Yang
Qiufeng Wang
Xin Geng
Venue: NeurIPS 2024 poster (2024)
Keywords: Model decomposition
Model initialization
Vision Transformer
Primary area: deep_learning_architectures

Abstract: Vision Transformers (ViTs) are widely used in a variety of applications, while they usually have a fixed architecture that may not match the varying computational resources of different deployment environments. Thus, it is necessary to adapt ViT architectures to devices with diverse computational overheads to achieve an accuracy-efficient trade-off. This concept is consistent with the motivation behind Learngene. To achieve this, inspired by polynomial decomposition in calculus, where a function can be approximated by linearly combining several basic components, we propose to linearly decompose the ViT model into a set of components called learngenes during element-wise training. These learngenes can then be recomposed into differently scaled, pre-initialized models to satisfy different computational resource constraints. Such a decomposition-recomposition strategy provides an economical and flexible approach to generating different scales of ViT models for different deployment scenarios. Compared to model compression or training from scratch, which require to repeatedly train on large datasets for diverse-scale models, such strategy reduces computational costs since it only requires to train on large datasets once. Extensive experiments are used to validate the effectiveness of our method: ViTs can be decomposed and the decomposed learngenes can be recomposed into diverse-scale ViTs, which can achieve comparable or better performance compared to traditional model compression and pre-training methods. The code for our experiments is available in the supplemental material.

Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models
Shuxia Lin Miaosen Zhang!2 12", Ruiming Chen 1.2"
Qiufeng Wangh 1.2 Xu Yang! and Xin
Gengl 1.2
'School of Computer Science and Engineering, Southeast University, Nanjing 21 210096, China Keyy Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Ministry of Education, China
{shuxialin, 230228501, 220232251, qfwang, xuyang-palm, xgeng) edu. cn
Abstract
Vision Transformers (ViTs) are widely used in variety of applications, while they usually have a fixed architecture that may not match the varying computa- tional resources of different deployment environments. Thus, is necessary adapt ViT architectures to devices with diverse computational overheads to achieve accuracy--fffcient trade-off. This concept is consistent with the motivation behind Learngene. To achieve this, inspired by polynomial decomposition in calculus, where function be approximated by linearly combining several basic components, we propose to linearly decompose the ViT model into a set of components called learngenes during element-wise training. These learngenes can then be recomposed into differently scaled, pre-initialized models to satisfy different computational resource constraints. Such decompositoon--ecmpoiiii strategy provides an economical and flexible approach to generating different scales of ViT models for different deployment scenarios. Compared to model compression or training from scratch, which require to repeatedly train on large datasets for diverse-scale models, such strategy reduces computational costs since it only requires to train on large datasets once. Extensive experiments are used to validate the effectiveness of our method: VITs can decomposed and the decom- posed learngenes can be recomposed into diverse-scale ViT's, which can achieve comparable or better performance compared to traditional model compression and pre-training methods. The code for our experiments is available in the supplemental material.
Introduction
The pre-training Vision Transformers (ViT's 13 have become fundamental to various applications, including image classification 7. object detection 29 52 semantic segmentation [46 67
and multimodal tasks 633 58). However, these VIT's typically have standard and relatively fixed architecture, which poses challenges for deployment in diverse real-world settings, i.c., devices in different application scenarios have different computational capabilities, making standard. fixed-size ViTs unsuitable for direct deployment. Furthermore, the storage demands a traditional ViT may exceed the capabilities of certain devices. For instance, the memory size for the VIT-B/16 model [I3] with 86M parameters approximately 320 MB, while some devices may find this prohibitive due inherent physical constraints, e.g., the limited RAM capacity of the Raspberry Pi with 256MB-512MG of RAM.
'These authors contributed equally.
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Teacher Model
M layer Transforer
laysr Transformer
Teacher Model
1.6
Parameter datapoint
Distillation
DecompossDiellatton)
Fitting Enne
Compose
Weight value
0.4
0.0
-0.4
-0.8
Finetune
Finetune
-1.2
8 Layer position
12
Figure I: Comparison between Knowledge Distillation (KD) (left) and our method (right). KD requires N times of training with the teacher model on data to produce N different scale models for N clients. In contrast, our method requires only single training to decompose the teacher model into different learngenes that can be economically and flexibly recomposed into models with diverse layers to meet different client needs.
Figure 2: After using PCA to re- duce the dimension of the parame- ters of a well-trained ViT, we find that the parameters of the most lay- ers have an approximately linear correlation with their layer position. More details are given in the sup- plementary material.
To address this limitation, researchers have developed various compression techniques 66 62 55
28
to reduce the model size to make them suitable for the application scenarios while meantime maximum preserving the model ability. Typical approaches including micro-architecture design 23 model pruning [14 (68) quantization [43]. and knowledge distillation 62). However, in these approaches, the architecture of the smaller model, such as the number of layers, is predetermined and thus lacks the flexibility to meet the diverse needs of real-world deployment scenarios. Consider the scenario depicted in Figure [ to obtain N models of different sizes for different devices, the knowledge distillation methods need to go through N separate trainings. Striking a balance between flexibility and accuracy is challenging and often requires complex training strategies ensure satisfactory model performance.
Then may ask, is there way to more flexibly and economically generate N smaller models from large one, while maintaining the performance of the smaller ones? In calculus, we know that a function can be approximated by linearly combining several basic components with corresponding weighted coefficients at various levels of precision, e.g., Taylor series [45), Chebyshev Polynomial of the First Kind (31), Fourier series [47 and so on. Such a mathematical theorem induces two engineering insights: (1) a function can be decomposed into series of basic components, i.e., the polynomial terms, and (2) we can combine an appropriate number of the basic components approximate a function with a certain degree of precision. Motivated by these insights, we explore whether a large ViT can be incrementally decomposed into basic components, which we term "Learngenes", mimicking the behavior of organismal genes as proposed by [49 [T]. In their innovative learning framework, critical knowledge is continuously condensed as learngenes during the evolution of the ancestry model, which are then inherited to initialize descendant models of varying sizes. Similarly, in our approach, we propose decompose large ViT into a set of learngenes, encapsulating critical knowledge from the original model. These learngenes can then be flexibly recomposed to initialize ViT models of different scales, adapting to diverse deployment scenarios.
Specifically, to achieve this, we propose that each layer W of the ViT can be decomposed into some basic learngenes: A1, .... AN. where each A, includes all submodules in W. such as Multiheaded Self-Attention (MSA), Multi-Layer Perceptron (MLP), Feed Forward Networks (FFN), and so on. However, ViT contains many layers and if we specify learngene space for each layer, then an L-layer VIT requires a total of N x L basic learngenes. which still requires a large amount of computation to decompose. Interestingly, as shown in of [54], the parameters of most layers in the well-trained ViT of [13] have an approximately non-decragiing trend after PCA dimension reduction. Among several possible fitting functions, the linear function is used here for its simplicity and effectiveness in approximating this trend. Thus we further assume that different layers of a VIT can also be linearly decomposed into the same learngene space, which means that we can also share the decomposed learngene across different layers in recomposition.

---

To summarize, we assume that the parameters of the 1-th layer can be got by
N W = Σ all,n) x A..
==1
where a(1, n.) are the linear combination coefficients, which are pre-defined and depend on the layer and learngene. Here, "pre-defined" denotes that a(l, n) satisfies a pre-defined role, e.g., we find that the first-kind Chebyshev polynomial formula is suitable one, and does require adjustment during training. Also, when recomposing, given the layer rank and which learngenes should be used, we can first compute the values of a(1, n) and then directly use the Equation 1 to construct the parameters of ViT with any layer.
Such decomposition--rcommpee such a decomposition strategy provides us with a novel training mechanism that allows us to iteratively train each learngene An in an incremental way. Specifically, to train the first learngene A1, we train a ViT where the parameter of the I-th layer is W = a(1, 1) x A, Then we begin to train the second learngene A₂ by constructing a ViT where Wi- a(1, 1) Ai + a(l, 2) x A2. During training, A is fixed and only the parameters of A₂ are updated. Then such a process is iterated a few times until all learngenes are trained. Compared to traditional pre-training methods, the decomposition approach not only generates a series of learngenes, but also yields variety of pre- pre-trained models with different scales during training. Importantly, the decomposed model achieves comparable performance the pre- pre-training method under the same parameters and training epochs.
strategy has the following two distinctive characteristics. First,
Second, after decomposition, the decomposed learngenes be flexibly recomposed into diverse scale ViTs with different depths without training. Meanwhile, we can use only an appropriate number of learngenes instead of all of them for recomposition achieve dynamic recuracy-efficieme offs. For example, we can use the previous 6 learngenes {Ai, As} to recompose a 4-layer ViT where the k-th layer is initialized as W = ===1 a(k,n) n.) x A., where a(k, n) is still pre-defined by satisfying a specific role, e.g., the first-kind Chebyshev polynomial coefficients. In contrast to model compression methods where each training only targets specific model size, which is not flexible enough to meet the needs of real-world deployment scenarios.
trade-
Finally, our experiments demonstrate the viability of linear decomposition and subsequent linear recomposition for ViT models. During decomposition, we observe that incrementally increasing the number of learngenes allows the performance to match that of a classically trained, integral ViT model. In recomposition, we can flexibly use an appropriate number of decomposed learngenes to construct ViTs of different layers. This method achieves performance comparable to other model compression methods, while facilitating the economical generation of models for diverse client requirements, e.g., reducing the training cost by 80x when generating the same number of models with diverse scales and initializations.
2 Related work
In this paper, present method that decomposes the parameters of the VIT model into series of learngenes combined by polynomial coefficients. Then, the decomposed learngenes are used to flexibly recompose new VIT' models with diverse scales. Thus, we discuss related works including model initialization and model compression which have similarities with the decomposition and recomposition of our method, respectively.
Model Initialization. Model initialization plays an important role in the training of deep neural networks, affecting both the rate of convergence and the quality of generalization. Fundamental methods, such as Xavier initialization [T3] and Kaiming initialization (2n] have been seminal in this domain. However, a recent trend is to use pre-trained models for initialization, as shown by studies such as 220 [5] 15 32 [50]. This practice revolves around using these pre- pre-trained models
starting point, and then fine-tume -tning them for specific tasks. While such pre-trained models offer a superior initialization point, often outperforming the likes of Xavier and Kaiming initialization and ensuring faster training convergence, they come with their own challenges. Their large architectural footprint makes them unsuitable for direct deployment across various application scenarios, especially given the varying computing capabilities of devices. Furthermore, these pre-training methods often require
3

---

large amounts of data and computational resources, making it difficult to build specialized models for individual tasks.
Model Compression Model compression 99 55 is key area in deep learning, especially for deployment on resource-constrained devices. Micro-architecture requires the design of specific model architectures to meet different computational resource requirements, but the optimal architecture often varies depending on the required model size. Model pruning [64] [65 is method that involves iteratively training the model, trimming less critical parameters. and subsequently fine-tuning for var- ious model sizes. Quantization [18 56 [14], which modulates the precision of model parameters also requires unique iterative adjustments and potential retuning for each accuracy target. Mcanwhile knowledge distillation [3 2]. a concept popularized by [22), uses a more comprehensive teacher model to guide the training of smaller student model. If we want to obtain N student models, need to train the model for N times. A common drawback of these techniques is the lack of flexibility: generating N models of different sizes typically requires N times the computational and time cost.
Prompt Tuning Visual Prompt Tuning (VPT) have also been proposed to improve parameter efficiency. For example, E22PP [17, which reduces tunable parameters through learnable prompts and pruning, and by works like Facing the Elephant in the Room [T6] that examine optimal conditions for VPT based on task and data distribution. Methods such as AdapterFusion 41 and Prefix- Tuning (27 use small, task-specific modules or tunable prefixes to allow efficient model adaptation without full retraining. In multimodal tasks, Learnabe In-Context Vector (L-ICV) 40) addresses in-context learning (ICL.) challenges by improving VQA performance with reduced computational costs by distilling task information into a single learnable vector. While these approaches reduce the number of tunable parameters, they still lack flexibility in deployment, For instance, generating N models of different sizes often requires N separate tuning or retraining processes, leading to significant computational overhead.
Therefore, we introduce novel training approach where we decompose the ViT model into different leamgenes. These decomposed learggenes can be recomposed to generate models of diverse scales, making them adaptable downstream tasks with different computational resource requirements. Additionally, by using these learngenes, the recomposed models also provide a good initialization. This results in improved performance and faster convergence in downstream tasks compared to learning from scratch.
3 Method
We propose to decompose a Vision Transformer (ViT) into a series basic learngenes that each layer of the ViT can be linearly recomposed by these learngenes. Figure outlines outtines the whole pipeline of our method, where Section 3.1 shows how to decompose the ViT into the basic learngenes and in Section we detail how to get these learngenes during training. After decomposition, Section (3.3 demonstrates how to flexibly recompose these learngenes into diverse ViTs with different depths to achieve a the balance between parameter efficiency and model performance.
3.1 Decomposing the ViT
As discussed in Section D motivated by calculus, we propose to decompose a ViT into different basic learngenes where each layer of the ViT is the linear composition of these learngenes. Formally, given the basic learngenes Ai,....Ay the parameters of the 1-th layer Wi can be got by Equation [ Note that W; represents the general term of the parameters in the (-th layer of a ViT, which can be the modules such as the Multi-head Self Attention (MSA). Multi-layer Perceptron (MLP), and Layer Normalization (LN). In other words, each module of VIT layer can be decomposed into a series of basic learngenes. For the coefficients a(I, we assume that they satisfy pre-defined polynomial such as the Taylor series, the first kind Chebyshev polynomial, or the Fourier series. Through preliminary experiments presented in Section 1.2. we find that the first kind Chebyshev polynomial particularly well suited for our purposes. Therefore, we use as an example for the decomposition coefficients in the following content. The first kind Chebyshev polynomial, represented by (x))), is

---

Decomposition
Distillation
Teacher model
0-th stage
14th stage
stage
Layer L(W,
dL 1)4,
LL 1)4,
((L.2)A
A/L. 1
@(1,2)4
m),,
Layer 2(W,)
0/2, 1)44
0 1)4,
222
212, 1)4,
2 2)4z
222.
Layer 1(W
1)4,
(1. 114,
2)44
1)A,
2)A
4
A2
A
Recomposition
"Mil conse"
1)4
2342
Init
Init
W: a12 1)4, + -.- at2, 334,
Layer
Layer
Layer
act 134,
aLL, 3)4,
Layer
RCL1M,
2142
4)A,
Figure 3: Decomposition and Recomposition. The upper part illustrates the decomposition process, where a ViT model is gradually recomposed into several learngenes. At each stage, only the newly added learngenes
trained, while the previously trained leamgenes remain frozen. The lower part shows the recomposition process with two examples. The first initializes 2-layer ViT with three learngenes trained by "without constraints" while the second initializes 3-layer ViT with four learngenes trained by "with constraints". Note that the flame icons indicate that the parameters of the layer are trained, while the snowflake icons indicates that the parameters are frozen.
defined recursively as follows:
To(r) 1, Ti (x) F, -
- (r) -
for n 2 1.
ViT have an approximately linear increasing trend. We assume that different layers of VIT share the same learngene space, and the polynomial coefficients used to compose the learngenes are layer-dependent. This means that for Tn (z), where x = 7 and the coefficient a(I, n) defined as follows:
Figure 2 was observed that the parameters of each layer
all, n) Tx 1
To avoid confusion, we use example to show how to calculate these coefficients. Suppose the ViT model we decompose a 12-layer ViT, the other words, T. equal For the second layer and the third order learngene, the formula to calculate a(2,3) is as follows:
2 2-1
2
Ti 2 2-1
((2.3) 2
T2
(4)
12
12
12
And for the fourth layer and the third-order learngene, the formula to calculate a(4 is
4
((4,3) 2
T
(5)
12
12
12
Note that the first kind Chebyshev polynomial is recursive, so in each layer the coefficients of the subsequent learngenes are influenced by the coefficients of the previous learngenes.
3.2 Learning the Basic Learngenes
Here we introduce how to obtain the basic learngenes during training. Suppose we want to decompose L-th layer ViT into N learngenes. We iteratively use more learngenes, following Equation to
5

---

construct and train the VIT. During training, in each iteration, only the newly added learngenes are trained, and all the previously used learngenes are fixed. More specifically, in the first iteration, we only use the first learngene A to build the ViT following Equation [ i.e., W = a(l, 1) and train this ViT. After the first iteration, we add the second learngene A₂ into the Equation | i.e., Wi = a(I, A + al1, 2) A2, and then continue training this ViT on classification again. However, in this iteration, only the second learngene A₂ will be trained, while A₁ will be fixed. Then, we will iterate this process until all N learngenes are trained in turn.
When training each learngene, we use the training objectives following Min 62 where the total loss L contains two terms: the classification cross-entropy loss Lee and the distillation loss Cut The classification eross-entropy loss Lee as follows:
Lex ΣΣ Wi log()r(z))).
(6)
Here, V/j denotes the ground truth label for the i-th class of the j-th data instance, expressed in one-hot encoded format. The function & represents the softmax operation, and 2, is the logit output corresponding to the j-th data instance from the model comprising with learngenes. Next, we define the distillation loss as follows:
La = K (==((==a)
where ž denotes the logits output of the teacher model, 2, denotes the logits output of the model comprising with learngenes and K represents the Kullback-Leibler divergence loss.
Thus, the loss function for learngene-witg wise training integrates both cross entropy and distillation losses as follows:
full feee + Lal-
(8)
3.3 Recomposing the ViTs
After the training process, the ViT model is decomposed into several learngenes. Consequently, these learngenes linearly composed by polynomial coefficients are serve as initialization for VIT models for diverse downstream tasks. It is important to note that, the linear coefficients are pre-defined by Equation which means that these coefficients will not be updated during fine-tuning the recomposed models. The initialization of ViT models with layers of different depth use a suitable number of learggenes from Equation with the number of layers L adapted to fit the available computational resources. Based on the above decomposition formulas, the ViT models consisting of L layers can be initialized follows:
Wi = a(1, 1)4, + a(1,2)42 +... all, A),
where W/ denotes the parameters of the /-th layer in the ViT models, k is the number of learngenes used for initialization.
After initializing the ViT models, there are two ways to train the model as shown in the lower part of Figure 3 The first one is that Equation merely provides an initialization for the ViT models. In this way, the ViT models are trained without being constrained by Equation | the parameters of each layer Wi are updated independently, i.c., the number of trainable parameters correlates with the layer number of the recomposed models. The second one is that the parameters of each layer of the VIT models are still constrained by Equation | In other words, during training. the ViT models update the parameters of the l learngenes used for initialization, not the parameters of Wi Consequently. the number of trainable parameters in the model depends on the number of learngenes used. Therefore, not only can we choose an appropriate number of learngenes to initialize models with an appropriate number of layers, but we can also choose different training methods based computational resources as discussed in Section 4.2.3
4 Experiments
In this section, we first describe the implementation details of the decomposition and recomposition processes of VIT in Section 4.1 Followed by experiments to validate that the VIT model can be decomposed into a series of Tearngenes and the feasibility of the ViT models that are flexibly composed of these learngenes in Section 4.2

---

4.1 Implementation Details
4.1.1 Datasets
To train each learngene, we use ImageNet-IK [12. which contains approximately 1.2M training images across 1000 classes and 50K validation images. After recomposing ViT models of different layers with leargenes, we adapt them on 9 diverse downstream datasets, which include object classification tasks: CIFAR-10 10
26]. CIFAR- 100 [26. and Tiny-ImageNet [: 5 fine-gradd fine-grained classifi- cation tasks: iNaturalist-2011 [66. Food-10) 101 4. Oxford Flowers- 102 [35]. Stanford Cars [25] and
Oxford-IIIT Pets [38]; texture classification task: DTD [00. Each dataset presents unique chal- lenges, ranging from basic object recognition to more specialized classification based on fine-grained
visual differences and texture patterns.
4.1.2 Details of Decomposition and Recomposition
Decomposition. Our approach decompose the VIT model into different learngenes is based on the DeïT architecture [48]. To train the learngenes,
distillation strategy with the RegNet-16GF [42 as the teacher model. We train total 12 learngenes because there are 12 layers in DeiT. Constrained by computational resources, instead of training each learngene sequentially, we divided the training process into five phases, with learngene counts of {1, 2, 4, 8, 12). For example, during the third training phase, only the two newly added learggenes trained, while the two previously trained learngenes are frozen. In each phase, the model is trained for 100 epochs with an initial warmup epochs.
Recomposition. In the recomposition, we can choose different number of learngenes to flexibly generate the VIT models with different layers, such as (2, 4, 6, 8, 10, 12) layers. First, we determine the appropriate number of learngenes and number of layers of the ViT models based on computing resources. Then we use the pre-defined Equation 3o calculate the polynomial coefficients. Following the Equation initialized, they be trained in one of two ways as mentioned in Section 4.2.3
/ can be used to initialize the parameters of the ViT models. After the models are
Computing resources. The resource cost of our method includes the decomposition and recompo- sition. In decomposition, takes 500 epochs obtain 12 learngenes, while recomposition requires no additional training to use different number of decomposed learngenes to initialize different layer- based models. Since the number of layers of ViTs ranges from 1 12, and each layer can be initialized with 12 different numbers of learngenes, there are a total of 12x 12 combinations of ViTs with different sizes and different initializations. Then if we try to get all these student ViTs by pre- te-ttrrin/ddistilaattit example, training one student takes 300 epochs, then 12 12 x 300=43.2K epochs are required in total.
we need to respectively train each one individually. Taking [62] as an
4.2 Results and Analyses
In this paper, we have elementary assumptions of Equation 2 Firstly, a VIT be decomposed into a series of basic learngenes. Secondly, ViTs with diverse depths can be flexibly recomposed by parts all of these learngenes. Here, we first determine the polynomial coefficients for decomposition and recomposition based on the experiments in Section 4.2. Then we implement experiments to validate these two major assumptions Section 4.2.2 and Section 4.2.3 respectively.
4.2.1 Choosing Suitable Polynomial Coefficients
In the decomposition and recomposition, there are sev- eral choices of the polynomial coefficients for Equation 3. e.g., Taylor series, the first kind Chebyshev polyno- mial, Fourier series and Legendre series. To select the appropriate polynomial coefficients, we perform sim- ple experiment in which the VIT model decomposed into 12 learngenes. We then use these polynomial coefficients to compose these learngenes and train them simultaneously for 100 epochs. The results are presented in Section 4.2.T it can be observed that when using the first kind Chebyshev Polynomial coefficients, we have the highest performance:
Table Results of ViTs with different decomposition coefficients.
Taylor 74.80%
Chehyshev 77.73%
Pourier Legendra 73.13%
Accuracy
7

---

77.73% accuracy. Therefore, we use the first kind Chebyshev polynomial coefficients in Equations and the subsequent decomposition and recomposition experiments.
4.2.2 Decomposing the VIT
3
In Section 3.2 we show that the basic learn- genes decomposed from the ViT model, i.e., DeiT-Base, can be trained incrementally. a method named ICT. Figure [a) also shows the result of training all learngenes simulta- neously under the same training settings, a method called SCT. The process of incremen- tal training is visually represented Figure f where it is evident that as the number of learn-
10
KC
SCT
DupS CT 54 S4-het
BEmA
76
Leamgene Number
Pa Paramettrs (M)
(a) Compare with Deit-Bast (b) Compare with SN-Net
Figure 4: The accuracy of our method is compared with that of DeiT-B and SN-Net on ImageNet- IK.
genes increases, the accuracy of the ViT model composed of these learngenes not only improves, but also surpasses the results of SCT.
In particular, 12 learngenes, the accuracy of the ViT model reaches 83.70 33.70% which is comparable to the performance of DeiT-Base, where the parameters in different layers are initialized independently. The results shown by DeiT in Figure [a)] which are the same as the training method used in the decomposition, are obtained by training with the loss function in Equation 8 with 500 epochs of training. Thus, after separately learning each linear learngene of the ViT, the performance can achieve the results of the classical ViT which is trained as whole, confirming the earlier assumption in Section that the parameters of ViT can be linearly decomposed.
Moreover, we also compare the decomposition results with SN-Net 37 in Figure 4(b) which describes the generation of differently scaled VITs on ImageNet- -k. The results show that the models obtained by decomposition outperform those stitched by SN-Net from DeiT-Tiny, DeiT--Small and DeiT-Base. The computational cost for both approaches is as follows: our method totals 800 epochs, consisting of 300 epochs for training DeiT-Base and 500 for the decomposition. In contrast, SN-Net incurs 1 1150 epochs, with each of DeiT-Tiiy/Smaal/Baas requiring 300 x 3 epochs, plus an additional 50 x 5 epochs for five stitched models. Thus, our method demonstrates not only better performance but also greater efficiency in computational cost.
4.2.3 Recomposing the VITs
After training, these learngenes can be flexibly recomposed into ViT models with different layers according Equation To validate the effectiveness of recomposing ViTs with diverse depths by using different numbers of learngenes, we perform the following experiments.
Evaluating Learngene Number and Layer Depth Effeets on Model Performance. As mentioned in Section ! the decomposed learngenes can be used to initialize VIT models with different layers to satisfy the computational resource requirements. Additionally, Section 3.3 introduces two ways for training the reconstituted VIT models. The first method, named "w/o const" means that Wi calculated by Equation Daaatioooonnly provides an initialization, and the parameters of each layer are independent. The second method, named "w/ const" is that the parameters of each layer still satisfy Equation 9 Due to space limitations, some experimental results are presented in the Appendix.
In Figure 5 we have two dimensions: the depth of the network and the number of the learngenes used to recompose. From these figures, first, we can find that when recomposing deeper ViT or using more basic learngenes to recompose ViTs, the performance will also improve in most cases, whether "ww/ const" or "w/o const" are used. In addition, increasing the number of layers tends to have a greater impact on the preform of the models. Second, on each graph, there is a boxed area bounded by red contours that represents 95% of the maximum value on each graph. Therefore, if the computer resources of clients are limited, they can choose to build VITs with fewer layers and initialize with fewer learggenes achieve better resource- resource-performance performance trade-off off. For example, in Figure ((g) the 6-layer ViT model initialized by only 1 learngene can achieve 95% of the best performance.
Figures 5(t) and 5(g) also show a comparison between the "w/ const" and "w/o const" training methods. However, it should be noted that when the number of layers exceeds the number of learngenes, the model trained by the "w/o const" method consumes more memory than the "w/
8

---

Lemggrt Number
Lost Number
81:15
(6.88
$7.99
11:39
66.M4
00.00
91.9
58,02
14.43
55.17
55.44
$9.67
19.50
15
66.53
99.8
Laargenc
90.09
.4
20.8
56,16
162
85.75
Loaragen
72.39
$5.0
92
56
93.71
$1.25
$5,91
55.22
75
19.05
19.30
21117
33.86
65.47
06.14
90:34
95.90
12.55
51.15
21.22
55.76
$5.15
70.16
75009
75.44
11.00
00.57
07.37
25.11
85.54
65.41
58:24
09.57
06.66
29:32
50.30
09.50
8
22
Depth
Deych
Depth
Depth
(a) Tiny-IMNET
(b) Food-101
(c) CIFARIO
(d)
Flowers
50.4
12.62
12.51 -1.00
Ioangene Narber
45:41
Leamgron Number
757
69.24
01
Learggeee Nunber
59.26
06.63
00.79
$8.50
Lampore Number
01.11
40-42
65.29
10.56
52.15
16.3
36.00
19.11
55.25
59.24
91.12
10.01
21.01
59.29
52.54
55.57
92331
42.00
-7
-3.00
1:05
16.04
-557
16.95
10,00
-27.0
15.14
-3.54
+13.5
38:12
73.97
87.6
77.04
97:42
95.13
97.25
67.1
18.50
272
Depch
2
Depth
2
22
Derth
12
Depth
(e) Miny-IMEET
(I) Food-101
(g) CIFARIO
(h) Flowers
Figure 5: Performance of ViTs with different layers initialized with different learngenes on down- stream datasets. The first row shows the results with the "woo const" training method, the second row is trained with "w/ const". The second row also shows the differences between the two, where "_" indicates improvement and " "_" " indicates degradation. The boxed areas bounded by the red contours represent regions where the accuracy is within 95% of the maximum value of each graph.
80
70
Accuracy
70
Accuracy
Accuracy
80-
Accuracy
60-
60
60
60
50
2if oint Oure hage nonort
Out cers."
25 50
75
25 58 75
25 50 75
25
50
75
Parameters (M)
Parameters (M)
Parameters (M)
Parameters (M)
(a) INAT
(b) Cars
(c) Pets
(d) DTD
Figure 6: Comparative results. The ViT' are initialized with two different recomposition cases and trained by corresponding methods, i.e., one trained with constraint and the other is trained without constraint, and then compared with the performance of differently scaled pre-trained models and models obtained by model compression methods.
const". Therefore, for downstream clients with limited computational resources who want better performance. They can construct ViT model with a higher number of layers, initialize it with smaller number of learngenes, and then use "w/ const" training method. For example, a 12-layer ViT model initialized with two learngenes requires 86MB of memory using the "w/o const" method, but only 14.3MB using the "w/ const" method. On Food-101 const" and 90.44% with "w/o const". Despite the slightly lower performance, the "w/ const" method significantly reduces memory usage by about 12 times.
achieves 89.34% performance with "w/
Comparative Analysis. We compare our method with pre- training & fine-tuning (Pre-Fin) and model compression methods. In particular,
(1) Pre-Fin: We use the pre-trained DeiT models including DeiT-Tiny, DeiT-Small and DeiT-Base in [88. and then fine-tune the model for downstream tasks. (2) Model compression:
The methods used include Mini-DeiT-B in [62 and Efficient- ViT 30
different learngene configurations recompose models with different numbers of trainable parameters. For the "w/ const", the parameters of the model depend on the number of learngenes
For two training strategies in recomposition,
in Section 3.3 Therefore, we use the 12-layer VIT model recomposed by { 21 4, 6, 8, 10, 12) learngenes. For the "w/o const", the trainable parameters correlate with the layer number of the model. Here, the number of learngenes is kept at 12 and ViT models are initialized over (2. 4, 6, 8, 10, 12) layers. Additional dataset results and corresponding numerical data are in the Appendix due space limitations.
From Figure we find that when different trainable parameters are used, "w/ const" outperforms the other methods such as pre-fine fine and model compression on downstream datasets. For example, in

---

when the model with 7.8M parameters is recomposed from single learngene and trained with constraints, the accuracy is 72.3%. This performance exceeds that of Efficient ViT, which has 8.8M parameters, by 6.3%. Moreover, "w/o const" performs slightly worse than other methods with less trainable parameters because the networks have fewer layers. As in Figure 66b) a 4-layer model with 28M parameters trained with "w/o const" performs over 9% worse than DeiT-Sman with 22M parameters. However, as the number of layers increases, the performance of models trained with "w/o const" improves. For example, 6-layer model with the same number of parameters as MiniViT which a 12-layer model, shows performance difference of only 1.4%, indicating competitive results. These comparisons validate that when using suitable training strategy, the recomposed diverse-scale ViTs have good initialization that by simply training. they can achieve comparable or better performance than Pre-Fin and model compression methods. Furthermore, by our strategy. an appropriate number of decomposed learngenes can be selected based on available computational resources to initialize the models without training from scratch, achieving flexible accuracy and efficiency trade-offs.
Figure 6(a)
5 Conclusion
In summary. this paper proposes a novel training method for efficiently generating VITs of varied sizes meet diverse computational needs. By employing linear decomposition, ViT can be decomposed into basic learngenes, termed as "learngenes", which encapsulate critical knowledge from the original model. These learngenes can then be selectively and linearly recomposed to form ViTs of various layers and sizes. This flexible recomposition provides an economical and adaptive solution for creating a series of small or medium ViTs tailored to different deployment environments. Our experiments confirm the effectiveness of this barngene-based decomposition-rrcopppon method and show that these recomposed ViT's maintain performance comparable to traditional model compression techniques while offering greater flexibility and efficiency.
Acknowledgments and Disclosure of Funding
This research supported by the National Science Foundation of China (62125603 62076063), the Key Program of Jiangsu Science Foundation (BK20243012), the Fundamental Research Funds for the Central Universities (2242024k30035) and the Big Data Computing Center of Southeast University.
References
[1] Tiny ImageNet Visual Recognition Challenge. http:://cs23n.ssanttee edu// MM-DD.
Accessed: YYYY-
[2] Nima Aghli and Eraldo Ribeiro. Combining weight pruning and knowledge distillation for compression. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 3191-3198, 2021.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 106 08254, 2021.
[4] Lukas Bossard, Mathhiu Guillaumin, and Lue Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 12 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Scott Lundberg, Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303. 2023.
[6] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Proceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pages 535-541 2006.
[7] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 357-366, 2021.
10

---

[8] Xinlei Chen, Saining Xie. and Kaiming He. An empirical study of training sell-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9640-9649, 2021.
[9] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv: 2017.
[10] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3606 36166666332766 2014.
[1I] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
Ieee, 2009.
[13] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /9999, 2020.
[14] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems. 35:4475- 4488, 2022.
[15] Xavier Glorot and Yoshua Bengio. Xavier initialization. J. Mach. Learn. Res, 2010.
[16] C Han, Q Wang. Y Cui, W Wang. L Huang, S Qi, and D Liu. Facing the elephant in the room: visual prompt tuning or full finetuning? arxiv. Preprint]. 2024.
[17] Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang. Siyuan Qi, and Dongfang Liu. E 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv: 13770, 2023.
[18] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint
2015.
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9729-9738, 38, 2020.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv 2015.
preprint
[23] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint ar /.ti::/774.44880 2017.
[24] Sasan Karamizadeh, Shahidan M Ahdullah, Azizah A Manaf, Mazdak Zamani, and Alireza Hooman. An overview of principal component analysis. Journal of Signal and Information Processing, 4(3B):17,, 2013.
[25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-gra -ne-ggrined categorization. In Proceedingy of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[26] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[27] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 2011:2101.001 90, 2021.
[28] Yawei Li, Kamil Adamczewski, Wen Li. Shuhang Gu, Radu Timofte, and Luc Van Gool. Revisiting random channel pruning for neural network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 191-201, 2022.

---

[29] Jingyu Lin, Yan Yan. and Hanzi Wang. A dual-path transformer network for scene text detection. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pages 1-5, IEEE, 2023.
[30] Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuging Yang. Han Hu, and Yixuan Yuan, Efficientvit: Memory efficient vision transformer with cascaded group attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[31] John C Mason and David C Handscomb. Chebyshev polynomials. CRC press, 2002.
[32] Xuran Meng, Difan Zou, and Yuan Cao. Benign overfitting in two-layer relu convolutional neural networks for xor data. arXiv preprint arXiv:2310.0195, 2023.
[33] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 5191-5198. 2020.
[34] James O' Neill. An overview of neural network compression. arXiv preprint arXiv:2006.03669, 2020.
[35] Maria-Elee Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes, 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 729-729. IEFE, 2008.
[36] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szaframiec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacidin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 2rXiv:2304.07199, 2023.
[37] Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Stitchable neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16102-16112, 2023.
[38] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505, IEEE, 2012.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perffrmance deep learning library. Advances in neural information processing systems, 32, 2019.
[40] Yingzhe Peng, Chenduo Hao, Yang, Jiawei Peng, Xinting Hu, Xin Geng, Learnable in-context vector for visual question answering. arXiv preprint arXiv:2406. 13185, 2024.
[41] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning, arXiv preprint arXiv:2005.002: 2020.
[42] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428-10436, 2020.
[43] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifi- cation using binary convolutional neural networks. In European conference on computer vision, pages 525-542. Springer, 2016.
[44] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[45] James Stewart. Essential calculus: Early transcendentals. Brooks/Cole, part of the Thomson Corporation, 2007.
[46] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVE international conference on compuler vision, pages 7262-7272. 2021.
[47] Georgi Tolstov, Fourier series. Courier Corporation, 2012.
[48] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
12

---

[49] Qiu-Feng Wang, Xin Geng, Shu-Xia Lin, Shi- Yu Xia, Lei Qi. and Ning Xu. Learngene: From open-worls to your learning task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8557-8565, 2022.
[50] Qiufeng Wang, Xu Yang. Haokun Chen, and Xin Geng. Vision transformers as probabilistic expansion from learngene. In Forty-first International Conference on Machine Learning.
[51] Qiufeng Wang. Xu Yang, Shuxia Lin, Jing Wang, and Xin Geng. Learngene: Inheriting condensed knowledge from the ancestry model descendant models. arXiv preprint arXiv:2305. 022299 2023.
[52] Zhehui Wang. Tao Luo, Miqing Li, Joey Tianyi Zhou, Rick Siow Mong Goh, and Liangli Zhen. Evolu- tionary multi-objective model compression for deep neural networks. IEEE Computational Intelligence Magazine, 10-22, 2021.
[53]
Ross Wightman et al. Pytorch image models, 2019.
[54] Shiyu Xia, Miaosen Zhang, Xu Yang, Ruiming Chen, Haokun Chen, and Xin Geng. Transformer as linear expansion of learngenc. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16014-16022, 2024.
[55] Canwen Xu and Julian McAuley. survey on model compression and acceleration for pretrained language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10566-10575, 2023.
[56] Jiwei Yang. Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 7308-7316 2019.
[57] Xingyi Yang. Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. Advances in neural information processing systems, 35:25739-253 -97733,,575,,... 2022.
[58] Chao Yi. De-Chuan Zhan, and Han-Jia Ye. Bridge the modality and capacity gaps in vision- -aiiuale model selection. arXiv preprint arXiv:2403.13799 2024.
[59] Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Zi- Zi-Hang Jiang, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEFF/CVF international conference on computer vision, pages 558-567, 2021.
[60] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEF/CVF international conference on computer vision, pages 6023-6032, 2019.
[61] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 17.00944 2017.
[62] Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12145-12154, 2022.
[63] Yi-Kai Zhang. Shiyin Lu, Yang Li, Yanqing Ma. Qing- Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Wings: Learning multimodal Ilms without text-only forgetting. arXiv preprint arXiv:2406. 03496.03466.204 2024.
[64] Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. Advances in Neural Information Processing Systems, 35:18309-18826, 2022.
[65] Kaiqi Zhao, Animesh Jain, and Ming Zhao. Automatic attention pruning: Improving and automating model pruning using attentions. In International Conference on Artificial Intelligence and Statistics, pages 10470-10480 10486. PMLR, 2023.
[66] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9728,, 2020.
[67] Hongyuan Zhu, Fanman Meng, Jianfei Cai, and Shijian Lu. Beyond pixels: A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation. Journal of Visual Communication and Image Representation, 34:12-27 27. 2016.
[68] Weiqi Zou, Yang Wang. Xueyang Fu, and Yang Cao. Dreaming to prune image deraining networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2022.
13

---

Amareringue
(a) MOCO
(b) DINO
(C) MAE-B
(d) BEITv2-B
Figure 7: The parameters of each layer of the pre-trained model and their corresponding layer position relationships.
A Appendix supplemental material
In the following section, we present the process of dimensionality reduction applied to the parameters of each layer in the Vision Transformer (ViT) models in order to observe the interrelationships between the parameters across different layers. We then supplement the experimental content presented in the main text, including the experimental results on decomposition and recomposition, as well implementation details. Finally, this section concludes with a discussion of the limitations and societal implications of our method.
A.1 Dimensionality Reduction and Inter-Layer Parameter Relationships in ViTs
Our analysis starts with the aggregation of parameter matrices from the Multihead Self-Attention (MSA), Multilayer Perceptron (MLP), and Layer Normalization (LN) modules of an extensively trained Transformer model. Specifically, within the MSA module and given an input æ, first perform the computation of (x to determine the attentional outputs for each head. This then facilitates the derivation of W for the MSA as follows:
W W W
(10)
where W and W², belonging to DxD act as transformation parameter matrices for keys and queries, respectively, where D denotes the dimensionality of the intermediate embeddings. Subse- quent operations involve the bias vectors for keys and queries, proceeding according to Equation 10 Together with other parameter matrices in the MSA module, we reshape these into vectors and subject them to L2 normalization.
In the MLP module, flatten and normalize weight matrices and bias vectors from two linear transformations. similar approach employed in the LN module, where weight matrices and bias vectors associated with transformation parameters normalized. Normalized vectors, specific to each parameter type, are concatenated alignment with their corresponding layer, culminating in a synthesized matrix S E NDx,, where D,, symbolizes the normalized vector dimension for each parameter type, and N signifies the depth of the Transformer.
Principal Component Analysis (PCA) [24] is then utilized to reduce the dimension of each vector in to one-dimensional scalar, primarily for analytical convenience. These reduced vectors are then combined to form the matrix H R. N with K indicative of the number of parameter types. Each row of / interpreted as the representative vector the corresponding layer. Another iteration of of H into one-dimensional domain, resulting in vector
PCA is performed to compress the U E N This final vector is used to illustrate the relationship between the sequential positioning of the layers in the transformer and their respective parameter values, thus illustrating the nuanced interplay of parameters across the architectural layers.
We also show the graphs for MoCov3 8. DINOv2 36. MAE-B [19 and BEITv2-B 3 in Fig- ures (() to 7 which show that the parameters of the layers have an approximately linear correlation
with the layer position.
A.2 Experiment
A.2.1 Decomposition
14

---

Origin
learngene- learngene-2 learngene-3
learngene-4
learngene-5
learngene-6
Plant
Bird
Insect
(a) Qualitative Visualization of Each learngene. Uses each of the first six leamgennss initialize the model separately, showing different characteristics and areas of focus for each learngene.
Origin
lcarngeme
2- genes
earnggenss
learngenes
5. learngenes
6- learngenes
Plant
Bird
Insect
(b) Qualitative visualization of the ViT model initialized by incremental learngenes.
Figure 8: Qualitative visualization of decomposed learngenes
Qualitative Visualization of Learngenes. After training, we obtain the decomposed learngenes of the Vision Transformer (ViT). To analyze their characteristics, we employ Gradient-weighte Class Activation Mapping (Grad-CAM) E4,, which is pivotal in the classification process, visualize the feature maps in two scenarios: one where each learngene is used independently to initialize ViT, and the other where learngenes are added incrementally to initialize the ViT model.
From Figure 8(a) we find that each learngene focuses on uniquely specific regions, indicating that the functionalities of the ViT model are decomposed into distinet learngenes. Specifically, the first learngene focuses on basic shapes and contours. The second learngene detects textures and colors, such as highlighting leaf textures and bird feather details. The third learngene identifies spatial features such as insect wings, while the fourth learngene detects finer details such as bird eyes or insect antennae. The fifth learngene emphasizes image contrast, and the sixth distinguishes between foreground and background, focusing on depth perception. As shown in Figure 8(6) early learngenes capture basic shapes and edges, while later learngenes refine the focus on textures, colors, and specific details such as leaf veins or bird feathers. As learngenes are added, the attention of model shifts to finer features such bird beaks, insect wings, and antennae, indicating a progression from general to detailed feature recognition that is likely to lead to more accurate image classification.
A.2.2 Recomposition
Evaluating Learngene Number and Layer Depth Effects on Model Performance. The experi- mental results in Figure on these two datasets further verify the findings in the text. The results in the appendix confirm the observation that deeper network configurations or larger number of
15

---

1
12A0
KLLS
$1.00
DMAT
11.00
17.30
11 HAUL 24
14-2
14:00
NLLS 11.20
11.00 10.00
(10
LongRR Narter
MUT
50.00
DLIP
7244
2016
1000
(a) CIFAR100 w'w (b) INAT / "w/o (c) Pets wwwo const" (d) Cars wwwoo const" (c) DTD /"woo const" const"
const"
1.54
0.
0.41
1.21
10.9
MLAA
10.00
56.00
20144
11:21
13.3
(744
10:4
73.44 2213
8611
41:4
(f) CIFARIO0 //w/ (g) INAT /"w/ const" (h) Pets //w/ const" const"
(i) Cars /"w/ const"
(i) DTD /"w/ const"
Figure 9: Performance of ViTs with different layers initialized with different learngenes down- stream datasets.
10
Accuraty
-0
sccuracy
Securagy
Accuracy 16
$5.0
92.5
50
22
52
Parameters (H)
Parameters
Parameters (M)
Parameters
(a) Tiny-IMNET
(b) CIFAR 100
(c) CIFARIO
(d) Food-101
(e) Flowers
Figure 10: Comparative results. The ViTs are initialized with two different recomposition cases and trained by corresponding methods, i.e., one trained with constraint and the other is trained without constraint, and then compared with the performance of differently scaled pre-trained models and models obtained by model compression methods.
decomposed learngenes generally improve ViT performance, whether trained by "w/o const" or "w/ const". The results also show stable performance beyond certain learngene and layer thresholds, and provide strategies for optimizing ViT models under resource constraints, such as using fewer layers and learngenes to achieve near-optimal performance. We also compare the "w/const" and "w/o const" training methods and find that performance differences decrease as model complexity increases. In particular, models with higher layers show minimal performance variance between the two methods, but differ memory consumption, with "w/o const" requiring more. We recommend "w/o const" for efficiency in resource-constraing scenarios, as shown by the 12-layer VIT model initialized with fewer learngenes, which achieves comparable performance with significantly less memory consumption.
Comparative Results. The experimental results presented in Figure 10 show that 12-layer Vision Transformer (ViT) initialized with different numbers of learngenes and trained using the "with constraint" method, represented by the orange lines in the graphs, outperforms traditional approaches such as pre-fine tuning and model compression under the same parameter constraints. In addition, as the number of layers in the model increases to a certain threshold, models initialized with 12 learngenes and trained using the "without constraint" method also outperform these basclines. Thus, method strikes a balance between flexibility and model efficiency.
Numerical Results for Comparative Results. Table 2 provide the numerical performance on 9 downstream tasks to compare our methods with pre-training fine-tuning and model compression methods. The same results also shown in Figures and 10 of the main paper.
16

---

Table Comparative experiments. Two cases of initializing models with learngenes and their corre- sponding training approaches, compared with the pre-traiinn pre-training fine-tuning and model compression methods.
Model
Para
CIFARIO
CIFARTOD
Tiny-ImageNe ImageNet
Food
Cars
Flowers
Model Pretrained
Deit-T
5M
97.85
86,36
76.15
70.1
86.54
79.32
64.69
90.43
65.48
Deit-S
22M
98.63
89.59
80.66
74.88
89.16
88.01
85.92
92.53
67.23
Deit-B
86M
98,92
90.03
87.5
77.43
91.44
91.97
92.54
93.54
72.87
Mini- Deit and Efficiett VIT
Mini-Deit
43M
98.57
90.29
81.54 74.95
73.85
90.94
89.68
92.15
93.71
68 83
EfficientVViT
8.8M
97.1
86.63
86.90 96
81.2
62.75
89.35
63.67
learngenes used recomposee different layers ViT' trained with "w/a const" training method
2layer 12learngene
14.3M
88,75
66.05
59.48
51.19
72.05
22.35
21.43
46.02
40.64
4layer- 2learngene
96.33
82.35
72.32
76.36
43,99
78.64
6layer-1 Sepeer-12leare 1learngene
43M
98.03
87.75
78.65
69.47
90
88.28
73.67
92.62
68.14
Slayer-1 Zlearngene
57.3M
98.36
89.49
82.1
71.98
90.91
91.27
89.19
94.2
72.5
10layer- I 2Zeaargenee
71.6M
98.76
90.6
88.01
75.03
91.61
91.99
91.59
94.47
73.67
12layer- 2leamgeene
86M
98.97
91.05
89.16
77.89
92.02
92.68
93.33
94,99
74.73
12 layered models recomposed by different number of learngenes, trained with "w const" training method
12layer- learrgene
7.8M
98.54
88.37
$1.85
84,61
66.98
90.95
12layer 2learngene
14.3M
98.65
89.24
83.13
75.21
89.34
89.62
88.98
93.19
67.13
Slaayer-llarrgenee
28.7M
98.69
90.91
86.63
76.14
90.83
91.95
92.88
94.69
72.23
12layer 6learngene
43M
98.75
90.69
86.18
76.95
91.37
91.99
93.2
94.63
72.66
57.3M
98.93
90.73
88.44
76.93
91.16
92.43
94.93
94.58
73.35
Pllaycr-10lcamgee
71.6M
98.86
90.31
88.01
92.42
95.24
73.4
12layer- 22eeamgene
86M
98.96
91.96
88.45
78.51
91.65
92.68
95.8
95.59
74.52
Table 3: Overview of Classification Datasets
Datasct Name
Categorics
Training Samples
Test Samples
CIFAR-
10
50,000
10,000
CIFAR-I00
100
50,000
10,000
Tiny-InageNet
100,000
10,000
INAT-2019
1,010
268,243
64,401
25,250
Stanford Curs
196
8,144
8,041
Oxford Flowers
102
2,000
818
Oxford-IIFT Pets
3,680
3,569
DTD
3,760
1,880
A.3
Implementation Details
A.3.1 Code
We implement the model using PyTorch [39 and the Timm library [53]. The decomposition process trained over 500 epochs four NVIDIA RTX 3090 GPUs, and the recomposed models are trained over 100 epochs on two NVIDIA RTX 3090 GPUs for each downstream task.
A.3.2 Datasets and Pre- prepprocgs
Datasets We adapt the recomposed models 9 diverse downstream datasets, covering wide range classification challenges. These datasets include object classification tasks: CIFAR-10
26],
CIFAR-101 (26). and Imny-ImageNet [1; 5 fine-grained classification tasks: iNtturalist-2019
66
Food-101 4. Oxford Flowers- 102 35 Stanford Cars (25,, and Oxford-IIfT Pets 38 texture
classification task: DTD 10]
The details of these datasets in Table
we train and evaluate the decomposition and recom- position processes on all datasets at resolution of 224 x 224. The data augmentation techniques employed include RandAugment [I]] Cutmix [60], Mixup [6]], and random erasing.
Data Process Following previous works 62
A.3.3 Hyper-parameter
Decomposition
We train each learngene using the following setting:
17

---

We employ distributed training across 4 GPUs, with each GPU handling 128 data instances, resulting in overall batch size of 512.
Batch Size:
Optimizer: The training of each learngene is optimized using AdamW, with initial learning rate of 0.0008 and a weight decay of 0.05.
Learning Rate Schedule: We apply a cosine learning rate decay, with a warm-up period of epochs.
Recomposition
We train the recomposed models on downstream tasks using the following setting:
Batch Size: We employ distributed training across 2 GPUs, with each GPU handling 128 data instances, resulting in an overall batch size 256.
Optimizer: The training of each learngene optimized using Adam W and weight decay of 0.05.
Learning Rate Schedule: We apply a cosine learning rate decay, with a warm-up period of epochs.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims stated in the abstract and introduction are fully supported by the experimental results presented in the paper, which confirm the effectiveness of our methodology in adapting Vision Transformers diverse computational settings.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of our method in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification: Our method is inspired by polynomial decomposition in calculus. This paper primarily investigates the feasibility of the approach, and this stage we are unable to provide a comprehensive theoretical underpinning.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our paper provides code in the Supplementary Material and describes the datasets and hyperparameters used the Experiments section the main text and in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide code in the Supplementary Material, and the datasets used in the paper are all public datasets, which are described in the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https::/niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wttpcc/ccccccccccccocooooo httpa//gggudeee/deuuuuubbiiiiiii wwweiisssooPoPoiiiioooo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: Yes]
Justification: We provide experimental setting Experiment section and the hyper- parameters in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Due the large size of the datasets, significance tests were not performed, but consistency was maintained by fixing the random seed during training and evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The appendix details the types and quantities of machines used, although specific execution times are not provided due the numerous experiments conducted.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics stttiisshttts////tt
Answer: [Yes]
Justification:
The research strictly conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses the societal impact of our work in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not involve the release of models or data, hence there are no associated risks requiring safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The experimental section and appendix provide detailed references to the datasets used, citing the original sources. In addition, all experimental settings are properly referenced, ensuring full compliance with the licensing requirements of the assets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No release of new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human subjects research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing or human subjects research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "jElPJbkD2G", "reviewer": "Reviewer_WjQY", "summary": "Inspired by polynomial decomposition in calculus, the authors propose linearly decomposing the ViT model into a set of components during element-wise training. These components can be recomposed into differently scaled, pre-initialized models to meet different computational resource constraints. This decomposition-recomposition strategy offers an economical and flexible approach for generating diverse scales of ViT models for various deployment scenarios. Unlike model compression or training from scratch, which require repeated training on large datasets for different-scale models, this strategy reduces computational costs by training on large datasets only once. Extensive experiments demonstrate the method's effectiveness, showing that decomposed ViTs can be recomposed to achieve comparable or superior performance to traditional model compression and pre-training methods.", "questions": "1.Can you demonstrate that your method is applicable to other well-known transformer variants such as Swin and MobileViT?\n\n2.Can you compare the performance of your proposed models, decomposed and recomposed from DeiT-B, against other state-of-the-art models with the same parameter settings?", "limitations": "I shared my concerns in the weakness section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors successfully decompose and recompose the layers of transformers, achieving competitive performance across various component configurations, parameter settings, and datasets.", "weaknesses": "There are many variants of pure vision transformers, such as Swin and MobileViT, which significantly outperform traditional vision transformers. It would be valuable to retrain these models for scenarios requiring different model sizes and datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To 1: Swin Transformers and MobileViT\n\nThank you for your question, which inspired us to explore the applicability of our method to Swin Transformers and MobileViT. \nSwin Transformers consist of four stages, where different stages have transformer layers with different resolutions, resulting in different components decomposed from layers across different stages. Due to time constraints, we focused on decomposing the layers in stage 3 because it has a larger number of layers. We visualized the relationship between the layer parameters and the corresponding layer index in stage 3. In Fig.A(a) and Fig.A(b) of PDF, we present the results for Swin-T and Swin-B, respectively. We observed a linear relationship between the parameters and the layer index, similar to that shown in Fig.2 in paper for plain transformers. Therefore, we adapted our method for Swin transformers by treating stage 3 as a plain transformer during decomposition. We have successfully decomposed one component from stage 3 of the Swin-Tiny model, achieving a result of 77.61\\%. This is close to the Swin-Tiny's performance on ImageNet-1k, which is 81.2\\%, and the initial results are promising. We will include the full experimental design and results in the revision.\n\nFor MobileViT, in Fig.A(c) we also visualize the transformer layers across different ViT blocks. We observed that within each block, there is a linear relationship between the layer parameters and the layer index. However, since MobileViT has relatively fewer parameters, it may not be economical and efficient to decompose it. Given the linear relationship, it is still possible that the transformer layers in MobileViT could be initialized with pre-trained components. We will perform further experiments to validate this possibility and incorporate these results in revision.\n\n\nTo 2: More comparative experiments\n\nThank you for your question. For decomposition, as in Fig.4(a), our decomposed models outperform the original DeiT-B model with the same number of parameters under the same training conditions. Fig.4(b) further shows that our decomposition approach leads to better performing models in different parameter regimes compared to the SN-Net family.\n\nFor the recomposition, we compared our recomposed models to different-sized DeiT models and other compression methods such as Mini-DeiT-B and Efficient-ViT. The parameters for these models were downloaded from their official sources and fine-tuned under the same training epochs as ours over 9 downstream tasks. The results in Fig.6 demonstrate the effectiveness of our method. In addition, we provide detailed numerical performance in the supplementary material (Tab.2).\n\nFurthermore, we conducted additional comparison experiments with the 12-layer ViT, based on the DeiT-Base structure, initialized with 4 components and trained using the \"with constraint\" method (28.7M parameters), compared to the pre-trained Swin-Tiny (28M parameters), as shown in Tab.A of the PDF. The results indicate that the model initialized with components performs comparably to the pre-trained Swin-Tiny on downstream tasks with a similar number of parameters. This further demonstrates the feasibility of using components to initialize ViTs of different scales. We will provide the full experimental results in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WQ1F97velK", "reviewer": "Reviewer_DK4A", "summary": "This paper proposes a novel method to efficiently adapt Vision Transformer (ViT) models to devices with diverse computational resources by linearly decomposing a ViT into basic components that can be flexibly recomposed into ViTs of various scales. The key ideas are:\n\n- Inspired by polynomial decomposition, a ViT can be decomposed into basic components where each layer is a linear combination of the components using polynomial coefficients.\n- The components are trained incrementally, where in each iteration only the newly added components are trained while previously trained ones are fixed. This element-wise training reduces compute cost.\n- The trained components can then be flexibly recomposed into ViTs of different depths using an appropriate number of components to achieve accuracy-efficiency trade-offs for different deployment scenarios.\n- Compared to repeatedly training diverse-scale models from scratch, this decomposition-recomposition strategy is more economical.\n\nExperiments demonstrate the effectiveness of the method - the decomposed components can recompose ViTs of various scales that achieve comparable or better performance than traditional model compression methods.", "questions": "Please see weakness section.", "limitations": "Please see weakness section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The linear decomposition-recomposition idea, inspired by mathematical concepts, provides an elegant and principled framework to adapt ViTs to diverse computational constraints. The assumption that ViT layers can be decomposed into and recomposed from basic components is well-motivated and validated.\n2. The incremental component-wise training is a smart strategy to reduce training cost while enabling the components to capture essential functionalities of the ViT. Limiting training to only the newly added components in each phase is efficient.\n3. The flexible recomposition allows generating a spectrum of ViT scales by varying the number of layers and components used. This provides a smooth accuracy-efficiency trade-off to cater to different deployment requirements, which is a key practical advantage.\n4. Extensive and rigorous experiments convincingly demonstrate the effectiveness of the method. The decomposed models perform competitively or better compared to strong compression baselines. Qualitative visualizations provide good insight into the progressive feature learning by the components.\n5. Overall, the framework attacks an important problem and proposes an innovative, principled, and effective solution. The writing is mostly clear and easy to follow.", "weaknesses": "I have three major concerns, I will consider adjusting the score based on the quality of the rebuttal provided.\n\n1. the recomposition process can be explained more clearly, especially the two types of coefficient initialization and the two training strategies (\"with constraint\" and \"without constraint\"). A figure illustrating the different recomposed architectures will help understanding. Regarding the statement in line 198 that the 'without constraint' training method still adheres to the constraints of Equation 9, could you clarify if this means the polynomial coefficients remain unchanged and maintain their predefined values during the training process?\n\n2. In addition to the linear relationship shown in Fig 2, it would be helpful to see several other models to demonstrate the generalizability of this pattern.\n\n3. The paper claims that the first-kind Chebyshev polynomial coefficients work best for recomposition, but limited rationale is provided for this choice.  Some theoretical analysis on why Chebyshev polynomials are particularly suitable would strengthen the motivation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To 1: Details of recomposition\n\nThank you for your feedback. In the revision, we will provide a more detailed explanation of the recomposition process and update Fig.3 in the paper to illustrate the initialization of different model architectures and the various training methods.\nIn Fig.D of the PDF, we provide two examples. At the top of Fig.D, a 2-layer ViT is initialized with 3 components, and at the bottom, a 3-layer ViT is initialized with 4 components. Regardless of the training method, different model structures are initialized using Eq.9. As shown in Fig.D, for the \"without constraint\" training, Eq.9 is used solely for initialization. During training, the parameters of each layer ($W_l$) are updated independently, similar to a typical ViT, without adhering to Eq.9. In contrast, the \"with constraint\" training method updates the parameters of the component $A_n$ during training and maintains the polynomial coefficients unchanged, meaning the components remain shared across all layers, and each layer's parameters continue to satisfy Eq.9.\n\n\nTo 2: More linear relationship examples for other models\n\nThank you for the suggestion. The supplementary material provides results for MoCo and DINO models using the same process as described in the paper. Additionally, in Fig.B, we show the relationship between each layer's parameters and the corresponding layer index for MAE-B[1], BEITv2-B[2], and RoBERTa-B[3] models. It is evident that these models also exhibit a linear relationship between the parameters and the layer index. In the revision, we will include these results to further demonstrate the generalizability of this pattern.\n\nTo 3: Chebyshev polynomials\n\nThank you for your observation. In Tab.1 of paper, our experimental results demonstrate that Chebyshev polynomials outperform other polynomials in recomposition. This may be because Chebyshev polynomials are known to provide the best uniform approximation of continuous functions [4]. And each layer of the ViT model can be viewed as a function, the parameter matrix of each layer can be decomposed into components and represented as a weighted sum using Chebyshev polynomials. In the revision, we will expand on this theoretical motivation and explicitly highlight the desirable mathematical properties of Chebyshev polynomials.\n\n\n[1]Masked Autoencoders Are Scalable Vision Learners. \n\n[2]BEIT: BERT Pre-Training of Image Transformers\n\n[3]RoBERTa: A Robustly Optimized BERT Pretraining Approach\n\n[4]Trefethen, L. N. (2013). Approximation Theory and Approximation Practice. SIAM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVua0agdOZ", "reviewer": "Reviewer_GH5B", "summary": "This paper proposes a novel approach to generate diverse-scale Vision Transformer (ViT) models with varying computational complexity, aiming to address the deployment challenges posed by the fixed architectures of standard ViT models. The key idea involves linearly decomposing a pre-trained ViT model into basic components and then recomposing these components to initialize ViT models of different scales, effectively enabling the economical generation of models tailored to diverse computational resource constraints.", "questions": "Please see the Weaknesses.", "limitations": "Please refer to the Weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized and logically structured, with the introduction, methodology, and experiments seamlessly connected, making it easy to understand and evaluate.\n\nThe proposed decomposition-recomposition strategy is interesting which offers two key advantages: (1) an iterative training mechanism that generates a series of pre-trained models with different scales during the decomposition process, and (2) the flexibility to recompose the decomposed components into diverse-scale ViTs without additional training, enabling dynamic accuracy-efficiency trade-offs.\n\nThe provision of code in the supplementary material facilitates reproducibility of the experiments.", "weaknesses": "The paper could explain the component sharing approach in more detail. Is it similar to MiniViT, where components are shared across layers? Or are the component parameters only shared within each layer? Also, please clarify if the forward process is the same as in traditional ViTs.\n\nFor the recomposition, could you provide more information on the initialization methods used for the 'with constraint' and 'without constraint' training approaches? Are the initialization methods the same, with the only difference being the parameters being optimized (layer parameters W vs. component parameters θ)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To 1: Components sharing approach\n\nThank you for your insightful question. The component sharing in our model is applied within each individual layer, unlike the cross-layer sharing seen in MiniViT. Additionally, the forward process follows the typical ViT architecture, such as DeiT, during training. In the revision, we will include more detailed explanations to clarify these points.\n\nTo 2: Training approaches in recomposition\n\nThank you for your question. For the two training approaches, the initialization methods used are the same. The key difference lies in how the parameters are optimized during training. In the \"without constraint\" approach, Eq.9 is only used to provide an initialization of layer parameters $W_l$ for ViTs. During training, the parameters of each layer $W_l$ are then updated independently, the same as the normal ViT fine-tuning, without being constrained by Eq.9. In contrast, the \"with constraint\" approach optimizes the component parameters directly, rather than the individual layer parameters $W_l$. This means that during gradient back-propagation, the gradient is applied to ${A_k}$ instead of ${W_l}$. Therefore, the parameters of each layer remain constrained by Eq.9, with components shared across all layers. We will clarify this further in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NZfU7A46zG", "reviewer": "Reviewer_caYK", "summary": "This paper presents a novel approach to adapting Vision Transformer (ViT) models for diverse computational environments. The key idea is to linearly decompose a large ViT model into a set of basic components during training, which can then be flexibly recomposed to create smaller, pre-initialized models suitable for various deployment scenarios.", "questions": "1. How to train each of the sub-components so the cumulative performance can be on par with the original model? Due to the parameter gap, given the same amount of data, smaller modules may not perform equivalently to the bigger model. With that said, a model of Wn may perform better than the cumulative power of {A1...An}. Did the authors make any effort to address this challenge?\n\n2. I have trouble understanding this statement: \"Interestingly, as shown in Figure 2 of [39], the parameters of most layers in the well-trained ViT of [10] have an approximately non-decreasing trend after PCA dimension reduction.\" What do the x and y axes represent? What is the figure telling us?\n\n3. In the empirical design, where can I see your arrangement of sub-module combinations {A1...An}? From lines 222 to 225, I only see layer-level combinations. Also, the empirical results should include the latency improvement for each scale as it is an important motivation of this paper.\n\n3. I have trouble understanding this statement: \"For example, during the third training phase, only the two newly added components are trained, while the two previously trained components are frozen. In each phase, the model is trained for 100 epochs with an initial 5 warmup epochs.\" Shouldn't this be one added component, not two?", "limitations": "Please refer to weakness and questions.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The work makes three key contributions: 1. The paper introduces the idea of linearly decomposing ViT layers into shared components, which is a novel approach in the field of model adaptation. 2. Unlike traditional compression methods that require separate training for each model size, this approach allows for flexible model generation from a single decomposition. 3. The flexibility in recomposing models of various scales addresses the challenge of deploying ViTs in environments with different computational constraints.", "weaknesses": "There are several concerns regarding this work:\n\n1. The training process for the decomposed modules requires a more detailed description.\n\n2. The implementation details of the reconstruction process are insufficiently explained.\n\n3. The empirical design does not fully support the proposed method. Many tunable components are not thoroughly discussed, such as the training and combination strategies for each layer's sub-modules, and the achieved latency savings at different scales.\n\n4. The literature review is superficial. Parameter-efficient paradigms should be comprehensively discussed to clearly differentiate the proposed solution from existing approaches.\n\n5. The work appears to be at an early stage of development. While the motivation and design show promise, the writing and formatting require significant improvement. The authors should enhance the overall writing quality and correct numerous typographical errors throughout the manuscript (e.g., in the caption of Figure 2, \"layer poaition\" is misspelled, and there should be a space before parentheses).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To W1 \\& Q1: Details of decomposition \n\nThank you for your question. Due to the parameter gap, the 12-layer model with only one component $A_1$ in the first stage does not perform as well as a typical 12-layer model. However, as more components are trained, the performance becomes comparable to the model with the same parameters.\n\nAs shown in Fig.C of the PDF, using DeiT-Base as an example, ${W_l}$ are the layer parameters of the model and are constructed by trainable parameters ${A_n}$ (include all modules in a ViT layer like MSA, MLP). In the 1-st stage, each layer $W_l$ initially contains one component, $A_1$, shared across all layers, with coefficients calculated using a predefined polynomial (Eq.3). The number of learnable parameters in the 12-layer model is approximately equal to $A_1$, achieving 76.6% performance, as in Fig.4(a).\n\nIn the 2-nd stage, a new component, $A_2$, is added to each layer, and $A_1$ and $A_2$ are combined using their respective coefficients to initialize each layer for forward propagation as Eq.1. During training, the parameters of $A_1$ remain fixed, and only the newly added component $A_2$ is updated. In this way, the learnable parameters still equals to the parameters of one component. This process is repeated until all components are fully trained. \n\nAs the number of added components increases, the performance of the 12-layer model improves, reaching 83.70\\% with 12 components, which is comparable to DeiT-Base trained as a whole, with a performance of 83.51\\% under the same training conditions and parameters. This confirms our assumption that each ViT layer can be linearly decomposed into a shared component {$\\{A_1, \\ldots, A_n\\}$}, and that a model composed of components {$\\{A_1, \\ldots, A_k\\}$} can perform comparably to the full model $W$ under the same parameters as in Fig.6. We will include these details in the revision for clarity. Should you have any further suggestions, we would be happy to address them.\n\nTo W2: Details of recomposition\n\nAfter all components are trained,  the components can be directly used to initialize model without training in the recomposition. In Fig.D of the PDF, we present two examples. Regardless of the training method, different model structures are initialized by Eq.9. Under the \"without constraint\" training method, Eq.9 is only used to initialize $\\{W_l\\}$; the trainable parameters of each layer ($\\{W_l\\}$) are updated independently during training, the same as the normal ViT fine-tuning. The \"with constraint\" training method decomposed the $\\{W_l\\}$ to trainable parameters $\\{A_k\\}$ while keeping the polynomial coefficients fixed using Eq.9. This means that, during the gradient back-propagation, the gradient is backed to $\\{A_k\\}$ instead of $\\{W_l\\}$. We hope this explanation clarifies the reconstruction, and we will include these details in the revision. \n\nTo W3 \\& Q3:  Empirical design of components \\& latency improvement\n\nThank you for your questions. In the decomposition and recomposition, all sub-modules within a ViT layer are treated uniformly. Therefore, in our paper, each component $A_n$ includes all sub-modules within a ViT layer $W$, such as MSA, MLP, FFN, and others. As in Fig.C of PDF, in the different stages of decomposition, the parameters of each layer are obtained by Eq.1. In the recomposition, as in Fig.D, ViT models of varying layer can be initialized with different numbers of components, with each layer’s parameters initialized by Eq.9.\n\nRegarding the latency improvements for different scales, in recomposition, models of different sizes can be initialized with components by Eq.9, without the need for repeated training as required in knowledge distillation, which contributes to our latency improvement. In Sec.4.1.2 \"Computing Resources\", we compare the computational savings of our method against other model compression methods, highlighting the efficiency of our approach.\n\nTo W4: Literature review\n\nWe appreciate the suggestion to expand the literature review. Currently our literature review includes model initialization and model compression (Micro-architecture design, Model pruning, Quantization, and Knowledge distillation). In the revision, we will also provide a detailed discussion of other parameter-efficient paradigms, including Low-Rank Factorization (LRF) and Parameter Sharing (PS). For LRK and PS, generating models of different sizes typically requires redesigning the factorization or sharing scheme for each target model size, which can be complex and resource-intensive. In contrast, our approach allows flexible and economical generation of models with different sizes without repeated training or redesign.\n\nTo W5: Writing\n\nThank you for the feedback. We'll improve the revision by enhancing writing quality, correcting typos, and refining formatting for a more polished presentation.\n\nTo Q2: Details of Fig.2\n\nThank you for your question. In Fig.2, the x-axis is the layer index of ViT, while the y-axis is the values of the parameters in each layer after performing PCA, reduced to one dimension (details are in the Suppl Material Sec.A.1). The figure shows a linear relationship between these two variables. This observation suggests that the parameters across layers in a well-trained transformer are not independent variables, but may be linearly related in a higher dimensional space, possibly correlated with the layer index. This implies that the parameters can be reconstructed using the form given in Eq.1, enabling flexible initialization for models of different scales. We will clarify this point in the revision to enhance understanding.\n\nTo Q4: Experimental design of decomposition training\n\nThank you. You've raised an important point; ideally, components would be added and trained one at a time, but that's time consuming. To balance efficiency and performance, we trained multiple components in stages. We will clarify this point in the revision to ensure better understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We greatly appreciate the insightful and constructive feedback provided by the reviewers. We have responded in detail to the concerns and questions raised, and have attached a PDF with figures and tables. Below is a brief overview of the PDF:

1. Fig. A: Shows the parameters of each layer of the pre-trained Swin Transformer/MobileViT and their corresponding layer position relationships (to Reviewer WjQY).

2. Fig. B: Presents additional examples of linear relationships for other models, illustrating the parameters of each layer and their corresponding layer position relationship (to Reviewer DK4A).

3. Fig. C: Demonstrates the decomposition process using DeiT-Base as an example (to Reviewer caYK).

4. Fig. D: Shows the recomposition process and two training methods with two examples: the first is a 2-layer ViT initialized with 3 components, and the second is a 3-layer ViT initialized with 4 components (to Reviewer DK4A, caYK). 

5. TABLE A: Contains comparative experimental results comparing the 12-layer ViT initialized with 4 components to the pre-trained Swin-Tiny (to Reviewer WjQY).

Meta-review: The authors propose an parameter-efficient learning technique. The final score is 5767. 

Overall, the author's rebuttal has won unanimous approval from the reviewers, and this article has made its own contribution. After reviewing the reviewers' comments, the author's responses, and the AC's discussion, we finally decided to accept this article, but based on the following two points:
1. The author needs to fulfill his promise and discuss the two articles raised by the reviewer caYK in the discussion section.

2. The author needs to change the tone of the conclusion and claim in the paper, because the current technology may not be able to surpass the following models in all sizes: Inception transformer, multiscale vision transformer, cswin
Decision: Accept (poster)