Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models

Authors: shuxia  lin
miaosen  zhang
ruiming  chen
qiufeng  wang
xu  yang
xin  geng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 14:21:29

Abstract: Vision Transformers (ViTs) are widely used in a variety of applications, while they usually have a fixed architecture that may not match the varying computational resources of different deployment environments. Thus, it is necessary to adapt ViT architectures to devices with diverse computational overheads to achieve an accuracy-efficient trade-off. This concept is consistent with the motivation behind Learngene. To achieve this, inspired by polynomial decomposition in calculus, where a function can be approximated by linearly combining several basic components, we propose to linearly decompose the ViT model into a set of components called learngenes during element-wise training. These learngenes can then be recomposed into differently scaled, pre-initialized models to satisfy different computational resource constraints. Such a decomposition-recomposition strategy provides an economical and flexible approach to generating different scales of ViT models for different deployment scenarios. Compared to model compression or training from scratch, which require to repeatedly train on large datasets for diverse-scale models, such strategy reduces computational costs since it only requires to train on large datasets once. Extensive experiments are used to validate the effectiveness of our method: ViTs can be decomposed and the decomposed learngenes can be recomposed into diverse-scale ViTs, which can achieve comparable or better performance compared to traditional model compression and pre-training methods. The code for our experiments is available in the supplemental material.

Main paper: 
 introduction: 
 The pre-training Vision Transformers (ViTs) [13] have become fundamental to various applications, including image classification [59, 7] , object detection [29, 52] , semantic segmentation [46, 67] , and multimodal tasks [63, 58] . However, these ViTs typically have a standard and relatively fixed architecture, which poses challenges for deployment in diverse real-world settings, i.e., devices in different application scenarios have different computational capabilities, making standard, fixed-size ViTs unsuitable for direct deployment. Furthermore, the storage demands of a traditional ViT may exceed the capabilities of certain devices. For instance, the memory size for the ViT-B/16 model [13] with 86M parameters is approximately 320 MB, while some devices may find this prohibitive due to inherent physical constraints, e.g., the limited RAM capacity of the Raspberry Pi 1 with 256MB-512MB of RAM. To address this limitation, researchers have developed various compression techniques [6, 62, 55, 28] to reduce the model size to make them suitable for the application scenarios while meantime maximum preserving the model ability. Typical approaches including micro-architecture design [23] , model pruning [14, 68] , quantization [43] , and knowledge distillation [62] . However, in these approaches, the architecture of the smaller model, such as the number of layers, is predetermined and thus lacks the flexibility to meet the diverse needs of real-world deployment scenarios. Consider the scenario depicted in Figure 1 : to obtain N models of different sizes for different devices, the knowledge distillation methods need to go through N separate trainings. Striking a balance between flexibility and accuracy is challenging and often requires complex training strategies to ensure satisfactory model performance.Then we may ask, is there a way to more flexibly and economically generate N smaller models from a large one, while maintaining the performance of the smaller ones? In calculus, we know that a function can be approximated by linearly combining several basic components with corresponding weighted coefficients at various levels of precision, e.g., Taylor series [45] , Chebyshev Polynomial of the First Kind [31] , Fourier series [47] and so on. Such a mathematical theorem induces two engineering insights: (1) a function can be decomposed into a series of basic components, i.e., the polynomial terms, and (2) we can combine an appropriate number of the basic components to approximate a function with a certain degree of precision. Motivated by these insights, we explore whether a large ViT can be incrementally decomposed into basic components, which we term "learngenes", mimicking the behavior of organismal genes as proposed by [49, 51] . In their innovative learning framework, critical knowledge is continuously condensed as learngenes during the evolution of the ancestry model, which are then inherited to initialize descendant models of varying sizes. Similarly, in our approach, we propose to decompose a large ViT into a set of learngenes, encapsulating critical knowledge from the original model. These learngenes can then be flexibly recomposed to initialize ViT models of different scales, adapting to diverse deployment scenarios. Specifically, to achieve this, we propose that each layer W of the ViT can be decomposed into some basic learngenes: A 1 , ..., A N , where each A n includes all submodules in W , such as Multiheaded Self-Attention (MSA), Multi-Layer Perceptron (MLP), Feed Forward Networks (FFN), and so on. However, a ViT contains many layers and if we specify a learngene space for each layer, then an L-layer ViT requires a total of N × L basic learngenes, which still requires a large amount of computation to decompose. Interestingly, as shown in Figure 2 of [54] , the parameters of most layers in the well-trained ViT of [13] have an approximately non-decreasing trend after PCA dimension reduction. Among several possible fitting functions, the linear function is used here for its simplicity and effectiveness in approximating this trend. Thus we further assume that different layers of a ViT can also be linearly decomposed into the same learngene space, which means that we can also share the decomposed learngene across different layers in recomposition.To summarize, we assume that the parameters of the l-th layer can be got byEQUATIONwhere a(l, n) are the linear combination coefficients, which are pre-defined and depend on the layer and learngene. Here, "pre-defined" denotes that a(l, n) satisfies a pre-defined role, e.g., we find that the first-kind Chebyshev polynomial formula is a suitable one, and does not require adjustment during training. Also, when recomposing, given the layer rank l and which learngenes should be used, we can first compute the values of a(l, n) and then directly use the Equation 1 to construct the parameters of a ViT with any layer.Such decomposition-recomposition strategy has the following two distinctive characteristics. First, such a decomposition strategy provides us with a novel training mechanism that allows us to iteratively train each learngene A n in an incremental way. Specifically, to train the first learngene A 1 , we train a ViT where the parameter of the l-th layer is W l = a(l, 1) × A 1 . Then we begin to train the second learngene A 2 by constructing a ViT where W l = a(l, 1) × A 1 + a(l, 2) × A 2 .Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The claims stated in the abstract and introduction are fully supported by the experimental results presented in the paper, which confirm the effectiveness of our methodology in adapting Vision Transformers to diverse computational settings.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 In this paper, we present a method that decomposes the parameters of the ViT model into a series of learngenes combined by polynomial coefficients. Then, the decomposed learngenes are used to flexibly recompose new ViT models with diverse scales. Thus, we discuss related works including model initialization and model compression which have similarities with the decomposition and recomposition of our method, respectively.Model Initialization. Model initialization plays an important role in the training of deep neural networks, affecting both the rate of convergence and the quality of generalization. Fundamental methods, such as Xavier initialization [15] and Kaiming initialization [21] , have been seminal in this domain. However, a recent trend is to use pre-trained models for initialization, as shown by studies such as [20, 5, 57, 32, 50] . This practice revolves around using these pre-trained models as a starting point, and then fine-tuning them for specific tasks. While such pre-trained models offer a superior initialization point, often outperforming the likes of Xavier and Kaiming initialization and ensuring faster training convergence, they come with their own challenges. Their large architectural footprint makes them unsuitable for direct deployment across various application scenarios, especially given the varying computing capabilities of devices. Furthermore, these pre-training methods often require large amounts of data and computational resources, making it difficult to build specialized models for individual tasks.Model compression [9, 55, 34] is a key area in deep learning, especially for deployment on resource-constrained devices. Micro-architecture requires the design of specific model architectures to meet different computational resource requirements, but the optimal architecture often varies depending on the required model size. Model pruning [64, 65] is a method that involves iteratively training the model, trimming less critical parameters, and subsequently fine-tuning for various model sizes. Quantization [18, 56, 14] , which modulates the precision of model parameters,also requires unique iterative adjustments and potential retuning for each accuracy target. Meanwhile, knowledge distillation [33, 2] , a concept popularized by [22] , uses a more comprehensive teacher model to guide the training of a smaller student model. If we want to obtain N student models, we need to train the model for N times. A common drawback of these techniques is the lack of flexibility: generating N models of different sizes typically requires N times the computational and time cost.Prompt Tuning Visual Prompt Tuning (VPT) have also been proposed to improve parameter efficiency. For example, Eˆ2VPT [17] , which reduces tunable parameters through learnable prompts and pruning, and by works like Facing the Elephant in the Room [16] that examine optimal conditions for VPT based on task and data distribution. Methods such as AdapterFusion [41] and Prefix-Tuning [27] use small, task-specific modules or tunable prefixes to allow efficient model adaptation without full retraining. In multimodal tasks, Learnable In-Context Vector (L-ICV) [40] addresses in-context learning (ICL) challenges by improving VQA performance with reduced computational costs by distilling task information into a single learnable vector. While these approaches reduce the number of tunable parameters, they still lack flexibility in deployment. For instance, generating N models of different sizes often requires N separate tuning or retraining processes, leading to significant computational overhead.Therefore, we introduce a novel training approach where we decompose the ViT model into different learngenes. These decomposed learngenes can be recomposed to generate models of diverse scales, making them adaptable to downstream tasks with different computational resource requirements. Additionally, by using these learngenes, the recomposed models also provide a good initialization. This results in improved performance and faster convergence in downstream tasks compared to learning from scratch.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discussed the limitations of our method in the Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We propose to decompose a Vision Transformer (ViT) into a series basic learngenes that each layer of the ViT can be linearly recomposed by these learngenes. Figure 3 outlines the whole pipeline of our method, where Section 3.1 shows how to decompose the ViT into the basic learngenes and in Section 3.2, we detail how to get these learngenes during training. After decomposition, Section 3.3 demonstrates how to flexibly recompose these learngenes into diverse ViTs with different depths to achieve a the balance between parameter efficiency and model performance.As discussed in Section 1, motivated by calculus, we propose to decompose a ViT into different basic learngenes where each layer of the ViT is the linear composition of these learngenes. Formally, given the basic learngenes A 1 , ..., A N , the parameters of the l-th layer W l can be got by Equation 1. Note that W l represents the general term of the parameters in the l-th layer of a ViT, which can be the modules such as the Multi-head Self Attention (MSA), Multi-layer Perceptron (MLP), and Layer Normalization (LN). In other words, each module of a ViT layer can be decomposed into a series of basic learngenes. For the coefficients a(l, n), we assume that they satisfy a pre-defined polynomial such as the Taylor series, the first kind Chebyshev polynomial, or the Fourier series. Through preliminary experiments presented in Section 4.2.1, we find that the first kind Chebyshev polynomial is particularly well suited for our purposes. Therefore, we use it as an example for the decomposition coefficients in the following content. The first kind Chebyshev polynomial, represented by T n (x), is defined recursively as follows:T 0 (x) = 1, T 1 (x) = x, T n+1 (x) = 2xT n (x) -T n-1 (x) for n ≥ 1.(2) Figure 2 , it was observed that the parameters of each layer in the ViT have an approximately linear increasing trend. We assume that different layers of a ViT share the same learngene space, and the polynomial coefficients used to compose the learngenes are layer-dependent. This means that for T n = (x), where x = l-1 L , and the coefficient a(l, n) is defined as follows:EQUATIONTo avoid confusion, we use an example to show how to calculate these coefficients. Suppose the ViT model we decompose is a 12-layer ViT, in the other words, L is equal to 12. For the second layer and the third order learngene, the formula to calculate a(2, 3) is as follows:EQUATIONAnd for the fourth layer and the third-order learngene, the formula to calculate a(4, 3) isEQUATIONNote that the first kind Chebyshev polynomial is recursive, so in each layer the coefficients of the subsequent learngenes are influenced by the coefficients of the previous learngenes.Here we introduce how to obtain the basic learngenes during training. Suppose we want to decompose a L-th layer ViT into N learngenes. We iteratively use more learngenes, following Equation 1, to construct and train the ViT. During training, in each iteration, only the newly added learngenes are trained, and all the previously used learngenes are fixed. More specifically, in the first iteration, we only use the first learngene A 1 to build the ViT following Equation 1, i.e., W l = a(l, 1)A 1 , and train this ViT. After the first iteration, we add the second learngene A 2 into the Equation 1, i.e., W l = a(l, 1)A 1 + a(l, 2)A 2 , and then continue training this ViT on classification again. However, in this iteration, only the second learngene A 2 will be trained, while A 1 will be fixed. Then, we will iterate this process until all N learngenes are trained in turn.When training each learngene, we use the training objectives following MiniViT [62] , where the total loss L contains two terms: the classification cross-entropy loss L ce and the distillation loss L dl . The classification cross-entropy loss L ce is as follows:EQUATIONHere, y ij denotes the ground truth label for the i-th class of the j-th data instance, expressed in a one-hot encoded format. The function ϕ represents the softmax operation, and z j is the logit output corresponding to the j-th data instance from the model comprising with learngenes. Next, we define the distillation loss as follows:EQUATIONwhere z t denotes the logits output of the teacher model, z s denotes the logits output of the model comprising with learngenes and KL represents the Kullback-Leibler divergence loss.Thus, the loss function for learngene-wise training integrates both cross entropy and distillation losses as follows:EQUATIONAfter the training process, the ViT model is decomposed into several learngenes. Consequently, these learngenes linearly composed by polynomial coefficients are serve as initialization for ViT models for diverse downstream tasks. It is important to note that, the linear coefficients are pre-defined by Equation 3, which means that these coefficients will not be updated during fine-tuning the recomposed models. The initialization of ViT models with layers of different depth use a suitable number of learngenes from Equation 3, with the number of layers L adapted to fit the available computational resources. Based on the above decomposition formulas, the ViT models consisting of L layers can be initialized as follows:EQUATIONwhere W l denotes the parameters of the l-th layer in the ViT models, k is the number of learngenes used for initialization.After initializing the ViT models, there are two ways to train the model as shown in the lower part of Figure 3 . The first one is that Equation 9 merely provides an initialization for the ViT models. In this way, the ViT models are trained without being constrained by Equation 9, the parameters of each layer W l are updated independently, i.e., the number of trainable parameters correlates with the layer number of the recomposed models. The second one is that the parameters of each layer of the ViT models are still constrained by Equation 9. In other words, during training, the ViT models update the parameters of the k learngenes used for initialization, not the parameters of W l . Consequently, the number of trainable parameters in the model depends on the number of learngenes used. Therefore, not only can we choose an appropriate number of learngenes to initialize models with an appropriate number of layers, but we can also choose different training methods based on computational resources as discussed in Section 4.2.3.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The appendix details the types and quantities of machines used, although specific execution times are not provided due to the numerous experiments conducted.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experiments: 
 In this section, we first describe the implementation details of the decomposition and recomposition processes of ViT in Section 4.1. Followed by experiments to validate that the ViT model can be decomposed into a series of learngenes and the feasibility of the ViT models that are flexibly composed of these learngenes in Section 4.2.To train each learngene, we use ImageNet-1K [12] , which contains approximately 1.2M training images across 1000 classes and 50K validation images. After recomposing ViT models of different layers with learngenes, we adapt them on 9 diverse downstream datasets, which include 3 object classification tasks: CIFAR-10 [26] , CIFAR-100 [26] , and Tiny-ImageNet [1] ; 5 fine-grained classification tasks: iNaturalist-2019 [66] , Food-101 [4] , Oxford Flowers-102 [35] , Stanford Cars [25] , and Oxford-IIIT Pets [38] ; 1 texture classification task: DTD [10] . Each dataset presents unique challenges, ranging from basic object recognition to more specialized classification based on fine-grained visual differences and texture patterns.Decomposition. Our approach to decompose the ViT model into different learngenes is based on the DeiT architecture [48] . To train the learngenes, we use a distillation strategy with the RegNet-16GF [42] as the teacher model. We train a total of 12 learngenes because there are In Section 3.2, we show that the basic learngenes decomposed from the ViT model, i.e., DeiT-Base, can be trained incrementally, a method named ICT. Figure 4 (a) also shows the result of training all learngenes simultaneously under the same training settings, a method called SCT. The process of incremental training is visually represented in Figure 4 , where it is evident that as the number of learngenes increases, the accuracy of the ViT model composed of these learngenes not only improves, but also surpasses the results of SCT.In particular, at 12 learngenes, the accuracy of the ViT model reaches 83.70%, which is comparable to the performance of DeiT-Base, where the parameters in different layers are initialized independently. The results shown by DeiT in Figure 4(a) , which are the same as the training method used in the decomposition, are obtained by training with the loss function in Equation 8, with 500 epochs of training. Thus, after separately learning each linear learngene of the ViT, the performance can achieve the results of the classical ViT which is trained as a whole, confirming the earlier assumption in Section 1 that the parameters of ViT can be linearly decomposed.Moreover, we also compare the decomposition results with SN-Net [37] in Figure 4 (b), which describes the generation of differently scaled ViTs on ImageNet-1k. The results show that the models obtained by decomposition outperform those stitched by SN-Net from DeiT-Tiny, DeiT-Small, and DeiT-Base. The computational cost for both approaches is as follows: our method totals 800 epochs, consisting of 300 epochs for training DeiT-Base and 500 for the decomposition. In contrast, SN-Net incurs 1150 epochs, with each of DeiT-Tiny/Small/Base requiring 300 × 3 epochs, plus an additional 50 × 5 epochs for five stitched models. Thus, our method demonstrates not only better performance but also greater efficiency in computational cost.After training, these learngenes can be flexibly recomposed into ViT models with different layers according to Equation 9. To validate the effectiveness of recomposing ViTs with diverse depths by using different numbers of learngenes, we perform the following experiments.Evaluating Learngene Number and Layer Depth Effects on Model Performance. As mentioned in Section 1, the decomposed learngenes can be used to initialize ViT models with different layers to satisfy the computational resource requirements. Additionally, Section 3.3 introduces two ways for training the reconstituted ViT models. The first method, named "w/o const" means that W l calculated by Equation 9only provides an initialization, and the parameters of each layer are independent. The second method, named "w/ const" is that the parameters of each layer still satisfy Equation 9. Due to space limitations, some experimental results are presented in the Appendix.In Figure 5 , we have two dimensions: the depth of the network and the number of the learngenes used to recompose. From these figures, first, we can find that when recomposing a deeper ViT or using more basic learngenes to recompose ViTs, the performance will also improve in most cases, whether "w/ const" or "w/o const" are used. In addition, increasing the number of layers tends to have a greater impact on the preform of the models. Second, on each graph, there is a boxed area bounded by red contours that represents 95% of the maximum value on each graph. Therefore, if the computer resources of clients are limited, they can choose to build VITs with fewer layers and initialize with fewer learngenes to achieve better resource-performance trade-off. For example, in Figure 5 (g), the 6-layer ViT model initialized by only 1 learngene can achieve 95% of the best performance.Figures 5(f ) and 5(g) also show a comparison between the "w/ const" and "w/o const" training methods. However, it should be noted that when the number of layers exceeds the number of learngenes, the model trained by the "w/o const" method consumes more memory than the "w/ const". Therefore, for downstream clients with limited computational resources who want better performance. They can construct a ViT model with a higher number of layers, initialize it with a smaller number of learngenes, and then use "w/ const" training method. For example, a 12-layer ViT model initialized with two learngenes requires 86MB of memory using the "w/o const" method, but only 14.3MB using the "w/ const" method. On Food-101, it achieves 89.34% performance with "w/ const" and 90.44% with "w/o const". Despite the slightly lower performance, the "w/ const" method significantly reduces memory usage by about 12 times.Comparative Analysis. We compare our method with pre-training & fine-tuning (Pre-Fin) and model compression methods. In particular, (1) Pre-Fin: We use the pre-trained DeiT models including DeiT-Tiny, DeiT-Small and DeiT-Base in [48] , and then fine-tune the model for downstream tasks.(2) Model compression: The methods used include Mini-DeiT-B in [62] and Efficient-ViT [30] .For two training strategies in recomposition, we use different learngene configurations to recompose models with different numbers of trainable parameters. For the "w/ const", the parameters of the model depend on the number of learngenes as in Section 3.3. Therefore, we use the 12-layer ViT model recomposed by {1, 2, 4, 6, 8, 10, 12} learngenes. For the "w/o const", the trainable parameters correlate with the layer number of the model. Here, the number of learngenes is kept at 12 and ViT models are initialized over {2, 4, 6, 8, 10, 12} layers. Additional dataset results and corresponding numerical data are in the Appendix due to space limitations.From Figure 6 , we find that when different trainable parameters are used, "w/ const" outperforms the other methods such as pre-fine and model compression on downstream datasets. For example, in Figure 6 (a), when the model with 7.8M parameters is recomposed from a single learngene and trained with constraints, the accuracy is 72.3%. This performance exceeds that of Efficient ViT, which has 8.8M parameters, by 6.3%. Moreover, "w/o const" performs slightly worse than other methods with less trainable parameters because the networks have fewer layers. As in Figure 6 (b), a 4-layer model with 28M parameters trained with "w/o const" performs over 9% worse than DeiT-Small with 22M parameters. However, as the number of layers increases, the performance of models trained with "w/o const" improves. For example, a 6-layer model with the same number of parameters as MiniViT, which is a 12-layer model, shows a performance difference of only 1.4%, indicating competitive results. These comparisons validate that when using suitable training strategy, the recomposed diverse-scale ViTs have good initialization that by simply training, they can achieve comparable or better performance than Pre-Fin and model compression methods. Furthermore, by our strategy, an appropriate number of decomposed learngenes can be selected based on available computational resources to initialize the models without training from scratch, achieving flexible accuracy and efficiency trade-offs. 
conclusion: 
 In summary, this paper proposes a novel training method for efficiently generating ViTs of varied sizes to meet diverse computational needs. By employing linear decomposition, a ViT can be decomposed into basic learngenes, termed as "learngenes", which encapsulate critical knowledge from the original model. These learngenes can then be selectively and linearly recomposed to form ViTs of various layers and sizes. This flexible recomposition provides an economical and adaptive solution for creating a series of small or medium ViTs tailored to different deployment environments.Our experiments confirm the effectiveness of this learngene-based decomposition-recomposition method and show that these recomposed ViTs maintain performance comparable to traditional model compression techniques while offering greater flexibility and efficiency. A Appendix / supplemental materialIn the following section, we present the process of dimensionality reduction applied to the parameters of each layer in the Vision Transformer (ViT) models in order to observe the interrelationships between the parameters across different layers. We then supplement the experimental content presented in the main text, including the experimental results on decomposition and recomposition, as well as implementation details. Finally, this section concludes with a discussion of the limitations and societal implications of our method. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discusses the societal impact of our work in the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not involve the release of models or data, hence there are no associated risks requiring safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The experimental section and appendix provide detailed references to the datasets used, citing the original sources. In addition, all experimental settings are properly referenced, ensuring full compliance with the licensing requirements of the assets used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: No release of new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No crowdsourcing or human subjects research.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: No crowdsourcing or human subjects research.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our analysis starts with the aggregation of parameter matrices from the Multihead Self-Attention (MSA), Multilayer Perceptron (MLP), and Layer Normalization (LN) modules of an extensively trained Transformer model. Specifically, within the MSA module and given an input x, we first perform the computation of (xW Q )(xW K ) T to determine the attentional outputs for each head. This then facilitates the derivation of W A for the MSA as follows:EQUATION)where W K and W Q , belonging to R D×D , act as transformation parameter matrices for keys and queries, respectively, where D denotes the dimensionality of the intermediate embeddings. Subsequent operations involve the bias vectors for keys and queries, proceeding according to Equation 10. Together with other parameter matrices in the MSA module, we reshape these into vectors and subject them to L2 normalization.In the MLP module, we flatten and normalize weight matrices and bias vectors from two linear transformations. A similar approach is employed in the LN module, where weight matrices and bias vectors associated with transformation parameters are normalized. Normalized vectors, specific to each parameter type, are concatenated in alignment with their corresponding layer, culminating in a synthesized matrix S ∈ R N ×D L , where D L symbolizes the normalized vector dimension for each parameter type, and N signifies the depth of the Transformer.Principal Component Analysis (PCA) [24] is then utilized to reduce the dimension of each vector in S to a one-dimensional scalar, primarily for analytical convenience. These reduced vectors are then combined to form the matrix H ∈ R N ×K , with K indicative of the number of parameter types. Each row of H is interpreted as the representative vector of the corresponding layer. Another iteration of PCA is performed to compress the rows of H into a one-dimensional domain, resulting in a vector U ∈ R N . This final vector is used to illustrate the relationship between the sequential positioning of the layers in the transformer and their respective parameter values, thus illustrating the nuanced interplay of parameters across the architectural layers.We also show the graphs for MoCov3 [8] , DINOv2 [36] , MAE-B [19] and BEITv2-B [3] in To analyze their characteristics, we employ Gradient-weighted Class Activation Mapping (Grad-CAM) [44] , which is pivotal in the classification process, to visualize the feature maps in two scenarios: one where each learngene is used independently to initialize a ViT, and the other where learngenes are added incrementally to initialize the ViT model.From Figure 8 (a), we find that each learngene focuses on uniquely specific regions, indicating that the functionalities of the ViT model are decomposed into distinct learngenes. Specifically, the first learngene focuses on basic shapes and contours. The second learngene detects textures and colors, such as highlighting leaf textures and bird feather details. The third learngene identifies spatial features such as insect wings, while the fourth learngene detects finer details such as bird eyes or insect antennae. The fifth learngene emphasizes image contrast, and the sixth distinguishes between foreground and background, focusing on depth perception. As shown in Figure 8 (b), early learngenes capture basic shapes and edges, while later learngenes refine the focus on textures, colors, and specific details such as leaf veins or bird feathers. As learngenes are added, the attention of model shifts to finer features such as bird beaks, insect wings, and antennae, indicating a progression from general to detailed feature recognition that is likely to lead to more accurate image classification.Evaluating Learngene Number and Layer Depth Effects on Model Performance. The experimental results in Figure 9 on these two datasets further verify the findings in the text. The results in the appendix confirm the observation that deeper network configurations or a larger number of decomposed learngenes generally improve ViT performance, whether trained by "w/o const" or "w/ const". The results also show stable performance beyond certain learngene and layer thresholds, and provide strategies for optimizing ViT models under resource constraints, such as using fewer layers and learngenes to achieve near-optimal performance. We also compare the "w/const" and "w/o const" training methods and find that performance differences decrease as model complexity increases. In particular, models with higher layers show minimal performance variance between the two methods, but differ in memory consumption, with "w/o const" requiring more. We recommend "w/o const" for efficiency in resource-constrained scenarios, as shown by the 12-layer ViT model initialized with fewer learngenes, which achieves comparable performance with significantly less memory consumption.Comparative Results. The experimental results presented in Figure 10 show that a 12-layer Vision Transformer (ViT) initialized with different numbers of learngenes and trained using the "with constraint" method, represented by the orange lines in the graphs, outperforms traditional approaches such as pre-fine tuning and model compression under the same parameter constraints. In addition, as the number of layers in the model increases to a certain threshold, models initialized with 12 learngenes and trained using the "without constraint" method also outperform these baselines. Thus, our method strikes a balance between flexibility and model efficiency.Numerical Results for Comparative Results. Table 2 provide the numerical performance on 9 downstream tasks to compare our methods with pre-training fine-tuning and model compression methods. The same results are also shown in Figures 6 and 10 of the main paper. We implement the model using PyTorch [39] and the Timm library [53] . The decomposition process is trained over 500 epochs on four NVIDIA RTX 3090 GPUs, and the recomposed models are trained over 100 epochs on two NVIDIA RTX 3090 GPUs for each downstream task.Datasets We adapt the recomposed models on 9 diverse downstream datasets, covering a wide range of classification challenges. These datasets include 3 object classification tasks: CIFAR-10 [26], CIFAR-100 [26] , and Tiny-ImageNet [1] ; 5 fine-grained classification tasks: iNaturalist-2019 [66] , Food-101 [4] , Oxford Flowers-102 [35] , Stanford Cars [25] , and Oxford-IIIT Pets [38] ; 1 texture classification task: DTD [10] . The details of these datasets are in Table 3 .Data Process Following previous works [62] , we train and evaluate the decomposition and recomposition processes on all datasets at a resolution of 224 × 224. The data augmentation techniques employed include RandAugment [11] , Cutmix [60] , Mixup [61] , and random erasing.Decomposition We train each learngene using the following setting:• Batch Size: We employ distributed training across 4 GPUs, with each GPU handling 128 data instances, resulting in an overall batch size of 512. • Optimizer: The training of each learngene is optimized using AdamW, with an initial learning rate of 0.0008 and a weight decay of 0.05. • Learning Rate Schedule: We apply a cosine learning rate decay, with a warm-up period of 5 epochs.Recomposition We train the recomposed models on downstream tasks using the following setting:• Batch Size: We employ distributed training across 2 GPUs, with each GPU handling 128 data instances, resulting in an overall batch size of 256. • Optimizer: The training of each learngene is optimized using AdamW and a weight decay of 0.05. • Learning Rate Schedule: We apply a cosine learning rate decay, with a warm-up period of 5 epochs.NeurIPS Paper ChecklistJustification: The research strictly conforms to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).n=1 a(k, n) × A n , where a(k, n) is still pre-defined by satisfying a specific role, e.g., the first-kind Chebyshev polynomial coefficients. In contrast to model compression methods where each training only targets a specific model size, which is not flexible enough to meet the needs of real-world deployment scenarios. 


Decision: The authors propose an parameter-efficient learning technique. The final score is 5767. 

Overall, the author's rebuttal has won unanimous approval from the reviewers, and this article has made its own contribution. After reviewing the reviewers' comments, the author's responses, and the AC's discussion, we finally decided to accept this article, but based on the following two points:
1. The author needs to fulfill his promise and discuss the two articles raised by the reviewer caYK in the discussion section.

2. The author needs to change the tone of the conclusion and claim in the paper, because the current technology may not be able to surpass the following models in all sizes: Inception transformer, multiscale vision transformer, cswin

Reviews: review: paper_summary: paper_summary: Inspired by polynomial decomposition in calculus, the authors propose linearly decomposing the ViT model into a set of components during element-wise training. These components can be recomposed into differently scaled, pre-initialized models to meet different computational resource constraints. This decomposition-recomposition strategy offers an economical and flexible approach for generating diverse scales of ViT models for various deployment scenarios. Unlike model compression or training from scratch, which require repeated training on large datasets for different-scale models, this strategy reduces computational costs by training on large datasets only once. Extensive experiments demonstrate the method's effectiveness, showing that decomposed ViTs can be recomposed to achieve comparable or superior performance to traditional model compression and pre-training methods.
strength_weakness: strength_weakness: There are many variants of pure vision transformers, such as Swin and MobileViT, which significantly outperform traditional vision transformers. It would be valuable to retrain these models for scenarios requiring different model sizes and datasets.
questions: questions: 1.Can you demonstrate that your method is applicable to other well-known transformer variants such as Swin and MobileViT?

2.Can you compare the performance of your proposed models, decomposed and recomposed from DeiT-B, against other state-of-the-art models with the same parameter settings?
limitations: limitations: I shared my concerns in the weakness section.
paper_summary: paper_summary: This paper proposes a novel method to efficiently adapt Vision Transformer (ViT) models to devices with diverse computational resources by linearly decomposing a ViT into basic components that can be flexibly recomposed into ViTs of various scales. The key ideas are:

- Inspired by polynomial decomposition, a ViT can be decomposed into basic components where each layer is a linear combination of the components using polynomial coefficients.
- The components are trained incrementally, where in each iteration only the newly added components are trained while previously trained ones are fixed. This element-wise training reduces compute cost.
- The trained components can then be flexibly recomposed into ViTs of different depths using an appropriate number of components to achieve accuracy-efficiency trade-offs for different deployment scenarios.
- Compared to repeatedly training diverse-scale models from scratch, this decomposition-recomposition strategy is more economical.

Experiments demonstrate the effectiveness of the method - the decomposed components can recompose ViTs of various scales that achieve comparable or better performance than traditional model compression methods.
strength_weakness: strength_weakness: I have three major concerns, I will consider adjusting the score based on the quality of the rebuttal provided.

1. the recomposition process can be explained more clearly, especially the two types of coefficient initialization and the two training strategies ("with constraint" and "without constraint"). A figure illustrating the different recomposed architectures will help understanding. Regarding the statement in line 198 that the 'without constraint' training method still adheres to the constraints of Equation 9, could you clarify if this means the polynomial coefficients remain unchanged and maintain their predefined values during the training process?

2. In addition to the linear relationship shown in Fig 2, it would be helpful to see several other models to demonstrate the generalizability of this pattern.

3. The paper claims that the first-kind Chebyshev polynomial coefficients work best for recomposition, but limited rationale is provided for this choice.  Some theoretical analysis on why Chebyshev polynomials are particularly suitable would strengthen the motivation.
questions: questions: Please see weakness section.
limitations: limitations: Please see weakness section.
paper_summary: paper_summary: This paper proposes a novel approach to generate diverse-scale Vision Transformer (ViT) models with varying computational complexity, aiming to address the deployment challenges posed by the fixed architectures of standard ViT models. The key idea involves linearly decomposing a pre-trained ViT model into basic components and then recomposing these components to initialize ViT models of different scales, effectively enabling the economical generation of models tailored to diverse computational resource constraints.
strength_weakness: strength_weakness: The paper could explain the component sharing approach in more detail. Is it similar to MiniViT, where components are shared across layers? Or are the component parameters only shared within each layer? Also, please clarify if the forward process is the same as in traditional ViTs.

For the recomposition, could you provide more information on the initialization methods used for the 'with constraint' and 'without constraint' training approaches? Are the initialization methods the same, with the only difference being the parameters being optimized (layer parameters W vs. component parameters θ)?
questions: questions: Please see the Weaknesses.
limitations: limitations: Please refer to the Weaknesses.
paper_summary: paper_summary: This paper presents a novel approach to adapting Vision Transformer (ViT) models for diverse computational environments. The key idea is to linearly decompose a large ViT model into a set of basic components during training, which can then be flexibly recomposed to create smaller, pre-initialized models suitable for various deployment scenarios.
strength_weakness: strength_weakness: There are several concerns regarding this work:

1. The training process for the decomposed modules requires a more detailed description.

2. The implementation details of the reconstruction process are insufficiently explained.

3. The empirical design does not fully support the proposed method. Many tunable components are not thoroughly discussed, such as the training and combination strategies for each layer's sub-modules, and the achieved latency savings at different scales.

4. The literature review is superficial. Parameter-efficient paradigms should be comprehensively discussed to clearly differentiate the proposed solution from existing approaches.

5. The work appears to be at an early stage of development. While the motivation and design show promise, the writing and formatting require significant improvement. The authors should enhance the overall writing quality and correct numerous typographical errors throughout the manuscript (e.g., in the caption of Figure 2, "layer poaition" is misspelled, and there should be a space before parentheses).
questions: questions: 1. How to train each of the sub-components so the cumulative performance can be on par with the original model? Due to the parameter gap, given the same amount of data, smaller modules may not perform equivalently to the bigger model. With that said, a model of Wn may perform better than the cumulative power of {A1...An}. Did the authors make any effort to address this challenge?

2. I have trouble understanding this statement: "Interestingly, as shown in Figure 2 of [39], the parameters of most layers in the well-trained ViT of [10] have an approximately non-decreasing trend after PCA dimension reduction." What do the x and y axes represent? What is the figure telling us?

3. In the empirical design, where can I see your arrangement of sub-module combinations {A1...An}? From lines 222 to 225, I only see layer-level combinations. Also, the empirical results should include the latency improvement for each scale as it is an important motivation of this paper.

3. I have trouble understanding this statement: "For example, during the third training phase, only the two newly added components are trained, while the two previously trained components are frozen. In each phase, the model is trained for 100 epochs with an initial 5 warmup epochs." Shouldn't this be one added component, not two?
limitations: limitations: Please refer to weakness and questions.

Comments: comment: Thank you very much for your thoughtful feedback. We greatly appreciate the time and effort you have put into reviewing our work. Your insights are invaluable, and we will take your suggestions into account as we plan to conduct further experiments to provide a more comprehensive evaluation.
The authors address parts of my questions. With the current experiments, it is hard to agree that the proposed method outperforms recent transformer models in various sizes such as Inception transformer, multiscale vision transformer, cswin. So I will keep the current rating.
Thank you very much for your kind words and for taking the time to review our manuscript. We greatly appreciate your thoughtful feedback and are pleased that our responses have addressed your concerns. Your recommendation and positive comments are highly encouraging, and we are delighted that you find our work to be a valuable contribution to the field.
Thank you for your well-crafted rebuttal, which has effectively addressed the majority of the concerns I raised in my initial review. After carefully considering the comments from other reviewers and your responses, I stand by my original recommendation to accept this manuscript. The strengths of the paper remain clear, and I believe it makes a valuable contribution to the field. Congratulations on this solid piece of work.
Thank you for your thoughtful feedback and support throughout the review process. Your insights are invaluable in helping us improve our work. We appreciate your time and effort, and we're grateful for the final rating.
Glad to hear our conversation has been helpful. I've completed the final rating—good luck!
Thank you for your insightful comments and for pointing out the relevance of the two seminal works. We agree that these works are indeed crucial in the context of parameter-efficient learning. E^2VPT introduces a method for efficient fine-tuning of large-scale transformer models through learnable prompts and pruning techniques, reducing the number of tunable parameters while maintaining performance. Facing the Elephant in the Room analyzes when and why Visual Prompt Tuning (VPT) is effective, offering insights into its applicability depending on task objectives and data distributions. Both are closely aligned with our approach to parameter-efficient learning, and we will include discussions of these works in our revision to provide a more comprehensive overview of existing approaches in the field of parameter-efficient learning.
Thank you for the detailed response. In general, this work can be categorized as parameter-efficient learning. With that said, the authors may also benefit from discussing the two seminal works [1][2].

[1] E^ 2VPT: An Effective and Efficient Approach for Visual Prompt Tuning;
[2] Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?