Initializing Variable-sized Vision Transformers from Learngene with Learnable Transformation
Authors: Shiyu Xia
Yuankun Zu
Xu Yang
Xin Geng
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter initialization
Learngene
Primary area: deep_learning_architectures

Abstract: In practical scenarios, it is necessary to build variable-sized models to accommodate diverse resource constraints, where weight initialization serves as a crucial step preceding training. The recently introduced Learngene framework firstly learns one compact module, termed learngene, from a large well-trained model, and then transforms learngene to initialize variable-sized models. However, the existing Learngene methods provide limited guidance on transforming learngene, where transformation mechanisms are manually designed and generally lack a learnable component. Moreover, these methods only consider transforming learngene along depth dimension, thus constraining the flexibility of learngene. Motivated by these concerns, we propose a novel and effective Learngene approach termed LeTs (Learnable Transformation), where we transform the learngene module along both width and depth dimension with a set of learnable matrices for flexible variablesized model initialization. Specifically, we construct an auxiliary model comprising the compact learngene module and learnable transformation matrices, enabling both components to be trained. To meet the varying size requirements of target models, we select specific parameters from well-trained transformation matrices to adaptively transform the learngene, guided by strategies such as continuous selection and magnitude-wise selection. Extensive experiments on ImageNet-1K demonstrate that Des-Nets initialized via LeTs outperform those with 100-epoch from scratch training after only 1 epoch tuning. When transferring to downstream image classification tasks, LeTs achieves better results while outperforming from scratch training after about 10 epochs within a 300-epoch training schedule.

Initializing Variable-sized Vision Transformers from Learngene with Learnable Transformation
Shiyu Xia, Yuankun Zu, Xu Yang, Xin Geng"
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China {shiyu_xia, zyk0418, xuyang-palm, xgeng>sseu edu.cn
Abstract
In practical scenarios, it necessary to build variable-sized models to accommodate diverse resource constraints, where weight initialization serves as a crucial step preceding training. The recently introduced Learngene framework firstly learns one compact module, termed learngene, from a large well-trained model, and then transforms learngene to initialize variable-sized models. However, the existing Learngene methods provide limited guidance on transforming learngene, where transformation mechanisms manually designed and generally lack learnable component. Moreover, these methods only consider transforming learngene along depth dimension, thus constraining the flexibility of learngene. Motivated by these concerns, we propose a novel and effective Learngene approach termed LeTs (Learnable Transformation), where transform the learngene module along both width and depth dimension with a set of learnable matrices for flexible variable- sized model initialization. Specifically, the compact learngene module and learnable transformation matrices, enabling both components to be trained. To meet the varying size requirements of target models, we select specific parameters from well-trained transformation matrices adaptively transform the learngene, guided by strategies such as continuous selection and magnitudewwise selection. Extensive experiments on ImageNet- IK demonstrate that Des-Nets initialized via LeT's outperform those with 100-epoch from scratch training after only epoch tuning. When transferring to downstream image classification tasks, LeT's achieves better results while outperforming from scratch training after about 10 epochs within 300-epoch training schedule.
construct auxiliary model comprising
Introduction
Vision Transformer (ViT) models have gained widespread attention due to their impressive perfor- mance on diverse vision tasks 1 4 6]. In practice, models of various sizes are often deployed and trained under diverse resource constraints, ranging from edge devices with limited computational resources computing clusters with sufficient resources, which may exhibit considerable diversity. Obviously, we could train each target model from scratch for specific tasks across different resource settings. However, such method underestimates the importance of weight initialization, which could significantly affect the training process and the final model quality 17 8 9 10) [1] [2] [3] 14). Moreover, the training and storage costs of such method grow linearly with the number of poten- tial scenarios. Consequently, fundamental research question arises: how to efficiently initialize variable-sized models while considering both the model performance and resource constraints.
"Co-corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

o
Ioomable width transformation
OOO
o learngene
C
learngene
Ancestry model condense
unkranatble transformation
icomable transformation
learngene
TB rewlyadted lnyer
newl added nouron
Initiaiize
DesoO newly-atiled layer
Descendant model
Descendant model
Descendant model
[a] Learngene
(b) Previous Mechanisms
LeTs
Figure 1: (a) Learngene paradigm. (b) Manually-crafied and depth-only transformation. (c) Learnable transformation along both width and depth dimension.
Nowadays, pre-training on large-scale datasets provides an excellent initialization for fine-tuning models across wide range of downstream tasks 15 16 17 18
19
20 211]. Nevertheless, such
method generally requires transferring the entire pretrained model parameters repeatedly without considering the available resources of different downstream tasks. Take the pretrained model family SimMIM [22] as an example, even the smallest model 86M ViT-B [23] may be deemed excessively large for some resource-conee condtrinee environments. One direct solution involves firstly pre- pre-training the target model on large-scale datasets under specific resource constraints before training it on target task. However, this process is not only time-consuming and computationally expensive, but also necessitates the access to the datasets used for pre-training and lacks the flexibility to initialize variable-sized models.
Recently, a novel learning paradigm termed as Learngene 24 25 has been proposed, which firstly learns one compact module, termed learngene, from a large well-trained network termed as ancestry model (Ans-Net). Then learngene transformed to initialize variable-sized descendant models (ees-Net,, after which they fine-tuned diverse downstream tasks, as shown in Fig.) Grad- LG 24 selects a few high-level layers as learngene, after which they are stacked with randomly- initialized layers construct Des-Nets. TLEG (26 linearly expands two integral learngene layers to initialize variable-depth Nes-Nets. LearngenePoo (77 distills multiple small models whose layers compose learngene instances and then stitches them construct Des-Nets. However, there exist several limitations in previous studies. Firstly, the learngene learning process provides well-trained learngene parameters but lacks the structural knowledge required transform the learngene for initializing Des-Nets. Secondly, existing transformation mechanisms are manually designed (e-g., stacking hundomly-niiializen layers over learngene layers) but lack a learnable component, as shown in (b).. Thirdly, existing studies typically focus on deepening well-trained learngene while overlooking the exploration of transforming learngene along another crucial dimension, i.e., the width dimension (88) let alone considering both width and depth dimension.
Motivated by the above limitations, we propose Learnable Transformation (LeTss, novel and effective Learngene approach for efficient model initialization, as shown in Fig cc). LeTs enables the simultaneous training of a compact learngene module and of learnable transformation parameters, with the latter encoding structural knowledge for transforming the former. Specifically, introduce and train auxiliary model (Aux-Net) constructed from the compact learngene through a series of learnable transformation matrices T, comprising a set of width transformation matrices F and one depth transformation matrix G. For the width transformation matrices F. we utilize two series of matrices Fin and Fout which perform in-dimension and out-dimension transformation on learngene matrices respectively. In the case of depth transformation matrix G. we divide the aatth-transformed learngene layers into multiple groups and utilize the entry of G as the coefficient to combine the layers within each group, thereby constructing new layers. During the training of Aux-Net, F and G' learn to capture structural knowledge about adding neurons and layers respectively. To meet the varying size requirements of target models, we first select specific parameters from the well-trained transformation matrices T to construct target ones de guided by strategy such as continuous selection and magnitude- selection. Then we employ T to transform learngene for initializing target Des-Nets, which are lastly fine-tuned different downstream tasks.

---

With comprehensive experiments, we show the superionity of LeTs: (1) Compared to 100-epoch from scratch training (Scratch) on ImageNet-1K (29) Des-Nets initialized via Le'Ts performs better after only 1 epoch tuning. (2) Evaluation performance on ImageNet-1K without any tuning after initialization implies that Le'Iss significantly enhances initialization quality, e.g., +3.5% with Des- H12-L.12 (86.6M) compared to TLEG [26]. (3) When transferring to downstream image classification tasks, LeTss presents better performance and training efficiency. For example, LeTs outperforms the pre-training and fine-tuning method (Pre-Fin) by 1.5% on CIFAR- 100 with Des-H12-L12. Furthermore, within a 300-epoch training schedule on Food-101 (0), LeTs outperforms the final performance of Scratch after about 10 epochs. For semantic segmentation tasks, LeT's outperforms Pre-Fin by 3.4% on ADE20K [31 with Des-H6-L12. (4) Compared to Pre-Fin, LeT's performs better while reducing around 2 initialization parameters when initializing variable-sized models.
Our main contributions are summarized as follows: (1) We introduce a novel and effective Learngene approach, termed Le'Ts, for efficient ViT-based model initialization, which is the first to utilize learnable matrices to adaptively transform the compact learngene. (2) We propose to transform learngene along both depth and width dimension for initialization, to our knowledge, has not been explored in the Learngene literature. (3) Comprehensive experiments under various initialization settings demonstrate the efficiency of LeTs.
2 Related Work
Parameter Initialization methods have been extensively developed, such as default initialization from Timm library [32]. Xavier initialization [7] and Kaiming initialization [9] A plenty of studies demonstrate that parameter initialization significantly affects the training process and the final model quality 7 8 TO 12 [3] 14). Appropriate initialization facilitates model convergence 33 while improper initialization may hinder the optimization process [10] 11. Nowadays, the pre-training and fine-tuning method involves transferring model parameters pretrained on large-scale datasets for fine-tuning on specific downstream tasks, during which the model architecture is maintained [34]
35
36 37). However, such method requires transferring the entire pretrained model parameters repeatedly, without considering the varying resource availability of different downstream tasks. Moreover, when a pretrained model of the target size is unavailable, we may firstly pre-train the target model on large-scale datasets. This process is not only conecconuumin and computationally expensive, but also requires access to the datasets used for pre-training and lacks the flexibility for initializing variable-sized models. Recently, [38 39 have focused on transferring large pretrained model parameters to initialize small ones. Besides, Matformer [40] allows for training one universal model which can be used to extract many smaller sub-models. In contrast, we propose to transform one compact learngene with learnable transformation matrices for initialization.
18
Learngene is a two-stage framework 24
25
26
27
45 46
47 which firstly
learns one compact module, termed learngene, from large well-trained network called ancestry model (Ans-Net), and then transforms learngene to initialize variable-sized descendant models (Des- Net), after which they are fine-tuned normally, as shown in Fig F (a.. Grad-LG (24) selects a few high-level layers as learngene based on the gradient information of Ans-Net, after which they are stacked with andomly-initialinn leamgene layers to initialize variable-depth Des-Nets. SWS 41 extracts learngene in multi-stan stage weight sharing fashion and duplicates learngene in its stage during initialization. WAVE (45) trains multiple weight templates as learngenes, enabling efficient initialization for variable-sized models via Kronecker Product. Rather than manually designing learngene transformation strategies, we seek to use learnable matrices, which contain structural knowledge, to transform the compact learngene for initializing variable-sized models. Furthermore, LeTs enables learngene to be transformed along both depth and width dimension, significantly enhancing the transformation flexibility.
layers build Des-Nets. TLEG 26 linearly expands two integral
Network Expansion, a prevalent training acceleration framework pioneered by 31
involves employs Net2Net
48
incrementally increasing the size of neural networks 44 50
52
53)
4%
function preseevinn expansions to increase the width by copying neurons and the depth by introducing identity layers. Bert2Bert [54] extends this concept by proposing funetion-preesevvng preserving width expansion
specifically for Transformers. Expansion (53) introduces a width expansion strategy for convolutional neural networks using orthogonal filters, as well as depth expansion strategy for Transformers through the corresponding exponential moving average model. LiGO [2 learns to linearly map
the parameters of a smaller model to initialize a larger one. While LeTs is inspired by these
3

---

Stage 1: Learning Process of Aux-Net
1
construct the Aux-itt intallee
Stage 2: Initialization from Learngene
PC
learnable matrices
O oo learngene
finettgge
Server
Ancestry Model
Auxiliary Model
Descendant Model
Figure In stage layers using a series of learnable transformation matrices. During training, F and G learn to capture structural knowledge about how add new neurons and layers into the compact learngene respectively. In stage 2, given the varying sizes of target Des-Nets, we select specific parameters from well-trained transformation matrices to transform learngene for initialization, which fine-tuned lastly under different downstream scenarios.
construct and train Aux-Net which is transformed from compact learngene
differs in least two key aspects. In terms objective, we focus on flexibly initializing variable-sized models from single compact learngene. Methodologically, such as compared to LiGO [52]. first transform the learngene layers along width dimension, after which we divide the tudth-transformed learngene layers into multiple groups and linearly combine the layers within each group to construct new layers. Additionally, one crucial aspect LeTs that selecting specific parameters from well-train celltrraind transformation matrices for subsequent initialization.
studies 48 52
3 Proposed Approach
2ig22 depicts the overall framework LeTs. In the first stage, we construct an auxiliary model (Aux-Net) comprising compact learngene and series of learnable transformation matrices. Then we train it by distilling knowledge from well-trained ancestry model (Ans-Net). In the second stage, given the varying size requirements of target models (e.g., layer numbers), we select specific parameters from well-trained transformation matrices guided by strategy such continuous selection and magnitude-win selection. Then we use these selected parameters to transform learngene for initializing Des-Nets, which are fine-tuned lastly.
3.1 Learnable Transformation for Learngene
We denote the parameters of the learngene module with L layers as O = Wi..., Wz] where Wi E RRd. xdout, din and daut denote the input and output dimension respectively. In the first stage, design auxiliary model (Aux-Net) whose parameters transformed from the compact learngene parameters with a series learnable transformation matrices T. We configure set of width transformation matrices F and one depth transformation matrix G for composing T. Specifically, we enlarge the input and output dimension of learngene matrices with transformation matrices F divide these tratt--transfrmmed learngene layers into
containing Fin and Fout Afterwards, multiple groups and utilize depth transformation matrix G to combine the layers within each group construct new layers. In the following, we detail the width transformation matrices F. depth transformation matrix G, the construction and training of Aux-Net.
Width transformation matrices. For each learngene layer Wi, introduce FM and Fout to perform in-dimension and out-dimension transformation respectively. In particular, we perform in-dimension transformation W. by multiplying Fin. after which insert the transformed part into the original learngene via
w = Concal (W, Fi" Wi).
where W/ represents the in-dimension transformed learngene and Concal represents the concatena- tion operation. Similarly, perform out-dimension transformation W, by multiplying Fout,

---

MiLe
F1
Av
PISert into
W
Mop
Gas Wi25
Gur
W:
Miu
step1: in-dimension transformation
W
Mil
insert into
Mis
Pia
1922
Maa Mix
15.3
+-6g M13
Say Wis
Pia
Mix 193
Max Maz
Win
W
step2: oit-eimnnioonn traroformation
(a) width transformation
(b) depth transformation
Figure 3: (a) Firstly, perform in in-dimension transformation Wi by multiplying F7". after which we insert the transformed part into the original learngene. Afterwards, we perform out-dimension transformation on W/ similarly. (b) After width transformation, we divide these width tramsformed learngene layers into groups. Take the jo-th group with two width-transformed learngene layers (Wet Jo.1 and W as an example, we construct W via G1,1 W 2o,2)
fo.1 + G1,2 War. omit the original subscripts and superscripts for the entry of all matrices.
Jo.2- For simplicity,
after which we insert the transformed part into W, via
Winh Concat W/ W, Foo(T),
(2)
where W represents the andth-transformet learngene. Width transformation process is illustrated
in Fig 3(a.. We could also first perform out out-dimension transformation followed by in-dimension transformation. Besides, we explore performing in-dimension and out-dimension transformation on Wi by directly multiplying F and Firat via W.t F!n Wi Wi referred to as LeTs(DE). Empirically, we first perform in-dimension and then out-dimension transformation in line with [52]. To keep the parameter efficiency, we share transformation matrices for different model components.
Depth transformation matrix. After the width transformation, we divide these hidth-transformed learngene layers into M groups and set the number width-transformed learngene layers the j-th group as T.y where 1,.., M. We denote the parameter matrices of the k-th indth-transformed learngene layer in the j-th group as W the layers within the ju-th group to construct the i-th layer of the target model via
j.k where L,..., Ljj Then we utilize matrix G combine
Li
W dt
G.&WWi Ao
A=
where Wh represents the i-th During the new layer constructions, we repeatedly select some learngene groups, as discussed in Sec 4.3 The process of depth transformation is illustrated in Fig 3bb. Moreover, adopt parameter sharing strategy between rows of G to provide guidance for initializing descendant models.
target layer and G. represents the (i, k:)-th entry.
Construction and Training of Aux-Net. We construct the Aux-Net from the learngene module using width and depth transformation matrices. Then we train the Aux-Net via prediction based distillation [55] for simplicity to distill knowledge from the Ans-Net. This involves minimizing the cross-entropy loss between the probability distributions of output predictions from both the Ans-Net and the Aux-Net similar to previous studies [26] 27]. Specifically, we introduce
distillation loss Ldistan = (CE(((r,T)) ((r,// /T)),, where т, and r. represent the logits of the Aux-Net and those of the pretrained Ans-Net respectively, CE((,)) represents soft cross-entropy loss, represents the temperature for distillation and e represents the softmax function. Additionally, we can seamlessly integrate advanced distillation techniques (56 57] into training. Besides, we also introduce one classification loss Lets = CF(o(r,)) a.r,),,,. y), where y represents the ground-truth label. Overall, our training loss is defined
Call = (1 Xot + ACdistill.
where À represents the trade-off coefficient. Noteworthy, we train the Aux-Net to obtain learngene parameters and set transformation matrices. During training, the width transformation matrices and depth transformation matrix capture structural knowledge about how to add new neurons and layers respectively, preparing for the subsequent initialization. Next, we elaborate how to use these well-trained transformation matrices to adapt learngene for initializing variable-sizer models.
5

---

3.2 Initialization from Learngene with Well-trained Transformation Matrices
Different from previous manual and depth-only transformation, LeTss enables learngene to deep- ened and widened with well well-trained transformation matrices. To meet the diverse size requirements of Des-Nets (e.g., layer numbers), we select specific parameters from well-trained transformation matrices to construct target ones. Specifically, we design several parameter selection strategies including continuous selection, magnitude-wi
selection and sequential selection. For width trans- formation matrices across different model components, we maintain the consistency of selection position to preserve the integrity of the learned neuron connections. After initialization, we fine-tune the descendant models on different downstream tasks without distillation. Next, we firstly introduce the selection strategy for width transformation matrices.
Continuous selection. For the initialization along width, we propose selecting continuous rows from F and Foot form the target transformation matrices. Notably, consistency preserved throughout the row selection process for different model components to maintain connectivity between neurons. Empirically, we default to selecting the first n rows from F and Fout.
Magnitude-wist selection. Parameter magnitude serves as an effective metric for assessing im- portance in model pruning, where the significance of one weight is determined by its magni- tude 55 59 60]. In our case, we adapt this metric for parameter selection from width transformation matrices. Take Fin as an example, we propose selecting n rows whose -i-norm or L2-norm ranks top-n, abbreviated as top-n LI/Lg-norm selection. Intuitively, this selection strategy enables us to prioritize parameters that contribute most significantly to the model performance, ensuring that the most impactful connections are preserved. By concentrating on the most critical parameters, LeTss enhances the initialization quality of different Des-Nets. Empirically, we demonstrate its effective- ness by comparing it with bottom-n Ta/T---norm selection which select rows whose I -norm or T2-norm ranks bottom-n, as discussed in Sec 4.3
For the depth transformation matrix, we introduce the sequential selection strategy.
Sequential selection. We propose to sequentially select learngene groups in predefined order, where different orders emphasize distinct groups. Each learngene group corresponds to several coefficient groups (rows of G). While selecting an approximately equal number of coefficient groups for each learngene group, we allocate more coefficient groups for shallower learngene group.
Experiments
4.1
Experimental Settings
We perform our main experiments on ImageNet-IK KK (2)]. several downstream image classification
datasets including CIFAR- CIFAR-100 611. Food-101
30) and Cars-196 (62). and several
semantic segmentation datasets including ADE20K 31 Pascal Context 63 and Cityscapes [64).
We report Top-1 classification accuracy (Top- (00))((%) for classification tasks, and Intersection over Union (mloU(%) averaged over all classes for segmentation tasks following (65]. Besides, we also report Params(M) and FLOPs(G) as the number of model parameters and indicators of theoretical complexity of model. In the first stage, we configure the learngene module with 6 heads (head dimension is 64) and 8 layers whose number of parameters is 15M, and transform it to construct Aux-Net with 12 heads and 16 layers based on DeiT [66]. Then we train the Aux-Net on ImageNet- 1K for 300 epochs to obtain learngene and transformation matrices. We choose Levit-384 [67] as the ancestry model. In the second stage, we set several variants of Des-Net where we change the layer and head numbers based on DeiT, egg., we name the Des-Net with 12 heads and 12 layers as Des-H12-L112. Please see more details in A.3
4.2
Main Results
Compared to training from scratch on ImageNet-1K, LeTs performs better while reducing large training costs. We compare LeT's with: (1) Scratch that training models of variable widths and depths from scratch for 100 epochs; (2) TLEG (66 that linearly expands learngene to initialize
models and finetunes them for 40 epochs; (3) SWS [41 that duplicates learngene in its pre-defined
stage to initialize models and finetunes them for 10 epochs. As shown in Fig 4 and Tab compared
to Scratch, LeTs performs better only after 1 epoch tuning, which reduces around 7 7x total training

---

Top-1 Aca7))
PED
642
Top-1 AccEN)
80 26.
750
Top-2 Acc1%)
80
765
60
60
50.
5
Scrochh
Scrothh
Scrach
Scrtch
TLEG
TLEO
20
20
TLES
Lets 10urst
Lets 10uns)
Lata (Dura)
Leth
05
20
40
60
80
100
05
20
40
60
80
160
20
40
60
80
100
20
40
6b
80
100
Epochs
Epocho hs
Ecochs
Epochs
(a) Des H8s-HH6-88 (15.0M)
(b) Des-H6-Lon (18.5M)
(c) Des-H6-L12 (22.1M)
(d) Des-H6-L14 (25.6M)
Top-1 Aod5o
Tap-1 Acc(3)
PEB
730
75.4
AcceN)
6 8
20
64.
745
756
Top-1 Acc(8)
60
80
613
25.
728
127
5
304
Scrochh
8
40
TLEG
Scratch
Scratch
20
Scratch
Le's (0urs)
0
Lels (Dura)
Lels (Duns)
0.41
Leh: (Dur))
20
d0
ab
100
055
4b
80
160
20
60
300
05
20
6b
100
Epochs
Epochs
Epochs
Epochs
(e) 1es-H6-L116 (29.2M)
(f) Des-H8-L8 (26.2M)
(g) Des-H8-L10 (32.5M)
(h) Des-H8-L12 (38.8M)
7: 1ea T8C
1003
611 124 14.
20
74.6 m2
20
766
12
Top-1 Acc(%)
61.
Top-1 Acc(%))
Access
60
Top-1 AcceN))
66.
5
16
Scratch
Scrazch
20
Scratch
20
Scratch
Lach: [Dunst
(Durs)
Lafk (Dun)
0:5%
Lafk (Oun)
05
20
40
Epochs
60
30
100
Epochs
05
20
40
Epochs
60
30
100
05
20
Epochs
50
80
100
(i) Des-I18-L14 (45.
(i) Des- (51.5M)
(k) Des-I110-L8 (40.7M)
(1) Des-I110-L10 (50.5M)
Top-1 Acco50
05.0
2255
231
780
Top-1 ACCeN)
10:3
76: 214
Acc(%)
26.
Top-1 Acci
nn
26 8
30
30
Scretch
Scrocch
Scretch
105
Scrrich
TLEG
LeTs 10urst
(Dum))
(Dun)
Lets 00uns
05
20
40
Epochs
60
30
100
Epocha
Epachs
100
05
20
40
Epochs
60
80
100
(m) Des-H100-112 (60.3M) (n) Des-H10-L114 (70.2M) (0) Hes-H10-L16 (80.0M) (p) Des-H12-1. (58.2M)
Top-1 ACC(8
Acc(S
Top-1 Ncc%%)
50
Scratch LLE
12.
Scratch
40
Scrwich
40
Scratch TIES
TLEG
TLBS
LE's (Durs)
Le5 (0urs)
LeTs (Dun)
Letk (0uun)
03
20
40
60
ab
100
0.
20
4b
<0
80
1do
100
100
130
Epochs
Epochs
Epochs
Epochs
(q) H12-H12-L10 (72.4M) (r) Des-H12-L12 (86.6M) (8) H12- (100.7M) (1) (114.1
Figure 4: Performance comparisons on ImageNet-IK Number in bracket represents Params(M).
costs for 24 models. Compared TLEG and SWS, LeTs also demonstrates superior performance after just epoch tuning in most cases, which highlights the effectiveness of introducing learnable transformation matrices. Notably, the efficiency of Le''s becomes increasingly obvious as the number of Des-Nets grows, since we only need to train the learngene and transformation matrices once for most Des-Nets. Morcover, LeTs could ficxibly initialize variable-sized models that are independent of the size of learngene and Aux-Net, as shown in Fig 5 Please see more detailed results in Tab
LeTs exhibits better performance and training efficiency when transferring to downstream datasets. We compare LeTss with: (1) Pre-Fin Fin that pre-training on ImageNet- 1 K and fine-tuning on downstream datasets; (2) Scratch; (3) Grad-LG (24): (4) TLEG (26]; (5) SWS 41.. As shown in Fi))))))-(h), LeTs consistently outperforms these baselines on downstream image classification datasets, demonstrating the effectiveness of leveraging well-trained transformation matrices for model initialization. Take Des-H6-L.12 as example, LeT's outperforms Pre-Fin by 2.3%, 1.9% and 3.5% on 100, Food-10 101 and Cars- 196, respectively. Notably from Fig 7 we observe that LeTss outperforms the final performance of Scratch after about 10 epochs within 300-epoch training
7

---

80
78
ImageNet-1] Top-1
00
74-
Leis
72-
Scratch
MatFormer
Params (MI Learngene 15.0
Params (M) or Aux-Net 37.7
50
100
150
200
250
300
Params (M)
Figure 5: LeTs could flexibly initialize variable-sized models that are independent of the size of learngene and Aux-Net. Compared with Scratch and MatFormer [40], LeTss demonstrates more initialization efficiency.
Top-2 Acci))
29.2
07.4
88.
Accet))
100
56.2
98.3
052
90.8
Tog-1 AcctS)
20
03.5
84.5
$5.0
329
10
Tal
TM
205
Leb
Gre R
ureft
Sunt Greet
noo g00
(a) CIFAR-100
(b) CIFAR-10
(c) Food-101
(d) Cars-196
Tap-2 Ace50)
89.5
96.0
Top Acc(50)
2E5 32.5
88.3
89.3
14.8
Nai
Gord
Drd S0 gose
(c) CIFAR-100
(f) CIFAR-10
Food-101
(h) Cars-
221
41.4
50.9
78.5
75.2
76.4
72.4
72.
yoo
pm
tuidth)
(i) ADE20K
ii) Paseal Context
(k) Cityscapes
(I) Linear Probe
Figure 6: Performance of (a)-(d): Des-H6-L112 and (e)-(h): Des-H12-L12 on downstream image classification datasets. We report the results of LeTss under the linear probing (LP) protocol in (I). Besides, we evaluate LeT's on semantic segmentation tasks in (i)-(k), where we set two variants of LeTs as LeTs(0ep) and LeTs(Sep). LeTs(Oep) represents that initializing the backbone with learngene and LeTs(5ep) represents that further fine-tuning on ImageNet-I IK for 5 epochs after initialization.
30 which is about 30x faster. We also report the results of LeT's under the linear probing protocol in 61) Moreover, we report the results LeTs on downstream semantic segmentation tasks. Specifically, we follow the training and model setting provided in (65), where we adopt Des-H6 Hoss-H6-12 and Des-H12-LI2 as the backbone and mask transformer as the decoder. As shown in o LeTsp)ep) outperforms Pre-Fin by 3.4%, 1.7% and 1.6% ADE20K, Pascal Context and Cityscapes respectively with Des-H6-Li1 as the backbone. Please see more details in A.3
schedule on Food- 101
Results evaluation on ImageNet- 1K without any tuning after initialization implies that LeTs greatly enhances initialization quality. To validate the initialization quality, we compare LeTs
with (1) Scratch; (2) Grad-LG 24); (3) TLEG (26); (4) SWS 41: (5) IMwLM (39). In Tab LeTs outperforms all baselines by a large margin in most cases. For example, LeTs outperforms TLEG

---

Model
Params(M)
FLOPs(G)
Scratch
IMwLM
Grad-LG
TLEG
SWS
LeTs
Des-H12-L8
58.2
11.7
77.2
9.5
0.1
69.8
74.4
74,1
Des-H12-L9
65.3
13.1
78.0
16.8
0.1
73.8
76.5
77.6
Des-
72,4
14.6
78.2
25.1
0.1
75.7
78.0
78.7
Des-I112-LI1
79.5
16.0
79.0
36.1
0.1
76.4
78.9
Des-I1l2-L12
86,6
17.5
79.6
48.1
0.1
76.6
79.3
80.1
Des-HI2-L11
93.7
18.9
80.3
76.7
80.5
Des-H12-L114
100.7
20.4
80.4
68.3
0.1
76.5
80.1
80.8
Des-H12-L.15
107.8
21.8
80.3
0.1
76.0
80.5
81.1
Des-H12-L16
114.9
23.3
80.4
78.7
0.1
75.5
80.7
81.5
Table Direct evaluation performance on ImageNet-Iie without any tuning after initialization.
Params (M)
Pre-Fin
LeTs Top- (9))
Params (M)
Pre-Fin S-P(M) Top- (op-((()
LeTs
Model
Model
S- S-P(M) Top- Top-1(%) ((%)
Top-1
Des-H8-L8
25.8
25.7
87.5
88.5
Des-HI2-La
57.5
57.4
88.4
89.3
D00-
32.1
32.0
88.2
89.2
Des-HI2-LI0
71.7
71.6
88.0
89.7
Des-H8-L12
38.4
38.3
88.6
89.9
Des-H10-L16
79.5
79.4
88.2
89.6
Des-H8-L14
44.7
44.6
88.4
89.8
Des-H12-L12
85.9
85.8
88.5
90.0
Des-I18-L16
51.0
50.9
89.1
90.1
Des-I112-L14
100.1
100.0
88.2
90.0
Des-H10-L12
59.8
59.7
88.6
90.0
Des-H12-L16
114.2
114.1
88.2
90.1
Table Performance comparisons on CIFAR-100 variable-sized Des-Nets. Pre-Fin transfers the pretrained parameters (S-P(M)) to initialize, which totally requires about 758M for 12 Des-Nett LeTs only needs to store 37.7M parameters (15.00 ... learngene) to initialize, which significantly reduces the parameters stored for initialization by 20x (758M vs. 37.7M).
by 4.3%, 3.0% and 3.5% on Des-H12-L8. Des-H12-L110 and Des-H12-L11. Importantly, Le'Ts can also achieve comparable performance with well-trained models (Scratch). For instance, LeTss outperforms Scratch by 0.5% and 0.5% on Des-H12-L10 and Des-H12-L12. The above results imply that LeTs provides effective initialization variable-sized models.
Compared to Pre-Fin, LeTs significantly reduces the pretraining efforts and initialization parameters when initializing variable-sized models for downstream tasks. As shown in Tab LeTs demonstrates superior performance while reducing around 3 x pre-training costs and decreasing around 20x (758M vs. 37.7M) initialization parameters, as compared Pre-Fin. Furthermore, LeTs only needs to train learngene and transformation matrices once, whereas Pre-Fin requires individual pre-training for each Des-Net. Clearly, the efficiency gains of LeTs become more pronounced with an increasing number of Des-Nets for different downstream tasks.
4.3 Ablation and Analysis
Selection strategies. We evaluate the performance LeTs (Des-H8-L12 epochs tuning) using various selection strategies. As shown in Tab we observe that continuous selection achieves slight better performance than magnitude-wise selection. Moreover, selecting n rows whose L Ln-norm or T2-norm ranks top-n (top-n Tn / nnom)) is better than those whose L,-norm Z2-norm ranks bottom-n (bottom-n L1/ Ln-rm)) Besides, we observe that selecting different learngenc groups for initialization achieves similar performance, where selecting more coefficient groups for shallower leargenne groups performs better than else.
Width and depth transformation. We investigate the effectiveness of our proposed width and depth transformation by comparing LeTs with one state-of-thh the expansion method LiGO [52]. Specifically, we adopt the expansion strategy proposed in LiGO into our two-stage initialization pipeline and use our proposed selection strategy for initializing different Des-Nets, referred to as LeTs (LiGO). From Tab E we observe that LeT's outperforms LeT's (LiGO) in most cases, which indicates that our proposed transformation pipeline are more suitable for initializing variable-sized models. Moreover, explore transforming the learngene matrices directly but not insert the transformed

---

Params (M)
Selection Strategy
Top-1 (%)
Learngene Group Selection
Top-1
Initialization Component
Top-1 (%)
Model
(%)
continuous
80.1
I.I.I.12222222333333.44
80.1
w/ MSA
71.7
top-n L2-norm
79.3
1.1.222222233334
79.6
w/o MLP
54.3
Des-H8-L12
38.8
bottom-n L2-norm
78.8
111.1.1222233333344
79.9
w/o LN
79.3
top-n Li-norm
79.4
L.I.I.222.33334444
79.9
w/o PE
78.5
bottom-n Li -oom
78.7
L11222222.234344
w/o Pos
76.6
Table 3: Performance ImageNet-IK IK when using different selection strategies, selecting different learngene groups and initializing Des-Nets without certain components.
LeTss (LiGO)
LeTss (11.4M)
LeTss (LiGO)
LeT's (DE)
Model
LeTs
Model
LeTs
Model
LeTs
Des-116-L12
77.9
78.2
79.5
Des-118-L8
78.0
78.1
Des-I112-L10
57.0
80.1
Des-H8-L12
78.7
78.9
80.1
Des-H8-L10
78.1
79.3
Des-H12-L12
57.6
80.8
Des-H10-L112
79.1
79.4
80.4
Des-H8-L14
78.8
80.5
Des-H12-L14
59.3
81.1
Des-H12-L12
79.1
79.8
80.8
Des-H8-L.16
79.4
80.9
Des-H12-L116
61.0
81.6
LeTss
LeTss (w/o ws)
LeTss (w/o ws)
LeTss (w/o dis)
Model
LeTs
Model
LeTs
Model
LeTs
(200ep)
Des-H6-L12
78.9
78.3
79.5
Des-H8-L8
77.5
78.1
Des-H12-L10
78.0
80.1
Des-H8-LIE 79.3
79.1
80.1
Des-H8-L10
78.4
79.3
Des-H12-L12
78.4
80.8
Des-H10-L112
79.5
79.8
80.4
Des-H10-L10
79.1
79.9
Des-H12-L144
79.1
81.1
Des-H12-L12
79.8
80.2
80.8
Des-H12-LI0
79.6
80.1
Des-H12-L16
79.3
81.6
Table 4: Performance on ImageNet-1K when using depth and width expansion strategies proposed in
LiGO [52] (named as LeTs(LiGO)), training smaller learngene module (named as LeTs(1 (..M)) direct transforming learngene matrices to compose target ones (named as LeTs)DE)), not sharing weights between rows of G' (named as LeTs(w/o ws)), training Aux-Net for 200 epochs (named as LeTs(220cp)) and not adopting distillation in the first stage (LeTs(w/o dis)).
matrices into the learngene, referred to (LeTssDE)). In addition, also explore the role of the parameter sharing strategy adopted on the rows of G (LeTs w/o ws). From Tab we observe that LeTs(DE) and LeTs(w/o ws) still enhances the initialization efficiency but is inferior our complete version.
Initialization components. We investigate the performance of LeTs (Des-H8-L12, 5 epochs tuning) by excluding one of the following components within ViT models: Patch Embedding (PE), Multi- head Sclf-Attention (MSA), Multi-Layer Perception (MLP), Layer Normalization (LN) Position Embedding (Pos). From Tab we find that omitting MSA or MLP from initialization results in significant performance degradation and initializing all components is necessary.
Size and training setting of learngene and transformation matrices. We evaluate the perfor- mance of Des-Nets initialized from well-trained learngene modules under different training settings. Specifically, we train smaller learngene module with 11.4M parameters (LeTs(1 (1TT((1..MM)) 1.4M)), shorten the training epochs of Aux-Net to 200 (LeTTs (200ep)))) or train the Aux-Net without distilling from the Ans-Net (LeTs(w/o dis)). From Tab we observe that LeTs(1 1. 1.M),, LeTsp200ep) and LcTs(w/o dis) still enhances the initialization effectiveness but is inferior to complete version.
Conclusion
In this paper, we proposed well-motivated and highly effective Learngene approach termed LeTs where we transform the learngene module with a set of leamnale transformation matrices for variable- sized model initialization, enabling adaptation to diverse resource constraints. LeTs is the first to adopt learnable matrices to transform the compact learngene along both the depth and width dimension, which significantly enhances the flexibility of learngene for model initialization. We demonstrated the efficiency of LeTs under various initialization settings empirically.
10

---

Acknowledgements
This research is supported by the National Science Foundation of China (62125602, 62076063), Key Program of Jiangsu Science Foundation (BK20243012) the Fundamental Research Funds for the Central Universites((2222433335355 and the Big Data Computing Center Southeast University.
References
[1] Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. Advances in Neural Information Processing Systems, 35:14541-14554, 2022.
[2] Changlong Jiang, Yang Xiao, Cunlin Wu, Mingyang Zhang, Jinghong Zheng, Zhiguo Cao, and Joey Tianyi Zhou. a2j-transformmr: Anchor-to-joint transformer network for 3d interacting hand pose estimation from single rgb image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8846-8855, 2023.
[3] Shaoru Wang. Jin Gao, Zeming Li, Xiaoqin Zhang, and Weiming Hu. A closer look at self- supervised lightweight vision transformers. In International Conference on Machine Learning, pages 35624-35641, PMLR, 2023.
[4] Haoyu Xie, Changqi Wang, Jian Zhao, Yang Liu, Jun Dan, Chong Fu, and Baigui Sun. Prel: Probabilistic representation contrastive learning for semi-supervised semantic segmentation. International Journal of Computer Vision, pages 1-19, 2024.
[5] Yang Qin, Yingke Chen, Dezhong Peng, Xi Peng, Joey Tianyi Zhou, and Peng Hu. Noisy- correspondence learning for text-to-image person re-identification. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 27197-27206. 2024.
[6] Chuang Lin, Yi Jiang, Lizhen Qu, Zehuan Yuan, and Jianfei Cai. Generative region-language pretraining for open-ended object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13958 13988-1398,,00 2024.
[7] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249 -26.. JMLR Workshop and Conference Proceedings. 2010.
[8] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807-814, 2010.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference computer vision, pages 1026-1034 1034, 2015.
[10] Dmytro Mishkin and Jiri Matas, All you need is good init. arXiv preprint arXiv:1511.06422 2015.
[11] Devansh Arpit, Víctor Campos, and Yoshua Bengio. How to initialize your network? robust initialization for weightnorm & resnets. Advances in Neural Information Processing Systems, 32, 2019.
[12] Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In International Conference on Machine Learning, pages 1475-4483. PMLR, 2020.
[13] Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernet- works. arXiv preprint 2ttiv::3//.083999 2023.
[14] Emily Dinan, Sho Yaida, and Susan Zhang. Effective theory of transformers initialization. arXiv preprint arXiv:2304.02034 2023.

---

[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMIR, 2021.
[16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVE conference computer vision and pattern recognition, pages 16000-16009, 2022.
[17] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[18] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
[19] Haoyu Hc, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity visual parameter-efficient fine-tuning, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11825-11835, 2023.
[20] Yi-Kai Zhang, Shiyin Lu, Yang Li, Yanging Ma, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Wings: Learning multimodal Ilms without text-only forgetting. arXiv preprint arXiv:2406.03499 2024.
[21] Chao Yi, De-Chuan Zhan, and Han-Jia Ye. Bridge the modality and capacity gaps in vision- language model selection. arXiv preprint 2rXiv:2403.13797, 2024.
[22] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2021.
[24] Qiu-Feng Wang, Xin Geng, Shu-Xia Lin, Su Xia, Lei Qi, and Ning Xu. Learngene: From open-world to your learning task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8557-8565, 2022.
(25] Qiufeng Wang, Xu Yang, Shuxia Lin, and Xin Geng. Learngene: Inheriting condensed knowledge from the ancestry model to descendant models. arXiv preprint 1xXiv:3305.02279 2023.
[26] Shiyu Xia, Miaosen Zhang, Xu Yang, Ruiming Chen, Haokun Chen, and Xin Geng. Trans- former as linear expansion of learngene. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16014-16022, 2024.
[27] Boyu Shi, Shiyu Xia, Xu Yang, Haokun Chen, Zhiqiang Kou, and Xin Geng. Building variable-sized models via learngene pool. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14946 14954, 2024.
[28] Lorenzo Noci, Chuning Li. Mufan Li, Bobby He, Thomas Hofmann, Chris Maddison, and Dan Roy. The shaped transformer: Attention models in the infinite depth-and-width limit. Advances in Neural Information Processing Systems, 36, 2024.
[29] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[30] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mingg discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
12

---

[31] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127:302-321, 2019.
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, begh-performacc deep learning library. In Proceedings of the 33th Annual Conference on Neural Information Processing Systems, pages 8024-8035, 2019.
[33] Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.
[34] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[35] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 15750-15758, 2021.
[36] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efffcient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, 2023.
[37] Hengcan Shi, Son Duy Dao, and Jianfei Cai. Llmformer: Large language model for open- vocabulary semantic segmentation. International Journal of Computer Vision, pages 1-18, 2024.
[38] Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. Weight subcloning: direct initialization of transformers using larger pretrained ones. arXiv preprint arXiv:2312.09299 2023.
[39] Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones. In International Conference Learning Representations (ICLR), 2024.
[40] Fnu Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham M Kakade, Ali Farhadi, et al. Matformer: Nested transformer for elastic inference. Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ @ NeurIPS 2023).
2023.
[41] Shi- Yu Xia, Wenxuan Zhu, Xu Yang, and Xin Geng. Exploring learngene via stage-wise weight sharing for initializing variable-sized models. arXiv preprint arXiv:2404. 16897, 2024.
[42] Qiufeng Wang, Xu Yang, Haokun Chen, and Xin Geng. Vision transformers as probabilistic expansion from learngene. In Forty-first International Conference on Machine Learning, 2024.
[43] Fu Feng, Jing Wang, Congzhi Zhang, Wenqian Li, Xu Yang, and Xin Geng. Genes intelligent agents. arXiv preprint arXiv:2306.10225. 2023.
[44] Fu Feng, Jing Wang, and Xin Geng. Transferring core knowledge via learngenes. arXiv preprint 24X.v22401.8113, 2024.
[45] Fu Feng, Yucheng Xie, Jing Wang, and Xin Geng. Wave: Weight template for adaptive initialization of variable-sized models. arXiv preprint arXiv:2406. 17503, 2024.
[46] Yucheng Xie, Fu Feng, Jing Wang, Xin Geng, and Yong Rui. Kind: Knowledge integration and diversion in diffusion models. arXiv preprint arXiv:2408.0733 2024.
[47] Yucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang, and Xin Geng. Fine: Factorizing knowledge for initialization of variable-sized diffusion models. arXiv preprint arXiv:2409. 19289, 2024.
[48] Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowl- edge transfer. arXiv preprint
2015.
13

---

[49] Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the transformer growth for progressive bert training, Proceedings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguisties; Human Language Technologies, pages 5174-5180, 2021.
[50] Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. In International Conference on Machine Learning, pages 11893-19900 PMLR, 2022.
[51] Peihao Wang, Rameswar Panda, and Zhangyang Wang. Data efficient neural scaling law via model reusing. In Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 36193-36206 PMLR, 23-29 Jul 2023.
[52] Peihao Wang. Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang. and Yoon Kim. Learning to grow pretrained models for efficient transformer training. International Conference on Learning Representations, 2023.
[53] Ning Ding, Yehui Tang. Kai Han, Chao Xu, and Yunhe Wang. Network expansion for practical training acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20269-20279) 2023.
[54] Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang. Zhi Wang. Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2134-2148, 2022.
[55] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1eX3v05503.0255 2015.
[56] Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12145-12154, 2022.
[57] Sucheng Ren, Fangyun Wei, Zheng Zhang, and Han Hu. Tinymim: An empirical study of distilling mim pre-trained models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3687-3697, 2023.
(58) Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv: 2017.
[59] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint WxXiv22033003360 2020.
[60] Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference on Machine Learning, pages 26809-26823, PMLR, 2022.
[61] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technique Report, 2009.
[62] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[63] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891 891-898, 2014.
14

---

[64] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[65] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7262-7272, 2021.
[66] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[67] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12259-
12269, 2021.
[68] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, ICLR, 2018.
[69] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. EMNLP, 2020.
[70] Wenhui Wang, Furu Wei, Li Dong, Hangho Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-577 2020.
[71] Shiming Chen, Ziming Hong, Guo-Sen Xie, Wenhan Yang, Qinmu Peng, Kai Wang, Jian Zhao, and Xinge You. Msdn: Mutually semantic distillation network for zero-shot learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 7612-7621, 2022.
[72] Kan Wu, Jinnian Zhang. Houwen Peng, Mengehen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In Computer Eccon-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXI, pages 68-85. Springer, 2022.
[73] Yutong Bai, Zeyu Wang, Junfei Xiao, Chen Wei, Huiyu Wang, Alan L Yuille, Yuyin Zhou, and Cihang Xie. Masked autoencoders enable efficient knowledge distillers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24265.222265. 2023.
[74] Thanh Nguyen-Dut Trung Le, He Zhao, Jianfei Cai, and Dinh Phung. Adversarial local distribution regularization for knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4681-4690, 2023.
[75] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02388, 2024.
[76] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transform- ers: State-of-the-art natural language processing. arXiv preprint 1900v2111, 2019.
15

---

A Appendix
In the appendix, we present more details about the proposed LeTs for variable-sized ViT-based model initialization in this paper, including:
In Subsection A.1
we present how to adopt weight sharing within width transformation transformation matrix for LeTs.
matrices and depth
In Subsection
A.2
we present the comparison between LeTss and model expansion.
In Subsection
A.
we provide more experimental details.
Subsection
A.4
4
we analyze that LeT's can significantly enhance training efficiency when
transferring to downstream datasets.
In Subsection
A.5
discuss the comparison between LeTs and knowledge distillation.
In Subsection
A.6
provide the comparison between LeTs and online models.
In Subsection
A.
we give the comparison between LeT's and meta learning.
In Subsection
A.
we discuss the limitations and future work of this paper.
In Subsection
A.9
we present the broader impacts of this paper.
A.1 Adopt Weight Sharing within Transformation Matrices for LeTs
To keep the parameter efficiency for transformation matrices, we propose to adopt weight sharing width transformation matrices and depth transformation matrix inspired by [52]. However, with the transformation pipeline,
maintain compared to [52], as we only need to add width transformation matrices to the shallower learngene modules, rather than to the deeper Aux-Net. We focus leveraging LeT's for ViT models [66), which mainly contains the following weight components: patch embedding, layer norm (LN), multi-head self-attention (MHA) and multi-layer perception (MLP).
compact transferred part
Width transformation matrices. For patch embedding, we adopt learnable matrice emut For MHA (W/, & E {Q, VQ,KV we adopt learnable matrices Fin. and Frrut where & E {Q, K, V,O). Here, we set F/ =ouutV F. E [Q, K, V, O). For LN, the learnable matrices are the associated out-dimension transformation fel
Fout,emb and Fiink = Fout,.mb where
matrices. For MLP (W/fel w we adopt learnable matrices Friik and Fous. (k E {fcl, fc2)) for the first and second MIP layer. Similarly, we Fin.fc2 Fout.fel F Fout,fe2 Fout,emb and Fin.fel emb, Learnable matrices for bias are shared in the same way as those for weights.
Depth transformation matrix. For depth transformation matrix G. we propose share the parameters within different rows of G. Such sharing strategy provides depth expansion guidance for transforming learngene along depth dimension.
A.2 Comparison between LeTss and Model Expansion
Model Expansion, a widely adopted training acceleration framework pioneered by [48]. involves the incremental enlargement of neural networks [49 50 52 53). Net2Net [48] utilizes function- preserving transformation to expand width by duplicating neurons and depth by incorporating identity layers. Bert2Bert [54] builds on this concept by proposing function- prreeeriipneeiinggg width expansion specifically for Transformers. Staged-training [50] achieves width expansion by doubling the dimensions of all matrices and depth expansion through rero-initializing normalization parameters. Expansion [53] introduces a width expansion strategy for convolutional neural networks using orthogonal filters, and a depth expansion strategy for Transformers based on the corresponding exponential moving average model. LiGO [52 learns to linearly map the parameters of smaller model to initialize larger one.
While LeTs is inspired by these studies, it differs in several key aspects:
Our goal is on flexibly initializing variable-sized models from a single compact learngene, rather than solely accelerating the training of models larger than learngene.
Objective:
16

---

SWS
LeTs
Model
Params(M)
FLOPs(G)
Scratch
TLEG
10 ep
1 ep
5 cp
D6
11.4
2.3
68.6
69.5
70.2
64.0
70.2
Des-H6LL8
15.0
3.1
71.7
72.3
73,4
76.0
77.6
Des-I16-L10
73.8
73.9
75.1
77.5
78.9
Des-H6-L12
22.1
4.6
75.0
75.1
75,8
78.2
79.5
Des-H16-L14
25.6
76.5
76.1
80.0
Des-H6-L116
29.2
6.1
76.7
76.4
76.7
79.3
80.4
Des-H8-L8
26.2
75.1
78.1
Des-H8-L10
32.5
6.6
76.6
78.3
79.3
Des-H8-L11
38.8
77.7
79.4
80.1
Des-H8-LE
45.1
9.3
78.5
80.0
80.5
Des-H8-L16
51.5
10.6
79.4 (125ep)
80.5
80.9
Des-H10-L8
40.7
8.2
76.6
77.3
78.6
Des-1110-L10
50.5
10.2
77.8
79.2
79.9
Des-H11-LL1
60.3
12.3
79.0
79.9
80.4
Des-H10-L14
70.2
14.3
79.4 (125ep)
80.4
80.8
Des-H10-L116
80.0
16.3
80.3 (125ep)
80.9
81.1
Des-H12-L8
58.2
11.7
77.2
78.1
79.0
78.2
79.0
Des-H12-L10
72.4
14.6
78.2
79.1
79.7
79.8
80.1
Des-H12-L12
86.6
17.5
79.6
79.9
80.1
80.5
80.8
Des H12-L14
100.7
20.4
80.4 (125ep)
80.4
80.4
80.9
81.1
1
114.9
23.3
80.4 (150ep)
80.6
80.6
81.3
81.6
Des-H12-LI8 129.1
26.2
79.1 (300ep)
79.7(0ep)
Des-H112-L24
171.6
34.9
79.5 (200ep)
81.2
81.4
Des-H16-L24
304.3
78.3 (200ep)
80.0
80.9
Table 5: The numerical results for Fig and 5 in original paper. The number of epochs indicated in brackets within the "Seratch" column, with the default being 100 epochs when no brackets are present.
Overall pipeline: Taking one state-of-the expansion method LiGO [22 as an example, LiGO firstly combines all layers of a smaller model to reconstruct each layer of a larger model along depth dimension, subsequently equipping each reconstructed layer with width expansion matrices. In contrast, we first transform the learngene layers along the width dimension and then partition the tenth-transformed layers into multiple groups, subsequently combining the layers within each group to construct new layers. With the "first-width-therr depth" transformation pipeline, we can maintain a compact transferred part, as we only need to add width transformation matrices to the shallower learngene modules, rather than the deeper Aux-Net. In contrast, research on model expansion does not prioritize keeping the transferred part compact, their objective is not transfer fewer parameters for initializing models varying sizes.
Parameter selection: Another crucial aspect of Le''s involves selecting specific parameters from well-trained transformation matrices for subsequent initialization.
A.3 Experimental Details
In this section, describe the experimental details of main results, present numerical results and model settings for Fig Fig
Fig 6 Tab Tabī and Tab in our original paper.
Training Details for Aux-Net. In general, we follow the training setting and hyperparameters provided in DeiT [66]. We make several modifications on DeiT: (1) We remove the [class] token; (2) We attach the model with global average pooling layer and fully-connected layer for image classification. We train Aux-H12-L16 for 300 epochs with 5 warm-up epochs on ImageNet- I We choose LeViT-384 [67] the ancestry model to employ distillation. Specifically, we use the Adam W (88 optimizer with weight decay 0.05 and cosine scheduler, where batch size is set 128
and À is set to 1.0. All models are implemented by PyTorch (32). and trained on NVIDIA Tesla V100
17

---

100
08.0
96.1
98.5
58.7
98.5
98.7
88
88.8
98.9
28.8
100
57.9
88.6
59.2
Top-1 Acc(%)
74.6
50.3
24.4
292
904
51.5
Top-1 Acc(%)
74.8
60
60.8
40
11.2
Scratch
Scratch
Lels (Ours)
Lets (Ours)
30
60
90
120
150
180
210
240
270 500
30
120
150
180
210
240
270 300
EDochs
EDochs
(a) CIFAR-10 10 (Des H12-L12)
(b) CIFAR 100 (Des H12-L12)
87.2
89.
100
97.5
98.0
98.
95.4
96.4
96.5
95.8
96.7
98.8
88.8
Top-1 Acc
30
75.
79.5
70.6
Top-1 Acc(%)
75.1
755
638
55.2
60.8
60
53.8
40
29.3
Scratch
Scratch LeTo (Ours)
20
LeTs (Ours)
60
90
120
150 180
300
150
210 240 270 300
Epochs
Epochs
(c) Food-101
(Des- (12--111
(d) CIFAR-10 (Des- (ees-11--112)
Top-1 Acc(%)
82.7
86.5
87.8
88.2
882
88.5
Top-1 Acc(%)
88.5
MI
13.3
53.3
MA
15.
75.3
25.5
44.2
50.
40.4
50.9
25.0
Scratch
25.2
Scratch
Lets (Ours)
Lets (Durs)
120
270
Epochs
Epochs
(e) CIFAR-100 (Des-H6-L.12)
(f) Food-101 (Des-H6-L.12)
Figure 7: Compared to Scratch, LeTs could significantly enhance training efficiency. Take Des-H12- L12 on Food- 101 an example, LeT's outperforms Scratch after 10 epoch tuning, which is about 30 faster.
Datasets
Model
Params(M)
Pre-Fin
Scratch
Grad-LG
TLEG
SWS
LeT's
Des-H6-L12
21.7
86.9
69.3
70.8
86.3
87.4
89.2
CIFAR-100
Des-HI12LL12
85.9
88.5
74.8
79.2
86.9
89.6
90.0
Des-H6--L12
21.7
98.2
86.7
88.0
98.0
98.3
98.8
CIFAR-10
Des-I112-L12
85.8
98.5
91.6
95.2
98.2
98.9
98.9
Des-H6-L12
21.7
88.9
75.3
77.9
88.5
89.2
90.8
Food-101
Des-H12-L12
85.9
89.0
79.6
83.4
89.3
90.6
91.2
Des-H6-L12
21.7
83.6
78.9
84.5
87.1
Cars-196
Des-H12-L12
86.0
87.6
88.1
89.9
90.2
Table 6: The numerical results for 6 in original paper.
GPUs and NVIDIA GeForee RTX 3090 GPUs. And GPU hours are approximately measured based on Tesla V100 GPUs and RTX 3090 GPUs.
Experimental Details, Model Settings and Numerical Results for Fig Fig in Our Original Paper. We report the numerical results of Fig 4 and 5ig55in Tab For Scratch, we train all the Des- Nets for 100 epochs (some for more than 100 epochs) with 5 warm-up epochs on ImageNet- [29] with timm default initialization
[32), where we use the Adam W [68] optimizer with weight decay cosine scheduler following (66). For TLEG, we follow the experimental
0.05, batch size 128 details and results of (26. For SWS, we follow the experimental details and results of [41]. For Matformer, we refer to the results presented Figure 4(a) of their original paper [40] For LeT's, we
initialize Des-Nets with continuous selection and fine-tune them for 5 epochs. We use the AdamW
18

---

Datasets
Backbone
Params(M)
Pre-Fin
LeTs (0ep)
LeTs (5ep)
Des-H6-L12
26.5
36.8
40.2
41.4
ADE20K
Des-H12-L12
103.3
42.0
43.2
Des-H6-L12
26.3
47.0
48.7
49.9
Pascal Context
Des-H12-L12
103.1
49.8
50.9
Des-H6-L12
26.9
72.9
74.5
76.5
Cityscapes
Des-H12-L12
104.2
75.3
76.4
Table 7: The numerical results for Fig i)-(k) in our original paper.
optimizer with weight decay 0.05, batch size 128 and cosine scheduler for all Des-Nets. We also inherit the parameters of patch projection and classification head to initialize Des-Nets. comparison to Scratch, we calculate GPU hours for training these 24 Des-Nets and find that LeT's significantly reduces total training GPU hours by approximately 7x (6K GPU hours for Scratch versus 0.8K GPU hours for LcTs). For Res-HH,-L24, we construct and train one Aux-Net with 16 heads and 24 layers from Des-H Hes-HI2-L116 (initialized) for epochs to obtain the well-trained transformation matrices, after which we the Aux-Net to initialize the Des- H14--42 When training ImageNet- K, Des-H16-L22 underperforms compared to Des-H12-L12, where the relatively small number of training samples in ImageNet-1K 1K may be the cause.
Experimental Details, Model Settings and Numerical Results for Fig Our Original Paper. We report the numerical results of Fig 6 Tablaaan Tab For all methods and all downstream image classification datasets in Fig ([aa a)), we fine-tune the initialized Des-H6-L.12 and Des-H12-L112 for 300 epochs. We use the Adam W 688 optimizer with batch size 128 or 256, and cosine scheduler. For Cars- 196 [62], Scratch fails to converge, where the relatively small number of training samples may be the cause. For baseline methods, use the hyperparameter settings similar to those of LeTs. For all semantic segmentation datasets in Fig ())(kk, we follow the training setting of [65.
Experimental Details and Model Settings for Tab and Tab in Our Original Paper. For downstream transfer experiments, we acknowledge the necessity of retraining a specific task head when transferring well-trained parameters from task A to task B. However, in Tab both task A and task for all baselines and Le'T' involve ImageNet- IK. Therefore, we also inherit the classification head parameters from either the first stage or the pre-training stage to initialize the Des-Nets for ImageNet-1K K. For IMwLM (39) we use one 300ep-pre model with heads and 18 layers (129.IM) to initialize models of different depths. Specifically, we use the consecutive selection strategy and select the first V layers to initialize target models as introduced in IMWLM [39], where N represents the layer number of target models. In Tab for Pre-Fin, we use the models pretrained on ImageNet-IK to finetune on CIFAR-100. In Tab 4 about LeTs(LiGO), we firstly train a small model with 6 heads and 8 layers for 300 epochs. Then proposed in LiGO to construct and train the Aux-Net for 200 epochs, where the number of Aux-Net parameters is 60.2M. Lastly, we use the well-trained transformation matrices to initialize the Des-Net by our proposed parameter selection strategy.
depth and width transformation strategy
A.4 Training Efficiency when Transferring Downstream Datasets
From 7 we observe that LeTs not only achieves better performance than Scratch, but also is
significantly faster. Specifically, LeTs generally outperforms Scratch after about 10 epochs or epochs tuning, which is about 30x 60 faster on CIFAR-I0, CIFAR-100 [61 and Food-101 (30). In summary, LeTs provides a well initialization for variable-sized target models, which not only boosts the final model quality but also speeds up the model training.
A.5 Discussion about LeTss and Knowledge Distillation
Conceptual Comparison between LeTss and Knowledge Distillation. considerable body of literature has emerged, focusing on exploring techniques for knowledge distillation [55 69
70 66 71
72 56
73 74 75). DeiT (66) facilitates the training of ViT with guidance from a Conv Net teacher by introducing distillation token. MiniViT (56) transfers knowledge from large-scale ConvNets
19

---

to weight-muliplexed ViT' through weight distillation. DMAE 73 minimizes the discrepancy between the intermediate feature maps of the teacher and student models, alongside optimizing pixel reconstruction loss. The commonality among these methods is the requirement for multiple forward passes through a pretrained teacher during the training of new students, accommodating various resource constraints. Consequently, this entails additional resource consumption for the storage and computation of teacher models every time new models are trained for different scenarios.
Completely different from knowledge distillation, LeTs distills knowledge from the pretrained ancestry model to one auxiliary model once comprising one compact learngene module and series of learnable transformation matrices. Subsequently, LeT's selects target transformation matrices from these well-trained ones given the different sizes of target model under the guidance of our proposed selection strategies. Lastly, LeT's can flexibly initialize variable-sized models from learngene with target transformation matrices, while simultaneously eliminating the need for the pretrained ancestry model. As can be seen, Le'T' is also not a simple combination of knowledge distillation and transformation matrices. Obviously, if we simply combine them, we can obtain just only one target model, which is completely contrary to our research focus: transforming well-trained learngene to initialize variable-sized models.
A.6 Comparison between LeTTs and Online Models
Despite the widespread availability of online models in various sizes, it is important to note that these models are pre-trained by other institutions, which also incurs substantial training costs. contrast, Le'T's offers more effective and lightweight choice, which transforms one compact learngene with series of learnable transformation matrices to initialize variable-sized models, eliminating the need for repetitive online model downloads. Furthermore, LeTss flexibly construct and initialize models with finer granularity, unconstrained by layer numbers and head numbers, unlike online models available on platforms such as HuggingFace [76] and Timm 33.. This obvious flexibility allows LeT's to offer greater customization for diverse downstream scenarios with varying resource constraints.
A.7 Comparison between LeTs and Meta Learning
Regrading objective, Meta learning mainly focuses on parameter initialization of one target model, but rarely considers initializing variable-sized models to fit diverse resource constraints, the latter of which is our main focus in this paper. For methodology, LeTs highlights transforming one compact learngene with learnable transformation matrices for initialization. In contrast, Meta learning often uses techniques like learning to leamn
A.8 Limitations and Future Work
This paper develops an effective approach termed LeTss transform one compact learngene with series of learnable transformation matrices for variable-size model initialization. Despite LeTss achieves satisfactory performance under different initialization settings, we do not carefully select the hyperparameter settings for the construction and training process of Aux-Net, and the finetuning process of each initialized models with different sizes, such as the initialization order. Besides, we only adopt prdiction-basse distillation to train the compact learngene and transformation matrices, which may affect the quality of both of them.
In future work, we consider how to efficiently select specific hyperparameter settings and add more advanced distillation techniques to train the compact learngene and transformation matrices. Besides, we consider employing LeT's for language tasks and combine LeTss with other Learngene methods.
A.9 Broader Impacts
This paper focuses the fundamental research question of initializing variable-sized models for accommodating diverse resource constraints. The goal is using learnable transformation matrices to transform one compact learngene for initialization. Generally, there are no negative societal impacts involved in this paper.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the Introduction 1 which clearly states the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer the subsection A.8 of the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experimental results this paper are reproducible. Please refer to experimental details in our original paper and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: All source code for conducting the experiments will be made publicly available with a license that allows free usage for research purposes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https i
//nips.cc/pp wccppublicgguuuiecceeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer experimental details in our original paper and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We do not have error bars to show.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to experimental details in our original paper and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics to make sure the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In our research, we ensured that all assets used comply with the corresponding license requirements and adhere clear usage terms during
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Of3vN9IZ2l", "reviewer": "Reviewer_JzzD", "summary": "This research adopts the learngene learning paradigm; the core idea of learngene is to transform a well-trained ancestry model (Ans-Net) to initialize variable-sized descendant models (Des-Net). The authors pointed out two limitations of previous works: (1) the original learngene paradigm lacks the provision of structural knowledge which is not favorable for later transformation and (2) existing strategies to craft Des-Net are not learnable and they overlook the width dimension.\n\nTo address those limitations, the paper proposed LeTra, standing for Learnable Transformations:\n- Learnable transformation parameters with structural knowledge in Ans-Net that later facilitates descendant transformation. To this end, they train an auxiliary model (Aux-Net) to learn the transformation $T=P_{depth}F_{width}$ from Ans-net, where $P_{depth}$ and $F_{width}$ are learnable and structured matrices respectively responsible for depth and width dimensions.\n\n- Given different size of target Des-Net, some columns/rows of $P_{depth}$ and $F_{width}$ will be selected to construct the transformation $T^{des}$. Ans-Net is then transformed using $T^{des}$ to initialize the Des-Net\n\nThe authors conducted extensive experiments, showcasing the advantages of LeTra, not only in terms of performance but also in terms of fine-tuning speed.", "questions": "Those are related to the weaknesses above:\n\n- Is it possible to combine LeTra with other Learngene strategies to overcome the scaling limitation?\n\n- May the authors consider more complex downstream task like semantic segmentation or object detection? \n\n- Given the strong results of LeTra even without any fine-tuning (c.f. Table 1), I'm curious of the linear probing results, i.e. fine-tuning only a simple head for the downstream task; that protocol is common in self-supervise learning", "limitations": "I believe this is a solid work with significant contributions suitable for the conference. However, I have a few questions and suggestions that I would greatly appreciate being addressed. My current recommendation is quite positive.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper presentation is very clear and easy to understand. I enjoy reading it.\n\n- The motivation is clear and convincing. Extensive experiments validate the effectiveness of the proposed LeTra, showcasing the benefits of learnable transformation.", "weaknesses": "- The choice of Des-Nets is limited as those can be seen as sub-nets of the pretrained Aux-Net, i.e. one cannot scale Des-Net bigger than Aux-Net\n\n- Downstream experiments were only conducted on small classification datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1) Scaling Des-Net bigger than Aux-Net.**\n\nPlease see the relevant discussions in G1 of General response.\n\n**2) Combination with other Learngene strategies.**\n\nThank you for your insightful question!\n\nWe could combine LeTra with other Learngene strategies. For instance, we could replace the depth transformation strategy of LeTra with linear expansion strategy proposed by TLEG [23].\n\n**3) More complex downstream tasks.**\n\nThank you for your nice concern!\n\nIn the revision, we are committed to expanding our experimental scope to include more diverse tasks and datasets.\n\n**4) Linear probing results.**\n\nAs shown in the table below, we can observe that LeTra achieves better performance than Pre-Fin under the linear probing protocol. For example, LeTra outperforms Pre-Fin by 3.43% and 1.86% on CIFAR-100 and CIFAR-10 with Des-H12-L12.\n\n| Des-H12-L12 | Pre-Fin | LeTra |\n| :---------: | :-----: | :---: |\n|  cifar100   |  72.08  | 75.51 |\n|   cifar10   |  90.08  | 91.94 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lQbMdmpZLS", "reviewer": "Reviewer_i78Y", "summary": "To avoid unafforadable trainining cost, a new training paradigm like Learngene framework is proposed. Unlike previous work that mainly focus on the depth, the authors proposed Learnable Transformations, which is able to  adjust the learngene module along both depth and width dimension for flexible variable-sized model initialization. Experimental results indicate that the proposed method is able to achieve strong performance in one 1 epoch fine-tuning.", "questions": "Figure 1 (d) raises questions. It is commonly believed that depth is more crucial than width in deep networks. However, this work suggests that varying width can achieve better performance than increasing depth. Could the authors elaborate on this finding?", "limitations": "See above.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A new framework called LeTra is proposed, capable of transforming the learngene module along both depth and width.\n\n2. Extensive experiments demonstrate that the proposed method achieves promising performance, even with just one epoch of tuning.", "weaknesses": "As shown in Table 4, training from scratch consistently improves performance with increased depth, but almost no improvements are observed for the proposed method. Could the authors explain this in more detail?\n\nHow much does distillation contribute to the final performance? Is most of the performance gain from distillation during fine-tuning?\n\nCould the authors clarify the results in Table 1? From my understanding, even when initializing a model from pre-training, performance would significantly drop without any fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1) Performance improvements with increased depth.**\n\nThank you for pointing this out!\n\nIn Table 4, we present the results of LeTra with 2-epoch tuning (L315), which aims to demonstrate the effectiveness of our proposed depth transformation rather than performance improvements with increased depth.\n\nInterestingly, during the early training phase (e.g., 2 epochs), model performance does not consistently improve with increased depth, whether using Scratch that trains randomly-initialized models or LeTra that trains learngene-initialized models. To validate this claim, we provide the results of Scratch with 5-epoch training and LeTra with 2-epoch tuning below.\n\nFurthermore, extending the training of LeTra-initialized models to more epochs reveals a consistent improvement in model performance with increased depth.\n\n|   Model    | Params(M) | FLOPs(G) | Scratch (5ep) | Scratch (100ep) | LeTra (2ep) | LeTra (20ep) |\n| :--------: | :-------: | :------: | :-----------: | :-------------: | :---------: | :----------: |\n| Des-H6-L13 |   23.8    |   5.0    |     10.4      |      76.4       |    78.1     |     79.3     |\n| Des-H6-L14 |   25.6    |   5.3    |     10.7      |      76.5       |    78.0     |     79.5     |\n| Des-H6-L15 |   27.4    |   5.7    |     10.0      |      77.1       |    78.2     |     79.8     |\n\n**2) Contribution of distillation to the final performance.**\n\nThank you for your nice concern!\n\nWe remove the distillation process in the first stage and present the performance of LeTra in Figure II of our uploaded PDF.\n\nComparing \"LeTra (without distillation in first stage) (5 epoch)\" with \"LeTra (with distillation in first stage) (5 epoch)\", we observe that the distillation process in the first stage has a marginal improvement on the performance of Des-Nets.\n\nAdditionally, comparing \"LeTra (without distillation in first stage) (5 epoch)\" with \"Scratch (5 epoch)\", we find that LeTra's proposed initialization process significantly enhances Des-Nets' performance.\n\nConsequently, we can safely conclude that the most substantial performance gains stem from our proposed initialization process using LeTra rather than the distillation process in the first stage.\n\nFurthermore, it is important to note that we did not employ distillation during the fine-tuning of Des-Nets in the second stage.\n\n**3) Explanation of Table 1.**\n\nWe acknowledge the necessity of retraining a specific task head when transferring well-trained parameters from task A to task B. However, in Table 1, both task A and task B for all baselines and LeTra involve ImageNet-1K. Therefore, we also inherit the classification head parameters from either the first stage or the pre-training stage to initialize the Des-Nets for ImageNet-1K.\n\n**4) Discussion about importance of depth and width for deep networks.**\n\nThank you for your insightful question! While it is widely accepted that depth significantly impacts deep network design (as shown in the left table below, where model performance increases with depth), we empirically find that configuring width could also enhance model performance (as shown in the right table below). This discovery motivates us to explore transforming learngene across both depth and width dimension. Furthermore, recent studies (see [i]) emphasize the importance of simultaneously considering both width and depth dimensions in neural network design.\n\n[i] \"The shaped transformer: Attention models in the infinite depth-and-width limit.\", NeurlPS 2023.\n\n|    Model    | Params(M) | FLOPs(G) | Scratch (100ep) |      |    Model    | Params(M) | FLOPs(G) | Scratch (100ep) |\n| :---------: | :-------: | :------: | :-------------: | ---- | :---------: | :-------: | :------: | :-------------: |\n| Des-H12-L7  |    51.0     |   10.3   |      76.5       |      | Des-H7-L12  |   29.9    |   6.2    |      76.4       |\n| Des-H12-L8  |   58.2    |   11.7   |      77.2       |      | Des-H8-L12  |   38.8    |    8.0     |      77.7       |\n| Des-H12-L9  |   65.3    |   13.1   |       78.0        |      | Des-H9-L12  |    49.0     |    10.0    |      77.8       |\n| Des-H12-L10 |   72.4    |   14.6   |      78.2       |      | Des-H10-L12 |   60.3    |   12.3   |       79.0        |\n| Des-H12-L11 |   79.5    |    16.0    |       79.0        |      | Des-H11-L12 |   72.9    |   14.8   |      78.4       |\n| Des-H12-L12 |   86.6    |   17.5   |      79.6       |      | Des-H12-L12 |   86.6    |   17.5   |      79.6       |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HhDekYbzJ9", "reviewer": "Reviewer_8CMe", "summary": "This work builds on top of a learning paradigm called Learngene (introduced in an earlier work), which focuses on providing effective initializations for training multiple target models of different sizes. In this paradigm, a compact module, referred to as learngene, is first leaned from a large well-trained network, and then learngene is transformed to initialize different models of varying size. This work introduces a set of learnable transformation parameters into the learngene paradigm and trains the compact learngene module and the transformation parameters simultaneously. The learngene module and transformation parameters are trained by using them to create an auxiliary network which is trained using distillation from a large well trained model. Once trained, the transformation parameters can be used to transform learngene module parameters in order to initialize target networks of different sizes. \n\nExperiments are conducted using ImageNet and multiple other transfer learning datasets and the results show that the proposed approach improves training compute efficiency of target models significantly when compared to training from scratch.", "questions": "Some things are unclear from the paper. Please see the questions in 'weaknesses' section.\n\nComparisons with some highly relevant approaches are missing in the paper.", "limitations": "No specific negative societal impact.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The paper focuses on the problem of obtaining effective target model initializations cheaply (with less compute). This is a practically useful problem.\n\nThe experimental results show that the proposed initialization significantly improves training efficiency when compared to training from scratch.", "weaknesses": "Presentation: \nPresentation of the paper can be improved. \n\n* Overall, I feel that sec 3.1 has a lot of notations and all of them are not strictly necessary to clearly describe the method. I encourage the authors to think about simplifying this section.\n\n* Fig. 3(a) is confusing: Once we have chosen F_{l,in}^des and F_{l,out}^des (by selecting appropriate rows from F_{l,in} and F_{l,out}), the multiplication of three matrices F_{l,in}^des x W_l^LG x (F_{l,out}^des)^T should give the weights for the destination network layer. I do not understand why there are additional insert steps in Fig. 3(a). These steps do not seem to match with Eq. (1). Which process is actually used when performing width modifications? Is it Eq.(1) or the process depicted in Fig 3(a)? If it is the one in Fig. 3(a), is the same process used for constructing auxiliary network during training?\n\n* Small correction: I think the size of u_{ij} in line 158 should be D_in^L x D_in^L instead of D_in^aux x D_out^aux since u_{ij} is multiplied with the weights W_j^L of j^th learngene layer which is of size D_in^L x D_out^L.\n\n\n\nExperimental results:\n\n* Some things are unclear to me:\n    * What is the difference between scratch-1 and scratch-2 trainings for one model?\n    * What is the teacher network used for distillation when training auxiliary network?\n\n* Comparison missing with highly relevant alternative strategies:\n    * Based on my understanding of the selection process, the final parameter matrices of the DesNet are effectively sub-matrices of the parameter matrices of the first L^des layers of the trained auxiliary network. Such a selection process can be applied to any pretrained transformer network. For example, one can train a standard ViT of the same size as the auxiliary network using distillation with the same teacher and then directly sample weights from this trained model to initialize Des-Nets. Initializing smaller transformer models from larger pretrained transformer models has been recently studied in [36].\n    * Another relevant approach is which is not mentioned in the paper is \"MatFormer: Nested Transformer for Elastic Inference\".\n* Unfair comparison with from scratch training:\n    * This following statement in line 259-261 is incorrect: \"Compared to Scratch training, LeTra reduces around 3.6× total training costs (12×100) epochs vs. 320+12×1 epochs).\"\nComparing training costs simply in terms of number of epochs is not meaningful. LeTra trains a larger auxiliary model (H14-L15) and also uses distillation for training which requires running a teacher model during training. So, 1 epoch training of LetTra auxiliary model will be significantly costlier than one epoch training of any of the smaller Des-Net models. \n    * In terms of performance comparison, it is unfair to compare LeTra that uses distillation directly with training from scratch without distillation. \n\n* Experimental results presented only on Des-Nets whose size is in between Auxiliary network and Learngene network. How effective is the initialization for Des-Nets which are smaller than learngene?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Presentation]**\n\n**1) Simplification of notations in Sec 3.1.**\n\nThank you for pointing this out! In the revision, we will simplify the notations of Sec 3.1 to describe the method more clearly.\n\n**2) Initialization process in Figure 3.**\n\nThank you for raising this confusion!\n\nDuring the first training stage of LeTra, we construct the Aux-Net according to Eq.(1).\n\nFor the second initialization stage of LeTra, we first select certain rows/columns from well-trained **$F_{l,in}$** to form target matrices **$F_{l,in}^{des}$**. Subsequently, we use **$F_{l,in}^{des}$** to perform matrix multiplication with learngene matrices. The multiplication results are then inserted into original learngene matrices to derive **$W_{l}^{'}$**.\n\nSimilarly, we select certain rows/columns from well-trained **$F_{l,out}$** to construct target matrices **$F_{l,out}^{des}$**. Then we use **$F_{l,out}^{des}$** to multiply **$W_{l}^{'}$** and insert the multiplication results into **$W_{l}^{'}$** to obtain **$W_{l}^{des}$**.\n\nThus, the first training stage and the second initialization stage of LeTra are not inherently interconnected. In the revision, we will add these descriptions in the caption of Figure 3 to avoid any misinterpretation.\n\n**[Experimental results]**\n\n**1) Difference between Scratch-1 and Scratch-2.**\n\nScratch-1 involves training models from scratch with sizes identical to those used in LeTra (L544-545). Scratch-2 entails training models from scratch with sizes similar to those in Scratch-1 because these models only vary in depth (L546-547). Additional model details can be found in Table 5.\n\n**2) Teacher network.**\n\nWe choose LeViT-384 [48] as the teacher/ancestry model (L536-537).\n\n**3) Comparison with IMwLM [36] and Matformer [i].**\n\nWe employ DeiT-base distilled as the larger pretrained model for IMwLM [36]. For Matformer [i], we refer to the results presented in Figure 4(a) of their original paper. From Figure I of our uploaded PDF, compared to IMwLM [36] which initializes smaller models from larger pretrained ones and Matformer [i], we observe that LeTra achieves superior performance. Notably, LeTra demonstrates the capability to initialize models whose sizes are independent of larger pretrained models.\n\n[i] Kudugunta, Sneha, et al. \"Matformer: Nested transformer for elastic inference.\", NeurlPS 2023.\n\n**4) Comparison with scratch training in training costs.**\n\nThank you for pointing this out! In comparison to Scratch-1, we calculate GPU hours for training 12 Des-Nets and find that LeTra significantly reduces total training GPU hours by approximately **7×** (1322 GPU hours for Scratch-1 versus 190 GPU hours for LeTra).\n\n**5) Performance of Des-Nets without distillation in the first stage.**\n\nFirstly, it is important to note that we did not employ distillation during the fine-tuning of Des-Nets in the second stage.\n\nSecondly, we omit the distillation process in the first stage and present the performance of Des-Nets in Figure II of our uploaded PDF.\n\nWhile the distillation process in the first stage marginally enhances the final performance of Des-Nets, the most substantial performance improvement arises from our proposed initialization process using LeTra.\n\n**6) Performance of Des-Nets which are smaller than learngene.**\n\nPlease see the relevant discussions in G1 of General response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VobuD2IVgh", "reviewer": "Reviewer_p972", "summary": "This work proposes Learnable Transformations (LeTra) for improving learngene-based model initialization. In particular, a set of width transformations is learned to produce weight matrices of varying dimensions, and a set of depth transformations is learned to change the number of layers in the model. In the learngene learning stage, an auxiliary model is distilled from a large model, so that the learngene and the transformations can be optimized. After that, a set of variable-sized models can be initialized based on the optimized learngene and the transformations, which can serve as good starting points for efficient fine-tuning. Experiments on various image classification datasets including ImageNet and CIFAR with the Vision Transformer (ViT) architecture demonstrate the efficacy of the proposed method.", "questions": "1. [Transformation selection] In addition to the selection strategies introduced in Section 3.3, there could be other heuristics that consider the importance of each weight row/column, and select them accordingly. For instance, would it be helpful to rank the weights by magnitude and select the largest rows/columns?", "limitations": "The authors have adequately discussed the limitations and potential societal impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It is intuitive to apply both depth transformations and width transformations to learngene when constructing new models. This work shows the benefits of combining both transformations.\n\n2. The empirical results on various image datasets including ImageNet shows that LeTra can provide strong model initialization weights, significantly outperforming prior methods TLEG and Grad-LG.", "weaknesses": "1. [Presentation] The description of the proposed approach is not very clear:\n\n    - Overall, if we consider the ViT weights as a set of matrices, each width transformation linearly transforms the rows and columns of a weight matrix, and the depth transformations construct new layers by linear combining layers from the learngene. Section 3 could be improved by highlighting the core idea behind the two types of transformations and simplifying the notations.\n\n    - Section 3.3 is not easy to follow because the description is a bit vague. For instance, it is not specified how to choose the start and step size in \"step-wise selection.\" The difference between \"random selection\" and \"random selection (wo)\" is also unclear.\n\n    - The caption of Figure 3 could include more details.\n\n2. [Models larger than Aux-Net] The proposed approach learns width and depth transformations during Stage 1. However, such transformations cannot \"extrapolate,\" or in other words, we cannot build descendant models that are deeper or wider than the Aux-Net. This somehow limits the application scenarios.\n\n3. [Learngene training costs] When comparing LeTra with baselines, it would be helpful to also include the training costs of Stage 1 in which a more complex set of learngene + learnable transformations is optimized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Presentation]**\n\n**1) The description of transformations and notations.**\n\nThank you for pointing this out!\n\nIn the revision, we will simplify the notations of Section 3 and highlight the core idea behind the two types of transformations.\n\n**2) How to choose the start and step size in \"step-wise selection\".**\n\nIn practice, we usually choose the starting row as 1 and calculate the step sizes as $\\lfloor\\frac{D_{in}^{(aux)}}{D_{in}^{(des)}}\\rfloor$ and $\\lfloor\\frac{D_{out}^{(aux)}}{D_{out}^{(des)}}\\rfloor$.\n\n**3) Difference between \"random selection\" and \"random selection (wo)\".**\n\nFor random selection, we first randomly generate a set of indices, based on which we select corresponding rows/columns.\nDuring the selection process, we ensure that selecting the same rows/columns for all transformation matrices.\n\nIn the case of random selection (wo), as opposed to random selection, we opt to select distinct rows/columns for each transformation matrice (L238-239).\n\n**4) Caption of Figure 3.**\n\nIn the revision, we will add more descriptions in the caption of Figure 3 to avoid any misinterpretation.\n\n**[Models larger than Aux-Net]**\n\nPlease see the relevant discussions in G1 of General response.\n\n**[Comparisons between LeTra and baselines in training costs]**\n\nThank you for pointing this out!\n\nCompared to Scratch-1, LeTra reduces around 3.6× total training costs (12$\\times$100 epochs versus 320+12$\\times$1 epochs), where \"320\" means the training epochs of stage 1.\n\nAdditionally, in comparison to Scratch-1, we calculate GPU hours for training 12 Des-Nets and find that LeTra significantly reduces total training GPU hours by approximately 7× (1322 GPU hours for Scratch-1 versus 190 GPU hours for LeTra).\n\n**[More choices of transformation selection]**\n\nThank you for your insightful question!\n\nWe rank the weights by magnitude and select the largest rows/columns as the target transformation matrices, a strategy we refer to as \"rank\". \nAccording to the table below, we observe that \"rank\" achieves performance comparable to LeTra (continuous selection), thereby validating the robustness of our trained transformation matrices.\n\n|   Model    | Params(M) | FLOPs(G) | Scratch (100ep) | LeTra | step-wise | rank |\n| :--------: | :-------: | :------: | :-------------: | :---: | :-------: | :--: |\n| Des-H8-L13 |    42.0     |   8.6    |      78.1       |  80.0   |   79.6    | 79.8 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### General response

**G1. Size diversity of Des-Nets.**

We appreciate the valuable comments regarding the size diversity of Des-Nets, *e.g.*, scaling bigger than Aux-Net or smaller than learngene.

Firstly, we would have to emphasize that the primary focus of this paper is on initializing variable-sized models from learngene using well-trained transformations, rather than simply scaling model sizes beyond a certain threshold (*e.g.*, Aux-Net).

Secondly, it is important to note that we can initialize models whose sizes are independent of both learngene and Aux-Net. In our empirical setups, the size of Des-Nets ranges from 29.9M to 109.8M parameters, encompassing those of Aux-Net (78.6M), as shown in Figure 4.
Furthermore, we can initialize Des-Nets with parameters smaller than those of learngene, as well as Des-Nets deeper and wider than Aux-Net (H14-L15), utilizing well-trained learngene and transformations:

- For Des-Nets smaller than learngene, we directly select rows/columns from well-trained learngene matrices using our proposed selection strategies to initialize the target matrices of Des-Nets.

- For Des-Nets deeper and wider than Aux-Net, we first select rows/columns from well-trained transformation matrices using our proposed strategies, then integrate these selections into the original transformation matrices to achieve the desired size. Subsequently, we employ these expanded matrices to transform learngene for Des-Net initialization.

As shown in Figure I of our uploaded PDF, LeTra could flexibly and efficiently initialize *variable-sized* models that are independent of the sizes of learngene and Aux-Net.

Meta-review: The paper presents an improved learngene strategy that simultaneously trains a set of learnable transformations and compact learngene modules for subsequent model initialization. 

The paper initially received mixed reviews. While the reviewers considered the method well-motivated (JzzD, p972), novel (i78Y) and achieved good results on multiple datasets (JzzD, i78Y, 8CMe, p972), they also raised several major concerns on the paper: 1) lack of clarity in presentation and method details (8CMe, i78Y, p972); 2) restriction on the generated model sizes (JzzD, 8CMe, p972); 3) missing comparisons with related work, e.g., MatFormer (8CMe); 4) unfair or missing comparison settings (JzzD, i78Y, 8CMe,p972). 
The author's rebuttal provided further clarifications, additional comparisons, and empirical results, which addressed most of the concerns from Reviewer JzzD, 8CMe, p972, including 1)-4). During the post-rebuttal discussion, Reviewer 8CMe and p972 raised their ratings to the positive side, while Reviewer i78Y remained unconvinced by the experimental evaluation.

After reading the paper, the rebuttal and the reviewers' comments, the AC tended to agree with the majority of the reviewers. While the paper still has room for improvement, such as lacking experiments on larger benchmarks, the rebuttal enhanced the experimental validation with clear evidence and explanations. Overall, given the additional results, the effectiveness and improvement of this work over the prior art are convincingly demonstrated, and therefore the AC recommends it for acceptance.  The author should take into account the rebuttal and the reviewers' feedback, especially regarding the presentation and the added results, when preparing the final version.
Decision: Accept (poster)