Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers
Authors: Qiufeng Wang
Xu Yang
Fu Feng
Jing wang
Xin Geng
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision Transformer
Model Initialization
Learngene
Primary area: deep_learning_architectures

Abstract: In recent years, the merging of vast datasets with powerful computational resources has led to the emergence of large pre-trained models in the field of deep learning. However, the common practices often overgeneralize the applicability of these models, overlooking the task-specific resource constraints. To mitigate this issue, we propose \textbf{Cluster-Learngene}, which effectively clusters critical internal modules from a large ancestry model and then inherits them to initialize descendant models of elastic scales. Specifically, based on the density characteristics of attention heads, our method adaptively clusters attention heads of each layer and position-wise feed-forward networks (FFNs) in the ancestry model as the learngene. Moreover, we introduce priority weight-sharing and learnable parameter transformations that expand the learngene to initialize descendant models of elastic scales. Through extensive experimentation, we demonstrate that Cluster-Learngene not only is more efficient compared to other initialization methods but also customizes models of elastic scales according to downstream task resources.

Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers
Qiufeng Wangh?, Xu Yang! Fu Feng!?, "ongg Jing Wang
and Xin Gengl."
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Ministry of Education, China {qfwang, xuyang-palm, fufeng, wangjing91, xgeng)@seu. edu. cn
Abstract
In recent years, the merging of vast datasets with powerful computational resources has led to the emergence of large pre-trained models in the field of deep learning. However, the common practices often overgeneralize the applicability of these models, overlooking the task-specific resource constraints. To mitigate this issue, we propose Cluster-Learngenn, which effectively clusters critical internal modules from large ancestry model and then inherits them to initialize descendant models of elastic scales. Specifically, based on the density characteristics of attention heads, our method adaptively clusters attention heads of each layer and position- wise feed-forward networks (FFNs) in the ancestry model as the learngene. Moreover, we introduce priority weight-sharing and learnable parameter transformations that expand the learngene to initialize descendant models of clastic scales. Through extensive experimentation,
demonstrate that Cluster- Learnen- not only is more efficient compared to other initialization methods but also customizes models elastic scales according to downstream task resources.
Introduction
The evolution of deep learning has been profoundly influenced by the confluence of expansive data sources and robust computational capabilities. This collaboration has given rise to large pre-trained
foundation models [II JO 41 5 particularly those built upon the Transformer [47 [1 such as
the Vision Transformers (ViTs) [ The pre-trained foundation models, being widely deployed in various devices like smartphones edge devices, serve the initialization point [T6]
49) 50)
for diverse downstream applications. However, this dominant methodology implicitly assumes that cre-size-fit-aall approach, i.e., the whole foundation model is universally apt for every application, neglecting the specific resource constraints (e.g., memory, FLOPs, or latency) inherent to certain downstream tasks. Furthermore, not all tasks demand the full power of these extensive foundation models. This naturally raises pivotal question: Can we extract and harness the condensed part of these foundation models to achieve a harmonious balance between accuracy and resource efficiency?
To achieve the goal of efficiently initializing models, 49 50 introduce the innovative Learngene framework inspired by the observation of genes (cf. Fig. (a)). As showcased in Fig. (b), Learngene framework is designed in two pivotal stages. In the first stage, the significant knowledge is condensed from large ancestry model into more compact part termed as learngene. In the next stage. this
learngene is inherited to initialize the descendant models of elastic scales. Previous works 49 50
"Corresponding authors.
"The terms "foundation model" and "ancestry model," well as "downstream model" and "descendant model," are interchangeably utilized unless distinctions are explicitly mentioned.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Evalution
Ancestry Model
120
Mean Artertton Distance
1oo
Learrgene
Gane
Infiialzzo
Layer
(8)
(b)
(c)
(d)
Figure 1: (a) The ancestry of biological organisms condenses evolutionary information into
information-dent genes to initialize their diverse descendants [72. (b) The Learngene frame- work condenses the sigmificant knowledge from an ancestry model into a more compact part termed learngene and then inherited to initialize the descendant models of elastic scales. (c) The density of attention heads across the different layers of the ancestry model, which employs the DeiT-B [46]. (d) An illustration of our idea.
predominantly focus on extracting few integral layers as the learngene and manually stacking them with the randomly initialized layers.
However, such approaches struggle with inherent limitations: (i) The strategy of extracting certain integral layers overlooks the potential existence of learngene within these layers, leading to the preservation of many redundant weights. (ii) The approach of manually stacking the learngene with randomly initialized layers lacks the adaptability to scale the model, preventing the initialization of downstream models with custom dimensions.
As mentioned earlier, the Learngene framework aims to preserve the most generalizable part of the ancestry model while eliminating redundant weights that weaken representational capacity. Recent studies 42 [6)] have visualized the mean attention distance of ViTs, offering deeper insights into weight redundancy among attention heads across different layers. As illustrated in Fig. (c), the lower layers focus on both local and global perspectives, leading to more sparse density of attention heads. Conversely, the higher layers prioritize global context, resulting compact density. A notable observation is the repetitive functionality across many attention heads especially in the higher layers, which inevitably leads to weight redundancy.
Inspired by the above observation, we propose the Cluster-Learngene, an innovative approach that adaptively extracts internal modules in ViTs as the learngene, e.g., attention heads and position-wise feed-forward networks (FFNs). Firstly, to extract the cluster centroids of the attention heads (i.e., head centroids) across each layer of the ancestry model, we cluster the attention heads within each layer of the ancestry model based on their density characteristics. As depicted in Fig. (c-d), the attention heads in the first layer exhibit sparse density, resulting in five clusters, whereas the attention heads in the last layer cluster more compactly, forming single group. Furthermore, our method includes clustering FFNs by assessing the distance density of head centroids in adjacent layers of the ancestry model. Specifically, when the distance density of head centroids in adjacent layers is similar, we inherit the FFN from the shallower of these adjacent layers (i.e., FFN centroid) as the learngene. As illustrated in Fig. (c). the similar densities of attention heads in the 7-th and 8-th layers lead to proximate head centroids, enabling these layers to only require the inheritance from the 7th layer the FFN centroid. Overall, Cluster-Lean Learngene extracts critical parameters containing significant knowledge, as the extracted part represents attention heads/FFNs with similar semantics.
In the inheriting stage, to achieve the initialization of descendant models with varying number of attention heads, we adopt the priority weight-sharing. We start by ranking the head centroids based on the size of their respective clusters, arranging them in descending order of priority. Subsequently. by distributing these head centroids to initialize the attention heads of the descendant models. If the number of attention heads in a specific layer aligns perfectly with the number of centroids, they are evenly shared. However, if they fail to align perfectly, any remaining centroids are shared according to the remainder. Moreover, we apply learnable parameters to transform FFN centroids into multiple FFNs, thus enabling the initialization of descendant models with clastic scales.
we perform

---

Our contributions can be summarized as follows: (i) We propose the adaptive clustering to extract the head and FFN centroids as the learngene, ensuring the preservation of significant knowledge within the ancestry model. (ii) To achieve the initialization of descendant models, we introduce priority weight-sharing that favors head centroids within larger clusters and employ learnable parameters to transform the FFN centroids into multiple FFNs. (iii) Comprehensive experimental evaluations across datasets of different scales reveal that Learngen not only outperforms traditional initialization strategies but also stands toe-to-toe with more resource-demanding fine-tuning methodologies.
2 Related Work
Model Initialization:
Over the years, various initialization techniques have been proposed including the popular random initialization, Xavier initialization [3 and the Kaiming initialization [19].
Recently, the use of pre-trained foundation models has gained prominence as an initialization strategy before funent for specific tasks [0 41
57 37 SS
61 22 15
53 33 32 18 39 54 29 25
26 51
28
55 59
40
58 36
21 60). However, such an approach necessitates pre-training
separate models for each downstream task, which can lead to substantial computational resource consumption. In contrast, Cluster-Learngene presents a unique model initialization method that alleviates the need for multiple pre-training steps.
Density-based Clustering: Clustering aims to group similar data points together while separating dissimilar ones. A wide array of approaches has been explored, including partitioning-bant cluster- ing [14 27,, hierarchical clustering 35
52 and density- clustering 23 45 and so on. In particular, density-based clustering operates by taking into account the density relationships between data points to form clusters. Inspired by this, our method adopts a similar principle by assessing the density of attention heads to retain essential head centroids that represent significant knowledge.
3 Methodology
Learngene framework is primarily divided into two phases in Fig. [ the significant knowledge is condensed from an ancestry model into more compact part termed as learngene and then inherited to initialize the descendant models of assorted scales. Specifically, in phase 1, our Cluster-Learngeen selects mean attention distance as the density metric and uses it to cluster the attention heads each layer and FFNs in the ancestry model as the learngene, because they can effectively represent attention heads/FFNs with similar semantics. The pseudocode for this phase presented in Algorithm [ In phase 2, to initialize the descendant models, we employ priority weight-sharing of head centroids for varying number of attention heads as illustrated in Fig Fig[ leverage learnable parameters to expand the FFN centroids into multiple FFNs. Next, we briefly introduce some preliminaries related to ViTs.
3.1 Preliminary
In the ViT architecture, an input image is first divided into N non-overlapping patches, and each patch is linearly embedded into a flat vector of size D. The ViT encoder consists of alternating layers of multi-head self-attention (MSA) and FFN blocks. Let H denote the total number of heads in each layer. For the th head, the query Q E IRN xdv key Kr. E R xds, and value V2 E IR- xd, are linearly generated through leamed weight matrices W Dxd,, w E R Dxd,, and W E R Dxd, where de and d, are the dimensions of the key and value vectors, respectively. The SA mechanism of the i-th head can be represented as:
Qu.Kn
A h Attention(Q)) K,, Vi.) softmax
Vn-
(I)
Vik
MSA allows the model to jointly attend to information at different positions from different represen- tational subspaces at different positions:
MultiHead Q K, V) Concat( A
A'WWO,
(2)
where W E IR HDD is a learned weight matrix. Besides, the FFN can be formulated as: FFN(x) ReLU(xW, + b,) + b2
(3)
3

---

Algorithm 1: Adaptively Cluster for MSA
Input:
Number layers as I, set of attention heads in the 1 layer as Sj, radius as Eps, density threshold as Min/Ids, and distance function as Dist.
2 Output: The centroids of attention head in all clusters. Initialize all attention heads as unvisited and an empty list for clusters
for 1 = I....... L do
foreach attention head St do if a is not visited then Mark a as visited, Neighbor Hds all attention heads within Eps distance of end
if number of Neighbor Hds 2 dss then
10
C < new cluster, Add a to cluster C foreach attention head in Neighbor Hds do // Expand neighborhood
// Start new cluster
11
if b is not visited then
Mark b as visited
Neighbor H ds' all attention heads within Eps distance of b end
16
number of Neighbor H 2 MinHds then
Neighbor ds Neighbor Hs Neighbor Hds end
18
if b is not yet member of any cluster then Add to cluster C
20
end
21
Add C to the list of clusters
// Consolidate clusters
22
end
23
end
end
where x E NR is the input, W E Dxds, and W2 E Rd are the weight matrices, and b E Rdd and by E RD the bias vectors. djf y is the dimension of the intermediate layer.
3.2 Adaptively Learngene Clustering
3.2.1 Density metric on attention heads
Given a pre-trained ancestry model with L layers and H. attention heads per layer, let the attention weights for the th head in the 7th layer be denoted by the matrix (L.A) E NN The element A (L.A) represents the attention weight from position to position j. The distance matrix T E R Nx defined with the Euclidean distance between any two positions i and j in the sequence, given by Tú, (mx-aa))) -
+ (y: カو)² The mean attention distance for the 4 head in the 4h layer, encapsulating the weighted distance for each position across the sequence, is given by:
1
(Lh)) AJ
T
(4)
To deduce this metric for every head across all layers, iterate the above computation for every I E {1...., L} and {1...., Ha ). As depicted in Fig. and Appendix A. while the lower layers simultaneously attend to both local and global features, leading to a more dispersed distribution of attention heads, the higher layers predominantly focus on global aspects, causing tighter concen- tration of attention heads. As a result, there is significant overlap in the semantic representations among many attention heads, especially in the higher layers, leading to weight redundancy.
3.2.2 Cluster for MSA
Motivated by the empirical observations, we extract cluster centroids 43 in ViTs the learngene inherited into the descendant models, thus aggregating similar semantics
attention heads

---

into the head centroids. To realize this, we select d as a density metric for adaptively clustering the attention heads of the ancestry model each layer, without setting the number of clusters in advance. This realization prompts the formulation of the definitions and lemmas, which scaffold our adaptive clustering approach.
Definition Sips-neighhoroood of an attention head). The Eps-neighborhood of an attention head a, denoted as N FFps(a), is defined as: Naps(a) = {b S Dist(a, b) S Eps}, where
Dist(a, b) = d - d(b) denotes the difference in values between attention heads a and b.
Our approach could require for each head in cluster that there are least Minimum number of Heads (MinHds) ds) in an Eps-neigbborhood of that head.
Transitioning from the neighborhood concept, attention head considered density-rrachaabl from another head with respect to Eps and dinHds if there is a sequence of heads .i..... an such that a1 b, an a, and each head in this sequence lies within the Epss-neighborhood of its preceding head.
Definition 2
Definition connstte-conectedd). Broadening our purview, attention heads a and are labeled density-connectt with respect Eps and dss ds if there exists an intermediary head from which both a and are censity-reachalee
Considering all attention heads in layer I as St, a cluster C' based on Eps and MinHds is identified as a non-empty subset St that satisfies the conditions: (i) Maximality: For any heads and in from a dictated by Eps and MinHds.
the sequence, if a resides within C' and b is density-reachute then b seamlessly becomes part C. (ii) Connectivity: Within C, each pairing a, maintains a density-comnection, anchored by Eps and dss
Therefore, upon satisfying these two conditions, we cluster all attention heads of each layer into different lists of clusters. The pseudo-code for this process is summarized in Algorithm | For each cluster in the list of clusters, we select the attention head Cig closest to the cluster centroid as the learngene. This is because it effectively represents the functionality of all attention heads in the cluster. Formally, the formula can be expressed follows:
C arg min cEC
d - -C)
(5)
where d(()) is computed as the average d across all attention heads within this cluster C. The lemma presented below is pivotal in substantiating the correctness of our clustering algorithm.
Lemma 1 Presuming an attention head a belongs to St and satisfies the condition |N (a)) > dss Then, the set O {o E St and is cersity-reachable from a with respect to Eps and MinHds] collectively shapes a cluster.
3.2.3 Cluster for FFN
Based on the distances between attention heads discussed earlier, further cluster FFNs as the learngene. For adjacent layers sharing the same number Ne of head centroids, a key criterion is established: if the average distance across all head centroids between these layers is less than the threshold e, it suggests redundant similarity in their representational capabilities. This criterion is formulated as:
No Dist (Ci,k, C4+1,) <.
(6)
NG
Under this condition, we enhance model efficiency by preserving only the FFN from the shallower of these adjacent layers as the FFN centroid, thereby maintaining essential functionality while eliminating redundancy. Moreover, we also select those FFNs that do not fit this established criterion as the learngene because they exhibit representational capacities independent of adjacent layers.
3.3 Learngene Inheriting
3.3.1 Expanding head centroids with priority weight-sharing
Since the head centroids have been extracted as the learngene, we further expand them for initializing the descendant models with varying number of attention heads. For an ancestry model with L
5

---

Layer1
Weight Share
Layer
Weight Share
Expand
Expand
Head Centrolds
Multi head eelffattetion
Centroids
Mulli head
selfattention
Figure 2: Illustration of priority weight-sharing. The darker the color, the larger the cluster size
associated with the head centroid.
layers, the t layer has or head centroids of weight A Al.11, ..... A (l.e)) Importantly, the head centroids at each layer are sorted in descending order based on the size of their respective cluster, i.., centroids representing more attention heads in the ancestry model are ranked higher. These head centroids condense significant knowledge and ensure the initialization of descendant models without performance degradation. Assume the descendant model has Ha attention heads for each layer. To achieve the desired expansion of heads to initialize the descendant models, we adopt the priority weight-sharing and Fig. illustrates two scenarios:
When Ha is divisible by cr: The weights of head centroids are shared Ha times in sequence. For instance, centroids of weights A (L,1) and A (L.2) each share their weights across four attention heads, which are then directly assigned to eight attention heads of the descendant model in layer L.
When Ha is not divisible by e;: The weights of the head centroids are sequentially shared Ha
times, followed by appending A (1.1), A H. mod 4) the end. As illustration, we share the centroids of weights A (1.1), A (1.3) once and then append A (2.1) ,.... A (1,3), thus initializing eight attention heads of the descendant model in the first layer.
Ha to denote For the attention heads in the descendant models, we introduce the hyperparameter w Ha the factor by which the number of attention heads is reduced compared to the ancestry model. In addition to uniformly setting the number of attention heads for each layer with the hyperparameter w, we also explore two other possibilities in Appendix A.7: incrementing and decrementing the count of attention heads with layer depth. According to the adjustments in the number of attention heads, the weights Wo of the projection layer are also proportionally pruned and then inherited by the descendant models.
3.3.2 Expanding FFN
As outlined in Section we retain the shallowest FFN determined by distance- clustering, For this FFN centroid, introduce learnable parameters to transform it into multiple FFNs, facilitating the initialization of descendant models at elastic scales. This process is summarized in the formula:
FFN, (x) = (ReLU((x(W) + b,) b.)W.)W. W2 + ba) W..
Here, W. denotes the newly introduced learnable parameters to expand the (th FFN. These parame- ters are designed for minimal learning overhead, yet they are highly effective in quickly adapting descendant models to downstream tasks.
4 Experiments
4.1 Experimental Setting
Datasets. To condense the learngene, we employ the ImageNet-IK,, collection of 1.2 million training images and 50,000 validation images distributed across 1,000 classes part of the ILSVRC2012
competition 99. After initializing the descendant models with the learngene, we proceed to fine-tune these models diverse downstream tasks. These tasks include iNaturalist-2019 455, 1ood101 [4] Oxford Flowers [38], Stanford Cars [12), CIFAR-10 [24), CIFAR- 100 (24), CUB-200-24 2010 48 For
detailed dataset descriptions, Appendix A.2.
Please Appendix A.3 for details.

---

Training settings. During the learngene clustering, We set Eps, 10, MinHds = ensuring that each attention head included unique cluster. the learngene inheriting phase, we train the descendant models on downstream tasks for 500 epochs, including a 10-epoch warm-up period, except for NNaturalist-2019 where we train for 100 epochs with a 5-epoch warm-up. The initial learning rate is set to 5 x 10-1 for most tasks, except for Stanford Cars where it is 5 x 10 and weight decay of 0.05.
Architectures. Both the ancestry model and descendant models are variants derived from DeiT 46 In terms of width, there are three types DeiT: Tiny, Small, and Base. Furthermore, as detailed in Section 4.2.4 we implement experiments on Swin Transformer [30] to demonstrate the applicabilityy of our method across various backbones. The learnable parameters in Eqns. (7) are implemented through a nonlinear mapping such as neural network with the rectified linear units (RcLU).
4.2 Main Results of Model Initialization
In this section, we validate the capabilities of Cluster-Leamgene in efficiently initializing models and measure model performance with Top-1 accuracy.
4.2.1 Initializing Descendant Models of Elastic Scales
As illustrated Fig- 3 for the Tiny-scale descen- dant models with only 32 attention heads, Cluster- Learngene outperforms ImageNet. Moreover, the performance of Cluster- Learngene improves with an increase the num- ber of attention heads and FFNs. This shows that Cluster- Leasnere can adaptively initialize the de- scendant models with varying number of attention heads and FFNs, catering to downstream resource constraints. In contrast, Pretraining-Finetumii quires retraining for each model variant, signifi- cantly raising storage and training costs, particularly when dealing with models of clastic scales. There- fore, our method resolves the limitations of the one- size-fits-al approach seen in
Table 1: Comparisons of performance ImageNet-1K K between models trained From- Scratch with 100 epochs and those initial- ized via Cluster-Learngene fine-tuned for 50 epochs.
Cetraaniing-Fnettunin
on
Model
Ha
La
Params (M)
FLOPs (G)
From-Scratch
Ours
1.3
0.3
50.06
52.73
2
9
1.9
0.4
54.64
59
2.5
0.5
re-
Tiny
6
3
0.6
58.16
59.58
0.9
12
5.7
1.2
61.44
70.28
7.3
1.4
63.17
70.90
9.5
1.9
64.25
73,20
Small
11.4
2.3
61.91
67.72
16,8
3.4
67.02
Moreover, we initialize 22 descendant models ImageNet-IK, each with different configurations, such the number of attention heads Ha per layer and the quantity of FFNs La in descendant mod- els. As shown in Tab. [ Cluster-lLernngene swiftly initializes models of varying scales and proves com- petitive in overall performance. For example, the Small-scale descendant model with Ha = 6 and
4.5
29,9
5.9
70.32
70.41
26.9
7.4
72.04
72.13
41
8.8
73.73
73.95
Base
58.2
11.7
76.14
76.90
65.3
13.1
76.46
77.11
14.6
77.99
79.5
16
77.03
78.82
12
86.6
17.5
77.22
79.50
La 12 illustrates this point. Cluster- Learngee not only outperforms From-Seratch by 9.87 % but also manages to reduce 2x training times.
4.2.2 Efficiently Initializing Large Models on ImageNet
The results in Fig. reveal that Cluster-Learngene outperforms inherited parameters. For example, in the case of Small-scale descendant models, Cluster-Learngenn inherits only 15. parameters and attains a performance of 78.43%, while Eetraaning--Fintuninn with 18.0M parameters achieves less than 75%. These results demonstrate the advanced initialization ability of Cluster-Leamgenee as the clustered learngene retains the critical generalizable knowledge.
with fewer
4.2.3 Initialization Results on Different Downstream Tasks
We conduct comparative analysis of approach for initializing descendant or downstream models, as follows: (i) Frtraaninnn-imetuninggg This approach pre-trains DeiT on ImageNet and
Please Appendix A.6 for more analysis.
7

---

(a) Tiny
(b) Small
(c) Base
Accuracy
70
Lusteei L12
77.5
Accuracy
80
70
75.0
Pretraining Finetunng
76
100
200
300
100
200 300
100
200 300
Total number of attention heads
Total number of attention heads Total number of attention heads
Figure 3: Initializing descendant models of elastic scales. "L.6/9/12" denote descendant models
with 6, 9, and 12 layers, respectively. For fair comparison, the downstream models in Pretraining- Finetuning inherit parameters from 12 layers of the pre-trained model, with the inherited number of attention heads matching those in Cluster-Learngena We fine-tune 50 epochs for all models. In (a). the hyperparameter w takes values ranging from 1 to 8 (i.e., the number of attention heads in descendant models is eight times that of the ancestry model). In (b), w ranges from 2 to 4. Continuing this pattern, in (c), w ranges from a maximum of 4 to a minimum of 2'
15
Tiny
75
30
78,43
(b) Small
80
(c) Base
85
iPorams (M)
10
70
65,78
4.5
64,53
4.6
64, 44
4.5
6% 70
10p Accii
P (N)
74,48
74,14
74.06
70
79
Aceiio
Perams (M)
70
79.5
76,57
71.3
78,14
71.3
77,85
21.3
00
75
70m
65/2
Front Middle Last Figure 4: Efficiently initializing large models on ImageNet. "Front/midddl//last" refer to inheriting parameters from the front, middle, or last 10 layers of pretrained model to initialize 12-layer descendant models. All approaches are fine-tuned for 50 epochs. "I-Params" means the number of Inherited parameters in the downstream/ddsscedann models, measured in MB.
Durs
Durs
Front
Middle
Last
60
Ours
front
Middle
Last
60
Dell-Smaa
(b) Dei--ooee
Sodn-Small
Swinn-use
80.0
37.5
1r Speeduc
80.0
Bix Specdks
Accommoy
$2.5
30.0
Nal
80.0
3,00 Spoedup
75.0
Frstiirting Iintuniing
77.5
37.5
100
2010
300
100
200
300
100
200
300
100
200
300
Epoch
Spach
Epech
Epach
Figure 5: Faster convergence. Different points represent results for varying epochs and the hyperpa- rameter w is set to 1.0 for our method.
Table 2: DeiT-Small Results on downstream datasets. t represents the performance improvement achieved by (luster-Leamgentt when compared to the best method excluding Pretraining- Tretraining-Finetunnnng All results are derived from the 6-layer downstream models.
Method
I-Params
iNa9-2019
Food-
Flowers
Cars
CIFAR-10
CIFAR-IU
CUB-200
CourainingFFinctunn
10.5
68.48
87.8
91.13
86.81
97.59
84.43
78.13
Srom-Scratch
50.79
74.64
72.91
71.63
92.49
73.32
62.75
Heuristie- Learngee-
5.6
53.21
77.09
82.84
81.52
93.12
78.13
72.64
59.83
86.37
85.01
93.67
75.98
70.28
Auto Leangene
10.5
59.92
80.25
87.02
84.98
93.58
79.49
73.31
Cluster- Learngen
7.5
71.09(111.17)
89.53)77.74)
92.31(15.29)
(1.87))4.86)
97.7%(14.12)
85.38((5.89)
75.98(12.67)
subsequently fine-tuned the entire model on downstream tasks. (ii) From-Scratch: We commence with a randomly initialized DeiT model on the downstream datasets. (iii) Heuristic-Leamgenne 44
This strategy involves extracting the last six layers from a DeïT model pre-trained on ImageNet and then stacking them with randomly initialized lower layers to construct a new model. (iv)
Weight-Transffrmatior 663] This method employs Weight Transformation to pre-train DeiT on ImageNet, followed by fine-tuning the entire model to adapt it to specific downstream tasks. (v)
Auto-Lcamgene [50]: The first six layers are extracted from the DeiT and then stacked with randomly initialized higher layers to initialize the descendant models.
As illustrated in Tab. 2 our Cluster-Learngene significantly outperforms both From-Scratch and Weight-Traie Transformation. When compared to other Learngene methods, such as Auto-Learneen.. Cluster-Learngens exceeds by 11.17% on the Naturalist-2019 (iN9t.2019). These results highlight the superior capability of Learner- in efficiently initializing descendant models. Moreover, on six datasets, the performance of Cluster-Learngeene outperforms that of Pretraining--Fietuunng where the entire model is fine-tuned. This phenomenon can be attributed to the more universally significant knowledge within learngene, allowing it to adapt effectively to various downstream tasks.
8

---

4.2.4 Faster Convergence
We provide a detailed comparison of training efficiency between approach and From-Scratch on ImageNet. As shown in Fig. (a), Cluster-Learngene requires only 4.0 x less training overhead compared to From Scratch Small-scale descendant models. A key advantage of our approach is that descendant models initialized with the learngene achieve faster convergence, owing to superior initialization point.
4.2.5 Higher Data Efficiency
Table 3: Initialization of descendant mod- els with diverse training samples. The sym- bol denotes the performance gap between our approach and the From-Scratch method. Cluster-Learngenn initializes the descendant model over 50 training epochs. In contrast, Prom-Scratch results are achieved after 300 training epochs.
We further conduct experiments on Base-scale de- scendant models over different percentages of train- ing data from ImageNet- IK (IN-IK). As shown in Tab. while our method does not outperform the From-Scratch on the entire dataset, its performance exhibits greater stability the amount of training data decreases. For instance, with only 25% of the training data, Cluster- Leanggn- outperforms From- Scratch by 7.09%. % while requiring only of the training cost. This higher data efficiency of our method is attributed to the significant knowledge within the learngene, which helps descendant mod- els mitigate overfitting, especially in scenarios with limited data.
Training From-Semich
Cluster- Leamgene
100% IN-IK IN-1K 25% IN-IK
81.80
78.43 76.41((1.71)
65.73
72.82("7.099
4.3 Analysis and Ablation
In this section, we provide further analysis and ablation of Cluster-Learnngne. Unless otherwise specified, conduct experiments on CIFAR-100 and use Small-scale DeiT the ancestry model.
4.3.1 Comparison of the Clustering Method
Additionally, we compare the results of k-means clustering 20 of attention heads with cluster centroids (k) set at 1, 2, and 3. Tab. 4 shows that Cluster-Learneene not only outperforms in clustering efficiency but also adaptively adjusts the count of cluster centroids for each model layer, unlike k-means that requires predefined the numbers of cluster centroids.
Table 4: Comparison of the clustering method. All results are from the 6-layer downstream models.
k-means, k 1 k-means, k = k-means, k = 3 Ours
80.12
81.02
81.41
85.38
4.3.2 Comparison of Priority Weight-sharing
In Table our priority weight-sharing is compared against three alternative weight-sharing methods: following the original sequence of heads, by ascending d of heads, and by descending d of heads. The results validate the effectiveness of priority weight-sharing, as our method accounts for the fact that the larger the cluster a head belongs to, the richer the critical knowledge represents.
Table 5: Comparison of priority weight-sharing, All results are derived from the 6-layer down- stream models.
Origin
Increasing d
Decreasing d
Ours
83.98
84.24
84.63
85.38
4.3.3 Qualitative Visualization
We illustrate attention representations in Fig. to explain which significant knowledge inherited by learngene. To clarify the visualization, we apply power exponent of 0.25. In the first layer of the ancestry model, a head centroid is clustered from heads 1, 2, 4, and 5 to initiate the head I in
Please Appendix A.9 and A.10 for more visualization.

---

First layer alncestr
Head
Head
Head
Head 4
Head 5
Head 6
model
100
100
100
100
100
100
0.8
04
Head
Head
Head
Head
First layer descendant
Last layer of descendant model
100
100
100
100
model
100
100
100
Head
Head
Head
Head
Head
Last byee Maanestryy model
100
100
100
100
100
100
100
100
Figure 6: Visualization of attention representations (197 197). We perform the following normalization operation on all attention heads A of the ancestry model and descendant model:
AA..)
The descendant model is trained for 50 epochs, and is set to
255
the descendant model, and so forth. Then, nright--haring is applied to expand head centroids, e.g., sharing twice to initialize the descendant model.
In the first layer, heads 1, 2, 4, and 5 with similar semantics form the largest cluster, mainly focusing on attention patterns along the main diagonal. Additionally, heads 3 and present distinct semantics, with head 4 reflecting more abstract and high-level features akin to the final layer. Thus, the first learngene layer integrates three principal representation patterns from the ancestry model. In contrast, the representations in the final layer of the ancestry model exhibit significant repetition, leading to clustering of a single-head centroid for initializing the attention heads of the descendant model.
5 Conclusion
We propose Cluster-Learngenn to adaptively cluster attention heads, extracting head centroids and FFN centroids as the learngene. Subsequently, we adopt the priority weight-sharing of head centroids for varying number of attention heads and leverage learnable parameters expand the FFN centroids into multiple FFNs, enabling adaptation to diverse downstream resource constraints. Extensive experiments validate the efficiency and scalability of our initialization method.
Acknowledgments and Disclosure of Funding
This research was supported by the National Science Foundation of China (62125602, 62076063), the Key Program of Jiangsu Science Foundation (BK20243012), the Fundamental Research Funds for the Central Universities (2242024k30035) and the Big Data Computing Center of Southeast University.
References
[I] Mohiuddin Ahmed, Raihan Seraj. and Syed Mohammed Shamsul Islam. The k-means algorithm: comprehensive survey and performance evaluation. Electronics, 129): 2020.
[2] Devansh Arpit, Victor Campos, and Yoshua Bengio. How to initialize your network? robust initialization for weightnorm & resnets. Advances in Neural Information Processing Systems, 32, 2019.
[3] Panthadeep Bhattacharjee and Pinaki Mitra. A survey of density based clustering algorithms. Frontiers of Computer Science, 15:1-27. 2021.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-100 101 mil-mining discriminative components with random forests. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
10

---

[5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee. Yin Tat Lee, Yuanzhi Li. Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303.12712. 2023.
[6] Adil Abdu Bushra and Gangman Yi. Comparative analysis review of pioneering dbscan and successive density-based clustering algorithms. IEFE Access, 9:87918-87935, 2021.
[7] Shiming Chen, Ziming Hong, Guo-Sen Xie, Wenhan Yang. Qinmu Peng, Kai Wang. Jian Zhao, and Xinge You. Msdn: Mutually semantic distillation network for zero-shot learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7612-7621, June 2022.
[8] Vincent Cohen-Addad, Varun Kanade, Frederik Mallmann- Trenn, and Claire Mathieu. Hierarchical clustering: Objective functions and algorithms. Journal of the ACM (JACM), 66(4):1-42, 2019.
[9] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 245.255. leee, 2009.
[10] Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume oLgg and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties.
[11] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[12] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine-grained car detection for visual census estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[13] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings. 2010.
[14] Greg Hamerly and Charles Elkan. Learning the k in k-means. Advances in neural information processing systems, 16. 2003.
[15] Yizeng Han, Gao Huang, Shiji Song, Le Yang. Honghui Wang, and Yulin Wang. Dynamic neural networks:
A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11))7433-77) 2021.
[16] Boris Hanin and David Rolnick, How to start training: The effect of initialization and architecture. Advances in Neural Information Processing Systems, 31, 2018.
[17] Uri Hasson, Samuel A Nastase, and Ariel Goldstein. Direct fit to nature: an evolutionary perspective on biological and artificial neural networks. Neuron, 105(3)441--34, 2020.
[18] Haoyu He, Zizheng Pan, Jing Liu, Jianfei Cai, and Bohan Zhuang, Efficient stitchable task adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28555 28565, June 2024.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Rea, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Abiodun M Ikotun, Absalom E Ezugwu, Laith Abualigah, Belal Abuhaija, and Jia Heming. K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big data. Information Sciences, 622:1 78-210, 2023.
[21] ZHAO Yongxin ZHANG Shenglin GONG Zican JI Yuhe, HAN Jing. Log anomaly detection through gpt-2 for large scale systems. ZIE Communications, 21) 2023.
[22] Runhao Jiang, Jie Zhang, Rui Yan, and Huajin Tang. Few-Shot Learning in Spiking Neural Networks by Multi-Timescal Optmizztiinn Neural Computation, 3339):2439-247,, 08 2021. ISSN 0899-7667. doi: 10. 10.1ne2/neco 01423.
[23] HlassPPtee Kriegel, Peer Kröger, Jörg Sander, and Arthur Zimek. Density- Density-based clustering. Wiley interdisciplinary reviews: data mining and knowledge discovery, 1(3):231-22 12400 2011.
11

---

[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] Lujun Li. Peijie, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wisd wise pruning of large language models. In NeulPS, 2024.
[26] Wei Li, Lujun Li, Mark Loe, and Shengjic Sun. Als: Adaptive layer sparsity for large language models via activation correlation assessment. In NeulPS, 2024.
[27] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Peng. Contrastive clustering, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 8547-8555, 2021.
[28] Shuxia Lin, Miaosen Zhang, Ruiming Chen, Xu Yang, Qiufeng Wang, and Xin Geng. Linearly decomposing and recomposing vision transformers for diverse-scale models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[29] Biao Liu, Ning Xu, and Xin Geng. Progressively selective label enhancement for language model alignment. arXiv preprint arXiv:2408. 02599, 2024.
[30] Ze Liu, Yutong Lin, Yuc Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[31] WANG Jiabo LIU Yang LUO Wenjian LIU Qinbe, JIN Zhihao. Msra-fed: communicationn-cfiiceen federated learning method based on model split and representation aggregate. ZTE Commanications, 20 (3):35, 2022.
[32] Xuran Meng and Jianfeng Yao. Impact of classification difficulty on the weight matrices spectra in deep learning and application carly-stopping. J. Mach. Learn. Res., 24(1), March 2024. ISSN 1532-4435.
[33] Xuran Meng, Difan Zou, and Yuan Cao. Benign overfitting in two-layer relu convolutional neural networks for xor data. In Forty-first International Conference on Machine Learning.
[34] Paul Michel, Omer Levy, and Graham Neubig, Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
[35] Fionn Murtagh and Pedro Contreras. Algorithms for hierarchical clustering: an overview. Wiley Interdisci- plinary Reviews: Data Mining and Knowledge Discovery, 2(1):86-9) 97,9, 2012.
[36] ZOU Xiaojing DOU Yutao Albert Y. ZOMAYA NAN Yucen, FANG Minghao. collaborative medical diagnosis system without sharing patient data. ZTE Communicationns 20(3):3, 2022.
[37] Zanlin Ni, Yulin Wang, Jiangwei Yu, Haojun Jiang, Yu Cao, and Gao Huang. Deep incubation: Training large models by divide-and-conquee conquurrin.. 2022.
[38] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
[39] Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Stitchable neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (6102-16111 June 2023.
[40] Yingzhe Peng, Chenduo Hao, Xu Yang, Jiawei Peng, Xinting Hu, and Xin Geng. Learnable in-context vector for visual question answering. arXiv preprint orxiv:2406. 13185, 2024.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Mcila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 2021.
[42] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Prucessing Systems, 14:12116-22128, 2021.
[43] Erich Schubert, Jörg Sander, Martin Ester, Hans Peter Kriegel, and Xiaowei Xu. Dbscan revisited, revisited: why and how you should (still) use dbscan. ACM Transactions on Database Systems (TODS), 42(3):1-23 221, 2017.
12

---

[44] Chongjie Si. Zhiyi Shi, Shifan Zhang, Xiaokang Yang. Hanspeter Pfister, and Wei Shen. Unleashing the power of task-specific directions parameter eflicient fine-tuning, arXiv preprint urXiv:2409.01035, 2024.
[45] Kiat Chuan Tan, Yulong Liu, Barbara Ambrose, Melissa Tulig, and Serge Belongie. The herbarium challenge 2019 dataset. arXiv preprint (rXX:::996005372 2019.
[46] Hugo Touvron, Matthieu Cord, Mathijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. International conference on machine learning, pages 10347-10351 PMLR, 2021.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech- birds-200-2011 dataset. 2011.
[49] Qiu-Feng Wang, Xin Geng, Shu-Xia Lin, Shi- Yu Xia, Lei Qi, and Ning Xu. Learngene: From open-world to your learning task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8557-8565, 2022.
[50] Qiufeng Wang, Xu Yang. Shuxia Lin, and Xin Geng. Leamgene: Inheriting condensed knowledge from the ancestry model to descendant models. arXiv preprint arXiv:2305.0223 022799, 2023.
[51] Qiufeng Wang. Xu Yang. Haokun Chen, and Xin Geng. Vision transformers as probabilistic expansion from learngene. In Forty-first International Conference on Machine Learning, 2024.
[52] Yu Wang, Xinjie Yao, Pengfei Zhu, Weihao Li, Meng Cao, and Qinghua Hu. Integrated Heterogeneous Graph Attention Network for Incomplete Multi-modal Clustering. International Journal of Computer Vision, 132( September 2024, ISSN 1573-1405. doi: 10.1007/s| 12611122--2006666
[53] Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x 16 words: Dynamic transformers for efficient image recognition. Advances neural information processing systems, 34:1 1960-11973,, 2021.
[54] Yongrong Wu, Jingyu Lin, Houjin Chen, Dinghao Chen, Lxqing Yang, and Jianhing Xiahou. A graph- transformer network for scene text detection. International Conference on Intelligent Computing, pages 680-690. Springer, 2023.
[55] Shiyu Xia, Miaosen Zhang, Xu Yang. Ruiming Chen, Haokun Chen, and Xin Geng. Transformer as linear expansion of learngene. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16014 16022, 2024.
[56] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14475-14485, 2023.
[57] Xingyi Yang. Daquan Zhou, Songhua Liu. Jingwen Ye, and Xinchao Wang. Deep model reassembly. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 25739-25753. Curran Associates, Inc., 2022.
[58] Xu Yang, Yingzhe Peng, Haoxuan Ma, Shuo Xu, Chi Zhang, Yucheng Han, and Hanwang Zhang. Lever Im: Configuring in-context sequence to lever large vision language models. arXiv e-prints, pages arXiv-2312, 2023.
[59] Yao Yao, Zuchao Li, and Hai Zhao. arXiv:2405. 12528, 2024.
Sirllm: Streaming infinite retentive 1lm.
arXiv preprint
[60] Hua Yuan, Yu Shi, Ning Xu, Xu Yang, Xin Geng, and Yong Rui. Learning from biased soft labels. Advances in Neural Information Processing Systems, 36, 2023.
[61] Wei Yuan, Hongzhi Yin, Ticke He, Tong Chen, Qiufeng Wang, and Lizhen Cui. Unified question generation with continual lifelong lcarning. In Proceedings of the ACM Web Conference 2022, WWW '22, page 871-881, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450399960 doi: 10. 10.1145/3485447.3511 4411555544..511999 1930.
[62] Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, 10(1):1-7, 2019.
13

---

[63] Jinnian Zhang, Houwen Peng. Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVE Conference on Computer Vixion and Puttern Recognition, pages 12145-12154, 2022.
[64] Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4388 4403, 2021.
[65] Pengfei Zhu, Xinjie Yao, Yu Wang, Binyuan Hui, Dawei Du and Qinghua Hu, Multi-view deep subspace clustering networks, arXiv preprint arXiv:I900 01978, 2019.
14

---

Dataset
Total
#Training
#Validation
#Testing
#Classes
Oxford Flowers
38
8,189
1,020
1,020
6,149
102
CUB-200-2011
48
11,788
5,394
600
5,794
200
Stanford Cars
12
16,185
7.329
815
8,041
196
CIFAR10
24]
65,000
50,000
5,000
10,000
10
CIFAR100
24
65,000
50,000
5,000
10,000
100
Food101
4
101,000
75,750
25,250
101
iNaturalist-2011
268,243
265,213
3030
1010
Table 6: Characteristics of the downstream datasets
A Appendix supplemental material
A.1 Mean Attention Distance in DefT-S and DeiT-Ti
Fig. illustrates the mean attention distance for two other variants of DeiT. In both variants, the lower layers exhibit a dual focus on both local and global aspects, resulting in a relatively sparse distribution of attention heads. Conversely, the higher layers prioritize the global context, leading to a more compact distribution of attention heads. Importantly, many attention heads in these layers exhibit repetitive functionality, contributing weight redundancy. In the process of FFN clustering, the shallowest layers are preserved as the FFN centroids across different configurations: in DeiT--Tiny, layers 110,11,12); and in DeiT-Base, to
this applies layers (2,3) and (11,12); in DeiT-Small layers (7,8) and 110,11,12).
(a) Small
(b) Tiny
120
100
Mean Attention Distance
Mean Attention Distance
80
80
60
60
layer
40
layer
40
layer
layer 5
layer
Tyye
layer
20
layer
layer
20
layer 10
layer
layer 11
2
5
10
Network depth Figure 7: The distribution density of attention heads across the different layers of the ancestry model, which employs the DeiT-S and DeiT-Ti [46].
Network depth
A.2
Downstream Datasets
Tab.
presents the details of all downstream tasks.
A.3
Projection Layer
According to adjustments in the number of attention heads, the weights Wo of the projection layer also proportionally pruned expanded with the hyperparameter and then inherited by
15

---

(a) DeiT-Tiny
(b) 5win-Tiny
Accuracy (%)
72.5
Accuracy (%)
4.0% Speedup
3.3x Speedup
80
70.0
Cluster-Leamgene
From-Scratch
67.5
Pretraining Finetuning
75
100
200
300
100
200
300
Epoch
Epoch
Figure 8: Faster convergence. Different points represent results for varying epochs and the hyperpa- rameter w is set to 1.0 for our method.
the descendant models. Additionally, we directly inherit the weights of layer normalization, patch embeddings, and position embeddings in the ancestry model, which constitute only small fraction of all weights.
A.4 Faster Convergence
We provide detailed comparison of training efficiency ImageNet for the Tiny-scale descendant models. As shown Fig. [ (b). Cluster-Learngene requires only 4.0 less training overhead compared to From Scratch Small-scale descendant models. A key advantage of approach is that descendant models initialized with the learngene achieve faster convergence, owing to superior initialization point.
A.5
Ablation on the Selection of Head Centroids
We conduct ablation assess whether inheriting parameters from the nearest module the cluster centroid or averaging parameters after clustering heads and FFNs is more effective. Table 7 shows Cluster-Leanngene excelling when inheriting from the closest heads/FFNs the centroid, a logical approach as these modules aptly represent similar semantics within cluster.
MSA
FFN
Acc (%)
average
average
85.12
argmin
average
84.29
average argmin
84.76
argmin argmin
85.38
Table 7: Ablation on the selection of head centroids. All results are derived from the 6-layer
downstream models, conduct experiments CIFAR-100 and use DeiT-Small the ancestry model.
A.6
Ablation on
Table g shows the results for different values of E. = 1 implies no FFN clustering, potentially causing negative transfer. & 100 means clustering all FFNs in adjacent layers with identical head centroid counts, resulting in an excessive cluster of FFNs and subsequent degradation in the performance initialized descendant models. Our Cluster-Learngenn (6 10) strikes good balance between these issues.
8
6
10
8
100
85.27
85.38
80.45
Table 8: Ablation on e. All results are derived from the 6-layer downstream models.
16

---

Model
Decrementing
Incrementing
Tiny
76.56
78.01
Small
79.47
81.29
Base
80.18
81.65
Table 9: Increment decrement the count of attention heads. "Decrementing" denotes halving the number of attention heads in the first four layers, reducing them by quarter in the middle four layers, and maintaining them in the last four layers relative to the ancestry model. Conversely, "Incrementing" represents the opposite pattern.
A.7 Variation in the Count of Attention Head with Model Depth
Tab. presents scenarios where the number of attention heads varies across different layers. Across all descendant model configurations, "Incrementing" consistently outperforms "Decrementing" by margin of 1.45% terms of accuracy. These findings align with previous research [34 30]. which suggests that setting more attention heads in higher layers can assist these layers in learning more abstract and high-level feature representations.
A.8 Initializing Descendant Models
of Elastic Scales
As illustrated in Fig. 3 Cluster-learngene incurs a total 300 + 50 10 = 800 epochs. In contrast, pre-trained models require retraining fine-tuning. Considering only the training expense of new models, training 10 such models would require at least 300 * 10 3000 epochs. Therefore, algorithm can reduce the computational cost by at least 3x. This efficiency demonstrates how our method overcomes the one-size-fits-al approach.
model of elastic scale for 300 epochs each, followed by
limitation inherent in the
A.9 Limitations
The performance of descendant models within Cluster-Learngene heavily depends on the quality of the ancestry model. If the ancestry model harbors inherent limitations biases, these issues may propagate and even amplify within the descendant models, potentially compromising their effectiveness. Therefore, ensuring the robustness and faireess of the ancestry model is paramount for maintaining the performance of descendant models in Cluster-Learngene.
17

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
Please see Section 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: Please see Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuia wwwiicccguie//eeeeeeeeeis CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpe:::////iipp// httpr/iss.....b.....hhhcccood..d
Answer: [Yes]
Justification: Please Appendix.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see the Experimtent Section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whw.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
22

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "wDS9UrVe5y", "reviewer": "Reviewer_GgZj", "summary": "This paper introduces Cluster-Learngene, a weight initialization method for initializing downstream models of various sizes with pre-trained models. The proposed method is based on Learngene and includes the following two improvements: MSA and FFN centroid, which extract critical parameters and reduce redundancy; priority weight-sharing, which is able to initialize downstream models with varying number of attention heads.", "questions": "(1)\tAfter Adaptively Learngene Clustering, the number of layers obtained is variable. How to deal with that if it is not equal to the number of layers in the downstream model?\n\n(2)\tIn priority weight-sharing (Figure 2), what if the number of head centroids is less than the number of head in the downstream model?\n\n(3)\tAre all parameters of the downstream model (inherited or not) updated (or fixed) during training?\n\n(4)\tIn L222, “The learnable parameters in Eqns. (7) are implemented through a nonlinear mapping such as a neural network with the rectified linear units (ReLU)”, does this mean that $\\hat{W}_t$ is more than just a parameter?", "limitations": "The limitations have been discussed.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1)\tThe paper is overall well-written with sound illustrations;\n\n(2)\tAs a weight initialization method, the proposed method makes sense and holds many potential applications;\n\n(3)\tCompared with Learngene, the proposed method can better adapt to downstream models of different scales.", "weaknesses": "(1)\tSection 3.1 and L183-196 seems to have a messy formatting of formulas; for example, some letters are missing. This reduces the readability and overall quality of the paper;\n\n(2)\tSome key experiments were insufficient. For example, the counterpart of this work is the pre-existing Learngene methods, whereas in Table 1, only training from scratch is compared and in Table 2 results on ImageNet are also lacking;\n\n(3)\tSome details of the proposed method are not explained clearly enough (see Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. Below, I will respond to your questions.\n\n> Q1: Section 3.1 and L183-196 seems to have a messy formatting of formulas; \n\nR1: We appreciate the reviewer's keen observation regarding Section 3.1 and lines 183-196. We apologize for any inconvenience caused and assure the reviewer that we will thoroughly revise this section to ensure all formulas are correctly formatted and legible. Below are the complete expressions of equations (1) and (2) in Section 3.1：\n\n​\t\t\t\t\t\t\t$\\begin{align}\\mathbf{A}^h=\\text{Attention}(\\mathbf{Q}_ h, \\mathbf{K}_ h, \\mathbf{V}_ h) = \\text{softmax}\\left(\\frac{\\mathbf{Q}_ {h}\\mathbf{K}_ {h}^\\top}{\\sqrt{d_ k}}\\right)\\mathbf{V}_ {h}\\end{align}$. \t\t\t\t\t\t\t    (1)\n\n​\t\t\t\t\t\t\t$\\text{MultiHead}(\\mathbf{Q}, \\mathbf{K}, \\mathbf{V}) = \\text{Concat}(\\mathbf{A}^1, \\ldots, \\mathbf{A}^H)\\mathbf{W}^O$.\t \t\t\t\t\t\t\t\t (2)\n\nHere are the revisions made to  L183-196：\n\n- When $H_d$ is divisible by $c_l$: The weights of head centroids are shared $\\frac{H_d}{c_l}$ times in sequence. For instance, centroids of weights $\\mathbf{A}^{(L,1)}$ and $\\mathbf{A}^{(L,2)}$ each share their weights across four attention heads, which are then directly assigned to eight attention heads of the descendant model in layer $L$.\n\n-  When $H_d$ is not divisible by $c_l$: The weights of the head centroids are sequentially shared $\\lfloor \\frac{H_d}{c_l}\\rfloor$ times, followed by appending $\\mathbf{A}^{(l,1)}, \\ldots, \\mathbf{A}^{(l,H_d \\mod c_l)}$ at the end. As an illustration, we share the centroids of weights $\\mathbf{A}^{(1,1)}, \\ldots, \\mathbf{A}^{(1,5)}$ once and then append $\\mathbf{A}^{(1,1)}, \\ldots, \\mathbf{A}^{(1,3)}$, thus initializing eight attention heads of the descendant model in the first layer.\n\n  \n\nFor the attention heads in the descendant models, we introduce the hyperparameter $\\omega = \\frac{H_a}{H_d}$ to denote the factor by which the number of attention heads is reduced compared to the ancestry model. ...  According to the adjustments in the number of attention heads, the weights $\\mathbf{W}^O$ of the projection layer are also proportionally pruned and then inherited by the descendant models.\n\n\n\n> Q2：Some key experiments were insufficient. \n\nR2: Thank you for the reviewer's suggestion. (1) We have added a strong comparative method called 'Pretraining-Finetuning' in Table 1, as shown in the table below： \n\n| Model | $H_d$ | $L_d$ | Params (M) | FLOPs (G) | From-Scratch | Pretraining-Finetuning | Ours      |\n| ----- | ----- | ----- | ---------- | --------- | ------------ | ---------------------- | --------- |\n| Tiny  | 3     | 12    | 5.7        | 1.2       | 61.44        | 66.36                  | **70.28** |\n| Small | 6     | 12    | 22         | 4.6       | 68.56        | 75.01                  | **78.43** |\n| Base  | 12    | 12    | 86.6       | 17.5      | 77.22        | 78.13                  | **79.50** |\n\nFrom-Scratch trains models for 100 epochs, while the Pretraining-Finetuning and our method only fine-tune the downstream models for 50 epochs.  Compared with the other two methods, our approach demonstrates the best initial performance across different scales.\n\n(2) In Table 2, we have supplemented the results on ImageNet as shown in the table below:\n\n| Method                 | I-Params | ImageNet  |\n| ---------------------- | -------- | --------- |\n| Pretraining-Finetuning | 10.5     | 68.85     |\n| From-Scratch           | 0        | 64.91     |\n| Heuristic-Learngene    | 5.6      | 66.44     |\n| Weight-Transformation  | 10.5     | 68.20     |\n| Auto-Learngene         | 10.5     | 67.78     |\n| Cluster-Learngene      | 7.5      | **71.36** |\n\nOur method continues to consistently outperform the baselines on the ImageNet dataset.\n\n\n\n> Q3: After Adaptively Learngene Clustering, the number of layers obtained is variable. How to deal with that if it is not equal to the number of layers in the downstream model?\n\nR3:  Your question is thoughtful. We have already addressed this issue in the **Appendix A.3**：\"According to the adjustments in the number of attention heads, the weights $\\mathbf{W}^O$ of the projection layer are also proportionally pruned or expanded with the hyperparameter $\\omega$ and then inherited by the descendant models.\"\n\n\n\n> Q4: In priority weight-sharing (Figure 2), what if the number of head centroids is less than the number of head in the downstream model?\n\nR4: Even if the number of head centroids is fewer than the number of heads in the downstream model, we can initialize the heads in the downstream model multiple times with priority weight-sharing of head centroids. Therefore, our priority weight-sharing strategy does not require any absolute size relationship between the number of head centroids and the number of heads in the downstream model.\n\n\n\n> Q5: Are all parameters of the downstream model (inherited or not) updated (or fixed) during training?\n\nR5: All parameters of the downstream model (inherited or not) update during training. However, due to leveraging  weight-sharing, the computational cost of our descendant models remains smaller compared to fine-tuning the entire pre-trained model.\n\n\n\n> Q6: In L222, “The learnable parameters in Eqns. (7) are implemented through a nonlinear mapping such as a neural network with the rectified linear units (ReLU)”, does this mean that 𝑊^𝑡 is more than just a parameter?\n\nR6: I apologize for any confusion caused by the description in my paper.  The symbol  $\\mathbf{\\widehat{W}}_ {t}$ indeed represents the newly introduced learnable parameters that are used to expand the $t^{th}$ feedforward network (FFN). However, in the implementation of this expanding FFN process, not only do we apply the simple transformation with these learnable parameters  $\\mathbf{\\widehat{W}}_ {t}$, but we also incorporate the ReLU (Rectified Linear Unit) activation function to provide some additional non-linear transformations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uhZtSGjqnq", "reviewer": "Reviewer_2s46", "summary": "The paper introduces Cluster-Learngene, a novel approach for initializing Vision Transformers (ViTs). This method clusters attention heads and position-wise feed-forward networks (FFNs) from a large \"ancestry\" model to form a condensed initialization core, termed \"learngene.\" By leveraging the density characteristics of attention heads and FFNs, Cluster-Learngene effectively customizes models of elastic scales to meet the resource constraints of various downstream tasks. The approach not only preserves critical knowledge but also enhances efficiency by reducing redundancy and facilitating weight-sharing in descendant models.", "questions": "1. The Cluster-Learngene method is primarily designed for models with full attention mechanisms. Given the emergence of attention variants aimed at reducing computational costs, such as sparse or low-rank attention, it remains to be seen whether this method can be effectively transferred and applied to these types of models.\n2. The hyperparameter Eps (ε) is currently set manually and plays a crucial role in the effectiveness of the clustering. Could the authors provide ablation study data to explore the robustness of the algorithm with varying settings of this parameter?", "limitations": "Please refer to weakness and questions", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Originality: The clustering approach for attention heads based on density characteristics is innovative.\n- Quality: The empirical validation is extensive, comparing the proposed method against several state-of-the-art initialization techniques.\n- Clarity: Despite some dense sections, the overall exposition of concepts is clear.\n- Significance: The method addresses critical issues in the scalability and resource efficiency of model deployment, making it highly significant for applications in resource-constrained environments.", "weaknesses": "Insufficient Downstream Task Evaluation: The dataset assessments for downstream tasks are currently limited to classification challenges. The generalizability of this method to other tasks such as segmentation and detection remains unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. Below, I will respond to your questions.\n\n\n\n> Q1: Insufficient Downstream Task Evaluation\n\nR1: Thanks for your suggestion. We conduct additional segmentation experiments on the ADE20K [A]. We set the base learning rate to $10^ {−3}$ and train for 160K iterations with a batch size of 8. The results are as follows:\n\n| Method                 | T-Params (M) | I-Params (M) | FLOP (G) | mIoU      |\n| ---------------------- | ------------ | ------------ | -------- | --------- |\n| Pretraining-Finetuning | 98.5         | 83.2         | 105      | 47.08     |\n| Heuristic-Learngene    | 57.8         | 42.5         | 105      | 40.12     |\n| Cluster-Learngene      | 74.7         | 59.4         | 105      | **48.30** |\n\n, where “T-Params\" and “I-Params\" correspond to the **T**otal number of parameters in the downstream/descendant models and the number of parameters **I**nherited into the downstream/descendant models, respectively. We validate the efficient cross task initialization capability of our Cluster-Learngene. On the other hand,  Fine-tuning exhibits inferior performance due to the risk of negative transfer compared to our Cluster-Learngene. [A] Bolei Zhou, , et al. \"Semantic understanding of scenes through the ADE20K dataset.\" Int. J. Comput. Vis., 127(3):302–321, 2019.\n\n\n> Q2: Given the emergence of attention variants aimed at reducing computational costs, such as sparse or low-rank attention, it remains to be seen whether this method can be effectively transferred and applied to these types of models.\n\nR2:  Thanks for your thoughtful question.  In general, sparse and low-rank attention mechanisms reduce the computational load through specific acceleration techniques. Sparse attention achieves this by limiting each query to compute attention with only a local set of keys. Meanwhile, low-rank attention decomposes the attention weight matrix into a low-rank form, such as $\\mathbf{A} \\approx \\mathbf{P}^{\\top} \\mathbf{Q}$, where $\\mathbf{P}$ and $\\mathbf{Q}$ are low-rank matrices. However, they still produce complete attention head outputs. Therefore, our Cluster-Learngene can still calculate the Mean Attention Distance and apply it to these sparse or low-rank attention mechanisms.\n\n\n\n> Q3: The hyperparameter Eps (ε) is currently set manually and plays a crucial role in the effectiveness of the clustering.\n\nR3:   In the **Appendix A.6**, we have analyzed the varying settings of the hyperparameter Eps (ε). The following table displays the results for different values of ε:\n\n| $\\varepsilon=1$ | $\\varepsilon=10$ | $\\varepsilon=100$ |\n| --------------- | ---------------- | ----------------- |\n| 85.27           | **85.38**        | 80.45             |\n\n$\\varepsilon=1$ implies no FFN clustering, potentially causing negative transfer. $\\varepsilon=100$ means clustering all FFNs in adjacent layers with identical head centroid counts, resulting in an excessive cluster of FFNs and subsequent degradation in the performance of initialized descendant models. Our Cluster-Learngene ($\\varepsilon=10$) strikes a good balance between these issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rIwVNXIki8", "reviewer": "Reviewer_SzCi", "summary": "This paper seeks to address the issue of overgeneralizing the applicability of large pre-trained models in deep learning, particularly when faced with task-specific resource constraints. The authors propose Cluster-Learngene, a novel method that clusters critical internal modules from a large ancestry model and uses them to initialize descendant models of various scales. By adaptively clustering attention heads and feed-forward networks (FFNs) based on their density characteristics, Cluster-Learngene creates a flexible and efficient initialization process. The method also incorporates priority weight-sharing and learnable parameter transformations to expand the learngene. Extensive experiments demonstrate that Cluster-Learngene outperforms other initialization methods in efficiency and in customizing models to fit the resources available for downstream tasks.", "questions": "See Weaknesses.", "limitations": "The authors have addressed the limitations and potential negative societal impacts of their work.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThe problem studied in this paper is interesting and valuable. The paper is well-structured and clearly written, making it accessible and comprehensible to a broad audience.\n\n2.\tThe approach of adaptively clustering attention heads and FFNs based on density characteristics is innovative and adds a novel dimension to model initialization techniques. The introduction of priority weight-sharing and learnable parameter transformations effectively addresses the need for models to adapt to varying resource constraints.\n\n3.\tThe authors have conducted an extensive set of experiments to validate the effectiveness of Cluster-Learngene, including initializing descendant models of elastic scales and evaluating initialization results on different downstream tasks, both of which show significant improvements compared to baselines.", "weaknesses": "1.\tThe paper lacks a complexity analysis or specific time cost assessments for Cluster-Learngene, which are crucial for understanding its computational efficiency.\n\n2.\tRegarding the choice of hyperparameters, the authors could provide an ablation study, particularly for parameters like ε, to better understand their impact on the model's performance.\n\n3.\tWill the code be available for open source in the future?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. Below, I will respond to your questions.\n\n> Q1: The paper lacks a complexity analysis or specific time cost assessments for Cluster-Learngene, which are crucial for understanding its computational efficiency.\n\nR1: For the largest model, DeiT-Base, the computational time for the clustering step is **less than 6 minutes** on a single GeForce RTX 3090 GPU, which is negligible compared to the several hours or even a full day required for fine-tuning downstream tasks.\n\n\n\n> Q2: Regarding the choice of hyperparameters, the authors could provide an ablation study, particularly for parameters like ε, to better understand their impact on the model's performance.\n\nR2: In the **Appendix A.6**, we have analyzed the varying settings of the hyperparameter Eps (ε). The following table displays the results for different values of ε:\n\n| $\\varepsilon=1$ | $\\varepsilon=10$ | $\\varepsilon=100$ |\n| --------------- | ---------------- | ----------------- |\n| 85.27           | **85.38**        | 80.45             |\n\n$\\varepsilon=1$ implies no FFN clustering, potentially causing negative transfer. $\\varepsilon=100$ means clustering all FFNs in adjacent layers with identical head centroid counts, resulting in an excessive cluster of FFNs and subsequent degradation in the performance of initialized descendant models. Our Cluster-Learngene ($\\varepsilon=10$) strikes a good balance between these issues.\n\n\n\n> Q3: Will the code be available for open source in the future?\n\nR3:  We expect to release our code by late Sep.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PZRhR5x7PJ", "reviewer": "Reviewer_ScsP", "summary": "Cluster-Learngene is an innovative approach for initializing Vision Transformer models. It works by inheriting adaptive clusters from a large pre-trained ancestry model. The key features of this method include:\nAdaptive Clustering: Cluster-Learngene adaptively clusters the attention heads and feed-forward networks within each layer of the ancestry model based on their density characteristics. This forms compact and information-dense modules called \"learngenes.\"\nEfficient Initialization: The method uses priority weight-sharing and learnable parameter transformations to expand these learngenes. This enables efficient initialization of descendant models with varying scales and resource constraints.\nResource Customization: Cluster-Learngene tailors the model initialization to the downstream task resources, offering a more efficient alternative to the standard pretraining and fine-tuning approach.\nEmpirical Validation: Extensive experiments on multiple datasets demonstrate that Cluster-Learngene outperforms traditional initialization strategies and is competitive with more resource-intensive fine-tuning methods.\nScalability and Efficiency: The approach achieves faster convergence and higher data efficiency compared to models initialized from scratch, making it effective for scenarios with limited data.\nBroad Applicability: The paper shows the effectiveness of Cluster-Learngene across different model scales and various downstream tasks, showcasing its versatility.\nIn summary, Cluster-Learngene presents a novel and efficient strategy for initializing Vision Transformer models, addressing the challenge of resource constraints in downstream tasks.", "questions": "1. Can you offer more experiments on other architectures(ConvNext, hybrid arch etc.), datasets(VTAB, FGVC etc.) or even modalities(like NLP tasks&models) to further validate the effectiveness of Cluster-Learngene? Which I believe will be powerful proof of its generalizability.\n\n2. There have been other methods in model merging, can you compare Cluster-Learngene with these methods[1,2] to highlight its advantages, in experimental and theoritical ways?\n[1]Yang, Enneng, et al. \"Adamerging: Adaptive model merging for multi-task learning.\" arXiv preprint arXiv:2310.02575 (2023).\n[2]Yang, Enneng, et al. \"Representation Surgery for Multi-Task Model Merging.\" arXiv preprint arXiv:2402.02705 (2024).\n\n3. How computationally expensive is the clustering step compared to the savings in downstream fine-tuning? Is there a break-even point in terms of number of downstream tasks where this becomes beneficial?", "limitations": "The authors provide a good discussion of limitations in the appendix, acknowledging that the performance of descendant models depends heavily on the quality of the ancestry model.\n\nThe discussion of broader impacts is quite limited. Given that the method aims to make transfer learning more efficient, some discussion of potential positive impacts (e.g. democratizing access to good models with less computational resources) as well as negative impacts (e.g. potential to amplify biases present in large pre-trained models) would be valuable. The authors could expand on mitigation strategies for some of these risks.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper contributes to the field of deep learning with several key strengths:\n\nOriginality: It proposes an innovative method for initializing Vision Transformer models by adaptively clustering attention heads and FFNs, introducing the concept of learngenes.\n\nQuality: The research is thorough, with a well-defined methodology and robust empirical validation across multiple datasets.\n\nClarity: The paper is clearly structured and articulated, making the complex methodology understandable.\n\nSignificance: It addresses an important problem in AI—efficient model scaling for different resource constraints, with potential far-reaching impacts in various application domains.\n\nCreative Combination: The paper successfully combines existing concepts in a novel way, leading to a unique and effective model initialization approach.\n\nRemoving Limitations: It offers a solution that overcomes the limitations of traditional model initialization, enhancing flexibility and efficiency.", "weaknesses": "Lack of comparison to other clustering methods: The paper uses a density-based clustering approach, but does not compare this to other clustering methods like k-means. \n\nExperiments focused on image classification: The experimental evaluation is primarily on image classification tasks. Testing on a broader range of vision tasks like object detection or segmentation would demonstrate the generalizability of the approach better.\n\nThe paper format can be improved: Table 2 should underline the highest result. Tables 4 and 5 should be three-line tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. Below, I will respond to your questions.\n\n> Q1: Lack of comparison to other clustering methods.\n\nR1: Your question is thoughtful. In **Section 4.3.1**,  we have compared clustering methods such as k-means and provided different values for the set (k) of cluster centroids in k-means. The results are as follows:\n\n| *k-means*,  $k=1$ | *k-means*,  $k=2$ | *k-means*, $k=3$ | Ours      |\n| ----------------- | ----------------- | ---------------- | --------- |\n| 80.12             | 81.02             | 81.41            | **85.38** |\n\nCluster-Learngene not only outperforms in clustering efficiency but also adaptively adjusts the count of cluster centroids for each model layer, unlike *k-means* which requires predefined numbers of cluster centroids.\n\n\n\n\n\n\n\n> Q2: Experiments focused on image classification.\n\nR2: Thanks for your suggestion. We conduct additional segmentation experiments on the ADE20K [A]. We set the base learning rate to $10^ {−3}$ and train for 160K iterations with a batch size of 8. The results are as follows:\n\n| Method                 | T-Params (M) | I-Params (M) | FLOP (G) | mIoU      |\n| ---------------------- | ------------ | ------------ | -------- | --------- |\n| Pretraining-Finetuning | 98.5         | 83.2         | 105      | 47.08     |\n| Heuristic-Learngene    | 57.8         | 42.5         | 105      | 40.12     |\n| Cluster-Learngene      | 74.7         | 59.4         | 105      | **48.30** |\n\n, where “T-Params\" and “I-Params\" correspond to the **T**otal number of parameters in the downstream/descendant models and the number of parameters **I**nherited into the downstream/descendant models, respectively. We validate the efficient cross-task initialization capability of our Cluster-Learngene. On the other hand,  Fine-tuning exhibits inferior performance due to the risk of negative transfer compared to our Cluster-Learngene. [A] Bolei Zhou, et al. \"Semantic understanding of scenes through the ADE20K dataset.\" Int. J. Comput. Vis., 127(3):302–321, 2019.\n\n\n> Q3: The paper format can be improved.\n\nR3：We thank the reviewer for their constructive feedback and will enhance the paper by underlining the highest result in Table 2 and reformating Tables 4 and 5 into three-line tables for improved clarity and readability.\n\n\n\n> Q4: Can you offer more experiments?\n\nR4: Thanks for your suggestion.    (1) We have further supplemented the result for the **hybrid architecture**, such as LeViT-192 [B], which integrates convolutional and transformer components. We perform clustering and expansion on the MSA and FFN in stages 1-3 of LeViT-192. The experimental results are presented in the table below:\n\n| Method                 | CIFAR-100 | ImageNet  |\n| ---------------------- | --------- | --------- |\n| Pretraining-Finetuning | 85.11     | 69.08     |\n| From-Scratch           | 74.06     | 65.12     |\n| Heuristic-Learngene    | 78.22     | 66.65     |\n| Auto-Learngene         | 80.96     | 67.91     |\n| Cluster-Learngene      | **86.27** | **71.60** |\n\nWe fine-tune all downstream models for 500 epochs on CIFAR-100 and 50 epochs on ImageNet. It can be observed that our method, when extended to the hybrid architecture, also achieves the best results on both CIFAR-100 and ImageNet.  [B] Graham, Benjamin, et al. \"Levit: a vision transformer in convnet's clothing for faster inference.\" *ICCV*. 2021.\n\n（2）For Section 4.2.3, in addition to the existing results on FGVC (CUB-200-2011, Stanford Cars, Oxford Flowers) and VTAB (CIFAR-100, Flowers102)   datasets, we have also supplemented results from **FGVC** datasets such as Stanford Dogs, NABirds, and four **VTAB** datasets which are SVHN, DTD, EuroSAT, and Resisc45. The results are presented in the table below:\n\n| Method                 | Stanford Dogs | NABirds   | SVHN      | DTD       | EuroSAT   | Resisc45  |\n| ---------------------- | ------------- | --------- | --------- | --------- | --------- | --------- |\n| Pretraining-Finetuning | 75.41         | 78.33     | 97.57     | 84.88     | 97.32     | 96.53     |\n| From-Scratch           | 61.45         | 64.99     | 91.23     | 73.74     | 92.49     | 90.60     |\n| Heuristic-Learngene    | 71.68         | 73.12     | 94.81     | 77.30     | 94.88     | 94.25     |\n| Auto-Learngene         | 73.53         | 73.65     | 95.16     | 79.85     | 95.02     | 94.37     |\n| Cluster-Learngene      | **76.37**     | **80.76** | **97.92** | **85.30** | **98.24** | **96.85** |\n\nOur Cluster-Learngene outperforms Pretraining-Finetuning and other previous Learngene methods, thereby demonstrating the strong initialization capability and generalizability of our approach.\n\n\n\n\n> Q5: Can you compare Cluster-Learngene with model merging[1,2]? \n\nR5: Thank you to the reviewer for providing these papers. We will cite them in the 'Related Work' section. At the conceptual level, Multi-Task Model Merging[1,2] aims to merge models for collaborative multi-task processing, whereas our Learngene is designed to **independently initialize** diverse models and generate **scalable scales** to meet specific task demands and resource limitations. In the experimental  way, we compare our method with the two Multi-Task Model Merging methods[1,2] on the cars dataset. Our method achieves a performance of **89.87%** on DeiT-Small, outperforming the Multi-Task Model Merging methods[1,2], which show performances of 69.6% and 72.0% on the larger-scale ViT-B/32, respectively.\n\n\n\n> Q6: Computationally expensive and break-even point. \n\nR6:  （1）For the largest model, DeiT-Base, the computational time for the clustering step is **less than 6 minutes** on a single GeForce RTX 3090 GPU, which is negligible compared to the several hours or even a full day required for fine-tuning downstream tasks. （2）Indeed, our Cluster-Learngene reaches a break-even point with as few as two downstream tasks, indicating that the conditions for our algorithm to be beneficial are quite lenient.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
> Q1: Experiments focused on image classification: The experimental evaluation is primarily on image classification tasks. Testing on a broader range of vision tasks like object detection or segmentation would demonstrate the generalizability of the approach better.

R1: Thanks for your suggestion. We conduct additional segmentation experiments on the ADE20K [A]. We set the base learning rate to $10^ {−3}$ and train for 160K iterations with a batch size of 8. The results are as follows:

| Method                 | T-Params (M) | I-Params (M) | FLOP (G) | mIoU      |
| ---------------------- | ------------ | ------------ | -------- | --------- |
| Pretraining-Finetuning | 98.5         | 83.2         | 105      | 47.08     |
| Heuristic-Learngene    | 57.8         | 42.5         | 105      | 40.12     |
| Cluster-Learngene      | 74.7         | 59.4         | 105      | **48.30** |

, where “T-Params" and “I-Params" correspond to the **T**otal number of parameters in the downstream/descendant models and the number of parameters **I**nherited into the downstream/descendant models, respectively. We validate the efficient cross task initialization capability of our Cluster-Learngene. On the other hand,  Fine-tuning exhibits inferior performance due to the risk of negative transfer compared to our Cluster-Learngene. [A] Bolei Zhou, , et al. "Semantic understanding of scenes through the ADE20K dataset." Int. J. Comput. Vis., 127(3):302–321, 2019.



> Q2: The hyperparameter Eps (ε) is currently set manually and plays a crucial role in the effectiveness of the clustering.

R2:  In the **Appendix A.6**, we have analyzed the varying settings of the hyperparameter Eps (ε). The following table displays the results for different values of ε:

| $\varepsilon=1$ | $\varepsilon=10$ | $\varepsilon=100$ |
| --------------- | ---------------- | ----------------- |
| 85.27           | **85.38**        | 80.45             |

$\varepsilon=1$ implies no FFN clustering, potentially causing negative transfer. $\varepsilon=100$ means clustering all FFNs in adjacent layers with identical head centroid counts, resulting in an excessive cluster of FFNs and subsequent degradation in the performance of initialized descendant models. Our Cluster-Learngene ($\varepsilon=10$) strikes a good balance between these issues.



> Q3: How computationally expensive is the clustering step compared to the savings in downstream fine-tuning? Is there a break-even point in terms of number of downstream tasks where this becomes beneficial?

R3:  （1）For the largest model, DeiT-Base, the computational time for the clustering step is **less than 6 minutes** on a single GeForce RTX 3090 GPU, which is negligible compared to the several hours or even a full day required for fine-tuning downstream tasks.（2）Indeed, our Cluster-Learngene reaches a break-even point with as few as two downstream tasks, indicating that the conditions for our algorithm to be beneficial are quite lenient. Considering that real-world applications always involve multiple tasks, our approach provides clear benefits.



> Q4: The discussion of broader impacts.

R4: In practice, learngene strives to preserve the core knowledge from the original model, avoiding the redundancy found in large pre-trained models. Consequently, descendant models initialized with the learngene reduce the biases that are present in the original large pre-trained models.

Meta-review: In this paper, the authors introduce Cluster-Learngene, a novel method for initializing vision transformers. After the discussion period, most of the reviewers tend to accept this paper. Reviewer GgZj still kept their negative rating. However, they didn't respond to the authors' feedback. After checking the paper, the reviews, and the discussions, I tend to accept this paper. The authors are required to include the discussions with the reviewers in their final version. Besides, the code shared during the discussion period should also be made publicly available.
Decision: Accept (poster)