Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers

Authors: qiufeng  wang
xu  yang
fu  feng
jing  wang
xin  geng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 19:56:00

Abstract: In recent years, the merging of vast datasets with powerful computational resources has led to the emergence of large pre-trained models in the field of deep learning. However, the common practices often overgeneralize the applicability of these models, overlooking the task-specific resource constraints. To mitigate this issue, we propose \textbf{Cluster-Learngene}, which effectively clusters critical internal modules from a large ancestry model and then inherits them to initialize descendant models of elastic scales. Specifically, based on the density characteristics of attention heads, our method adaptively clusters attention heads of each layer and position-wise feed-forward networks (FFNs) in the ancestry model as the learngene. Moreover, we introduce priority weight-sharing and learnable parameter transformations that expand the learngene to initialize descendant models of elastic scales. Through extensive experimentation, we demonstrate that Cluster-Learngene not only is more efficient compared to other initialization methods but also customizes models of elastic scales according to downstream task resources.

Main paper: 
 introduction: 
 The evolution of deep learning has been profoundly influenced by the confluence of expansive data sources and robust computational capabilities. This collaboration has given rise to large pre-trained foundation models [11, 10, 41, 5] , particularly those built upon the Transformer [47, 11] , such as the Vision Transformers (ViTs) [11] . The pre-trained foundation models, being widely deployed in various devices like smartphones or edge devices, serve as the initialization point [16, 2, 19, 64, 49, 50] for diverse downstream applications. However, this dominant methodology implicitly assumes that a one-size-fits-all approach, i.e., the whole foundation model is universally apt for every application, neglecting the specific resource constraints (e.g., memory, FLOPs, or latency) inherent to certain downstream tasks. Furthermore, not all tasks demand the full power of these extensive foundation models. This naturally raises a pivotal question: Can we extract and harness the condensed part of these foundation models to achieve a harmonious balance between accuracy and resource efficiency?To achieve the goal of efficiently initializing models, [49, 50] introduce the innovative Learngene framework inspired by the observation of genes (cf. Fig. 1 (a) ). As showcased in Fig. 1 (b), Learngene framework is designed in two pivotal stages. In the first stage, the significant knowledge is condensed from a large ancestry model into a more compact part termed as learngene. In the next stage, this learngene is inherited to initialize the descendant models of elastic scales. * Previous works [49, 50] predominantly focus on extracting a few integral layers as the learngene and manually stacking them with the randomly initialized layers.However, such approaches struggle with inherent limitations: (i) The strategy of extracting certain integral layers overlooks the potential existence of learngene within these layers, leading to the preservation of many redundant weights. (ii) The approach of manually stacking the learngene with randomly initialized layers lacks the adaptability to scale the model, preventing the initialization of downstream models with custom dimensions.As mentioned earlier, the Learngene framework aims to preserve the most generalizable part of the ancestry model while eliminating redundant weights that weaken representational capacity. Recent studies [42, 56] have visualized the mean attention distance of ViTs, offering deeper insights into weight redundancy among attention heads across different layers. As illustrated in Fig. 1 (c), the lower layers focus on both local and global perspectives, leading to a more sparse density of attention heads. Conversely, the higher layers prioritize a global context, resulting in a compact density. A notable observation is the repetitive functionality across many attention heads especially in the higher layers, which inevitably leads to weight redundancy.Inspired by the above observation, we propose the Cluster-Learngene, an innovative approach that adaptively extracts internal modules in ViTs as the learngene, e.g., attention heads and position-wise feed-forward networks (FFNs). Firstly, to extract the cluster centroids of the attention heads (i.e., head centroids) across each layer of the ancestry model, we cluster the attention heads within each layer of the ancestry model based on their density characteristics. As depicted in Fig. 1 (c-d ), the attention heads in the first layer exhibit a sparse density, resulting in five clusters, whereas the attention heads in the last layer cluster more compactly, forming a single group. Furthermore, our method includes clustering FFNs by assessing the distance density of head centroids in adjacent layers of the ancestry model. Specifically, when the distance density of head centroids in adjacent layers is similar, we inherit the FFN from the shallower of these adjacent layers (i.e., FFN centroid) as the learngene. As illustrated in Fig. 1 (c), the similar densities of attention heads in the 7-th and 8-th layers lead to proximate head centroids, enabling these layers to only require the inheritance from the 7th layer as the FFN centroid. Overall, Cluster-Learngene extracts critical parameters containing significant knowledge, as the extracted part represents attention heads/FFNs with similar semantics.In the inheriting stage, to achieve the initialization of descendant models with varying number of attention heads, we adopt the priority weight-sharing. We start by ranking the head centroids based on the size of their respective clusters, arranging them in descending order of priority. Subsequently, we perform weight-sharing by distributing these head centroids to initialize the attention heads of the descendant models. If the number of attention heads in a specific layer aligns perfectly with the number of centroids, they are evenly shared. However, if they fail to align perfectly, any remaining centroids are shared according to the remainder. Moreover, we apply learnable parameters to transform FFN centroids into multiple FFNs, thus enabling the initialization of descendant models with elastic scales.Our contributions can be summarized as follows: (i) We propose the adaptive clustering to extract the head and FFN centroids as the learngene, ensuring the preservation of significant knowledge within the ancestry model. (ii) To achieve the initialization of descendant models, we introduce priority weight-sharing that favors head centroids within larger clusters and employ learnable parameters to transform the FFN centroids into multiple FFNs. (iii) Comprehensive experimental evaluations across datasets of different scales reveal that Cluster-Learngene not only outperforms traditional initialization strategies but also stands toe-to-toe with more resource-demanding fine-tuning methodologies. 
related work: 
 Model Initialization: Over the years, various initialization techniques have been proposed including the popular random initialization, Xavier initialization [13] and the Kaiming initialization [19] .Recently, the use of pre-trained foundation models has gained prominence as an initialization strategy before fine-tuning for specific tasks [11, 10, 41, 57, 37, 5, 61, 22, 15, 53, 33, 32, 7, 18, 39, 54, 29, 25, 26, 51, 28, 44, 55, 59, 40, 58, 36, 31, 21, 60] . However, such an approach necessitates pre-training separate models for each downstream task, which can lead to substantial computational resource consumption. In contrast, Cluster-Learngene presents a unique model initialization method that alleviates the need for multiple pre-training steps.Density-based Clustering: Clustering aims to group similar data points together while separating dissimilar ones. A wide array of approaches has been explored, including partitioning-based clustering [14, 1, 27] , hierarchical clustering [35, 8, 65, 52] , and density-based clustering [23, 43, 6, 3] , and so on. In particular, density-based clustering operates by taking into account the density relationships between data points to form clusters. Inspired by this, our method adopts a similar principle by assessing the density of attention heads to retain essential head centroids that represent significant knowledge.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please see Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 Learngene framework is primarily divided into two phases in Fig. 1 (b): the significant knowledge is condensed from an ancestry model into a more compact part termed as learngene and then inherited to initialize the descendant models of assorted scales. Specifically, in phase 1, our Cluster-Learngene selects mean attention distance as the density metric and uses it to cluster the attention heads of each layer and FFNs in the ancestry model as the learngene, because they can effectively represent attention heads/FFNs with similar semantics. The pseudocode for this phase is presented in Algorithm 1. In phase 2, to initialize the descendant models, we employ priority weight-sharing of head centroids for varying number of attention heads as illustrated in Fig. 2 and leverage learnable parameters to expand the FFN centroids into multiple FFNs. Next, we briefly introduce some preliminaries related to ViTs.In the ViT architecture, an input image is first divided into N non-overlapping patches, and each patch is linearly embedded into a flat vector of size D. The ViT encoder consists of alternating layers of multi-head self-attention (MSA) and FFN blocks. Let H denote the total number of heads in each layer.For the h th head, the queryQ h ∈ R N ×d k , key K h ∈ R N ×d k , and value V h ∈ R N ×dv are linearly generated through learned weight matrices W Q h ∈ R D×d k , W K h ∈ R D×d k , and W V h ∈ R D×dv, where d k and d v are the dimensions of the key and value vectors, respectively. The SA mechanism of the i-th head can be represented as:EQUATIONMSA allows the model to jointly attend to information at different positions from different representational subspaces at different positions:EQUATIONwhere W O ∈ R Hdv×D is a learned weight matrix. Besides, the FFN can be formulated as: EQUATIONGiven a pre-trained ancestry model with L layers and H a attention heads per layer, let the attention weights for the h th head in the l th layer be denoted by the matrix A (l,h) ∈ R N ×N . The element A (l,h) i,j represents the attention weight from position i to position j. The distance matrix T ∈ R N ×N is defined with the Euclidean distance between any two positions i and j in the sequence, given by T i,j = (x i -x j )2 + (y i -y j ) 2 . The mean attention distance for the h th head in the l th layer, encapsulating the weighted distance for each position i across the sequence, is given by:EQUATIONTo deduce this metric for every head across all layers, iterate the above computation for every l ∈ {1, . . . , L} and h ∈ {1, . . . , H a }. As depicted in Fig. 1 and Appendix A.1, while the lower layers simultaneously attend to both local and global features, leading to a more dispersed distribution of attention heads, the higher layers predominantly focus on global aspects, causing a tighter concentration of attention heads. As a result, there is a significant overlap in the semantic representations among many attention heads, especially in the higher layers, leading to weight redundancy.Motivated by the empirical observations, we extract cluster centroids [43, 6, 3] of attention heads in ViTs as the learngene inherited into the descendant models, thus aggregating similar semantics into the head centroids. To realize this, we select d as a density metric for adaptively clustering the attention heads of the ancestry model at each layer, without setting the number of clusters in advance. This realization prompts the formulation of the definitions and lemmas, which scaffold our adaptive clustering approach. Definition 1 (Eps-neighborhood of an attention head). The Eps-neighborhood of an attention head a, denoted as N Eps (a), is defined as: N Eps Our approach could require for each head in a cluster that there are at least a Minimum number of Heads (M inHds) in an Eps-neighborhood of that head.Definition 2 (density-reachable). Transitioning from the neighborhood concept, an attention head a is considered density-reachable from another head b with respect to Eps and M inHds if there is a sequence of heads a 1 , . . . , a n such that a 1 = b, a n = a, and each head in this sequence lies within the Eps-neighborhood of its preceding head. Definition 3 (density-connected). Broadening our purview, attention heads a and b are labeled density-connected with respect to Eps and M inHds if there exists an intermediary head o from which both a and b are density-reachable.Considering all attention heads in layer l as S l , a cluster C based on Eps and M inHds is identified as a non-empty subset of S l that satisfies the conditions: (i) Maximality: Therefore, upon satisfying these two conditions, we cluster all attention heads of each layer into different lists of clusters. The pseudo-code for this process is summarized in Algorithm 1. For each cluster C in the list of clusters, we select the attention head C lg closest to the cluster centroid as the learngene. This is because it effectively represents the functionality of all attention heads in the cluster. Formally, the formula can be expressed as follows:EQUATIONwhere d(C) is computed as the average d across all attention heads within this cluster C. The lemma presented below is pivotal in substantiating the correctness of our clustering algorithm.Lemma 1 Presuming an attention head a belongs to S l and satisfies the condition |N Eps (a)| ≥ M inHds. Then, the set O = {o | o ∈ S l and o is density-reachable from a with respect to Eps and M inHds} collectively shapes a cluster.Based on the distances between attention heads discussed earlier, we can further cluster FFNs as the learngene. For adjacent layers sharing the same number N C of head centroids, a key criterion is established: if the average distance across all head centroids between these layers is less than the threshold ε, it suggests redundant similarity in their representational capabilities. This criterion is formulated as:EQUATIONUnder this condition, we enhance model efficiency by preserving only the FFN from the shallower of these adjacent layers as the FFN centroid, thereby maintaining essential functionality while eliminating redundancy. Moreover, we also select those FFNs that do not fit this established criterion as the learngene because they exhibit representational capacities independent of adjacent layers.Since the head centroids have been extracted as the learngene, we further expand them for initializing the descendant models with varying number of attention heads. For an ancestry model with L layers, the l th layer has c l head centroids of weight A (l,1) , . . . , A (l,c l ) . Importantly, the head centroids at each layer are sorted in descending order based on the size of their respective cluster, i.e., centroids representing more attention heads in the ancestry model are ranked higher. These head centroids condense significant knowledge and ensure the initialization of descendant models without performance degradation. Assume the descendant model has H d attention heads for each layer. To achieve the desired expansion of heads to initialize the descendant models, we adopt the priority weight-sharing and Fig. 2 illustrates two scenarios:• When H d is divisible by c l : The weights of head centroids are shared H d c l times in sequence. For instance, centroids of weights A (L,1) and A (L,2) each share their weights across four attention heads, which are then directly assigned to eight attention heads of the descendant model in layer L.• When H d is not divisible by c l : The weights of the head centroids are sequentially shared H d c l times, followed by appending A (l,1) , . . . , A (l,H d mod c l ) at the end. As an illustration, we share the centroids of weights A (1,1) , . . . , A (1,5) once and then append A (1,1) , . . . , A (1,3) , thus initializing eight attention heads of the descendant model in the first layer.For the attention heads in the descendant models, we introduce the hyperparameter ω = Ha H d to denote the factor by which the number of attention heads is reduced compared to the ancestry model. In addition to uniformly setting the number of attention heads for each layer with the hyperparameter ω, we also explore two other possibilities in Appendix A.7: incrementing and decrementing the count of attention heads with layer depth. According to the adjustments in the number of attention heads, the weights W O of the projection layer are also proportionally pruned and then inherited by the descendant models. †As outlined in Section 3.2, we retain the shallowest FFN determined by distance-based clustering. For this FFN centroid, we introduce learnable parameters to transform it into multiple FFNs, facilitating the initialization of descendant models at elastic scales. This process is summarized in the formula:EQUATIONHere, W t denotes the newly introduced learnable parameters to expand the t th FFN. These parameters are designed for minimal learning overhead, yet they are highly effective in quickly adapting descendant models to downstream tasks.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental setting: 
 Datasets. To condense the learngene, we employ the ImageNet-1K, a collection of 1.2 million training images and 50,000 validation images distributed across 1,000 classes as part of the ILSVRC2012 competition [9] . After initializing the descendant models with the learngene, we proceed to fine-tune these models on diverse downstream tasks. These tasks include iNaturalist-2019 [45] , Food101 [4] ,Oxford Flowers [38] , Stanford Cars [12] , CIFAR-10 [24] , CIFAR-100 [24] , CUB-200-2011 [48] . For detailed dataset descriptions, see Appendix A.2.Training settings. During the learngene clustering, We set Eps, ε = 10, M inHds = 1 ‡ , ensuring that each attention head is included in a unique cluster. In the learngene inheriting phase, we train the descendant models on downstream tasks for 500 epochs, including a 10-epoch warm-up period, except for iNaturalist-2019, where we train for 100 epochs with a 5-epoch warm-up. The initial learning rate is set to 5 × 10 -4 for most tasks, except for Stanford Cars where it is 5 × 10 -3 , and a weight decay of 0.05.Architectures. Both the ancestry model and descendant models are variants derived from DeiT [46] . In terms of width, there are three types of DeiT: Tiny, Small, and Base. Furthermore, as detailed in Section 4.2.4, we implement experiments on Swin Transformer [30] to demonstrate the applicability of our method across various backbones. The learnable parameters in Eqns. (7) are implemented through a nonlinear mapping such as a neural network with the rectified linear units (ReLU).In this section, we validate the capabilities of Cluster-Learngene in efficiently initializing models and measure model performance with Top-1 accuracy. As illustrated in Fig. 3 , for the Tiny-scale descendant models with only 32 attention heads, Cluster-Learngene outperforms Pretraining-Finetuning on ImageNet. Moreover, the performance of Cluster-Learngene improves with an increase in the number of attention heads and FFNs. This shows that Cluster-Learngene can adaptively initialize the descendant models with varying number of attention heads and FFNs, catering to downstream resource constraints. In contrast, Pretraining-Finetuning requires retraining for each model variant, significantly raising storage and training costs, particularly when dealing with models of elastic scales. Therefore, our method resolves the limitations of the onesize-fits-all approach seen in Pretraining-Finetuning. The results in Fig. 4 reveal that Cluster-Learngene outperforms Pretraining-Finetuning with fewer inherited parameters. For example, in the case of Small-scale descendant models, Cluster-Learngene inherits only 15.1M parameters and attains a performance of 78.43%, while Pretraining-Finetuning with 18.0M parameters achieves less than 75%. These results demonstrate the advanced initialization ability of Cluster-Learngene, as the clustered learngene retains the critical generalizable knowledge.We conduct a comparative analysis of our approach for initializing descendant or downstream models, as follows: (i) Pretraining-Finetuning: This approach pre-trains DeiT on ImageNet and ‡ Please see Appendix A.6 for more analysis. The first six layers are extracted from the DeiT and then stacked with randomly initialized higher layers to initialize the descendant models.As illustrated in Tab. 2, our Cluster-Learngene significantly outperforms both From-Scratch and Weight-Transformation. When compared to other Learngene methods, such as Auto-Learngene, Cluster-Learngene exceeds by 11.17% on the iNaturalist-2019 (iNat-2019). These results highlight the superior capability of Cluster-Learngene in efficiently initializing descendant models. Moreover, on six datasets, the performance of Cluster-Learngene outperforms that of Pretraining-Finetuning, where the entire model is fine-tuned. This phenomenon can be attributed to the more universally significant knowledge within learngene, allowing it to adapt effectively to various downstream tasks.We provide a detailed comparison of training efficiency between our approach and From-Scratch on ImageNet. As shown in Fig. 5 (a), Cluster-Learngene requires only 4.0 × less training overhead compared to From Scratch on Small-scale descendant models. A key advantage of our approach is that descendant models initialized with the learngene achieve faster convergence, owing to a superior initialization point. We further conduct experiments on Base-scale descendant models over different percentages of training data from ImageNet-1K (IN-1K). As shown in Tab. 3, while our method does not outperform the From-Scratch on the entire dataset, its performance exhibits greater stability as the amount of training data decreases. For instance, with only 25% of the training data, Cluster-Learngene outperforms From-Scratch by 7.09%, while requiring only 1 6 of the training cost. This higher data efficiency of our method is attributed to the significant knowledge within the learngene, which helps descendant models mitigate overfitting, especially in scenarios with limited data.In this section, we provide further analysis and ablation of Cluster-Learngene. § Unless otherwise specified, we conduct experiments on CIFAR-100 and use Small-scale DeiT as the ancestry model.Additionally, we compare the results of k-means clustering [20] of attention heads with cluster centroids (k) set at 1, 2, and 3. Tab. 4 shows that Cluster-Learngene not only outperforms in clustering efficiency but also adaptively adjusts the count of cluster centroids for each model layer, unlike k-means that requires predefined the numbers of cluster centroids. Table 4 : Comparison of the clustering method. All results are from the 6-layer downstream models. In Table 5 , our priority weight-sharing is compared against three alternative weight-sharing methods: following the original sequence of heads, by ascending d of heads, and by descending d of heads.The results validate the effectiveness of priority weight-sharing, as our method accounts for the fact that the larger the cluster a head belongs to, the richer the critical knowledge it represents. We illustrate attention representations in Fig. 6 to explain which significant knowledge is inherited by learngene. To clarify the visualization, we apply a power exponent of γ = 0.25. In the first layer of the ancestry model, a head centroid is clustered from heads 1, 2, 4, and 5 to initiate the head 1 in the descendant model, and so forth. Then, weight-sharing is applied to expand head centroids, e.g., sharing twice to initialize the descendant model.In the first layer, heads 1, 2, 4, and 5 with similar semantics form the largest cluster, mainly focusing on attention patterns along the main diagonal. Additionally, heads 3 and 4 present distinct semantics, with head 4 reflecting more abstract and high-level features akin to the final layer. Thus, the first learngene layer integrates three principal representation patterns from the ancestry model. In contrast, the representations in the final layer of the ancestry model exhibit significant repetition, leading to the clustering of a single-head centroid for initializing the attention heads of the descendant model. 
conclusion: 
 We propose Cluster-Learngene to adaptively cluster attention heads, extracting head centroids and FFN centroids as the learngene. Subsequently, we adopt the priority weight-sharing of head centroids for varying number of attention heads and leverage learnable parameters to expand the FFN centroids into multiple FFNs, enabling adaptation to diverse downstream resource constraints. Extensive experiments validate the efficiency and scalability of our initialization method. [24] 65,000 50,000 5,000 10,000 10 CIFAR100 [24] 65,000 50,000 5,000 10,000 100 Food101 [4] 101,000 75,750 25,250 0 101 iNaturalist-2019 [45] A Appendix / supplemental material A.1 Mean Attention Distance in DeiT-S and DeiT-Ti Fig. 7 illustrates the mean attention distance for two other variants of DeiT. In both variants, the lower layers exhibit a dual focus on both local and global aspects, resulting in a relatively sparse distribution of attention heads. Conversely, the higher layers prioritize the global context, leading to a more compact distribution of attention heads. Importantly, many attention heads in these layers exhibit repetitive functionality, contributing to weight redundancy. In the process of FFN clustering, the shallowest layers are preserved as the FFN centroids across different configurations: in DeiT-Tiny, this applies to layers (2,3) and (11, 12) ; in DeiT-Small, to layers (10, 11, 12) ; and in DeiT-Base, to layers (7, 8) and (10, 11, 12) .  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Please see the Experimtent Section. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Tab. 6 presents the details of all downstream tasks.According to the adjustments in the number of attention heads, the weights W O of the projection layer are also proportionally pruned or expanded with the hyperparameter ω and then inherited by the descendant models. Additionally, we directly inherit the weights of layer normalization, patch embeddings, and position embeddings in the ancestry model, which constitute only a small fraction of all weights.We provide a detailed comparison of training efficiency on ImageNet for the Tiny-scale descendant models. As shown in Fig. 8 (b), Cluster-Learngene requires only 4.0 × less training overhead compared to From Scratch on Small-scale descendant models. A key advantage of our approach is that descendant models initialized with the learngene achieve faster convergence, owing to a superior initialization point.We conduct an ablation to assess whether inheriting parameters from the nearest module to the cluster centroid or averaging parameters after clustering heads and FFNs is more effective. Table 7 shows Cluster-Learngene excelling when inheriting from the closest heads/FFNs to the centroid, a logical approach as these modules aptly represent similar semantics within a cluster. 7 : Ablation on the selection of head centroids. All results are derived from the 6-layer downstream models. we conduct experiments on CIFAR-100 and use DeiT-Small as the ancestry model.Table 8 shows the results for different values of ε. ε = 1 implies no FFN clustering, potentially causing negative transfer. ε = 100 means clustering all FFNs in adjacent layers with identical head centroid counts, resulting in an excessive cluster of FFNs and subsequent degradation in the performance of initialized descendant models. Our Cluster-Learngene (ε = 10) strikes a good balance between these issues. 9 : Increment or decrement the count of attention heads. "Decrementing" denotes halving the number of attention heads in the first four layers, reducing them by a quarter in the middle four layers, and maintaining them in the last four layers relative to the ancestry model. Conversely, "Incrementing" represents the opposite pattern.ε = 1 ε = 10 ε =Tab. 9 presents two scenarios where the number of attention heads varies across different layers. Across all descendant model configurations, "Incrementing" consistently outperforms "Decrementing" by a margin of 1.45% in terms of accuracy. These findings align with previous research [34, 30] , which suggests that setting more attention heads in higher layers can assist these layers in learning more abstract and high-level feature representations.As illustrated in Fig. 3 , Cluster-Learngene incurs a total of 300 + 50 * 10 = 800 epochs. In contrast, pre-trained models require retraining a new model of elastic scale for 300 epochs each, followed by fine-tuning. Considering only the training expense of new models, training 10 such models would require at least 300 * 10 = 3000 epochs. Therefore, our algorithm can reduce the computational cost by at least 3×. This efficiency demonstrates how our method overcomes the one-size-fits-all limitation inherent in the Pretraining-Finetuning approach.The performance of descendant models within Cluster-Learngene heavily depends on the quality of the ancestry model. If the ancestry model harbors inherent limitations or biases, these issues may propagate and even amplify within the descendant models, potentially compromising their effectiveness. Therefore, ensuring the robustness and fairness of the ancestry model is paramount for maintaining the performance of descendant models in Cluster-Learngene. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.† Corresponding authors. * The terms "foundation model" and "ancestry model," as well as "downstream model" and "descendant model," are interchangeably utilized unless distinctions are explicitly mentioned.† Please see Appendix A.3 for more details.§ Please see Appendix A.9 and A.10 for more visualization. 


Decision: In this paper, the authors introduce Cluster-Learngene, a novel method for initializing vision transformers. After the discussion period, most of the reviewers tend to accept this paper. Reviewer GgZj still kept their negative rating. However, they didn't respond to the authors' feedback. After checking the paper, the reviews, and the discussions, I tend to accept this paper. The authors are required to include the discussions with the reviewers in their final version. Besides, the code shared during the discussion period should also be made publicly available.

Reviews: review: paper_summary: paper_summary: This paper introduces Cluster-Learngene, a weight initialization method for initializing downstream models of various sizes with pre-trained models. The proposed method is based on Learngene and includes the following two improvements: MSA and FFN centroid, which extract critical parameters and reduce redundancy; priority weight-sharing, which is able to initialize downstream models with varying number of attention heads.
strength_weakness: strength_weakness: (1)	Section 3.1 and L183-196 seems to have a messy formatting of formulas; for example, some letters are missing. This reduces the readability and overall quality of the paper;

(2)	Some key experiments were insufficient. For example, the counterpart of this work is the pre-existing Learngene methods, whereas in Table 1, only training from scratch is compared and in Table 2 results on ImageNet are also lacking;

(3)	Some details of the proposed method are not explained clearly enough (see Questions).
questions: questions: (1)	After Adaptively Learngene Clustering, the number of layers obtained is variable. How to deal with that if it is not equal to the number of layers in the downstream model?

(2)	In priority weight-sharing (Figure 2), what if the number of head centroids is less than the number of head in the downstream model?

(3)	Are all parameters of the downstream model (inherited or not) updated (or fixed) during training?

(4)	In L222, “The learnable parameters in Eqns. (7) are implemented through a nonlinear mapping such as a neural network with the rectified linear units (ReLU)”, does this mean that $\hat{W}_t$ is more than just a parameter?
limitations: limitations: The limitations have been discussed.
paper_summary: paper_summary: The paper introduces Cluster-Learngene, a novel approach for initializing Vision Transformers (ViTs). This method clusters attention heads and position-wise feed-forward networks (FFNs) from a large "ancestry" model to form a condensed initialization core, termed "learngene." By leveraging the density characteristics of attention heads and FFNs, Cluster-Learngene effectively customizes models of elastic scales to meet the resource constraints of various downstream tasks. The approach not only preserves critical knowledge but also enhances efficiency by reducing redundancy and facilitating weight-sharing in descendant models.
strength_weakness: strength_weakness: Insufficient Downstream Task Evaluation: The dataset assessments for downstream tasks are currently limited to classification challenges. The generalizability of this method to other tasks such as segmentation and detection remains unknown.
questions: questions: 1. The Cluster-Learngene method is primarily designed for models with full attention mechanisms. Given the emergence of attention variants aimed at reducing computational costs, such as sparse or low-rank attention, it remains to be seen whether this method can be effectively transferred and applied to these types of models.
2. The hyperparameter Eps (ε) is currently set manually and plays a crucial role in the effectiveness of the clustering. Could the authors provide ablation study data to explore the robustness of the algorithm with varying settings of this parameter?
limitations: limitations: Please refer to weakness and questions
paper_summary: paper_summary: This paper seeks to address the issue of overgeneralizing the applicability of large pre-trained models in deep learning, particularly when faced with task-specific resource constraints. The authors propose Cluster-Learngene, a novel method that clusters critical internal modules from a large ancestry model and uses them to initialize descendant models of various scales. By adaptively clustering attention heads and feed-forward networks (FFNs) based on their density characteristics, Cluster-Learngene creates a flexible and efficient initialization process. The method also incorporates priority weight-sharing and learnable parameter transformations to expand the learngene. Extensive experiments demonstrate that Cluster-Learngene outperforms other initialization methods in efficiency and in customizing models to fit the resources available for downstream tasks.
strength_weakness: strength_weakness: 1.	The paper lacks a complexity analysis or specific time cost assessments for Cluster-Learngene, which are crucial for understanding its computational efficiency.

2.	Regarding the choice of hyperparameters, the authors could provide an ablation study, particularly for parameters like ε, to better understand their impact on the model's performance.

3.	Will the code be available for open source in the future?
questions: questions: See Weaknesses.
limitations: limitations: The authors have addressed the limitations and potential negative societal impacts of their work.
paper_summary: paper_summary: Cluster-Learngene is an innovative approach for initializing Vision Transformer models. It works by inheriting adaptive clusters from a large pre-trained ancestry model. The key features of this method include:
Adaptive Clustering: Cluster-Learngene adaptively clusters the attention heads and feed-forward networks within each layer of the ancestry model based on their density characteristics. This forms compact and information-dense modules called "learngenes."
Efficient Initialization: The method uses priority weight-sharing and learnable parameter transformations to expand these learngenes. This enables efficient initialization of descendant models with varying scales and resource constraints.
Resource Customization: Cluster-Learngene tailors the model initialization to the downstream task resources, offering a more efficient alternative to the standard pretraining and fine-tuning approach.
Empirical Validation: Extensive experiments on multiple datasets demonstrate that Cluster-Learngene outperforms traditional initialization strategies and is competitive with more resource-intensive fine-tuning methods.
Scalability and Efficiency: The approach achieves faster convergence and higher data efficiency compared to models initialized from scratch, making it effective for scenarios with limited data.
Broad Applicability: The paper shows the effectiveness of Cluster-Learngene across different model scales and various downstream tasks, showcasing its versatility.
In summary, Cluster-Learngene presents a novel and efficient strategy for initializing Vision Transformer models, addressing the challenge of resource constraints in downstream tasks.
strength_weakness: strength_weakness: Lack of comparison to other clustering methods: The paper uses a density-based clustering approach, but does not compare this to other clustering methods like k-means. 

Experiments focused on image classification: The experimental evaluation is primarily on image classification tasks. Testing on a broader range of vision tasks like object detection or segmentation would demonstrate the generalizability of the approach better.

The paper format can be improved: Table 2 should underline the highest result. Tables 4 and 5 should be three-line tables.
questions: questions: 1. Can you offer more experiments on other architectures(ConvNext, hybrid arch etc.), datasets(VTAB, FGVC etc.) or even modalities(like NLP tasks&models) to further validate the effectiveness of Cluster-Learngene? Which I believe will be powerful proof of its generalizability.

2. There have been other methods in model merging, can you compare Cluster-Learngene with these methods[1,2] to highlight its advantages, in experimental and theoritical ways?
[1]Yang, Enneng, et al. "Adamerging: Adaptive model merging for multi-task learning." arXiv preprint arXiv:2310.02575 (2023).
[2]Yang, Enneng, et al. "Representation Surgery for Multi-Task Model Merging." arXiv preprint arXiv:2402.02705 (2024).

3. How computationally expensive is the clustering step compared to the savings in downstream fine-tuning? Is there a break-even point in terms of number of downstream tasks where this becomes beneficial?
limitations: limitations: The authors provide a good discussion of limitations in the appendix, acknowledging that the performance of descendant models depends heavily on the quality of the ancestry model.

The discussion of broader impacts is quite limited. Given that the method aims to make transfer learning more efficient, some discussion of potential positive impacts (e.g. democratizing access to good models with less computational resources) as well as negative impacts (e.g. potential to amplify biases present in large pre-trained models) would be valuable. The authors could expand on mitigation strategies for some of these risks.

Comments: comment: Dear Reviewer,



Thank you for the insightful comments and constructive suggestions you provided during the review process. Your expertise and attention to detail have significantly enhanced the quality of our manuscript. We are confident that our final submission will reflect the improvements discussed and will be better aligned with the interests and expectations of the community. 

Once again, thank you for your time, efforts, and for providing us with the opportunity to improve our work.



Best regards,
Thank you for your detailed rebuttal and the efforts to address the concerns raised. I appreciate the clarity provided in your responses, and it helps in better understanding the intentions and execution of your work. However, I would like to emphasize a couple of points to ensure that the manuscript aligns with the real-world applicability and theoretical positioning of the proposed method.

Continuing Concerns:
Understanding of Generalization:
It appears there might be a misunderstanding regarding my previous comment on generalization. The success of widely adopted models like CLIP and SAM, which are finely tuned for various tasks including new categories in industrial applications, highlights a pivotal issue. Practically, most AI engineers would opt to fine-tune such established models rather than adopting a new method. The repeated emphasis on "generalization across different models" might be diverting from the practical effectiveness of widely used methods. A practical example, such as deploying a new industrial detection model, typically sees immediate application of methods like SAM. It may be beneficial to realign the discussion to how your method could complement or offer distinct advantages over these well-entrenched approaches in specific scenarios.

Critique of Over-Generalization in Deep Learning:
The critique of pretrained models' over-generalization in deep learning presented in your manuscript seems to create a problem that might not exist to the extent suggested. While academic research indeed isn't just about surpassing previous performance metrics, it's crucial to base arguments on clearly existing gaps or limitations in current methodologies. I believe it would strengthen your paper to reframe the narrative to emphasize the unique contributions of your method rather than positioning it against a seemingly exaggerated issue. This approach would likely resonate more authentically with the community and highlight the novel aspects of your work without undermining the established successes of deep learning generalization.

Conclusion:
Despite these concerns, I acknowledge the potential of the method and the effort to ensure reproducibility by providing code, albeit with some readability issues. Considering the comprehensive responses and revisions, I am inclined to adjust my score to a 5, reflecting my recognition of your method’s merits and my best wishes for your success in refining the paper to meet the community's standards and expectations.
Dear Reviewer 2s46,

This paper received mixed ratings. I would really appreciate it if you could check the authors' responses and post your further concerns (if there are still remaining concerns). Thank you so much!

Your AC
Dear Reviewer GgZj,

This paper received mixed ratings. I would really appreciate it if you could check the authors' responses and post your further concerns (if there are still remaining concerns). Thank you so much!

Your AC
Thank you very much for your timely response and feedback. The specific questions you raised are excellent, as they facilitate our further clarification and help to dispel any misunderstandings. Below, I will respond to your questions.

> Q1:Reproducibility Concerns.

R1: We appreciate your focus on the reproducibility of our work and understand the importance of meeting the community's standards in this regard. We apologize for any confusion caused by our initial submission and are committed to addressing your concerns. To ensure reproducibility, we have provided the complete code to the Area Chair as per the guidelines. We have taken care to include all necessary details and dependencies to allow for the replication of our experiments and results. 



> Q2:Lack of Practical Significance.

R2: Thank you for highlighting the practical significance of our proposed method. We recognize the importance of clearly presenting the unique advantages and potential impact of our approach. In our manuscript, we place a strong emphasis on the practical benefits of our method, which include:

(1) **Scalability**: Our approach is designed to initialize downstream models of various scales to meet the needs of different downstream tasks. This level of scalability, which allows for a high degree of customization and adaptation to specific task requirements, is a characteristic that technologies such as LoRA and Adapters do not possess, as demonstrated through our results in Tables 1 and 2 and Figure 3.

(2) **Resource Efficiency**: We underscore the resource optimization capabilities of our method, a critical feature for applications operating under constrained computational budgets, as demonstrated in Table 3 and Figure 5.



> Q3: Abstract and Generalization Concerns

R3: In the context of our work, by "overgeneralize ," we refer to the tendency to apply pre-trained models in a broad range of tasks without sufficient consideration for the unique requirements and constraints of specific applications. While models like CLIP have indeed shown remarkable versatility, our paper emphasizes the importance of customizing model initialization for downstream models of varying scales according to the particularities of each task, especially when there are resource constraints. We have designed experiments of different scales, as demonstrated in **Table 1** and **Figure 3** of the main paper, to showcase the effectiveness of our tailored initialization approach.



> Q4:Clarity and Coherence in Abstract

R4: We are grateful for your candid feedback on the clarity and coherence of our abstract. We have taken your comments to heart and will undertake a thorough revision of the abstract.  We will focus on defining the problem of task-specific resource constraints in the context of deep learning and succinctly describe how Cluster-Learngene offers a novel and effective strategy for model initialization that is both elastic in scale and sensitive to the needs of downstream tasks. In addition, we also hope that the explanations provided in R3 will aid in your understanding of our abstract and help clarify any misunderstandings.
While the method proposed has potential, I have concerns that I believe need to be addressed to meet the publication standards particularly regarding reproducibility and the practical significance of the work. Below, I outline my concerns and request further clarifications.

### Weaknesses:

1. **Reproducibility Concerns:**
  My previous concern was not a request for code per se but rather an expression of apprehension regarding the reproducibility from the submission as presented. Even though the authors have now provided the code to the Area Chair, the submission in its current form does not meet the standards for acceptance based on reproducibility. I will lower my confidence rating and defer to the Area Chair to make a final judgment based on the quality of the provided code.

2. **Lack of Practical Significance:**
   The practical significance of the proposed method is not sufficiently demonstrated. The manuscript fails to convincingly argue why the industry would adopt such a method in the face of increasingly popular large models like SAM and CLIP, which are becoming more efficient to fine-tune (e.g., through methods like LoRA and Adapters). The abstract does not adequately highlight the relevance and potential impact of the proposed method within this context.

### Specific Questions:

- **Q1: Abstract and Generalization Concerns:**
  The statement in the abstract, "However, the common practices often overgeneralize the applicability of these models, overlooking the task-specific resource constraints," is quite puzzling. Generalization is a cornerstone of success in deep learning. It appears there is a misunderstanding or miscommunication here, as models like CLIP are, in fact, increasingly being applied to specialized tasks with considerable success. Could you please clarify what is meant by overgeneralization in this context?

- **Q2: Clarity and Coherence in Abstract:**
  Frankly, the abstract is poorly written, with unclear problem statements and key solutions, which makes the entire paper somewhat perplexing. It is crucial for the abstract to clearly outline the challenges and articulate how the proposed method addresses these challenges. A revision to better communicate these aspects would significantly improve the paper's impact and understandability.

**Closing Remarks:**

I hope these comments are helpful for refining your paper. Enhancing these aspects could substantially improve the manuscript's clarity, practical relevance, and reproducibility, thereby strengthening its contribution to the field.
Thank you very much for your timely response and constructive suggestions. Below, I will address the questions that you have newly raised:

> Q1:Performance in Practical Settings

We compared the traditional industry-standard approach of pre-training followed by fine-tuning and found that although our approach inherits fewer parameters, the performance is better. Therefore, this result needs to be viewed comprehensively and the representative results are shown in the table below: 

| Method                 | Params (M) | iNat-2019 | Cars      | **NABirds** |
| ---------------------- | ---------- | --------- | --------- | ----------- |
| Pretraining-Finetuning | 10.5       | 68.48     | 86.81     | 78.33       |
| Cluster-Learngene      | **7.5**    | **71.09** | **89.87** | **80.76**   |

 In response to your request, we are pleased to report that in our exploration of real-world applications, we have extended our presentation beyond the classification results featured in the paper. Specifically, we have supplemented our experiments with segmentation results in the rebuttal phase, as per your guidance. Moreover, to initialize $N$ downstream models of different scales, Pretraining-Finetuning would require pre-training $N$ times. However, our method only needs to utilize **one** pre-trained model, from which we can cluster and expand to initialize $N$ downstream models of different scales, saving the pre-training time for $N-1$ models.

> Q2: Reproducibility Concerns

Due to the organizing committee's policy against providing anonymized code links directly in the rebuttal response, we have sent the anonymized code link to the Area Chair (AC) as per the instructions. We kindly ask the AC to forward it to you, in the hope that this will address your reproducibility concerns.
Thank you for addressing many of the concerns I initially had with your manuscript. However, I maintain my original score due to two persisting issues:

Performance in Practical Settings: While the theoretical contributions of your work are clear, the practical performance does not appear to significantly surpass the traditional industry-standard approach of pre-training followed by fine-tuning. The advantages of your method over existing techniques are not clearly demonstrated, and the problem statement and solution approach described in the abstract seem somewhat superficial. For your work to have a substantial impact, it would be beneficial to provide more concrete evidence or comparative analysis demonstrating its superiority in real-world applications.

Reproducibility Concerns: The complexity of your proposed method and the lack of critical implementation details raise concerns about reproducibility. Without these details, it may be challenging for readers to replicate your results or compare them with their work, which could undermine the credibility and utility of your paper. I would encourage you to include these essential details or simplify the approach to facilitate a better understanding and reproducibility by the wider research community.

I hope these points are helpful for refining your paper.