KIND: Knowledge Integration and Diversion for Training Decomposable Models
Authors: Yucheng Xie
Fu Feng
Ruixiao Shi
Jing Wang
Yong Rui
Xin Geng
Venue: ICML 2025 poster (2025)
Keywords: Model Initialization
Decomposable Pre-trained Models
Diffusion Transformers
Primary area: deep_learning->algorithms

Abstract: Pre-trained models have become the preferred backbone due to the increasing complexity of model parameters. However, traditional pre-trained models often face deployment challenges due to their fixed sizes, and are prone to negative transfer when discrepancies arise between training tasks and target tasks.
To address this, we propose **KIND**, a novel pre-training method designed to construct decomposable models.
KIND integrates knowledge by incorporating Singular Value Decomposition (SVD) as a structural constraint, with each basic component represented as a combination of a column vector, singular value, and row vector from $U$, $\Sigma$, and $V^\top$ matrices.
These components are categorized into **learngenes** for encapsulating class-agnostic knowledge and \textbf{tailors} for capturing class-specific knowledge, with knowledge diversion facilitated by a class gate mechanism during training.
Extensive experiments demonstrate that models pre-trained with KIND can be decomposed into learngenes and tailors, which can be adaptively recombined for diverse resource-constrained deployments. 
Moreover, for tasks with large domain shifts, transferring only learngenes with task-agnostic knowledge, when combined with randomly initialized tailors, effectively mitigates domain shifts.
Code will be made available at https://github.com/Te4P0t/KIND.

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Yucheng Xie Fu Feng Ruixiao Shi 12
Jing Wang" 12
Yong Rui
Xin Geng
Abstract
(a)
Traditional Pretraining
Knowledge Diversion
Dataset
Dataset
Pre-trained models have become the preferred backbone due to the increasing complexity of model parameters. However, traditional pre- trained models often face deployment challenges due to their fixed sizes, and are prone to negative transfer when discrepancies arise between train- ing tasks and target tasks. To address this, propose KIND, novel pre-training method de- signed to construct decomposable models. KIND integrates knowledge by incorporating Singular Value Decomposition (SVD) as a structural con- straint, with each basic component represented as combination of a column vector, singular value, and vector from U, Σ, and VT matri- ces. These components categorized into learn- genes for encapsulating class-agnostic knowledge and tailors for capturing class-specific knowl- edge, with knowledge diversion facilitated by a class gate mechanism during training. Exten- sive experiments demonstrate that models pre- trained with KIND can be decomposed into learn- genes and tailors, which can be adaptively re- combined for diverse resource-constraine ployments. Moreover, for tasks with large do- main shifts, transferring only learngenes with task-agnostic knowledge, when combined with randomly initialized tailors, effectively mitigates domain shifts. Code will be made available at https:/gittub..om//e4P/////
Fixed-size
Pretrained
Decomposable
Leard -gen
Model
Model
Model
(b)
Targe
Deploy
Direct
Recombine
Limited Resources
Wrong Size
(c)
Targe
All Knowledge
Task-agnostic
Lora XRedundancy (Random)
Tailor (Random)
Domain Shift
Figure I. (a) Traditional pre-training prioritizes maximizing per- formance on training datasets, often producing fixed-size models and making them prone to negative transfer. In contrast, KIND redefines the training objective pre-train models that hoth structure- and knowledge decomposable. (b) Consequently, KIND enables pre trained models to be adaptively restructured, facilitat- ing deployment in diverse resource-constraines scenarios. (c) Ad- ditionally. the task-agnostic knowledge encapsulated learngenes can effectively mitigate domain shifts.
knowledge-dcoompssbble
de-
stone of modern machine learning (Qiu et al., 2020; Han et al., 2021; Feng et al., 2025b). These pre-trained models have proven highly effective, especially when combined with parameter- efficciet fine-tuning (PEFT) techniques such LoRA (Hu al., 2022; Hayou al., 2024) and its vari- ants (Zhang et al., 2023; Valipour et al., 2023; Liu al.,
1. Introduction
2024). However, traditional pre-training approaches primar- ily focus on optimizing performance for specific training datasets, often neglecting their transferability to downstream tasks and adaptability to diverse deployment scenarios.
The increasing size of models has significantly increased computational costs, making pre-trained models corner-
'School Computer Science and Engincering, Southcast University, Nanjing, China Key Laboratory of New Genera- tion Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University). Ministry of Education, China 'Lenovo Research. Correspondence to: Jing Wang <wangjing91 @ @ww...du.ccn.... Xin Geng <xgeng aigeng@seu.cduu seu.edu.cn>.
As a result, pre-trained models typically have fixed, large size, designed to encapsulate much knowledge as possi- ble from the training data. This design, however, presents significant challenges for practical deployment, which is often constrained by factors like memory usage, processing power, and response time (Zhang et al., 2022). More im- portantly, when downstream tasks differ significantly from the pre-training datasets, the transferred knowledge can
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
become redundant (Feng al., 2024), biased (Ren et al., 2024), even harmful (Wang al., 2019; Rosenstein et al., 2005). These limitations underscore that traditional pre- trained models may not always serve as optimal backbones, as illustrated in Figure 1. This raises a critical question: Can we rethink the pre-training process to develop decom- posable pre-trained models that can be adaptively adjusted to meet the specific requirements of downstream tasks and deployment scenarios?
to meet storage and computational limits, maintaining per- formance without sacrificing performance. 3) Tasks with Large Domain Shifts: KIND transfers learngenes only, combined with randomly initialized tailors, enabling effi- cient adaptation via class-agnostic knowledge.
Our main contributions are as follows: 1) We redefine the pre-training objective by shifting the focus from solely max- imizing model performance diverting knowledge into class-agnostic knowledge and class- components, facilitating the construction of a more flexible and decom- posable backbone adaptable to various scenarios, 2) We pro- pose KIND, a novel pre-training method that integrates and diverts knowledge, marking the first application of learn- genes to image generation tasks. 3) We establish a new benchmark for evaluating transfer efficiency and flexibility in diffusion models. Extensive experiments demonstrate that KIND achieves aath-off performance while pro- viding flexible storage and computational efficiency.
Recently, a novel knowledge transfer framework called Learngene has been introduced (Wang et al., 2023). Un- like traditional transfer learning methods, Learngene en- capsulates task-agnostic knowledge into modular network fragments (Feng et al., 2023) known as learngenes, to en- hance the efficiency of knowledge transfer and improve network adaptability. Building upon the Learngene frame- work, we propose KIND, a novel pre-training method that performs Knowledge INtegration and Diversion during the pre-training process. KIND is designed to construct flexible and decomposable pre-trained models, facilitating adap- tive transformations to address the diverse requirements of downstream tasks and deployment scenarios.
2. Related Work
2.1. Initialization and Training of Variable-sized Models
KIND decomposes the weight matrix into basic components for knowledge integration, then associates class-specific and class-agnostic knowledge with distinct components to facil- itate knowledge diversion. For this decomposition, KIND employs Singular Value Decomposition (SVD), represent- ing each basic component as a combination of column vector, singular value, and row vector derived from the U, Σ. and VT matrices. These basic components are cat- egorized into two types: learngenes, which encapsulate class- agnostic knowledge, and tailors, which capture class- specific knowledge. Instead of directly applying SVD to pre-trained model weights (Han et al., 2023; Zhang & Pi- lanci, 2024; Robb et al., 2020), KIND incorporates SVD as a structural constraint during pre-training and trains the basic components rather than the full weight matrices. Such indirect training enables more explicit control over each class-specific component, guided by a class gate mecha- nism, thereby facilitating effective knowledge diversion.
Practical deployments often encounter constraints related to memory usage, processing power, and response time, necessitating models of variable sizes (Zhang et al., 2022). However, traditional pre-trained models are typically fixed in size, requiring retraining when a suitable model size is unavailable (Qiu et al., 2020; Han et al., 2021). While tra- ditional model compression techniques, such as knowledge distillation (Gou J, 2021; Muralidharan et al., 2024) and model pruning (Zhang al., 2024a; Castells et al., 2024), can generate models of variable sizes, they involve repeated operations for each model size, resulting in significant inef- ficiencies in both time and resource consumption.
The Learngene framework, inspired by the transfer of ge- netic information in nature (Feng et al., 2023), encap- sulates common knowledge into modular network frag- ments, termed "learngenes", and employs them to initialize variable-sized models (Wang et al., 2023). Notably, the process of condensing knowledge from pre-trained mod- els into learngenes incurs a one-time cost, eliminating the need for further training during model initialization. Cur- rent learngene-based methods, either direct transfer selected layers from pre-trained models (Wang et al., 2022; 2023), or apply predefined rules (e.g.. Kronecker products) to dis- till knowledge knowledge into learngenes (Xia et al., 2024; Feng et al., 2025a). However, these approaches neglect the alignment between model components and their correspond- ing knowledge, limiting their efficiency and adaptability.
We conduct experiments on class-conditional image genera- tion tasks to better demonstrate knowledge transfer, using Diffusion Transformers (DiTs) (Peebles & Xie, 2023) as the backbone for diffusion models. We prait DiT-B and DiT-L with KIND on ImageNet- KK resulting in decompos- able models that can be effectively divided into learngenes and tailors. Extensive experiments evaluate KIND across three scenarios. 1) General Tasks: Models pre-trained with KIND perform on par with traditional pre-trained models (often outperforming them) without additional computa- tional costs. 2) Resource-constrainee Scenarios: KIND facilitates flexible combinations of learngenes and tailors
In contrast, KIND enhances such alignment through knowl- edge diversion during pre-training, constructing decom- posable model that enables more flexible and efficient ini-
2

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
tialization across varying model sizes.
and value V RTxD, with parameter matrices Wi. 9 W and W E Dxd:
2.2. Parameter Efficient Fine-Tuning (PEFT)
Qike
The increasing complexity of model parameters has made fine-tuning all parameters of pre-trained models resource- intensive and time-consuming (Touvron et al., 2021; Achiam et al., 2023). To address this, PEFT techniques are devel- oped to adapt large pre-trained models to new tasks by fine-tuning only a small set of parameters (Hu et al., 2022; Houlsby et al., 2019; Hu et al., 2023; Chen et al., 2022). Re- cent approaches apply SVD to pre-trained weight matrices, fine-tuning models by adjusting singular values, a process known as spectral shift (Han et al., 2023; Robb al., 2020; Sun et al., 2022), by fine-tuning singular vectors (Zhang et al., 2024b; Zhang & Pilanci, 2024). However, existing PEFT methods rely on models pre-trained with traditional objectives and do not fully consider their adaptability as universal backbones across diverse tasks.
A softmax
VG, A e e Vd
(2)
MSA mechanism combines h self-attention heads A and projects the concatenated outputs using weight matrix Wa:
MSA = concat( (A1, A2,.., W, Wo E Ad
(3)
In the implementation of MSA, the matrices Wi, W and W Rxd for h attention heads are combined into three parameter matrices W Wk. and W. E Dxhd
PFF layer comprises two linear transformations W.n E DxD and Wout E D'DD with a GELU (Hendrycks & Gimpel, 2016) activation function:
PFF(2) = GELU(>,,,. in + b out + b2
(4)
In contrast, KIND decomposes pre-trained models into learngenes and tailors through knowledge diversion. The class-agnostic knowledge encapsulated in learngenes signif- icantly enhances transfer adaptability, particularly for tasks with large domain shifts compared to the training tasks.
where b, and by are the biases for the linear transformations, and D' denotes the hidden layer dimensions.
3.2. Knowledge Integration in Weight Matrices
FSGAN (Robb et al., 2020) directly applies SVD pre- trained model parameters and fine-tunes the singular val- ues for adaptation, achieving success in image segmenta- tion (Sun et al., 2022) and generation (Han et al., 2023) This shows that SVD can create a compact parameter space, facilitating efficient fine-tuning of pre-trained models.
3. Methods
3.1. Preliminary
3. LATENT DIFFUSION MODELS
Latent diffusion models transfer the diffusion process from the high-resolution pixel space the latent space by em- ploying an autoencoder E, which encodes an image into a latent code 2 = E(x). A diffusion model is then trained to generate the corresponding latent code in a denoising process, minimizing the following objective:
However, directly applying SVD to pre-trained parameter matrices decomposes them based on fixed orthogonaliza- tion rules, leading to poor interpretability and making it challenging to determine whether the knowledge in each basic component is class-specific. This limits the model's decomposability. risking the loss of valuable knowledge.
L = E. E...,,,.(lll [||s- eo (z:lc, 1)112]
To address this, we integrate knowledge by reconstructing weight matrices using the SVD-derived components U, Σ. and V, where each basic component is combination of column vector, singular value and row vector from U, Σ, and VT, We then explicitly associate each basic component with a specific type of knowledge (either class-specific or class- agnostic), which is achieved through a class gate mechanism to divert knowledge (Section 3.3).
Here, Eq is noise prediction network that predicts the noise added to 2, at timestep t, conditioned on
3.1.2. DIFFUSION TRANSFORMERS (DITs)
DiT is a transformer-based architecture for noise predic- tion, replacing the traditional UNet. Given an image x E H and its latent code 2 E R xhaxe encoded by ε, DIT divides the latent code = into patches, which are then mapped D-dimensional patch embeddings, with added position embeddings.
For the DiT architecture, the main weight matrices across the L-layers are = (w(()-L) (lLL) ((()) W(1+L) w-(1-2) oue W(111).1 Let w(( represent any weight matrix in layer 1, where S and S {g, k v, o, in, out} denotes the set of subscripts. The matrices U
The structure of DiTs resembles that of Vision Transformers (ViTs), which comprises L stacked layers, each containing Multi-Head Self-Attention (MSA) mechanism and a Point- wise Feedforward (PFF) layer. In each layer, a self-attention head A, performs self-attention using a query Q, key K,
(W(">)) denotes the set (W w ...., w. Similar notations throughout the paper follow this convention.
3

---

g (e E [Na. Nr, Nas Nr Nol, E S,L [1, L]} KIND: Knowledge Integration and Diversion for Training Decomposable Models
(a)
Knowledge Integration
Pointwise Feedforward
Timestep
Label
W
X
Σ
X
VT
Ng
Nay NNy
Multi-Head
DiT Block
X
X
Pointwise Feedforward
XN
(b)
Knowledge Diversion
0
0
Class Gate
VAE
C2
C3
C4
C dls
Loss L
Training Data
3
2t
Latent Space
Pixel Space
Figure 2. (a) For each weight matrix in DiTs, we integrate it into the product of matrices U. E and VT, formally inspired by SVD. The components of these matrices are then explicitly partitioned into the learngenes and tailors, which encapsulate class-agnostic and class-specific knowledge. respectively. (b) Knowledge is diverted through class gate ensuring each training image updates only the learngenes their corresponding class-related tailors, so that the class-agnostic knowledge can condensed into the learngenes, while knowledge specific each class is diverted into corresponding tailors.
si()) (i are the corresponding components that constitute (!()) which is calculated as:
NG is the number of basic components forming learngenes:
: U. 2 v. T
(7)
In this way, the basic components of each matrix are partitioned into Ng learngenes and Nets tailors, with the model parameters represented as = - Te.
(5)
Σ (1.)) (1.3) U2
(t.))
To encapsulate the class-specific knowledge the e-th class in the c-th tailor, we introduce a class gate G [O...., 0, 1,0,0..... Oj E Net for knowledge diversion dur- ing the training of DiTs, where only one the element at the c-th position is set to 1, corresponding to the class in- dex. This mechanism ensures that, for each training class, only the weight parameters of the learngene and relevant tailors are updated (See Algorithm for more details). The optimization objective defined
where (1) = diag(or) with jo. (1,1) (1.2)
(.r)).
..... o
Ul (1(11) (1,1) (1.2)
th (..)) E Rmixr, and (.)) [cr (1.1) (1.2) U. / E Rrxma The rank r and dimen- ...
sions my and m2 associated with Wil). Each basic component is represented (1.1) (ub. (1.3) ((.1)
3.3. Knowledge Diversion by Class Labels
Given dataset with Nats classes, objective is allocate knowledge of each class to the corresponding basic com- ponents while extracting class-agnostic knowledge shared across all classes, thereby achieving knowledge diversion.
Net
arg min La s.t. s.t G 9.T
Σ Te
(8)
where the loss function L is defined in Eq. (1).
We categorize basic components into learngenes and tai- lors, encapsulating class-agnostic and class-specific knowl- edge, respectively. Specificallyy, the components are parti- tioned based the number of classes Nats and matrix rank satisfying = Nats Ny + Ng. where Ny denotes the number of components per class, with the tailor for the e-th class To:
3.4. Decomposable Models for Diverse Scenarios
After training via knowledge diversion, obtain decom- posable model made of basic components, which can be adaptively reassembled to meet the target memory size and specific task requirements during deployment.
Te = 11 E [[c 1). Cr,c Nr1. S, e [1,
Recombination for Variable Model Sizes. In practice, not all knowledge in pre-trained models is applicable to
(6)

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
(a) Recombination for Variable Model Sizes
Table I. Performance of constructing variable-sized models on training classes. "Para." denotes the total number of model pa- rameters, which reflects the model size. "Time" is the additional training steps required to construct models of the target sizes.
Learngene Random Trainable
Target Classes
Frozen
Direct Select T Rog
Para.(M)
Methods
Time
FID
sFID
1St
Prec.
f
X
X
457.0
Trad. PT
9.68
6.15
72.22
0.69
0.47
DiT-B DIT-L
362.5
Heur-LG 100K
23.86
7.24
48.34
0.54
0.47
249.2
Laptop-diff' 100K
17.20
7.25
57.07
0.59
0.47
249.2
Auto-LG
100K
18.38
8.22
57.68
0.58
0,46
245.9
KIND
9.33
6.80
79.39
0.69
0.46
(b) Class-ognostic Knowledge for Large Domain Shift
129.7
Trad. PT
25.14
7.57
47.15
0.53
0.46
Target Classes
108.4
Hcur-LG
100K
41.53
8.93
34.29
0.42
0.47
76.5
Laptop-diff 100K
48.22
11.09
31.19
0.37
0.47
76.5
Auto-LG
100K
45.69
10.77
32.77
0.39
0.47
Add Random Tailors Frandom
70.2
KIND
21.14
8.85
58.18
0.55
0.44
X
minimize inter-class similarity, we merge certain similar classes based on their superclasses in WordNet (Miller, 1995), resulting in a final set of 611 classes. Among these, 150 classes used for pre-training the diffusion models, while the remaining 461 classes serve novel classes for constructing downstream tasks. Further details can be found in Appendix A.3. Additionally, we use datasets, includ- ing CelebA-HQ (Huang et al., 2018), Hubble (Weinzierl, 2023), MRI, and Pokémon, to simulate large domain shifts compared to the training data.
Figure 3. (a) For downstream tasks with pre-trained classes, it can dircetly select the tailors corresponding the target classes while discarding unrelated ones. (b) When encountering tasks with large domain shifts, only the learngene is transferred, combined with randomly initialized tailors for class-specific fine-tuning.
downstream tasks, and transferring excessive knowledge can be both memory-intensive and redundant. For down- stream tasks similar to parts of the training dataset, we can directly select the appropriate pre-trained tailors combined with learngenes. For instance, when deploying DiT pre- trained on ImageNet to a conturcce-cntttineeed device for generating images of "dogs", we can deploy only the tailor corresponding to "dog" (Taog) and the learngene (G). Sim- ilarly, for unknown classes, we can select closely related tailors for fine-tuning, adjusting the number of tailors based on the available memory.
4.2. Basic Setting
For pre-training DiT, we train class-conditional latent DiTs of sizes -B and -L, with latent patch size of p 2 at a 256 256 image resolution on training classes. All models are trained using Adam W with batch size of 256 and constant learning rate of 1 x 10-4 over 300K steps. An exponential moving average (EMA) of DiT weights is used with a decay rate of 0.9999, and results are reported using the EMA model. During image generation, a classifier- free guidance (cfg) scale of 1.5 is applied. Performance evaluated using Fréchet Inception Distance (FID) (Heusel et al., 2017), sFID (Nash et al., 2021), Fréchet DINO dis- tance(FDD) (Stein et al., 2023), Inception Score (Salimans et al., 2016) and Precision/Recall (Kynkäänniemi et al., 2019). Further details are provided in Appendix A.2.
Class-agnostic Knowledge for Large Domain Shift. Pre- trained models often encounter negative transfer when fac- ing large domain shifts, challenge that also affects the transfer of pre-trained tailors. In such cases, class-agnostic knowledge encapsulated in learngenes fully demonstrates its advantages. Thus, for tasks with large domain shifts, only learngenes need to transferred, along with randomly initialized tailors Trandom- During fine-tuning, we freeze the learngene and only update the tailors, enabling them to learn class-specifie knowledge from the downstream task, thereby achieving more efficient fine-tuning.
5. Results
5.1. Construction of Variable-Sized Pre-Trained Models
The models pre-trained by KIND are inherently decom- posable, consisting of learngenes that encapsulate class- agnostic knowledge and tailors that capture class-specific knowledge. This decomposition enables flexible deploy- ment of models across devices, as demonstrated in Table 1.
4. Experiments
4.1. Datasets
We conduct cansicondd generation on ImageNet-
Compared to traditional pre-trained models, KIND achieves comparable performance with the same number of training
1K (Deng et al., 2009), which contains 1, 1,000 classes. To
5

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Table 2. Performance of various PEFT and learngene methods on novel classes. All methods are fine-tuned for 50K steps on 18 downstream tasks involving novel classes. "Para." denotes the average number of trainable parameters, while "FLOPs" represents the average total floating-Pol point operations required during fine-tuning.
DiT-B/2
DiT-L/2
Methods
(ara.(M) (LOPP(()
FID
sFID
ISt
Prec.
Recall
Para.(M) FLOPPs()
FID4
sFID
IS1
Prec.
Recall
SVDiff
0.1
43.6
55.01
18.12
19.6
0.35
0.55
0.2
155.0
49.59
16.81
20.8
0.38
0.56
OFT
14.2
119.7
36.19
17.79
32.0
0.48
0.50
50.5
425.6
24.81
18.27
44.1
0.59
0,47
LoRA
12.8
50.1
36.70
16.28
31.6
0.44
0.57
45.3
178.2
22.55
14.00
46.3
0.55
0.56
PISSA
12.8
50.1
33.16
15.51
34.6
0.49
0.52
45.3
178.2
19.41
14.72
53.7
0.63
0.50
LoHa
12.7
87.1
42.38
17.37
27.3
0.40
0.58
45.3
309.6
29.79
15.17
35.8
0.49
0.59
DoRA
12.8
129.5
35.87
16.40
32.3
0.45
0.56
45.6
503.0
21.28
14.16
48.3
0.57
0.55
Heur-LG
129.6
43.6
55.45
22.14
24.4
0.33
0.48
456.8
155.0
41.83
19.23
30.9
0.40
0.51
Auto-LG
129.6
43.6
56.38
21.39
25.5
0.30
0.49
456.8
155.0
31.78
18.71
41.7
0.46
0.54
KIND
12.8
33.7
20.94
14.75
62.4
0.53
0.50
45.4
119.6
12.87
12.93
86.1
0.65
0.51
Full FT
129.6
43.6
26.49
15.08 45.1
0.51
0.55
456,8
155.0
14.51
13.16
69.1
0.63
0.55
steps, without increasing training complexity. Additionally, the decomposable nature of KIND allows for direct recom- bination tailored to specific deployment needs, with no fur- ther time-consuming steps required. In contrast to knowl- edge distillation and pruning (Zhang et al., 2024a), KIND offers significant advantages by avoiding the resource over- head of repeated distillation and pruning for each model size, which is required in distillation-bar based methods.
Table 3. Performance comparison of KIND and PEFT methods in transferring to downstream tasks with significant domain shifts, evaluated using FDD for image quality assessment.
CelebA HQ
Hubble
MRI
Pokemon
DFT-B
DiT-L
DiT-B
DiT-L
DFT-B
DiT-L
DiT-B
DiT-L
SVDiff'
0.622
0.388
0.385
0.305
0.187
0.148
0.605
0.469
OFT
0.343
0.226
0,255
0.168
0.056
0.046
0.469
0.321
LoRA
0.284
0.197
0.232
0.142
0.061
0.056
0.412
0.285
PISSA
0.281
0.195
0.211
0.152
0.057
0.051
0.418
0.295
Unlike traditional leamgenes,, such as Heur-LG (Wang et al., 2022) and Auto-I.G (Wang et al., 2023), which directly transfer certain layers from traditional pre-trained mod- els, KIND encapsulates task-agnostic knowledge into learn- genes and retains task-specific knowledge in tailors through knowledge diversion. This enables the direct combination of learngenes and tailors without additional training, ensuring both efficiency and adaptability across tasks.
LoHa
0.336
0.268
0,252
0.189
0.065
0.130
0.439
0.316
DoRA
0.282
0.203
0.589
0.330
0.043
0.048
0.396
0.333
KIND
0.201
0.152
0.124
0.109
0.042
0.040
0.343
0.262
In contrast, KIND diverts class-agnostic knowledge into learngenes, creating a flexible backbone for adaptation to downstream tasks with novel classes. The randomly ini- tialized tailors are adjusted via low-rank assumptions, com- bining with learngenes to meet task-specific needs, thereby improving transfer efficiency and enhancing the generaliz- ability of knowledge transfer. As shown in Figure 4 and Table 2, KIND -enerraed images outperform PEFT methods in both quality and performance metrics.
5.2. Performance on Tasks with Novel Classes
To evaluate KIND's adaptability, we use learngenes as the backbone with randomly initialized tailors and compare it to PEFT methods based traditional pre-trained models on tasks with novel classes. As shown in Table 2, KIND achieves state-of-the-art results on DiT-B and DiT-L, reduc- ing FID by 6.54 and sFID by 1.07, while using only 45.4M parameters and saving 35.4G FLOPs on DiT-L.
5.3. Performance on Tasks with Large Domain Shifts
KIND demonstrates significant advantages in adapting to tasks with novel classes, with these benefits becoming even more pronounced when dealing with tasks involving large domain shifts. As shown in Table and Figure 5, KIND out- performs PEFT methods on both DiT-B and DiT-L, achiev- ing substantial improvements in image generation quality.
Despite the efficiency of PEFT methods, significant per- formance gap remains compared to Full FT, highlighting the task discrepancy between training and novel classes. PEFT methods, which freeze pre-trained parameters, struggle to adapt to novel tasks. As shown in Figure 4, PEFT-generated images perform poorly in capturing class-specific knowl- edge due to limited trainable parameters and task mismatch. Existing learngene methods like Heur-LG and Auto-LG transfer partial knowledge from pre-trained models, but the transferability of each module, trained with traditional ob- jectives, is limited.
This further demonstrates that the knowledge encapsulated in learngenes is sufficiently class-agnostic, allowing to be shared effectively across various tasks. In contrast, PEFT methods based on traditional pre-trained models show disad- vantages, as the knowledge learned from ImageNet is often difficult to transfer to new domains, especially in specialized fields like Hubble and MRI. This highlights key limitation
6

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
SVDiff
OFT
LoRA
PiSSA
LoHA
DoRA
Heur-LG Auto-LG
Full FT
KIND
Raccoon
Swan
Fox
Koala
Bird
Figure Selected samples from tasks with novel classes, generated by KIND and other PEFT methods using the DiT-L/22 model, with
resolution of 256 x 256. All images generated using classifier-free guidance (cfig) scale of 3.0.
SVDiff
OFT
LoRA
PISSA
LoHA
DoRA
KIND
Table 4. Ablation study on different components of KIND.
MRI Celeba-HO
LG Tailor Gate FIDJ
sFID.,
IS1
Prec.
Recall
#1
60.28
19.96
20.4
0.30
0.49
DiT-B2
#2 #3
V
49.54
18.08
23.2
0.34
0.56
V
/
21.60
14.84
59.7
0.54
0.50
KIND
V
V
20.94
14.75
62.4
0.53
0.50
Pokemon Hubble
DiT-L/2
42.04
18.07
28.0
0.41
0.54
#2
33.53
15.55
32.2
0.46
0.59
#3
V
13.03
12.93
85.1
0.64
0.51
KIND
V
12.87
12.93
86.1
0.65
Figure 5. Selected samples from tasks with large domain shifts, generated by KIND and other PEFT methods using the DiT-L/2, with a resolution of 256 x All images are generated using fassifier-ffre guidance (cfg) scale of 1.5.
trained weights and randomly selects No singular vectors to form its backbone, followed by fine-tuning with LoRA. #2 replaces the backbone with learngenes extracted by KIND, based the structure in #1. #3 substitutes tailors for LoRA in fine-tuning the model, without using the class gate.
of current pre-training approaches, which aim to improve generalization by incorporating many domain-specific images as possible during training (Ramesh et al., 2022; Esser et al., 2024). While this may enhance performance, it leads to larger model sizes, reduced transfer flexibility, and increased computational overhead.
As shown in Table 4, the knowledge encapsulated in leam- genes, which undergoes knowledge diversion, is more class- agnostic, making it better suited for adaptation to down- stream tasks, especially when these tasks differ significantly from the training tasks (c.g.. #1 vs. #2). Additionally, tailors can also function as a PEFT method by integrating class- specific knowledge into pre-trained models or learngenes, thereby enhancing the model's ability to acquire new knowl- edge for downstream tasks (#2 vs. #3). Finally, the class gate further enhances this by helping the model distinguish class-specific knowledge, boosting the effectiveness of the tailors (#3 vs. KIND).
5.4. Ablation and Analysis
5.4.1. ABLATION EXPERIMENTS
To assess the effectiveness of learngenes, tailors, and the class gate, we conduct a series of ablation experiments. #1 performs Singular Value Decomposition (SVD) on pre-

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
OFT
Lora
Dora
PISSA Auto-L6
KIND
Tortoise
Bird
Swan
Panda
Cat
Raccoon
w/ Tailor w/o Tailor
10k
Training Process
20K
30k
Koala
Leopard
Fox
Starfish Flower Mountain
w/ Tailor w/o Tailor
40K
50K
Figure 6. Visualization of convergence speed of KIND and other methods downstream tasks. Each image is sampled every 10K steps to illustrate progress more clearly.
Figure 7. Visualization of KIND w/ and w/o Tailers (i.c., leamgene only) across 12 superclasses for 2 different seeds.
agnostic knowledge. To further investigate this, we analyze the properties of the class-agnostic knowledge encapsulated in learngenes. Table 5 compares learngenes themselves (i.e., w/o tailors) with pre-trained models on training tasks. The results reveal that learngenes demonstrate higher entropy. along with lower variance and kurtosis, suggesting that the class-agnostic knowledge they encapsulate widely ap- plicable across diverse classes. Such stability underscores that learngenes, backbone, offer better adaptability to unfamiliar classes than traditional pre-trained models.
Table Comparison pre-trained models learngenes when serving as backbones on training tasks.
Entropyt
Variance,,
Kurtosis.
Raw Images of ImageNet
1.458
6.414e-1
884.3
Pretrained Model
2.387
4.516e
780.1
Learngene
4.046
1.495e
544.9
5.4.2. STRONG LEARNING ABILITY BROUGHT BY LEARNGENES
We also visualize learngenes with and without tailors in Figure 7. The visualizations demonstrate that learngenes are not sensitive to category variations, consistently generating similar images across different class conditions. While these images may lack detailed semantic information on their own, combining them with class-speciff knowledge (i.e., tailors) enables the generation of images corresponding specific classes. This further underscores the inherent commonality of knowledge within learngenes.
As noted in (Wang et al., 2022; Xia et al., 2024), learngenes accelerate downstream model adaptation by transferring common knowledge, offering significant advantage over training from scratch. Beyond this, KIND further improves convergence speed compared to PEFT methods. Figure 6 illustrates the convergence speed of KIND, with images generated by models every 10K training steps.
The convergence speed is generally influenced by the num- ber of trainable parameters during fine-tuning, with PEFT methods focusing reducing this number using techniques like orthogonalization and low-rank constraints (Ding et al., 2023; Han et al., 2024). However, these methods often neglect the transferability of knowledge in pre-trained mod- els by directly fixing their parameters. contrast, KIND leverages learngenes that encapsulate class agnsoti knowl- edge the backbone, offering superior transferability while remaining lightweight. Meanwhile, the tailors capture task- specific knowledge, allowing KIND to achieve faster con- vergence and improved performance on downstream tasks.
6. Conclusion
In this study, we introduce KIND, pre-training method for constructing decomposable models. KIND employs knowl- edge diversion during pre- vre-training, separating plass-agnostie knowledge into learngenes and class-specific knowledge into tailors. This approach enables the adaptive assembly of variable-sized models by selectively integrating relevant tailors. The alass-igi knowledge within learngenes mit- igates the challenges of tasks with large domain shifts, par- ticularly when combined with randomly initialized tailors for task-specific fine-tuning. We demonstrate the effective- ness of KIND in resource-constraing scenarios and tasks with significant domain shifts, with further analysis and vi- sualizations illustrating the robustness of the class- agnostic knowledge encapsulated learngenes.
5.4.3. ANALYSIS ON ALNSSIAGNOSTIC KNOWLEDGE
As discussed earlier, learngenes provide a superior back- bone compared pre-trained models by encapsulating class-
8

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Acknowledgement
Feng, E,, Wang, J., Zhang, C., Li, W., Yang, X., and
Geng, X. Genes in intelligent agents. arXiv preprint arXiv:2306.1 2023.
We sincerely appreciate Freepik for contributing to the figure design. This research was supported by the Jiangsu Science Foundation (BK20243012, BG2024036, BK20230832), the National Science Foundation of China (62125602, U24A20324, 92464301, 62306073), China Postdoctoral Science Foundation (2022M720028), and the Xplorer Prize.
Feng, F., Wang. J., and Geng. X. Transferring core knowl- edge via learngenes. arXiv preprint urXiv:2401.081.33 2024.
Feng, F., Xie, Y., Wang, J., and Geng, X. Wave: Weight template for adaptive initialization of variable-sized mod- els. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR'25), pp. 1-10, 2025a.
Impact Statement
The broader impact of our work lies in how KIND redefines the training objectives of pre-trained models, enabling the construction of decomposable models that can be recom- bined to create models with variable sizes. This approach facilitates faster deployment, reduces resource consumption, and enhances adaptability across various tasks and datasets, offering significant value for both research and industrial applications in Al model scaling and transfer learning.
Feng, F., Xie, Y., Yang. X., Wang. J., and Geng, X. Re- defining <creative>> in dictionary: Towards an enhanced semantic understanding of creative generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR 25)), 2025b.
Gou J, Yu B, M. S. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789- 1819, 2021.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Han, L.,, Li, Y., Zhang, H., Milanfar, P., Metaxas, D., and Yang, F. Svdiff: Compact parameter space for diffusion fine- fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR'23), pp. 7323-7334, 2023.
Castells, T., Song. H.-K., Kim, B.-K., and Choi, S. Ld- pruner: Efficient pruning of latent diffusion models using task-agnostic insights. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'24) pp. 821-830, 2024.
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y. Zhang, A., Zhang, L., al. Pre-trained models: Past, present and future. Al Open, 2:225-250, 2021.
Han, Z.. Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-effiient fine-tuning for large models: A com- prehensive survey. arXiv preprint arXiv:2403. 2024.
Chen, S., Ge, C., Tong. Z., Wang, J., Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transformers for scalable visual recognition. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS 22), pp. 16664-16678, 2022.
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models. In Proceedings of Interna- tional Conference on Machine Learning (ICMI.'24) pp. 1-12, 2024.
Deng. J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVE International Confer- ence on Computer Vision (ICCV'09), pp. 248-255, 2009.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus), arXiv preprint arXiv:1606.08411 2016.
Ding, N., Qin, Y., Yang, G., Wei, F,, Yang, 7., Su, Y., Hu, S., Chen, Y., Chan, C.-M,, Chen, W., et al. Parameter- efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Proceed- ings of the Annual Conference on Neural Information Processing Systems (NeurIPS' pp. 1-12, 2017.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of International Confer- ence on Machine Learning (ICMI. 24)), pp. 1-13, 2024.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficcen transfer learning for nlp. In Proceedings of International Conference on Machine Learning (ICMI. 19), pp. 2790-2799, 2019.

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR'22) 22). Pp. 1-13, 2022.
Ren, H., Materzynska, J., Gandikota, R., Bau, D., and Tor- ralba, A. Art-free generative models: Art creation without graphic art knowledge. arXiv preprint arXiv:2412.00176. 2024.
Robb, E., Chu, W.-S., Kumar, A., and Huang, J.-B. Few- shot adaptation of generative adversarial networks. arXiv preprint arXiv:2010. 1000::0000..993,,,,,. 1943, 2020.
Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.-P., Bing, L., Xu, X., Poria, S., and Lee, R. Lim-adapters: An adapter fam- ily for parameter-efficient fine-tuning of large language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'23), pp. 5254 5276, 2023.
Rosenstein, M. T., Marx, Z., Kaelbling, L. P., and Diet- terich, T. G. To transfer or not to transfer. In NIPS 2005 Workshop on Transfer Learning, pp. 1-4, 2005.
Huang, H., He, R., Sun, Z., Tan, T., et al. Introvae: Intro- spective variational autoencoders for photographic image synthesis. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS' 18), pp- 1-12, 2018.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. In Proceedings of the Annual Conference Neural Information P Processing Systems (NeurlPS' 16), pp. 1-9, 2016.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metrie for assess- ing generative models. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS pp. 1-9,2019, 2019.
Stein, G., Cresswell, J. C., Hosseinzadeh, R., Sui, Y., Ross, B. L., Villecroze, V., Liu, Z., Caterini, A. L., Taylor, E., and
G. Exposing flaws of generative model evaluation metrics and their unfair treatment of dif- fusion models. In Proceedings of the Annual Conference Neural Information Processing Systems (NeurlPS 23), pp. 1-16, 2023.
Liu, S.-y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight- decomposed low-rank adaptation. In Proceedings of In- ternational Conference on Machine Learning (ICML'24) pp. 1-13, 2024.
Sun, Y., Chen, Q., He, X., Wang. J., Feng, H., Han, J., Ding, E., Cheng, J., Li, Z., and Wang, J. Singular value fine- tuning: Few-shot segmentation requires few-parameters fine-tuning. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS'22 pp. 37484-37496, 2022.
Miller, G. A. Wordnet: lexical database for english. Com- munications of the ACM, 38(11 1):1111394444 41, 1995.
Muralidharan, S., Sreenivas, S. T., Joshi. R. B., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. Proceedings of the Annual Conference on Neural Information P Processing Systems (NeurlPS 24), pp. 1-15, 2024.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transform- ers & distillation through attention. In Proceedings of In- ternational Conference on Machine Learning (ICML'11), pp. 10347-10357, 2021.
Nash. C., Menick, J.. Dicleman, S., and Battaglia, P. Gen- erating images with sparse representations. In Proceed- ings of International Conference on Machine Learning (ICMLI'I), pp. 7958-7968, 2021.
Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. Dylora: Parameter-efficient tuning of pre-trained mod- els using dynamic search-free low-rank adaptation. In Proceedings of the Conference of the European Chap- ter of the Association for Computational Linguisties, pp. 3274 3287, 2023.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV'23), pp. 4195-4205, 2023.
Wang, Q., Geng, X., Lin, S., Xia, S.-Y., Qi, L., and Xu, N. Learngene: From open-world to your learning task. In Proceedings of the AAAI Conference Artificial Intelli- gence (AAA1'22), pp. 8557-8565, 2022.
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., and Huang, X. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10): 1872-1897, 2020.
Wang, Q., Yang, X., Lin, S., and Geng, X. Learngene: Inheriting condensed knowledge from the ancestry model to descendant models. arXiv preprint arXiv:2305.02277 2023.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical cext-conditional image generation with clip latents. arXiv preprint arXiv:2204. 06/25, 1(2):3, 2022.
10

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Wang, Z., Dai, Z.., Póczos, B., and Carbonell, J. Characteriz- ing and avoiding negative transfer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR'24) pp. 11293-1130,, 2019.
Weinzierl, M. A. Esa hubble deep space images cap- co datasets/ Supermaaxmanness-auubbbaa hubble, 2023.
tions.
Xia, S., Zhang. M., Yang, X., Chen, R., Chen, H., and Geng, X. Transformer as linear expansion of learngene. In Pro- ceedings of the AAAI Conference on Artificial Intelligence (4AAI224,, pp. 16014-16022, 2024. 2024.
Zhang, D., Li, S., Chen, C., Xie, Q., and Lu, Laptop-diff: Layer pruning and normalized distillation for compress- ing diffuusin models. arXiv preprint arXiv:2404. 10098, 2024a.
Zhang, and Pilanci, M. Spectral adapter: Fine-tuning spectral space. arXiv preprint arXiv:240..13952, 2024.
Zhang, F. Li, L.. Chen, J., Jiang, Z., Wang, B., and Qian, Y. Increlora: Incremental parameter allocation method for parameter-efficien fine-tuning, arXiv preprint arXiv:2308. 12043, 2023.
Zhang, J., Peng, H., Wu, K., Liu, M., Xiao, B., Fu, J., and Yuan, L.. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'22), 22), pp. 12154, 2022.
Zhang, X., Wen, S., Han, L., Juefei-Xu, F. Srivastava, A., Huang. J., Wang. H., Tao, M., and Metaxas, D. N. Spectrum-aware parameter efficient fine-tum tunin for diffu- sion models, arXiv preprint 2055::2455.21050 2024b.

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
A. Training Details
Table 6. Hyper-parameters for KIND diverting knowledge train-
A.1. Details Knowledge Diversion
ing classes of ImageNet-Iee IK..
Algorithm presents the pseudo code for diverting class- agnostic knowledge into learngenes and sass-specific knowledge into tailors.
Training Settings
Configuration
optimizer
Adam lc-4
learning rate weight decay
Algorithm Diversion of Class-agnostic Knowledge and Class-specific Knowledge
batch size
256
training steps image size VAE
200,000
Input: DIT f. Training dataset {xx (9), ) Nat. classes, number epochs Nep batch size B, learning rate a Output: Learngene G
256x256
ema
DiT block Na (DiT-B/-L)
adaLN-Zero
I: Randomly initialize the weight matrices of f. as well as the matrices (, x!!), and vi())
318 424
Nr (DiT-B/-L)
3 4
2: for ep = 1 to Npp do
3: for each batch {(ax, (:)) ) do
4: Update 0 of with U..) x and V.( under the rule of Eq. (5)
merged into superclasses in ImageNetIK and their corre- sponding superclasses are listed in Table 10 and Table 11, while the rest remain the same as the classes in ImageNet- IK.
5: Initialize class gate G E Batt according to labels of images in this batch
6: For each æs, forward propagate d1 fas, G.0)
7: Calculate Lbuch
# > [(Ui, vi) according to
Eq. (1)
B. Additional Results
loss Lram compute the gradients with respect to U.. x and V,S; Ltuch, VyLtacc and Vy
Backward propagate
We provide more images of novel classes generated by KIND which is DiT-L/2 model composed of learngenes and tailors at 256 256 resolution, as shown in Figure 8-1.5
Update the leamgeness (i()) LG and vit).
U = U.l a Vu Loauth
G.. S.G
VILoach
ve. = vill a V
Update the tailors (i( x!(() and V:!0
10:
Tu = T., a G(V Lrench)
Σい (i)
-
C:
Vil = v(ll
a G(Vv v Loanch)
11: end for Tea
end for
A.2. Hyper-parameters
Table 6 presents the basic settings, including learning rate, training steps and the number of learngene components NG and tailor components Ny for KIND integrating and divert- ing knowledge. And Table 7 presents the typee-parameters of PEFT and other learngene methods on 18 downstream tasks. Apart from general hyper- parameters, the hyper-parameters specific to each method. Among them, the parameter r of Lora, PiSSA, Dora and LoHA denotes the rank and the r in OFT denotes the block number respec- tively.
also record
A.3. Details of Downstream Tasks
Table 9 presents the details 18 downstream tasks, which are sorted by the class numbers in each task. Each task is composed of [7] novel classes, where the classes
12

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Table 7. Hyper-parameters for PEFT and learngene methods when finc-tuning novel classes of ImageNet- IK
Training Steps
Learning Rate
(DIT-B 03//11 /L)
Batch Size
Methods
Rank Block Number
Task
#4
#6
#8
#10
#11
#12
#13
#14
#15
#16
#17
#18
ID
OFT
256
50K
le-4
6
Lora
512 50K
le-3
-B
21
39
54
60
69
72
78
78
78
84
84
87
90
90
93
99
102
105
-L
28
52
72
80
92
96
104
104
104
112
112
116
120
120
124
132
136
140
PiSSA
256 50K
le-3
-B
21
39
54
60
69
72
78
78
78
84
84
87
90
90
93
99
102
105
-L.
28
52
72
80
92
96
104
104
104
112
112
116
120
120
124
132
136
140
Dora
256 50K
le-3
-B
54
60
72
78
84
84
87
90
93
102
105
-L
28
52
72
80
92
96
104
104
104
112
112
116
120
120
124
132
136
140
LoHA
256 50K
le-3
-B
10
19
27
30
34
36
39
39
39
42
42
43
45
45
46
49
51
52
-L
14
26
36
40
46
48
52
52
52
56
56
58
60
60
62
66
68
SVDiff
256 50K
5e-3/3e-3
Heru-LG
le-4
Auto-LG
256 50K
lc-4
KIND
256 50K
le-3
Full FT
256
50K
le-4
Table 8.
Detailed FID of PEFT and learngene methods when fine-tuning each novel classes.
Task ID #9
Methods
#1
#2
#3
#4
#5
#6
#7
#8
#10
#11
#12
#13
#14
#15
#16
#17
#18
SVDiff
143.4
144.9
140.6
112.3
112.5
114.2
117.4
104.3
108.6
107.5
102.3
93.6
97.5
108.9
109.6
95.2
81.6
100.6
OFT
92.3
90.4
93.7
71.9
76.0
86.7
82.3
72.6
74.5
76.9
65.4
63.0
67.8
77.3
78.1
64.4
63.7
75.2
85.5
94.2
97.7
75.8
80.3
89.4
76.6
76.2
68.9
64.7
79.1
67.0
63.3
78.6
PiSSA
83.0
89.4
93.0
69.3
73.8
82.1
81.1
69.4
71.6
76.2
64.3
60.5
64.3
74.4
70.2
60.7
59.7
71.1
DrT-B
LoHa
94.9
100.8
108.3
84.3
88.2
95.8
97.5
85.5
86.6
90.6
78.9
73.2
79.3
88.8
88.0
76.4
69.4
86.9
Dora
82.9
91.6
94.0
73.1
77.8
87.2
87.8
73.9
75.4
79.0
67.8
64.2
69.6
77.0
78.6
65.2
62.2
77.0
Heru-LG
98.7
111.1
122.4
97.0
102.5
114.4
122.2
95.1
99.5
108.4
87.8
90.5
91.7
103.6
101.8
94.4
88.3
100.2
Auto-LG
107.8
113.7
129.3
105.6
100.1
117.7
112.3
100.5
100.3
105.7
89.9
91.4
93.7
105.1
101.7
99.1
87.3
99.9
KIND
55.0
73.4
70.4
52.7
58.3
65.2
59.7
47.8
51.9
56.7
42.7
43.7
44.6
56.3
62.8
43.5
39.8
52.0
Full FT
56.3
75.5
78.1
59.9
65.1
72.6
70.1
58.6
60.1
66.1
54.2
51.4
53.8
63.7
63.3
52.8
51.5
62.7
SVDiff
118.2
132.0
127.2
98.3
97.2
103.0
105.6
92.3
98.3
97.2
92.9
84.1
90.5
102.3
110.4
109.0
76.3
92.3
OFT
59.4
71.4
72.4
52.3
57.9
65.1
64.5
55.1
60.4
58.6
48.7
50.1
52.7
62.2
60.8
50.1
51.1
61.9
Lora
54.6
72.5
55.9
59.0
53.7
54.5
48.7
49.4
57.1
47.5
46.1
59.7
PiSSA
52.6
68.9
67.0
50.2
54.1
60.8
58.4
49.2
48.4
55.4
43.1
44.4
44.3
53.1
48.6
41.1
41.9
50.6
DELL
LoHa
65.3
78.7
83.3
63.9
69.6
77.6
78.3
66.1
66.7
73.8
62.2
59.0
62.5
68.6
68.1
59.3
56.1
72.5
Dora
52.2
71.3
68.0
52.9
56.7
64.3
62.4
52.2
51.1
58.7
46.9
47.0
47.9
56.0
55.7
44.9
45.1
56.2
Heru-LG
73.3
92.6
97.1
79.9
86.2
94.4
94.5
77.8
82.2
88.8
72.4
71.9
77.2
85.8
85.1
74.8
71.9
84.0
Auto-LG
66.5
81.1
82.6
69.3
70.0
80.4
76.3
66.6
67.4
72.8
58.8
59.3
61.0
70.9
69.3
64.9
58.1
70.2
KIND
39.0
66.2
61.8
44.2
46.0
54.7
47.5
39.1
40.0
46.3
33.2
36.3
34.7
45.9
43.9
31.5
30.9
40.8
Full FT
38.0
64.1
61.9
44.6
45.5
56.1
50.8
41.4
41.2
48.5
36.1
38.6
38.2
47.4444
43.4
35.3
34,9
44.1
13

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Table 9. Details of superclasses in each downstrcam task
Task|
Superclasses of ImageNet
#1
n02510455
n02509815
n01662784
n8213 n02083346 n02437616 n02457408
#2
n03187595 n03788365
n03933933
nO4273569 n03843555 n03400231
n03325584 n094725997
n03874293
n04591713
n03854065
n03868863
n07711569
#3
n07753592
n03763968
n03109150
n09399592
n03903868
n03720891
n02939185
n03908714
n04014297
n02804414
n06785654
n04131690
n02794156
m02971356
n02056570
m02965783
n04243546
n06359193
#4
n02877765 n04238763
n04009552
n03666591
n07614500
n09332890
n01629276
n04483307
n03291819
n02120997
n03717622
n04041544
n03873416
n04467665
n03394916
n03272010
104118538
n04367480
n04447861
n03775071
#5
nO4086273
m04141076
n03657121
n03379051
n02401031
n01503061
n03840681
n04380533
n03871628
nl 1819999
n04090263
n04557648
n03016953
n02808304
n02879718
n03724870
n04423845
n02917067
n03691459
n02672831
n04146614 m04525305
n04264628
#6
n03496892
n06874185
n04392985
n03485794
n03982430
n04540053
n03602883 n02871525
n02978881
n03961711
n04005630
n03065424
n04200800
n02823750
n03344393
n04325704
n03220513
n03498962
n04356056
n03347037
n09421951 n07760859
n04133789
n07565083
#7
n04332243
n02883205
n03405725
n03017168
n04553703
n03777568
n02951358
n07720875
n03637318 n02090827
n04265275
m03028079
n07920052
n03954731
n04141327
n03255030
n03447447
n00002684
n03530642
n03425413
n04524313 n03110669
n03764736
n12267677
n02676566
n03417042
#8
n03676483
n02865351
n03792972
n02974003
n02906734
n07860988
n03249569
n00021265
n02727426 n03782006
n02317335
n02815834
n03388043
nO3529860
n02817516
n03761084
n09246464
n03899768
n03970156
n04485082
n01769347
n07880968
n03197337
n03876231
n02699494
n03472232
#9
n02121808 m07734744
n03424325
n03494278
n03935335
no3690938
n03240683
n03467068
n02980441 n03450230
n02512053 n04517823
n02730930
n03133878
n03259280
n04376876
n03803284
n03920288
n02966193
n02814860
n02669723 m03000134
n02793495
nO2766320
nO3649909
n04125021
#10
n03985232
n
n03388549
nO4065272
n03633091
n02916936
n03201208
n04208210
n02988304
n09229709
n02769748
n02791270
n03814639
n03481172
n03692522
nO4501370
n03584829
n02843684
n04252225
n03196217
n02704792 n03384352
n03785016
n03459775
n03599486
n01806143
n03294048
n03995372
#11
n04341686
n03603722
n04081281
nO3623198
n03497657
n02690373
109193705
n04486054
n01986214
n01639765
nO31 10011
m03532672
n03540267
m02356798
n03662601
n04277352
mO4204238
n04204347
n04530566
n04033901
n03793489
n02268 148
nO4209239
n04266014
n01861778
n03062245
103179701
1939491
#12
n04111531
n04597913
n07932039
n04118776
n02859443
n04523525
n02077923
n03938244
n07707451
n04371430
n02797295
m04228054
nO3207743
n01882714
n07716906
n03216828
n04589890
n03063689
n03630383
n04252077
n02153203
m03207941
n03908618
n03796401
n07697313
n02898711
m04548362
n03290653
n02930766
#13
n03000247
m04040759
n04590129
m03492542
n03733805
n04044716
n01877812
n04418357
m09428293
n03045698
n03998194
n03443371
n03983396
n03902125
n03598930
n01844917
n04509417
n02441326
n02786058
n03134739
n03838899
nO4192698
nO2837789
n02074367
n02701002
n07717070
n03977966
n12992868
n03445777
n04162706
#14
n03538406
n03314780
n03916031
n04310018
n04074963
n04462240
n03250847
n01704323
n07753113
n04532106
n09288635
m04033995
n03929855
n03733281
n04562935
n03124043
n03682487
n04487081
n03743016
n03670208
n03980874
n04596742
n03457902
n04536866
n03085013
n03527444
n04099969
n04141975
n04326547
n02825657
#15
n04417672
n02966687
n03868242
n02692877
n04435653
n04039381
n02084071
n02776631
a02950826
n04350905
n04552348
n07831146
n04149813
nO3787032
n03791053
n04357314
n04476259
n02129604
n03791235
n03992509
n01604330
n03891332
nO4613696
n04592741
n02687172
m02782093
n04525038
n02835271
n01674464
n07742313
n02454379
#16
103910333
nU2323902
n03327234
n01726692
n03095699
n04443257
n04201297
n02667093
n04584207
n04328188
n02909870
m04311174
n04067472
n04270147
n04344873
n03777754
n03658185 185
n03706229
n07836838
n03770679
n03208938
n01976146
n02062744
n03697007
n03476684
n02469914
n04458633
n02274259
n10565667
n01872401
n03584254
n04019541
n03461385
#17
n03063599
n04576211
n03841143
n03617480
n02992211
nO4251 144
n04239074
n02131653
n04254120
n02979186
n01514668
n03476991
n04229816 16
n03776460
n04429376
n01696633
n01905661
n03594945
n04370456
n02
n04230808
n03141823
n00001930
m03485407
n04372370
n04285008
n03032252
n04286575
n02894605
n03709823
n02329401 163309
n03721384
n03857828
#18
n02870880
103127747
n02880940
n04346328
n04482393
n03800933
n04152593
n53051550
a03042490
n04317175
n03661043
m04548280
nO4235860
n02807133
n02790996
n03877472
n07892512
n07871810
n03866082
n07875152
n10148035
n04531098
n038 14986
n02927161
n04296562
n03729826
n04023962
n01768244
n00003553
n04127249
n04505470
n03825788
n03794056
nO3929660
n03742115
14

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Table 10. Details of superclasses in ImageNet-IK
Superclass
Classes of ImageNet
n02084071
n02085620
n02085782
n02085936
n02086079
n02086240
n02086646
n02086910
n02087046
n02087394
n02088094
n02088238
n02088364
n02088466
n02088632
n02089078
n02089867
n02089973
n02090379
n02090622
n02090721
n02091244
n02091467
n02091635
n02091831
n02092002
n02092339
n02093256
n02093428
n02093647
n02093754
n02093859
n02093991
n02094114 14
n02094258
n02094433
n02095314
n02095570
n02095889
n02096051
n02096177
n02096294
n02096437
n02096585
n02097047
n02097130
n02097209
n02097298
n02097474
n02097658
n02098108
n02098286
n02098413
n02099267
n02099429
n02099601
n02099712 12
n02099849
102100236
n02100583
n02100735
10210
n02101006
n02101388
n02101556
102102040
n02102177
n02102318
n02102480
n02102973
n02104029
n02104365
n02105056
n02105162
n02105251
n02105412
n02105505
n02105641
n02105855
n02106030
n02106166
102106382
n02106550
n02106662
n02107142
n02107312
n02107574
n02107683
n02107908
n02 108000
102108089
n02108422
1
1
n02109047
nO2 102109525
n02109961
n021 10063
n02110185
n02110341
n02110627
102110806
n02110958
102111129
n02111277
n02111500
n02111889
n02112018
n02112137
n02112350
n02112706
n02113023
n02113186
n02113624
H02113712
nO21 13799
n021 13978
n01503061
n11530575
178
nO1532829
nO1534433
nO1537544
nO1558993
561560
n01580077
n01582220
n01592084
n01601694
nO1608432
n01817953
n01818515
n01819313
n01820546
n01824575
n01828970
n01829413
n01833805
n01843065
n01843382
n02002556
n02002724
m02006656
n02007558
n02009229
n02009912
n0201 1460
n02012849
n02013706
n02017213
n02018207
n02018795
n02025239
n02027492
n02028035
n02033041
n020371
n02051845
n02058221
n02159955
n02165105
n02165456
n02167151
n02168699
n02169497
n02172182
n02174001
n02177972
192190166
n02206856
n02219486
n02226429
n02229544
n0223 1487
n02233338
n02236044
m02256656
102259212
n02264363
n02469914
n02481823
n02483362
n02483708
n02484975
n02486261
n02486410
n02487347
n02488291
n02488702
n02489166
n02490219
n02492035
n02492660
n02493509
n02493793
n02494079
n02497673
nO2500267
n01726692
n01728572
n01728920
n01729322
n01729977
n01734418
n01735189
n01737021
n01739381
n01740131
n01742172
n01744401
n01748264
n01749939
n01751748
n01753488
n01755581
n01756291
n02512053
n01440764
n01443537
n01484850
n01491361
n01494475
n01496331
n01498041
n02514041
n02526121
n02536864
n02606052
n02607072
n02640242
n02641379
n02643566
n02655020
n01674464
n01675722
n01677366
n01682714
n01685808
n01687978
n01688243
n01689811
nO1692333
n01693334
n01694178
n01695060
n02401031
n02403003
n02408429
n02410509
n02412080
n02415577
n02417914
n02422106
n02422699
n02423022
n01769347
n01770081
nO1773157
n01773549
n01773797
n01774384
n01774750
n01775062
n01776313
n02083346
n02114367
n02114548
n02114712
no2114855
n02115641
n02115913
n02116738
n02117135
n02441326
m02441942
n02442845
n02443114 14
n02443484
n02444819
n02445715
n02447366
n12992868
n12985857
112998815
n13037406
n1 n13040303
n13044778
n13052670
n1 3054560
n02153203
nO1795545
n01796340
n01797886
n01798484
n01806567
n01807496
n02120997
n02125311
n02127052
m02128385
n02128757
n02128925
n02130308
n00224259
n02276258
nO2277742
n02279972
n02280649
n02281406
n02281787
n04531098
n02795169
n02808440
n03950228
n04049303
n04398044
n04493381
n01629276
no1629819
n01630670
n01631663
n01632458
n01632777
n01662784
n01664065
nO1665541
nO1667114
n01667778
n91669191
n01905661
n01924916
n01950731
n01955084
n01990800
n02321529
n02121808
n02123045
n12123159
n02123394
n02123597
n02124075
n02329401
n02342885
n02346627
m02361337
n02363005
n02364673
n04341686
n03781244
n03788195
n03837869
n03877845
n03956157
15

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Table 11. Details of superclasses in ImageNet-IKK IK (continued)
Superclass
Classes of ImageNet
Superclass
Classes of ImageNet
n01976957
n01978287
n01978455
n01980166
n01981276
n02134971
n02137549
n02138441
n02118333
n02119022
n02119789
n02120079
n02120505
n02268148
n02268443
n02268853
n02131653
n02132136
n02133161
n02134084
n02134418
n03906997
n02783161
n03388188
n04530566
n02981792
n03947888
n04147183
n04612504
n01604330
n01614925
n01616318
n00021265
n07579787
n07583066
n075841
n0759061
n01696633
n01697457
n01698640
n01639765
n01641577
n01644373
n01644900
n01940736
n01943899
n01968897
n01844917
n01855032
n01855672
n01860187
n01942177
n01944390
n01945685
n01861778
n01871265
n02504013
n02504458
n02062744
n02066245
n0207120
n00002684
n01914609
n01917289
n09256479
n02090827
n02091032
n02091 134
n01976146
n01983481
n01984695
n01985 128
n02134971
n02137549
n02138441
n02323902
n02325366
n02326432
nO2328150
n02268148
n02268443
n02268853
n02335503
n02395406
n02396427
n02397096
n03906997
n02783161
n03388 183
n03472232
n02777292
n03535780
n03888605
n03001627
n02791122 124
n03376595
n03800933
n02787622
n02804610
n03884397
n00001930
n02799071
n09835506
n03791235
n02814533
n03100240
n03930630
n04235291
n02860847
n03218198
n03497657
n02869837
n03124170
n04259630
n04014297
n02895154
n03146219
n03405725
n03018349
n03337140
n04550184
n02883344
n03014705
n03127925
n04576211
n032275562
n03393912
n03895866
n03540267
n03026506
n(4254777
n04230808
n03534580
n03770439
n04136333
n03380867
n03047690
n03680355
n02888711
n0431 1004
n04366367
nO4532670
n03682487
n03075370
n03874599
n07707451
n07714571
n07716358
n07718747
n02766320
n03125729
n03131574
n0 A01604330
n01614925
n01616318
n0392811 116
n03452741
n04515003
n01696633
n01697457
n01698640
n04464852
n03478589
n04389033
n01940736
n01943899
n01968897
nO3985232
n03642806
n03832673
n01942177
n01944390
n01945685
nO4524313
n03673027
n04347754
n02062744
n02066245
n02071 1294
n03051540
n03710637
n03710721
n02090827
n02091032
n02091134
n04565375
n03773504
n04008634
n02880940
n03775546
n04263257
n03294048
n03924679
n04004767
n03327234
n03930313
n04604644
n02942699
n03976467
n04069434
n03603722
n04560804
n04579145
n07679356
n07684084
n07695742
n07717070
n07717410
n07717556
n00003553
n12057211
n n12620546
n13134947
n12144580
n13133613
16

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
D
Figure 8. Images of n02510455 generated KIND.
Figure Images of n02509815 generated by KIND.
17

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Figure 10. Images of n01882714 generated by KIND.
Figure 11. Images of n02120997 generated by KIND.
18

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Figure 12. Images of n01503061 generated by KIND.
Figure 13. Images of n09193705 generated by KIND.
19

---

KIND: Knowledge Integration and Diversion for Training Decomposable Models
Figure 14. Images of n09472597 generated by KIND.
Figure 15, Images of n09399592 generated by KIND.
20


Reviews:
[{"review_id": "ivlUkREsBe", "reviewer": "Reviewer_abHL", "summary": "This paper proposes a method to decompose a model’s learnable matrices into class-agnostic knowledge (learngenes) and class-specific knowledge (tailors) using Singular Value Decomposition (SVD). The learning process for tailors is regulated by a class gate, ensuring that only one class is activated per image. After training, the decomposed components can be flexibly recombined based on specific downstream tasks and resource constraints by selecting the required tailors.\n\nThe authors conduct experiments using a generative DiT model to demonstrate a better trade-off between the number of training parameters and generation quality. Additionally, the proposed method improves knowledge transferability to novel classes and datasets with larger domain shifts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer abHL, \n\nWe sincerely appreciate your recognition of our practicality and efficiency. \nDue to length constraints, **experimental tables and figures, are provided via anonymous links (permitted by ICML25)**.\n\n>**Q1:Class Gate and Sparse Gradients**\n\nParameter updates remain sufficient without excessive sparsity (see [Table](https://anonymous.4open.science/api/repo/a-8112/file/1.pdf)). The learngene, shared across all classes, receives updates from all samples, while tailors benefit from batch training, ensuring broad class coverage (batch size=256, total classes=150).\n\nThis setup also reduces gradient conflicts across classes, allowing KIND to achieve competitive performance with similar overhead while maintaining a flexible, decomposable structure.\n\n>**Q2:Task-agnostic Knowledge in Learngenes**\n\nFigure 7 shows that images generated solely from learngenes lack class-specific semantics (visually similar regardless of input labels under the same seed).\n\nTo quantify this, [Table](https://anonymous.4open.science/api/repo/a-8112/file/2.pdf) (i.e., Table 5) analyzes the classification distributions of InceptionNet across raw ImageNet images, pre-trained model outputs, learngene-generated images, and pure noise. Entropy, variance, and kurtosis are used to measure distribution uniformity, discreteness, and sharpness.\n\nResults show that learngene-generated images exhibit low correlation with all ImageNet classes and align statistically closer to noise, confirming their class-agnostic nature and lack of semantics.\n\n>**Q3:Specific Combinations of Training Class**\n\nThis concern arises only with extremely limited classes, e.g., shared features like fur may dominate between cats and dogs, but introducing a distinct class, such as turtles, reduces this effect.\n\n[Tabel](https://anonymous.4open.science/api/repo/a-8112/file/3.pdf) shows that beyond 100 classes, additional classes offer minimal benefit. As long as the class set is sufficiently diverse, its specific composition has little impact.\n\n>**Q4:Nature of Transferred Knowledge**\n\nBLIP-based VQA verification in [Tabel](https://anonymous.4open.science/api/repo/a-8112/file/4.pdf) confirms that learngene-generated images are natural images (only 5.1\\% are classified as 'noisy images').\nThis suggests that learngenes encode a general noise-to-image mapping, while tailors inject class semantics.\n\n>**Q5:Cross-class Validation**\n\nAs shown in [Figure](https://anonymous.4open.science/api/repo/a-8112/file/5.pdf), the 'panda' tailor fails to generate images for other classes, confirming its class-specific nature.\n\n>**Q6: Theoretical Assumptions**\n\nThe core assumption holds under large domain shifts. \nFrobenius norm in [Tabel](https://anonymous.4open.science/api/repo/a-8112/file/6.pdf) shows minimal weight perturbations ($||E^{[t]}||\\ll||W^*||$) when transferring learngenes across domains.\n\n>**Q7:Larger Number of Classes**\n\nA larger number of training classes is unnecessary, as 100 classes is sufficient for capturing class-agnostic knowledge (see Q3). \nOnce the class-agnostic knowledge has encapsulated in learngenes, additional classes can be integrated by training only corresponding tailors.\n\n>**Q8:Missing Details for Table 5**\n\nSee Q2.\n\n>**Q9:Related Works in NAS**\n\nBoth KIND and NAS support variable-sized models, but only KIND extracts class-agnostic knowledge for transferability, while NAS focuses solely on network structure.\n\n[Tabel](https://anonymous.4open.science/api/repo/a-8112/file/7.pdf) shows that NAS struggles with domain shifts. \n\n>**Q10:Tailor Initialization and Model Size**\n\nIn new tasks, the tailor is typically randomly initialized, which is simple and effective (see Q14). \n\nThe number of tailors and parameters varies with class count, with Table 2 reporting the **average parameters** across tasks.\n\nModel size is also influenced by task complexity, and adjusting each tailor's rank balances performance and efficiency (see [Tabel](https://anonymous.4open.science/api/repo/a-8112/file/8.pdf)).\n\n>**Q11:Relationship between Parameters and FLOPs**\n\nTraining parameters and FLOPs are related but not strictly proportional, as FLOPs depend on both parameter count and computational patterns like matrix multiplications.\n\n>**Q12:Performance Gains over FT**\n\nWe further compare full-parameter fine-tuning in [Tabel](https://anonymous.4open.science/api/repo/a-8112/file/9.pdf), confirming that KIND improves performance with lower computational cost.\n\n>**Q13:Multi-class Generation**\n\nMulti-class generation can be achieved by setting the class gate to 1 for desired classes (see [Figure](https://anonymous.4open.science/api/repo/a-8112/file/10.pdf)).\n\n>**Q14:Selection Criterion for Tailors**\n\nAs noted in Q10, randomly initializing tailors is simple and effective. \nFuture work may explore fine-tuning from similar-class tailors (e.g., Koala from Monkey in [Tabel](https://anonymous.4open.science/api/repo/a-8112/file/11.pdf)) or adaptively integrating multiple tailors via MoE.", "claims_and_evidence": "•  The claimed benefit of reduced training complexity (Lines 257–258) lacks sufficient experimental support. If training classes are isolated using the class gate, does this introduce sparse gradients, potentially making the training process more challenging?\n\n•  The claim that learngenes capture task-agnostic information is not well supported by the visualization in Figure 7. The generated images from learngenes exhibit recognizable object-like patterns, and different seeds produce distinct patterns. This observation contradicts the assertion that learngenes do not favor any specific class. Further clarification or empirical validation is needed to reconcile this discrepancy.\n\n•  The extent of class-agnostic information varies across different classes. For example, fur may be considered class-agnostic when training involves only cats and dogs, but this assumption may not hold in a more diverse dataset. This suggests that the knowledge encoded in learngenes is highly dependent on the specific combination of training classes. This dependency should be explicitly discussed.\n\n•  While learngenes are expected to be transferable across different domains, the exact nature of the transferred knowledge remains unclear. The paper does not provide a thorough validation of what aspects of knowledge are being effectively transferred. Additional experiments or analysis would help substantiate this claim.\n\n•  A cross-class validation is missing to support the claim that tailors are class-dependent. For instance, what happens if the tailors from class A are used to generate images for class B? Including such an experiment would help confirm whether tailors truly capture class-specific information and whether their utility is restricted to the classes they were trained on.", "methods_and_evaluation_criteria": "See the Claims And Evidence session.", "theoretical_claims": "The core assumption in Line 636 states that the learned weight is close to an underlying matrix and that the matrix decomposition holds when perturbations are sufficiently small. How can this assumption be systematically evaluated across different scenarios? In tasks with larger domain gaps, the distinction between learngenes and tailors becomes more pronounced. How can this difference be quantified to ensure that the assumption remains valid? Providing empirical or theoretical justification for this assumption across varying domain shifts would strengthen the paper’s argument.", "experimental_designs_or_analyses": "•  What happens when training with a significantly larger number of classes? Since the parameter size is directly linked to the size of tailors, as shown in Table 1, will the model expand and scale linearly? If so, does this suggest that the proposed method’s efficiency gains diminish as the number of classes increases? Clarifying the scalability of the approach would strengthen the discussion.\n\n•  The details for Table 5 are missing. Additional explanations regarding the setup, evaluation metrics, and key findings should be provided to ensure clarity and completeness.", "relation_to_broader_scientific_literature": "The key contributions are related to knowledge decomposition, parameter decomposition, and recombination. They might also relate to research areas such as domain shift and model personalization.", "essential_references_not_discussed": "The authors are encouraged to discuss some related papers in the area of NAS, which train one model but can be regrouped at no cost at inference [A]\n\n[A] Once-for-All: Train One Network and Specialize it for Efficient Deployment. ICLR'20.", "other_strengths_and_weaknesses": "see previous sessions.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "•  For tasks involving novel classes, how is the tailor initialized? Is the entire tailor randomly initialized, or only a portion of it? It is assumed that each task begins with an initialized tailor, but since different tasks involve varying numbers of classes, does this imply that the model size is variable? The parameter values reported in Table 2 appear to be fixed, clarification on this aspect is needed.\n\n•  Is there a direct relationship between the number of training parameters and FLOPS? If so, providing explicit details or empirical validation would strengthen the discussion.\n\n•  What is the exact improvement of KIND over FT in Table 3? Reporting the numerical difference would make the performance comparison more precise and informative.\n\n•  Can this method generate combinations of multiple classes? If so, how is the combination controlled or influenced by the tailors? If not, what are the key limitations preventing such functionality?\n\n•  How are closely related tailors selected for fine-tuning, as described in Line 217? What is the selection criterion, and how many tailors are chosen for fine-tuning? A more detailed explanation of this process would improve clarity and reproducibility.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cfZnnEvMvx", "reviewer": "Reviewer_QbMr", "summary": "## Summary\nThis work applies SVD on the weight matrices $W_q, W_k, W_v, W_o, W_{in}, W_{out}$ of pretrained diffusion transformers (DITs). Then finetunes the SVD decomposed matrices U, $\\Sigma, $V$ with target label information. The SVD decomposed matrices are futher splited into two parts to store 1) general information and 2) target label specified information. \n\n\n## strongthness \n\n- The idea of pretraining DiTs with SVD decomposition of weights is interesting. \n- The reconstruction quality of DiTs trained by the proposed method is clearly better than that of other methods. (e.g. Figure 4, 5). \n\n\n## weakness\n\n- Although the proposed method appears simple (as shown in Algorithm 1), the writing is somewhat hard to follow. I am confused about this paper. For example, line 123  \"KIND decomposes pre-trained models\" implies applying KIND after pretraining. However, the learning of learngenes and tailors (line 217) requires applying KIND during pretraining. Again, Table 1, 2, 3 compares KIND with other \"post-training\" methods, while line 251, section 5.1 says \"the model pretrained by KIND\". \n\nWhere does the KIND apply, pretraining or post-training? and What do you want to compare with, pretraining approaches or post-training approaches?\n\n- Assuming KIND is applied on model pretraining (line 251, section 5.1), did other post-training methods in table 1,2,3 use the KIND pretrained DiTs? or normally pretrained DiTs?  Keep in mind that, KIND pretraining benefits from the target label information. Did normally pretrained DiTs, if there is any, use the target label information. \n\n- Without a clarification about the concerns above, the fairness of experimental results is questionable. \n\n\n\n\n\n\n## suggestions\n- I suggest author to put Algorithm 1 in the main text to make the paper easy to understand. As a reader of an experimental work, I want to know 1) How does the algorithm work (algorithm 1) and 2) What are the results. The Algorithm 1 is so clear and so simple! It should be in the main text.\n- Line 150~155 shows the SVD reparameterization is applized on $W_q, W_k, W_v, W_o, W_{in}, W_{out}$. However, Figure 2 implies that the SVD is only applied on $W_{in}, W_{out}$. I suggest to update figure 2 accordingly. \n\n- I am positive on this paper because of the interesting reconstruction quality. However, it is important to have a fair comparison and a clear expression. \n\n### questions \n\n- Algorithm 1 line 1 sayes the initialization of both W and SVD matrices U, $\\Sigma$, V. Did you use both matrices, or one? I am confused. It seems that Algorithm 1 line 8 uses SVD matrices U, $\\Sigma$, V, rather than W. If this is the case, how do you constrain $U^{\\top}U=I$?", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QbMr, \n\nWe sincerely appreciate your valuable comments and recognition of our work’s innovation and performance. Below is our detailed response.\n\n>**Q1: Where does the KIND apply, pretraining or post-training? What do you want to compare with, pretraining approaches or post-training approaches?**\n\nKIND is a novel pre-training method for constructing decomposable models. Unlike traditional pre-training approaches, KIND explicitly partitions knowledge into class-agnostic and class-specific components, encapsulating them in learngenes and tailors, respectively. \nThis decomposition transforms fixed-size models into modular structures, where learngenes enhance transferability, and tailors adapt to specific tasks.\n\nKIND operates during pre-training, producing learngenes and tailors for flexible downstream deployment. \nAccordingly, we first compare it with pre-training methods such as traditional PT and Laptop-Diff, demonstrating that training a decomposable model incurs no extra computational cost while improving structural flexibility. \nTo highlight the strong transferability of class-agnostic knowledge, we also compare it with parameter-efficient fine-tuning methods (i.e., post-training approaches) such as LoRA and PiSSA, particularly in tasks with large domain shifts.\n\nIn summary, KIND is a pre-training framework that enhances post-training adaptability by enabling adaptive model scaling based on task demands and computational resources.\n\n>**Q2: Did other post-training methods in Table 1,2,3 use the KIND pretrained DiTs or normally pretrained DiTs? Did normally pretrained DiTs use the target label information?**\n\nAs noted in Q1, KIND is a pre-training framework for decomposable models that enhances post-training flexibility.\nTable 1 compares different pre-training strategies, including traditional PT and knowledge distillation (e.g., Laptop-Diff), in constructing models of various sizes. \nIn contrast, Tables 2 and 3 evaluate downstream performance, comparing KIND with parameter-efficient fine-tuning methods applied to **normally pretrained DiTs**.\n\nThis setup ensures a fair comparison, as normally pretrained DiTs also incorporate target label information.\nNormally pretrained DiTs generate images based on class-conditioned information and inherently depend on class labels during pre-training.\nTo further validate fairness, we compare the training performance of KIND-pretrained models with normally pretrained models (see Table below). \nThe results confirm that normally pretrained models perform comparably to KIND, demonstrating that improvements in Tables 2 and 3 solely stem from the class-agnostic knowledge encapsulated in learngenes.\n\n||Model|Steps|FID↓|sFID↓|IS↑|Prec.↑|Rec.↑  \n|-|-|-|-|-|-|-|-\n|Traditional PT|DiT-L|300K|9.68|**6.15**|72.22|0.69|**0.47**\n|KIND|DiT-L|300K|**9.33**|6.80|**79.39**|0.69|0.46\n\n>**Q3: Fairness of experimental results.**\n\nWe have addressed your concerns in detail in Q1 and Q2, which we believe sufficiently clarify this issue.\n\n>**Q4: Algorithm 1 line 1 says the initialization of both $W$ and SVD matrices $U$, $\\Sigma$, $V$. Did you use both matrices, or one? How do you constrain $U^\\top U=I$?**\n\nWe apologize for any confusion caused by the imprecise wording in Algorithm 1, line 1.\nAs you noted, during knowledge diversion, gradient updates are applied only to $U$, $\\Sigma$ and $V$, while $W$ is indirectly updated via Eq. (5) as $W=U\\Sigma V^\\top$.\n\nRegarding the constraint $U^\\top U=I$, we initially explored enforcing orthogonality using **Cayley parameterization** (details can be found in official PyTorch documentation), a transformation that maps a skew-symmetric matrix to an orthogonal matrix.\nSpecifically, we can construct $U$ as $U=(I+Q)(I-Q)^{-1}$, where $Q$ is a skew-symmetric matrix satisfying $Q=-Q^\\top$.\nWhile this guarantees orthogonality, it incurs substantial computational overhead ($\\sim7\\times$, see table below) due to the matrix inversion, without notable empirical benefits.\n\n||Model|Steps|GPU Time|FID↓|sFID↓|IS↑|Prec.↑|Rec.↑\n|-|-|-|-|-|-|-|-|-\n|w/ Cayley|DiT-B/4|200K|104.8 hour|56.99|**47.04**|24.7|0.38|**0.46**\n|w/o Cayley|DiT-B/4|200K|14.7 hour|**52.78**|49.66|**25.7**|**0.40**|0.45\n\nGiven these trade-offs, we do not enforce explicit orthogonality constraints. Instead, we employ class gates to associate distinct feature representations with corresponding singular vectors, thereby naturally mitigating correlations among the singular vectors. This enables $U$ and $V$ to approximate orthogonality through the learning process.\n\nAs shown in [Figure](https://anonymous.4open.science/api/repo/a-8112/file/12.pdf), the process of knowledge diversion through class gate allows the learned matrices to preserve orthogonality without requiring explicit constraints.\n\n>**Q5: Other suggestions.**\n\nThank you for your valuable suggestion. We will move Algorithm 1 to the main text and complete the missing details in Figure 2 in revision.", "claims_and_evidence": "check summary", "methods_and_evaluation_criteria": "check summary", "theoretical_claims": "no theory.", "experimental_designs_or_analyses": "yes.", "relation_to_broader_scientific_literature": "check summary", "essential_references_not_discussed": "no.", "other_strengths_and_weaknesses": "check summary", "other_comments_or_suggestions": "check summary", "questions_for_authors": "check summary", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Li7piteQEd", "reviewer": "Reviewer_Y7yt", "summary": "This manuscript proposes a novel pre-training method named KIND, aiming to address the adaptability issues of traditional pre-trained models in different tasks and deployment scenarios. KIND integrates and distributes knowledge by using SVD during the pre-training process, creating learngenes and tailors respectively, and achieves effective knowledge transfer through a class gate mechanism. Experiments verify that KIND can be flexibly deployed in various scenarios and significantly improve the transfer efficiency in tasks with large domain shifts. The contribution of KIND lies in providing a decomposable structure for pre-trained models, enabling the models to be dynamically adjusted according to task requirements.\n\n## update after rebuttal\nThe author's response has addressed my concerns, and I am inclined to accept this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Y7yt, \n\nWe sincerely appreciate your insightful feedback and your recognition of the innovation and practicality of our work. Below, we provide our detailed response.\n\n>**Q1: Lack of comparison of similar methods (e.g., FacT and BOFT).**\n\nFacT and BOFT leverage matrix decomposition techniques relevant to this work but are fundamentally Parameter-Efficient Fine-Tuning (PEFT) methods. \nThey focus on decomposing pre-trained weight matrices to identify compact parameter subspaces, that can be efficiently fine-tuned for adapting pre-trained models to novel tasks. However, these methods heavily rely on traditional pre-trained models, which are typically fixed in size, structurally inflexible, and risk negative transfer.\n\nIn contrast, KIND introduces **a novel pre-training paradigm** that explicitly decomposes knowledge into class-agnostic and class-specific components, encapsulated in learngenes and tailors, respectively. This approach results in decomposable pre-trained models, where the modular design enhances transferability while enabling task-specific adaptation, effectively addressing the limitations of traditional pre-training.\n\nTable 2 in the manuscript highlights the superior transferability of learngenes by comparing KIND with PEFT-based methods, including SVD-based approaches (e.g., SVDiff, PiSSA) related to FacT, and OFT-based methods related to BOFT.\nPer your suggestion, additional comparisons with FacT and BOFT (see Table below) further demonstrate that KIND consistently outperforms these PEFT methods, underscoring its ability to transfer only class-agnostic knowledge while avoiding the deployment challenges and the redundant, biased, or harmful transfer often associated with traditional pre-trained models on which PEFT approaches typically rely.\n\n|DiT-B|CelebA|Hubble|MRI|Pokemon\n|-|-|-|-|-|\n|FacT-TT|0.307|0.242|0.067|0.425     \n|BOFT|0.318|0.247|0.058|0.433      \n|KIND|**0.201**|**0.124**|**0.042**|**0.343** \n\n|DiT-L|CelebA|Hubble|MRI|Pokemon\n|-|-|-|-|-|\n|FacT-TT|0.240|0.168|0.081|0.299     \n|BOFT|0.213|0.155|0.051|0.296      \n|KIND|**0.152**|**0.109**|**0.040**|**0.262**\n\n>**Q2: Application in the limited resources scenario is not highlighted in experiments.**\n\nThe decomposable model pre-trained by KIND can be flexibly restructured to accommodate computational constraints, enabling efficient deployment on **mobile and edge devices**.\nAlthough direct evaluation on mobile hardware is beyond our current resources, we approximate its feasibility by analyzing FLOPs, memory footprint, and extrapolating inference latency on modern mobile chip.\n\nWe construct a mobile-compatible DiT using KIND and evaluate its efficiency across three state-of-the-art mobile chip: Apple A18, Kirin 9020, and Snapdragon 8 Gen3, which sustain 1907, 1720 and 2774 GFLOPS, respectively.\n\nAs shown in Table below, KIND-Mobile achieves a 3.3$\\times$ reduction in FLOPs and a 1.8$\\times$ reduction in memory usage compared to traditional pre-training, while maintaining strong generative performance (FID=21.14). \nNotably, inference latency remains under 4 seconds across all tested mobile chip, demonstrating KIND’s adaptability in resource-constrained environments.\n\n||Param.|FLOPs (G)|Memory (MB)|Apple A18 (s)|Kirin 9020 (s)|Snapdragon 8 Gen3 (s)|FID↓|IS↑\n|-|-|-|-|-|-|-|-|-\n|Traditional PT|129.7|43.62|518.8|11.43|12.68|7.86|25.14|47.15\n|KIND-Mobile|**70.2**|**13.22**|**280.8**|**3.47**|**3.84**|**2.38**|**21.14**|**58.18**\n\n>**Q3: Supplement the comparison of speed in Table 2.**\n\nTo further emphasize the computational efficiency of KIND on novel tasks, we report the GPU time of different methods in the Table below, following your suggestion.\n\nKIND achieves the best performance with the most efficient training compared with state-of-the-art PEFT methods. This is attributed to its encapsulation of class-agnostic knowledge into learngenes through knowledge diversion, thus enhancing structural flexibility.Notably, under large domain shifts, transferring only learngenes improves the adaptability of the pre-trained model while significantly enhancing transfer efficiency by reducing model parameters through the elimination of redundant class-specific knowledge encapsulated in tailors.\n\n|DiT-B|Para.|FLOPs|GPU Time\n|-|-|-|-\n|SVDiff|**0.1**|43.6|2.5\n|OFT|14.2|119.7|6.6\n|LoRA|12.8|50.1|2.9\n|PiSSA|12.8|50.1|2.9\n|LoHa|12.7|87.1|3.9\n|DoRA|12.8|129.5|6.2\n|Heur-LG|129.6|43.6|4.89\n|Auto-LG|129.6|43.6|4.89\n|Full FT|129.6|43.6|4.89\n|KIND|12.8|**33.7**|**1.6**\n\n|DiT-L|Para.|FLOPs|GPU Time|\n|-|-|-|-\n|SVDiff|**0.2**|155.0|6.6|\n|OFT|50.5|425.6|14.2|\n|LoRA|45.3|178.2|7.1|\n|PiSSA|45.3|178.2|7.1|\n|LoHa|45.3|309.6|12.8|\n|DoRA|45.6|503.0|22.2|\n|Heur-LG|456.8|155.0|10.0|\n|Auto-LG|456.8|155.0|10.0|\n|Full FT|456.8|155.0|10.0|\n|KIND|45.4|**119.6**|**6.3**|\n\n>**Q4: Introduce a README part in the code.**\n\nThank you for your suggestion. We will include a comprehensive README file in the future open-source release to facilitate the reproduction of KIND.", "claims_and_evidence": "The authors analyzed the existing problems and put forward the claim of \"rethinking the pre-training process to develop decomposable pre-trained models\".", "methods_and_evaluation_criteria": "The method is applicable to solving the problem of model decomposition, and the experimental verification indicators are reasonable.", "theoretical_claims": "The core theory is based on SVD, which is relatively easy to understand.", "experimental_designs_or_analyses": "The experimental design is reliable, and the datasets used in the experiments include ImageNet and several downstream datasets.", "relation_to_broader_scientific_literature": "The core decomposition method of this paper is quite similar to that of the paper \"FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer\" published in AAAI 2023, but no discussion is carried out.", "essential_references_not_discussed": "The core decomposition method of this paper is quite similar to that of the paper \"FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer\" published in AAAI 2023, but the authors did not conduct a comparison and discussion. In addition, the paper \"PARAMETER-EFFICIENT ORTHOGONAL FINETUNING VIA BUTTERFLY FACTORIZATION\" published in ICLR 2024 should also be compared and discussed.", "other_strengths_and_weaknesses": "Strengths\n\n1 The overall writing logic is relatively clear.\n\n2 The feature of \"no further time-consuming steps required\" is very friendly to the environment with limited computing resources.\n\nWeaknesses\n\n1 There is a lack of comparison and discussion of the decomposition methods in FacT of AAAI 2023 and BOFT of ICLR 2024.\n\n2 One of the authors' core arguments is the application in the Limited Resources scenario, but this point is not highlighted in the experimental part. For example, it is recommended to add experiments on mobile devices.", "other_comments_or_suggestions": "It is recommended to introduce a README part in the code of the supplementary material, otherwise it will cause inconvenience for reviewers to review.", "questions_for_authors": "The authors compared the number of parameters and the amount of computation in Table 2. It is recommended to supplement the comparison of speed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rtcWw1vwi4", "reviewer": "Reviewer_Dcjo", "summary": "This paper tårgets on training a better pre-trained model for downstream tasks. Concretely, they propose KIND (Knowledge Integration and Diversion). It utilizes SVD to yield basic components, and then classify them into two categories, learngenes and tailors. The former captures class-agnostic features, while the latter captures class-specific ones. With SVD, it trains basic components instead of the full-weight matrices. This method is reported to be the first to apply learngenes to image generation tasks. They establish a benchmark for evaluating the transferability of diffusion models. Extensive reported results prove the effectiveness of the proposed method.\n\n## After Rebuttal\nAfter reading the rebuttal and other reviews, I still tend to accept this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Dcjo, \n\nWe sincerely appreciate your insightful comments and your recognition of both the novelty and soundness of our methods, as well as the contribution of our benchmark to the community. Below, we provide a detailed response.\n\n>**Q1: Discussions on the limitations of the proposed method.**\n\nWe have briefly discussed the limitations in Section 7 of the original manuscript. Here, we further elaborate on these limitations and outline potential directions for future improvements:\n\n\n- **Limitations in Class-Conditional Generation.** \nTo illustrate the transferability of class-agnostic knowledge obtained via knowledge diversion, we focus on class-conditional generation tasks, where variations induced by different class labels naturally introduce downstream tasks with substantial domain shifts.\n\n    In contrast, text-conditional generation, which controls image content via prompts, is also widely adopted. \n    However, large-scale text-to-image diffusion models (e.g., Stable Diffusion 3) are pre-trained on a broad spectrum of internet images, making it challenging to define tasks with significant domain shifts. \n    Moreover, extending KIND to text-conditional generation requires transitioning class gate from discrete, countable class labels to an open-ended, uncountable prompt space, where **a binary class gate is insufficient**.\n\n    Future work will explore Mixture-of-Experts (MoE) techniques for knowledge diversion, enabling dynamic allocation of a limited set of tailors based on prompts.\n    We have conduct preliminary experiments using a PixArt-based model trained for 50,000 steps, see Tabel below. KIND may underperform compared to parameter-efficient fine-tuning (PEFT) methods such as LoRA, as pre-trained text-to-image diffusion models inherently bridge moderate domain differences.\n\n    |Dataset: MRI| CLIP Score↑|LPIPS↓\n    |-|-|-\n    |PixArt-Lora|**33.88**| 0.4252\n    |PixArt-KIND|33.20|**0.4213**\n\n\n    |Dataset: Pokemon|CLIP Score↑|LPIPS↓\n    |-|-|-\n    |PixArt-Lora|32.48|**0.4279**\n    |PixArt-KIND|**32.55**|0.4288\n\n- **Limitations in Structural Expansion.** \n    KIND has been applied exclusively to Transformer-based architectures, focusing on knowledge diversion in Multi-Head Self-Attention ($W_q$, $W_k$, $W_v$, $W_o$) and Pointwise Feedforward layers ($W_{in}$, $W_{out}$). \n    While DiT is becoming the dominant architecture for diffusion models, many classic diffusion models still rely on convolution-based UNets.\n\n    Extending KIND to architectures dominated by convolutional layers presents a key challenge.\n    Although convolutional weights can be represented as three-dimensional tensors and prior work has explored SVD-based decomposition for convolutional layers, the strong inductive biases of convolutional kernels pose unique difficulties. \n    Developing effective knowledge diversion strategies for convolutional networks remains an important direction for future research.\n    \n- **Limitations in Other Tasks (e.g., Image Classification).**\n    KIND has been primarily evaluated on image generation tasks, yet its knowledge diversion mechanism—encapsulating class-agnostic and class-specific knowledge into distinct network components—suggests broader applicability. \n   \n    A particularly promising direction is cross-domain few-shot learning, where models must generalize across domains with limited data. Traditional methods often struggle under large distribution shifts due to their reliance on prior knowledge from the source domain.\n    KIND offers a key advantage: learngenes serve as a transferable backbone for stable adaptation, while tailors enable task-specific fine-tuning with minimal data, improving generalization. \n    \n    However, unlike image generation, image classification lacks access to class labels during inference, requiring each image to traverse all tailors to extract features, leading to increased computational overhead. As the number of classes grows, classification complexity further escalates.\n    Preliminary experiments applying KIND to ViT for cross-domain few-shot learning (see Tabel below) demonstrate significant improvements over baselines, though a performance gap remains compared to state-of-the-art methods. Thus, developing an efficient learngene-tailer framework for classification remains an open research direction.\n\n    ||ChestX|ISIC|EuroSAT|CropDisease|Average\n    |-|-|-|-|-|-\n    |Vanilla ViT (Baseline)|26.3|46.1|88.6|94.6|63.9\n    |P>M>F|**27.3**|50.1|86.0|93.0|64.1\n    |StyleAdv|27.0|**51.2**|**90.1**|**96.0**|**66.1**\n    |KIND|26.5|47.9|88.8|95.0|64.6\n\nWe will provide a detailed discussion of these limitations and future research directions in the revised Appendix to inspire further advancements in KIND and broaden its application scope.", "claims_and_evidence": "One key idea in this method is to adopt SVD to categorize basic components into learngenes and tailors. This claim can be supported by previous works that SVD has been applied to disentangle class-agnostic and class-specific components.", "methods_and_evaluation_criteria": "I think the method is reasonable and new to me. Previous methods have applied SVD, but this method also shows enough novelty to me. \nI think the benchmark datasets cover some basic datasets in image generation, and plenty of criteria and scenarios have been incorporated in the experiment section.", "theoretical_claims": "I have checked A3. I have read A2 but I am not familiar with DK Theorem.", "experimental_designs_or_analyses": "I think the experiment section in this paper is good. It covers multiple datasets and settings, and reports enough criteria to validate the performance of methods. I think constructing such benchmark is beneficial to the community.", "relation_to_broader_scientific_literature": "I think this is an interesting paper. Besides the interesting method proposed in this paper, the benchmark they construct might be meaningful.", "essential_references_not_discussed": "N/A from my perspective.", "other_strengths_and_weaknesses": "The figures in this paper are good, illustrating the idea in a clear manner.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I want to see more discussions on the limitations of the proposed method. Some failure cases are also welcome.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces KIND, a pretraining method that decomposes model weights via SVD into class agnostic learngenes and class specific tailors, which enables modular adaptation and efficient transfer. KIND is particularly effective under domain shifts, where transferring only the learngenes yields strong performance with minimal overhead. Reviewers found the method technically sound and empirically strong, and the rebuttal addressed remaining key concerns around method's clarity, fairness of the comparison, and related work. Yet, there remains some limitations, including its narrow focus on transformer-based generative models, unknown scalability with many classes, and limited evaluation beyond vision tasks. Still, I found the the overall contribution as solid and impactful and recommend acceptance.
Decision: Accept (poster)