ECO: Evolving Core Knowledge for Efficient Transfer
Authors: Fu Feng
Yucheng Xie
Ruixiao Shi
Jianlu Shen
Jing Wang
Xin Geng
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge Transfer
Population-Based Adaptation
Biologically Inspired Computation
Primary area: deep_learning

Abstract: Knowledge in modern neural networks is often entangled and structurally opaque, making current transfer methods—typically based on reusing entire parameter sets—inefficient and inflexible. Efforts to improve flexibility by reusing partial parameters frequently depend on handcrafted heuristics or rigid structural assumptions, which constrain generalization. In contrast, biological evolution enables efficient knowledge transfer by encoding only essential information into genes through iterative refinement under environmental pressure. Inspired by this principle, we propose **ECO**, a framework that **E**volves **CO**re knowledge into modular, reusable neural components—termed *learngenes*—through similar evolutionary dynamics. To this end, we redefine learngenes as neural circuits and introduce Genetic Transfer Learning (GTL), a biologically inspired paradigm that establishes a genetic mechanism within neural networks in the context of supervised learning. GTL simulates evolutionary processes by generating diverse network populations, selecting high-performing individuals, and transferring their learngenes to subsequent generations. Through iterative refinement, GTL enables learngenes to accumulate transferable common knowledge. Extensive experiments show that ECO achieves efficient initialization and strong generalization across diverse models and tasks, while significantly reducing computational and memory costs compared to conventional methods.

ECO: Evolving Core Knowledge for Efficient Transfer
Fu Feng!a
Yucheng Xiel Ruixiao Shil.2 Jianlu Shen 1.2 Jing Wang! Xin Geng 1.2
"School of Computer Science and Engineering, Southeast University, Nanjing, China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University). Ministry of Education, China [ffenng xieyc, erie_rriiie eric_xiao, jlshen, wangjing91, xgeng ] seu.edu..c
Abstract
Knowledge in modern neural networks is often entangled and structurally opaque, making current transfer methods satttodds-yiicallly based reusing entire parameter sets iets-inefficient and inflexible. Efforts to improve flexibility by reusing partial parameters frequently depend handcrafted heuristics rigid structural assump- tions, which constrain generalization. In contrast, biological evolution enables efficient knowledge transfer by encoding only essential information into genes through iterative refinement under environmental pressure. Inspired by this princi- ple, we propose ECO, framework that Evolves COre knowledge into modular, reusable neural components-tenne termed learngenes-troough similar evolutionary dynamics. To this end, redefine learngenes neural circuits and introduce Genetic Transfer Learning (GTL). biologically inspired paradigm that estab- lishes a genetic mechanism within neural networks the context of supervised learning. GTL simulates evolutionary processes by generating diverse network pop- ulations, selecting high-performing individuals, and transferring their learngenes to subsequent generations. Through iterative refinement, GTL enables learngenes accumulate transferable common knowledge. Extensive experiments show that ECO achieves efficient initialization and strong generalization across diverse models and tasks, while significantly reducing computational and memory costs compared to conventional methods.
Introduction
Despite the remarkable progress of modern neural networks across a wide range of tasks 1 the internal organization of knowledge within these models remains largely opaque and poorly structured. As a result, conventional transfer learning approaches- -uch as full fine-tuning or paraaeter-efffcieet techniques like LoRA 17] rely on reusing entire parameter sets from large-scale pre-trained models 35). Although effective in reducing task-specific training overhead [7]. these methods implicitly assume that knowledge is globally entangled and uniformly distributed, with little consideration for its internal structure or modular organization. Consequently, they often suffer from limited adaptability across models of varying sizes and architectures [12 77,, and are highly susceptible to domain shifts, frequently resulting in biased [49 31] or even
61 13
detrimental transfer 55 73 32,,
as shown in Figure TT.
Biological systems, in contrast, offer a compelling model for efficient and generalizable knowledge transfer. Rather than replicating entire structural configurations, they encode essential knowledge into compact and inheritable units known as genes, which capture core functionality and support adaptation across generations and environments. These genes act as reusable blueprints [82 3 65] guiding the development of neural circuits that are both robust and efficient, as illustrated in Figure
"Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Malamute
Lion
(b)
Brain
Network
Transfer
Transfer
Extract
Initializs
Ertract
Initalize
I see dog in the snow, so I'm sure is malamute,
notice something in the may
CUOLO
Pretrained Model
also be malamute
Gene
Learngene
Figure 1: (a) Transferring excessive knowledge may introduce bias and lead to negative transfer. (b) In contrast, biological genes transfer only survival-essential knowledge to initialize certain innate neural circuits. Inspired by this, we propose encoding core transferable knowledge into modular neural circuits (termed learngenes, see Figure 2 to enable efficient knowledge transfer.
Motivated by this principle, recent work has explored biologically inspired approaches knowledge transfer, most notably through the Learngene framework [11 70) Unlike conventional parameter reuse, learngenes aim to support structured and composable transfer by encapsulating common knowledge into modular neural fragments, enabling flexible initialization across tasks and models of diverse scales. Early methods such as Heur-LG (68) and Auto-LG (70) identify transferable components using gradient-based heuristics or meta-learning strategies. More recent approaches, including TLEG [75] and WAVE [12), incorporate structural rriors-such as linear constraints or Kronecker decompositions- to enhance the modular organization of learned representations. Despite their improved flexibility and efficiency, they often depend on handcrafted heuristics or rigid structural assumptions, limiting their adaptability and generalization across diverse tasks and architectures.
In contrast, essential knowledge encoded in biological genes is not organized by predefined rules, but emerges through iterative refinement under environmental pressures such as mutation, selection, and inheritance. This perspective motivates shift toward understanding how transferable knowledge can be modularly organized within neural networks through similar evolutionary dynamics, rather than imposed through handcrafted heuristics or rigid structural assumptions (e.g., linear constraints). To this end, we propose ECO, biologically inspired framework that explores whether neural networks can autonomously condense transferable core knowledge and encapsulate into compact, reusable modules through population baaed adaptation and feedback-driven refinement.
At the core of ECO is a redefinition of the learngene structure as continuous neural circuits, such as interconnected kernel sets in CNNs, which mirror the structure of innate pathways shaped by genetic encoding in biological systems (see Figure [p). To automatically identify and encapsulate highly transferable core knowledge within neural circuits, we extend the evolutionary paradigm introduced in GRL [II] and propose Genetic Transfer Learning (GTL), an evolutionary framework that simulates genetic inheritance in neural networks in the context of supervised learning. GTL maintains a population of models, each trained on randomly sampled tasks. high-performing individuals are selected via tournament evaluation, and their learngenes are inherited and refined across generations, with mutation introduced to maintain diversity and promote exploration. This Lamarckian process progressively condenses core knowledge into learngenes, enabling efficient, scalable, and generalizable transfer across models and tasks.
We evolve learngenes for 250 generations in VGGI and ResNet12 on CIFAR-FS and minilmageNet, and for 100 generations in ResNet50 and MobileNetV3--arge on ImageNet. Extensive experiments demonstrate that ECO consistently outperforms existing methods across wide range of downstream architectures and tasks. Compared to full fine-tuning, ECO achieves 14.5x reduction storage, highlighting its efficiency in knowledge transfer. Additionally, ECO supports O(1) initialization across models of varying scales, enabling fast deployment without additional costs. Notably, ECO exhibits strong data efficiency, maintaining robust performance even in low-resource regimes.
Our main contributions are follows: We propose ECO, novel knowledge transfer method that adaptively identifies core transferable knowledge across architectures and tasks, without relying on handerafted heuristics or structural constraints. 2) We introduce Genetic Transfer Learning (GTL), a biologically inspired evolutionary paradigm for supervised learning that encapsulates core transferable knowledge into modular neural circuits (i.e., learngenes) through large-scale population- based training and inheritance. 3) We validate the scalability and generality of ECO across diverse tasks, model sizes, and architectures, achieving state-of-the-ar performance both accuracy and resource efficiency.

---

2 Related Work
Efficient Knowledge Transfer Transfer learning enhances performance on target domains by lever-
aging knowledge from source domains [6 23 Traditional methods, such as fine-tuning [18 [7].
are constrained by the fixed architecture and size of pre-trained models. Knowledge distilla- tion [57 67 [6 offers structural flexibility but remains computationally intensive. Recent methods aim to enhance efficiency by transferring compact and reusable knowledge representations (see Appendix A.Tp. Learngene-based approaches, including Heur-LG (68] and Auto-LG (70), identify
transferable components via heuristic strategies. Others, such as TLEG [75], WAVE [2],, and related studies (37 66 [8), integrate pre-trained knowledge via structural priors like low-rank decomposi-
tions. Alternative paradigms explore hypernetwork- based parameter generation [29] 30 or rule-based
[29]30
parameter reuse 80
33 3
In contrast, ECO extends the population-base based learning paradigm
of GRL 11 to CNNs in the context of supervised learning, adaptively evolving core transferable
knowledge through mutation-driven and survival-based selection. This process mitigates manual bias
and enhances generalization across architectures and domains.
Evolutionary Learning Evolutionary Learning (EL) solves optimization problems through stochas-
tic search inspired by biological evolution [60]. Related algorithms [56 [5|] 43 [53 [4] typically
encode candidate solutions such network parameters or architectures-into gene-like representa-
evolving them to maximize task-specific performance. While inspired by evolutionary principles, ECO diverges from traditional evolutionary learning by prioritizing knowledge inheritance over solution optimization. Notably, instead of searching for task-specific solutions, ECO establishes a genetic transfer mechanism that accumulates core knowledge transferable across diverse models and tasks. In this paradigm, learngenes function not as solution encodings, but as compact carriers of core knowledge condensed from diverse tasks. This shift enables scalable generalization without reliance on problem-specific heuristics.
tions (e.g., binary strings) 40
63
45
3 Methods
We reformulate CNNs in terms of kernel units and define the core operations of learngenes, in- cluding their representation, mutation, and inheritance. We then introduce the Genetic Transfer Learning (GTL) for extracting learngenes through evolutionary processes.
3.1 Preliminary
Consider a CNN W with Nz convolutional layers. The 1-th layer L1 E R XN xxxx comprises N filters Fi.y. where each filter contains N kernels Ki.f. E RE*X, These kernels capture spatial
features various levels of abstraction, enabling hierarchical representation learning. Accordingly, the trainable parameters of the entire network can be represented as a unified set of kernels:
N [KI, [1, [1,N[], E [1,
(1)
Given an input feature map T E DxHx to layer L,, the convolution produces an output I441 R x which serves as input to layer T4+1- To ensure valid feature propagation, CNNs enforce channel-wise consistency by matching the number of kernels in L4+1 with the number of filters in Li:
((+1) = (
(2)
3.2 Basic Operations for Learngenes
3.2.1 Form of Learngenes in CNNs
In biological neural systems, innate neural circuits are established birth under the guidance of genes, providing newborns with strong inherent learning abilities [74] 42 82). Motivated by this biological foundation, we interpret neural circuits convolutional neural networks (CNNs) as structured subnetworks [59, each composed of set of interconnected kernels that collectively implement continuous input-output transformation. These subnetworks serve as functional units responsible for localized computation and information flow within the network.
3

---

tnpur
Red Green Blue
D
Layer
Layer 6
Input
Layer 2
Layer
Output
Layer 3
Layer 4
Figure 2: Learngenes in ECO abstracted as complete neural circuits composed of selected kernels within filters. G represents the learngene kernel, while is normal kernel that is random initialized. denotes the continuous feature mapping path extracted from learngenes.
Building upon this abstraction, we define learngenes modular neural circuits within CNNs that encapsulate transferable and reusable knowledge. Formally, a learngene is defined as:
M N f E Fi, k K.}
where Fi and Kr denote the selected indices filters and kernels the 1-th layer, respectively.
To preserve the intervonnection among kernels and ensure uninterrupted continuity in feature transfor- mation, learngenes follow the structural alignment principle inherent CNNs (Eq. (2)) Specifically, we enforce the constraint:
K1+1 Fi
This condition preserves channel-wise consistency, enabling layer-wise modification through Fi while KG is automatically inferred via Eq. (4), ensuring coherent inter-layer connectivity of learngenes.
3.2.2 Mutation of Learngenes
In biological systems, evolution is driven by structural mutations that progressively refine genetic traits (27 24]. Analogously, learngene mutation also operates the structural level by modifying the arrangement of filters and kernels, enhance adaptability for encoding core knowledge.
Given the structural alignment in Eq. 4. mutations are applied primarily to the filter sets Fi each layer, with corresponding kernel indices K1+1 updated automatically to maintain connectivity, For each learngene, structural mutation performed independently at each layer with probability Pm- The probabilities of adding (p') )) removing (p filters in the 1-th layer of learngenes are given by:
FFl
Pi a
Pi
1-p
(5)
N
Fi
where a is balancing coefficient and may involve multiple filters per layer. The complete mutation procedure is detailed in Algorithm I
denotes cardinality. Mutations proceed layer-wise and
3.2.3 Inheritance of Learngenes
Learngene inheritance facilitates the transfer of core knowledge from source model target models with varying depths, widths, or architectures. To accommodate architectural differences during transfer, ECO employs the following strategies:
Zero-padding Incomplete Filters. Since learngenes may include only subset of kernels within each filter, direct insertion into wider filters introduce noisy, randomly initialized parameters.
To avoid disrupting the encoded knowledge, missing kernels are initialized as zeros, preserving functionality while enabling later adaptation (see Appendix Figure
Index Reordering for Narrower Networks. When transferring to narrower networks, mismatches in filter and kernel indices may arise. ECO resolves this by reordering indices according to their relative positions (see Appendix Figure 9. maintaining structural consistency and ensuring compliance with the alignment constraint in Fq.
Additional inheritance strategies, including the proposed partial identity mapping for depth expansion, detailed in Appendix B.4

---

Training Environment
(b)
Training Task
(c) Population
i Descendants
(i) Ancestral Learngenes
Classfication
Select
Train
Initialize
F2
Inherit
Es
Extract
Select
Inaresse
Miutate
Decrsode
Root
91
0-th
Unseen Environment
Initialize
93 X Elminote a Refresk
(d) Learngenes
Ps
1-th
Train
Evaluate
Tournament
Environment
(e) Critic Network
Winners
(g) Gene Tree
(h) Gene Pool
Figure The Genetic Transfer Learning (GTL) framework. GTL iteratively condenses knowledge by training populations (c) randomly sampled tasks (a. b) and selecting transferable core knowledge through mutation (d) and tournaments (f). The Gene Pool (h) and Gene Tree (g) store superior learngenes and track their kinship for inheritance (i. j), respectively.
3.3 Evolution for Learngene Extraction
Genetic Transfer Learning (GTL) is an evolutionary framework derived from GRL 11 that iteratively optimizes leamgenes by transferring them across generations of neural networks. Through this inheritance mechanism, GTL progressively condenses and refines core knowledge into compact, reusable learngenes that generalize across diverse models and tasks, as illustrated Figure and and formalized in Algorithm
Briefly, each generation begins with population of np neural networks, each inheriting learngenes from the previous generation and trained on a randomly assigned task. After training, learngenes extracted and undergo structural mutations to introduce diversity. A tournament-baat based selection mechanism is then employed to identify high-petee segh-performing learngenes, which are retained in the Gene Pool guide inheritance in the generation. This iterative process of inheritance, mutation, and selection continues across generations, progressively refining the quality of learngenes. The core components of GTL detailed below.
(1) Training the Population of Neural Networks. Let = Drain + Dvaal be the dataset with ne training classes and n., validation classes. In each generation, a population P (N, NN,,)) is created. Each network N; randomly samples & classes from Dirain to form k-way classification task Tú encouraging diverse learning environments (Figure Bl-c). As evolution progresses, & increases to simulate growing task complexity, emulating evolutionary pressure 41 (2
(2) Selecting Superior Learngenes. After training, each neural network Nf has updated its in- herited learngene G; by encoding task-specific experience, thereby refining the core knowledge. To promote adaptability and structural diversity, each G, then undergoes structural mutation (Figure 31 and Section 3.2.2). introducing controlled variations that further shape its knowledge representation.
Each mutated learngene G. is evaluated by initializing critic network C, which is trained on the vali- dation set Dval and the resulting accuracy defines the score for G; (Figure Bb). To select high-quality yet diverse learngenes, tournament selection is employed: in each round, learngenes are randomly sampled, and the one with the highest score is added to the winner set G = (9] " Gi...., G (Figure 3D, where = ng denotes the number of tournaments per generation.
(3) Storing Superior Learngenes and Tracking Kinship. Following tournament selection, the winner set G* is incorporated into the Gene Pool (GP), which retains high-qualit, learngenes for inheritance in future generations (Figure 3p). Initialized with top-performing learngenes from the first generation, the GP maintains up to p entries, denoted GP = { G ... .......99 g In each generation, a subset of learngenes from G admitted into the GP, replacing the lowest-performing entries to ensure both knowledge retention and evolutionary adaptability.
5

---

To trace inheritance across generations, GTL constructs a Gene Tree (GT) that records the evolutionary lineage of selected learngenes ( Each node in GT represents a superior learngene, with root nodes originating from the initial generation. New entries in GP are appended as leaf nodes, and the path length between nodes encodes their degree of kinship.
(4) Updating Learngene Scores. preserve ancestral excellence and guide future evolution, the scores of learngenes in the GP are updated after each generation. For every selected learngene g with score sf, its kinship is traced back from its corresponding Ieaf node to the root according to GT. Each ancestral learngene Gane along this path receives a score update:
Sanc
Sanc 17" s;
(6)
where 1) is the decay coefficient and 7 is the path length. See Algorithm 3 for more details.
(5) Generating the Next Generation of Learngenes. After refreshing the GP and updating learngene scores, the next generation of learngenes is sampled from the GP according score- proportional probability:
P:
where is the probability of selecting Gi as parent based on its score of (Figurd3).
The selected learngenes are then inherited by new population of networks P (see Figure 3 and Section ).2.3, initiating the next cycle evolution.
4 Experiments
Datasets. We conduct evolutionary experiments three datasets of increasing scale. CIFAR-FS 2 and minilmageNet [64] each contain 100 classes, split into 64 for training (Duain), 16 for validation (Dvai), and 20 for novel evaluation. ImageNet-1 K (10) contains 1,000 classes, divided into 640, 160, and 200 for the same purposes. We further evaluate the extracted learngenes on four downstream datasets: Oxford Flowers [46). CUB-200-2011
[66], Stanford Cars [15] and Food-101 (4]
Network Architectures. We evaluate four representative convolutional architectures: VGGII (52). ResNet12 20), ResNet50 (20] and MobilcNetV3--arge [21]. To assess the scalability and adaptabil- ity of learngenes in initializing models of varying capacity, we additionally evaluate width-adjusted variants of ResNet50 and MobileNett3-LLrgge with widths scaled from 0.5W to 0.9W, where W denotes the original width.
Evolutionary training is conducted independently across networks to support parallelism. For VGGI1 and ResNet12, learngenes evolve over 250 generations, each comprising 20 networks trained for 15 epochs. For MobileNerV3-Largge and ResNet50, evolution proceeds for 100 generations, with 6 networks per generation trained for 5 epochs. All experiments are executed on NVIDIA GeForce RTX 4090 GPUs, with total computational cost comparable to training a typical medium-scale model. Full hyperparameter configurations detailed in Appendix C.1
Training Details.
V6611, CIFAR-FS
ResNer12, miniImageNet
MobileNetV3-L, ImageNet
ResNe150, ImageNet
30
75
Porometerce (M)
Accuracy
20 160-
5 0KK
100
50-
7,64
10
22
225-
100 120 Generations
200 250
50 100
250
010 30
020
Generations
Generations
Generations
Figure 4: Accuracy (red curves) of networks on validation classes and parameter (red bars) of corresponding learngenes during evolution. Blue stars and bars represent performance and parameter transfer via direct fine-tuning Notably, models inheriting learngenes achieve comparable or superior results, especially in smaller networks, with significant reduction in transferred parameters.

---

5 Results
5.1 Evolutionary Performance of Learngenes Extracted by ECO
In biological evolution, beneficial mutations ac- cumulate through natural selection, leading to the progressive refinement of genes (24] 28).
Table I: Performance of networks inheriting learn- genes on validation and novelty classes across var- ious datasets. (ara.(M) represents the number of transferred parameters. N/A denotes that the crite- rion not applicable the respective method.
Similarly, ECO evolves learngenes by gradually refining core knowledge across generations.
As shown in Figure learngenes accumulate dominant mutations, reflected in the steady growth of their parameter counts. Consequently, networks initialized with evolved leargenes demonstrate consistently improved performance over time. Notably, ECO significantly outper- forms models trained from scratch, achieving performance comparable to pre-trained models while transferring significantly fewer parameters (14.5x reduction in VGGII).
VGG, 28. 18M
ResNet12, 12.44M
Dataset
CIFAR FS
minilmageNer
Methods
Para.
Valid
Novel
Para.
Valid
Novel
He- Inic [9]
0
63.13
66.05
0
67.19
65.55
GradInit 85
N/A
65.19
66.80
N/A
67.75
68.90
ECO- 10th
0.83
65.44
67.75
1.17
73.31
72.80
GHN-2
29
N/A
59.63
62.10
N/A
68.56
66.05
GHN-3 3D
N/A
57.56
61.45
N/A
77.56
77.55
ECO-50th
1.08
69.94
71.70
1.47
77.25
77.45
ECO-100th
1,30
71.56
73.25
1.71
77.88
78.15
Heur-LG 68
7.09
65.25
68.35
9.43
70.75
69.45
Table ! provides quantitative comparison across generations against baseline methods. Taking ResNet12 minilmageNet an il- lustrative example, sarly-generation learngenes already outperform direct initialization ap- proaches (e.g., GradInit (85)) and hypernetwork- based methods (e.g.. GHN-3 (30), benefit- ing from inherited transferable knowledge. At mid-generation, ECO surpasses advanced trans- fer methods such as Heur-LG (68) and Auto- LG 770,, transferring 4.8x fewer parame- ters highlighting its advantage in preserving convolutional feature hierarchies.
Auto-LG
70
9.19
68.00
71.15
9.51
75.25
75.55
ECO-150th
1.51
72.19
74.50
1.96
79.00
78.90
kEy, (39 ECO-200th
N/A
69.81
74.10
N/A
75.94
77.60
1.81
73.69
76.30
2.09
80,94
80.20
Wt Select 80
28.09
63.31
65.75
12.41
76.81
77.75
ECO-250th
1.94
75.00
76.95
2.47
82.00
81.60
Mobile L, 4.37M
ResNet50, 23.66M
Dataset
ImageNet
ImageNet
Methods
Para.
Valid
Novel
Para.
Valid Novel
Ini [19
38.54
38.70
0
$1.85
53.48
GradInit 85
N/A
41.86
41.13
N/A 52.06
51.94
ECO-10th
0.74
48.66
49.21
2.22
59.70
60.05
GHN-2
29
N/A
31.93
30.72
N/A 47.30
46.27
GHN-3 30
N/A
37.28
37.18
N/A
54.13
52.89
In later stages, ECO achieves a 2.6 performance gain over re-initialization methods like KE (59]. while reducing training time by xx. Compared to pre-training-base
BCO-30th
0.82
56.41
55.28
2.47
64.54
64.99
Heur-LG 68
1.76
43.55
43.07
14.99
$2.89
54.54
Auto-LG 70
1.77
50.76
50.13
15.21
55.10
55.60
ECO-50th
59.81
58.60
2.63
66.29
66.32
approaches (e.g., Weight Selection [80)., ECO reduces parameter trans- fer over 5.0x, with performance gains of 3.85, demonstrating both efficiency and the superior transferability of its core knowledge.
KE,,, [59
54.58
54.52
N/A
65.93
67.02
ECO-70th
0.92
63.39
62.08
2.82
68.35
67.92
Wt Select [0
2.94
56.03
55.84
23.45
69.91
69.29
ECO- 100th
0.98
66.50
65.22
3.13
70.73
69.91
5.2 Performance of ECO in Initializing Models of Variable Sizes
Neural networks deployed in diverse hardware environments often require models of varying sizes due to differing computational and storage constraints, but pre-trained models are typically fixed in size, making it impractical to have one for every possible configuration.
ECO addresses this challenge through scalable and size-agnostic learngenes, enabling efficient initialization of networks with variable sizes. As shown in Table 2 ECO consistently achieves superior accuracy across variable-sized models, significantly outperforming existing transfer methods.
Compared to direct pre-training (i.e., Direct PT), ECO reduces training time by least 2x, while transferring up to 2.5x and 3.7x fewer parameters than Weight Selection [80], highlighting its transfer efficiency. Moreover, ECO achieves a 3x reduction in FLOPs relative STKD [57]. Unlike distillation-based approaches whose computational cost scales linearly with the number of target models (O(n)) ECO directly initializes each model independently with no additional training overhead, enabling scalable deployment across diverse resource constraints.
7

---

Table 2: Performance of ECO and other methods for variable-sized model initialization. "Com." refers to the complexity of initializing n models, while "Time", "Para.(M)", and "FLOPs(G)" represent
training time, transferred parameters, and computational overhead, respectively. N/A denotes that the criterion is not applicable to the respective method.
MobileNer V3-- arge, 5.04M, ImageNet, Train
0.70W
0.75W
0.80W
0.85W
0.90W
Methods
Com. Time Para.
FLOPs
Acc.
Para.
FLOPs
Acc. Para. FLOPs
Acc.
Para.
FLOPs
Acc.
Para.
FLOPs
Acc.
Hc-Init 19
O(1) 1.0x
0
0.75
32.71
0
0.87
34.48)
0.92
34.54
1.04
36.131
0
1.15
37.07
GradInit 85
O(n) 1.0x
N/A
0.75
37.07
N/A
0.87
38.26
N/A
0.92
35.68
N/A
1.04
40.01
N/A
1.15
40.58
Direct
O(n) 2.0x
42.35
44.61
44.52
1.15
47.97
Wt Select 80
0(1) 1.xx 1.48
0.75
42.87
1,70
0.87
44,83
1,90
0.92
45.30
2.16
1.04
46.88
2.41
1.15
47.85
STKD [7
O(1) 1.xx
N/A
2.14
46.03
N/A
2.40
46.78
N/A
2.51
45.13
N/A
2.78
47.66
N/A
3.02
48.08
0(1) 10x%
0.98
46.07
0.98
0.87
48.09
0.98
0.92
48.15
0.98
48.73
0.98
1.15
49.13
ResNer50, 24.87M, ImageNet, Train
0.50W
0.55WV
0.60W
0.65W
0.70W
Methods
Com. Time Para.
FLOPs
Acc.
[Para. FLOPs
Acc.
|Para. FLOPs Acc.
Para. FLOPs
Acc.
Para.
FLOPS
Acc.
O(1) 1.0x
41.32
12.40
45.65
GradInit 85
O(n) 1.0x
N/A
6.82
37.28
N/A
8.00
39.28
N/A
9.33
41.09
N/A
10.84
42.68
N/A
12.40
43.97
Direct PT
O(n) 2.0x 0
6.82
49.85
8.00
51.66
9.33
53.59
10.84
54.54
12.40
55.89
Seleet
O(1) 1.0x
5.88
6.82
49.97
7,06
51.21
8.41
9.33
52.86
9.87
10.84
54.23
11.46
12.40
55.89
STKD
57
O(n) 1.0x
N/A
23.16
$1.86
N/A
25.78
$3.00
N/A
28.16
52.91
N/A
32.09
52.99
N/A
35.58
54.39
ECO
0(1) 10x
3.13
6.82
52.23
3.13
8.00
53.12
3.13
53.79
3.13
12.40
55,93
Table Performance ECO and other methods when transferring to downstream datasets. "Para." refers to the transferred parameters, with total parameters and FLOPs recorded for various architec- tures. N/A denotes that the criterion not applicable to the respective method.
Methods
Para.
Flower
CUB
Cars
Food
Aver:
Para.
Flower
CUB
Cars
Food
Aver
VGG11,
FLOPs=45.14G,
Para=28.72M
ResNet12, FLOPs=154.04.,
12.5=12.52M
Hc-Init
19
0
34.22
49.81
60.94
78.67
55.91
50.74
46.89
61.27
81.10
60.00
GradInit
85
N/A
36.38
49.36
66.86
78.07
$7.67
N/A
50.92
$5.61
72.45
81.39
65.09
GHN-2
29
N/A
46.32
49.05
68.18
67.57
57.78
N/A
54.92
49.36
59.21
74.10
59.40
GHN-3
30
N/A
44.09
50.55
69.39
66.49
$7.63
N/A
61.33
56.06
71.30
75.28
65.99
Heur-LG
68
7.09
38.84
55.54
71.67
78.65
61.18
9.43
$4.69
55.68
76.37
81.93
67.17
Auto-LG
70
9.19
50.64
58.54
74.93
78.60
65.68
9.51
61.90
58.99
81.68
82.15
71.18
Wi Select
80
28.09
62.87
60.08
76.21
78.26
69.36 39
12.41
S5,26
$7.92
77.74
81.68
68.15
ECO
1.94
64.42
60.20
78.12
79.47
70.55
2.47
81.48
64.14
84.28
82.41
78.08
MobileNetV3--Larre
FLOPs=
Para=
4.39M
ResNet50, FLOPs=24.62GG
Para=23.81M
Hc-Init [8
U
51.05
56.08
74.88
74,81
64.21
27.53
46.34
46.96
73.66
48.62
GradInit
885
N/A
56.38
56.33
71.67
54.49
59.72
N/A
46.09
48.15
$3.84
75.48
$5.89
GHN-2
22
N/A
46.07
47.53
56.96
66.35
54.23
61.68
55.66
66.91
66.79
62.76
GHN-3
30
N/A
42.75
45.06
54.40
58.54
50.19
NIA
49.18
$2.69
71.58
71.26
61.18
Heur-I.
68
1.76
56.12
58.23
75.43
74.31
66.02
14.99
53.78
$4.95
67.08
72.83
62.16
Auto-LG
700
1.77
56.77
59.15
77.98
74.69
67.15
15.21
55.42
57.11
71.14
73.28
64.24
Select
80
2.94
61.81
61.56
79.17
74.77
69.33
23.45
57.16
52.42
62.58
75.54
61.93
ECO
0.98
64.45
62.17
79.90
75.04
70.39
3.13
77.02
65.12
82.61
76.13
75.22
5.3
Performance ECO on Downstream Tasks
The core knowledge encapsulated learngenes exhibits strong generalizability. enabling effective transfer across wide range of downstream tasks. As shown in Table ECO consistently outperforms baseline methods, confirming its robust adaptability and transfer efficiency.
Hypernetwork-basee approaches (e.g.. GHN-2 [29], GHN-3 (30 achieve competitive results on small datasets but often underperform on larger ones due to their coarse architecture-lvvel parameter modeling. Methods like Hc (68) and Auto-LG [70] transfer pre-trained knowledge layer-wise, but this localized mapping can disrupt global feature hicrarchies, especially in compact architectures. Notably, Auto-LG lags behind ECO by 4.87 average with VGGI1, despite transferring 4.7x more parameters, reflecting inefficiencies in knowledge utilization.
Pre-trained model-based approaches offer extensive knowledge transfer. but their reliance on task- agnostic optimization can lead suboptimal adaptation. On Food-101 with VGGI1, Weight Selection achieves 78.26, in contrast to ECO's 79.47, highlighting the need for more targeted transfer mecha- nisms. These results underscore ECO's ability retain and apply transferable core knowledge more efficiently, offering scalable and task- task-adaptive solution for downstream model initialization.

---

Table 4: Accuracy of shw-shot classification. "-N" indicates narrower networks than normal ones.
CIFAR FS, VGGII
mindmmagenet, ResNer12
Methods
5-shot
10-shot
20-shot
5-shot
10-shot
20-shot
MAML([I4
63.4+=0.66
68.2+0.74
70.5++0.77
61.1=0.78
66.440.68
68.4+0.62
RelationNer 588
64.2++0.79
68.9+0.71
72.9+0.71
65.4+0.69
70.3+0.66
72.9+0.63
MatchingNel
59.9110.78
63.810.78
69.3.10.1
66.310.66
70.910.63
74.7100.99
P5otoNe([54]
65.9:10.55
69.320.79
73.120.69
66.5=0.71
72.440.60
74,9100559
Baseline++
64.910.78
71.31.0.3
75.3110.67
6..5
74,010060
78.2.10.51
ECO
69.9+-0.78 70.3
75.5+
78.5++0.63'
69.4--0711
75.4++0611
80.2++.52
BCO.
76.6110.65
80,5
0.70
76.8
81,7100,53
Table 5: Ablation study results across various architectures.
VGGI1
ResNet12
MobileNetV3-L.
ResNer50
Methods
Valid
Novel
Valid
Novel
Valid
Novel
Valid
Novel
He Init 19
63.13
66.05
67.19
65.55
38.54
38.70
51.85
53.48
Direct Select
70.06
72.30
77.50
76.85
52.46
52.14
60.63
61.23
w/o Tournament & GP
72.31
74.55
80.94
80.25
62.06
60.79
65.79
65.97
w/o Mutation
72.25
73.25
80.88
79.80
63.08
62.11
69.00
68.72
ECO
75.00
76.95
82.00
81.60
66.50
65.22
70.73
69.91
5.4 Improved Data Efficiency of ECO
Models initialized with learngenes demonstrate strong data efficiency, particularly in few-shot learning scenarios. As shown in Table 4 ECO consistently outperforms conventional few-shot learning methods, including those built upon fully pre-trained models such as Baseline++ [7]. underscoring the effectiveness of core knowledge encapsulated in learngenes under limited data conditions.
Notably, a narrower variant ("-N"), initialized with the same learngenes, achieves even higher accuracy. This be attributed to reduced parameter redundancy and stochasticity, indicating that learngenes in ECO not only encapsulate transferable representations but also enable more stable and efficient adaptation under both architectural and data constraints, thereby reinforcing its practicality and robustness in real-world, resource-constraing scenarios.
5.5
Ablation and Analysis
5.5.1
Effeets of Mutation
Mutations are essential to the selection and refinement of core knowledge, as they dynamically reshape the structure and semantics of learngenes. By introducing controlled variations, mutations promote population diversity and enhance the effectiveness of tournament-based selection.
As illustrated in Figure 5 learngenes within ResNet12 undergo continual structural evolution, pro- gressively condensing transferable knowledge while eliminating redundant components, facilitating the emergence more generalizable representations. Thus, the absence of mutation leads static learngene structures, which constrain learngene diversity and hinder knowledge accumulation, as demonstrated in Table
Filter ID
Filter
Layer ID NF5
Layer ID
Mutation
R:B 00
Initial Learngene
Final Learngene
Figure 5: Visualization of learngenes in ResNet12 pre- and post-evolution. is the filter of the
initial learngenes. is the filter that becomes a new part of learngene through mutation. indicates redundant filters removed during the evolution.

---

Vase
Trifle
Malamute
Lion
School Bus
Guitar
Retriever
Original Image
Random Knowledge
Random
ReaNet12
Entire Knowledge
Pretroin
Redhie112
miniZmogeNNet
Entire Knowledge
Pretrain
ResNe150
Pytorch Officia
Core Knowledge
Learngene
ReaNee11
miniZmageNNet
Figure 6: Visualization of core knowledge in learngenes, Networks have not undergone any learning or tie-tuning. ResNet55 (pre-true prainrained on ImageNet) already contains these classes.
5.5.2 Effects of Evolution
ECO leverages evolution to extract and refine core knowledge, enhancing transferability across model sizes and tasks. In contrast, traditional pre-trained models are optimized for specific objectives and lack mechanisms for isolating generalizable components. As a result, methods that directly extract fragments from such models (i.e., Direct Select) fail to capture broadly applicable knowledge, as shown in Table
Tournament selection plays critical role in identifying high-quality learngenes for preservation in the Gene Pool (GP), thereby facilitating the accumulation and iterative refinment of transferable core knowledge across generations. As evidenced in Table the removal of tournament selection and GP (i.e., w/o Tournament GP) disrupts this process, resulting the retention of redundant or suboptimal knowledge and weakening the influence of superior candidates across generations.
5.6 Visualization of Core Knowledge in Learngenes
To illustrate the core knowledge encapsulated in learngenes, visualize model attention using CAM (51) on sample images from novel classes in minilmageNet, which are not involved during learngene evolution.
As shown Figure randomly initialized models tend to focus on diffuse or irrelevant regions, while pre-trained models, though more focused, often highlight background areas. For instance, pre-trained ResNet12 fails to localize novel objects, and ResNe150, despite identifying relevant regions, exhibits background activation that may introduce bias (see Figure TT).
In contrast, models initialized with learngenes produce compact, focused attention maps, concentrat- ing on semantically meaningful and discriminative regions, even to unseen categories.
Conclusion
Inspired by biological knowledge transfer, we propose ECO, method that condenses core knowledge into learngenes for efficient transfer across models. Built upon the Genetic Transfer Learning (GTL) framework, ECO enables evolution of neural networks and the inheritance of learngenes in supervised tasks. Experiments show that learngenes impart strong learning capabilities, while ensuring scalability and adaptability across model sizes and tasks, providing an efficient and generalizable alternative for knowledge transfer through the inheritance of core knowledge.
10

---

Acknowledgement
We sincerely appreciate Freepik for contributing to the figure design. This research was supported by the Jiangsu Science Foundation (BG2024036, BK20243012), the National Natural Science Foundation China (62125602, U24A20324, 92464301, 62306073), China Postdoctoral Science Foundation (2022M720028, 2025T180432), the Xplorer Prize, the Fundamental Research Funds for the Central Universities (2242025K30024), and SEU Innovation Capability Enhancement Plan for Doctoral Students (CXJH SEU 26023).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal. Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint vrXiv:2303.08774, 2023.
[2] Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. Proceedings of the International Conference on Learning Representations (ICLR' (C),''9), pages 1-13, 2019.
[3] Johannes Bohacek and Isabelle M Mansuy. Molecular insights into transgenerational non- genetic inheritance of acquired behaviours. Nature Reviews Genetics, 16(1 1):641-6411222 2015.
[4] Lukas Bossard. Matthieu Guillaumin, and Luc Van Gool. Food-101 discriminative components with random forests. In Proceedings of European Conference on Computer Vision (ECCV"14), pages 446 461, 2014.
[5] Shuvam Chakraborty, Burak Uzkent, Kumar Ayush, Kumar Tanmay, Evan Sheehan, and Stefano Ermon. Efficient conditional pre- pre-training for transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR pages 2241-255,, 2022.
[6] Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Ad- versarial robustness: From self-supervised pre-training to fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'22 22), pages 699-708, 2020.
[7] Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Ku-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In Proceedings of the International Conference on Learning Representations (ICLR' 19), pages 1-12, 2019.
[8] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV'21), pages 9640-9649, 2021.
[9] Ashraf Darwish, Aboul Ella Hassanien, and Swagatam Das. A survey of swarm and evolutionary computing approaches for deep learning. Artificial Intelligence Review, 53(3):1767-1812, 2020.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV'09), pages 248-255, 2009.
[11] Fu Feng, Jing Wang, Xu Yang, and Xin Geng. Leamgene: Inheritable "genes" in intelligent agents. Artificial Intelligence, page 104421, 2025.
[12] Fu Feng, Yucheng Xie, Jing Wang, and Xin Geng. Wave: Weight template for adaptive initial- ization of variable-sized models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR'25) 25), pages 4819 2025.
[13] Fu Feng, Yucheng Xie, Xu Yang, Jing Wang, and Xin Geng. Redefining <creative> in dictionary: Towards an enhanced semantic understanding of creative generation. In Proceedings of the IFFE/CV Conference on Computer Vision and Pattern Recognition (CVPR'22 25), pages 18444- 18454, 2025.
11

---

[14] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. Proceedings of International Conference on Machine Learning I/),,,17),, pages 1135-1135, 2017.
[15] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine- grained car detection for visual census estimation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'77, pages 4502-4508, 2017.
[16] Yuxian Gu, Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR 24)), 2024.
[17] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In Proceedings of the International Conference on Machine Learning (ICML'24), 2024.
[18] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR'I9) pages 4918 4927, 2019.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance on imagenet classification. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR' 15), pages 1026-100 1034,, 2015.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR'IO) 16), pages 770-778, 2016.
[21] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobileneiv3. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'19), pages 1314-1324, 2019.
[22] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference Learning Representations (ICLR'22) 2022.
[23] Mohammadreza Iman, Hamid Reza Arabnia, and Khaled Rasheed. A review of deep transfer learning and recent advancements. Technologies, 11(2):40, 2023.
[24] Eva Jablonka, Marion J Lamb, and Eytan Avital. 'Lamarckian' mechanisms in Darwinian cvolution. Trends in Ecology & Evolution, 13(5):206-210, 1998.
(25) Khurram Javed and Martha White. Meta-learning representations for continual learning. In Proceedings of the Annual Conference on Neural Information P Processing Systems (NeurIPS' 19), pages 1-11, 2019.
[26] Mare TJ Johnson and Jason Munshi-South. Evolution of life in urban environments. Science, 358(6363) 2017.
[27] Motoo Kimura. The neutral theory of molecular evolution. Scientific American, 241(5))98-129, 1979.
[28] Motoo Kimura. The neutral theory of molecular evolution. Cambridge University Press, 1983.
[29] Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero Soriano. Parameter prediction for unseen deep architectures. In Proceedings of the Anmual Conference on Neural Information Processing Systems (NeurIPS'2) 22),, pages 19433-29448, 2021.
[30] Boris Knyazev, Doha Hwang, and Simon Lacoste-Julien. Can we scale transformers to predict parameters of diverse imagenet models? Proceedings of International Conference on Machine Learning (ICML 23), pages 17243-17259 2023.
12

---

[31] Zhiqiang Kou, Si Qin, Hailin Wang, Mingkun Xie, Shuo Chen, Yuheng Jia, Tongliang Liu, Masashi Sugiyama, and Xin Geng. Label distribution learning with biased annotations by learning multi-label representation. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI255, 25), pages 5545-5553, 2025.
[32] Zhiqiang Kou, Jing Wang, Yuheng Jia, and Xin Geng. Inaccurate label distribution learning. IEEE Transactions on Circuits and Systems for Video Technology, 34(10): 14130::0237710249 2024.
[33] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In Proceedings of the International Conference Learning Representations (ICLR (ICLR'20), pages
1-14, 2020.
[34] Shiyu Li, Edward Hanson, Hai Li, and Yiran Chen. Penni: Pruned kernel sharing for efficient enn inference. In Proceedings of International Conference on Machine Learning (ICML'20) pages 5863-5873, 2020.
[35] Suichan Li, Dongdong Chen, Yinpeng Chen, Lu Yuan, Lei Zhang, Qi Chu, Bin Liu, and Nenghai Yu. Improve unsupervised pretraining for few-label transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 21), pages 2021.
[36] Mingbao Lin, Bohong Chen, Fei Chao, and Rongrong Ji. Training compact cnns for image classification using dynamic-coded filter fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4548):10478 2023.
[37] Shuxia Lin, Miaosen Zhang, Ruiming Chen, Xu Yang. Qiufeng Wang, and Xin Geng. Linearly decomposing and recomposing vision transformers for diverse-scale models. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS') 24), 2024.
[38] Lin Liu, Lingxi Xie, Xiaopeng Zhang, Shanxin Yuan, Xiangyu Chen, Wengang Zhou, Houqiang Li, and Qi Tian. Tape: Task-agnostic prior embedding for image restoration. In Proceedings of European Conference on Computer Vision (ECCV'22), pages 447-464 2022.
[39] Yen-Cheng Liu, Chih- Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameter- efficient multi-task adaptation for dense vision tasks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS'22), pages 36889-36901, 2022.
[40] Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search. IEEE Transactions on Neural Networks and Learning Systems, 34(2):550-3) 2021.
[41] Luis López-Marry Samuel Marguerat, and Jürg Bähler. Tuning gene expression to changing environments: from rapid responses to evolutionary adaptation. Nature Reviews Genetics, 98):58335533 2008.
[42] Liqun Luo. Architectures of neuronal circuits. Science, 37366559):cabg7285, 2021.
[43] Seyedali Mirjalili. Genetic algorithm. In Evolutionary Algorithmss and Neural Networks: Theory and Applications, pages 55-55,275 2019.
[44] Seyedali Mirjalili, Jin Song Dong, Ali Safa Sadiq, and Hossam Faris. Genetic algorithm: Theory, literature review, and application in image reconstruction. pages 69-85, 2020.
[45] Vidyanand Mishra and Lalit Kane. survey of designing convolutional neural network using evolutionary algorithms. Artificial Intelligence Review, 56(6):5095-5132, 2023.
[46] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of Indian Conference on Computer Vision, Graphies & Image Processing, pages 722-729, 2008
[47] Kwanyoung Park, Junseok Park, Hyunseok Oh, Byoung- Tak Zhang, and Youngki Lee. Learn- ing tank-agi representation via toddler-inspired learning. In Proceedings of NIPS 2005 Workshop BabyMind, 2021.
13

---

[48] Jathushan Rajasegaran. Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incemeenal task-agnostic meta-learning approach. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR'20), pages 13588- 13597, 2020.
[49] Hui Ren, Joanna Materzynska, Rohit Gandikota, David Bau, and Antonio Torralba. Art-free gen- erative models: Art creation without graphic art knowledge. arXiv preprint urXiv:2412.00170 2024.
[50] Michael T Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G Dietterich. To transfer not to transfer. In Proceedings of NIPS 2005 Workshop on Transfer Learning (NIPSW pages 1-4, 2005.
[51] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of International Conference on Machine Learning (ICML' 17), pages 618-626, 2017.
[52] K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR' 2015.
[53] SN Sivanandam, SN Deepa, SN Sivanandam, and SN Deepa. Genetic algorithms. Springer, 2008.
[54] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS' 17), pages 4077-4087, 2017.
[55] Kenneth O Stanley, David B D' Ambrosio, and Jason Gauci. A hypercube-bart encoding for evolving large-scale neural networks. Artificial Life, 15(2):185-25) 2122, 2009.
[56] Kenneth O Stanley and Risto Müikkulaienn. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10(2):99-227, 2002.
[57] Tongtong Su, Jinsong Zhang. Zhaoyang Yu, Gang Wang, and Xiaoguang Liu. Stkd: Distilling knowledge from synchronous teaching for efficient model compression. IEEE Transactions on Neural Networks and Learning Systems, 34(12):00011-1006) 2023.
[58] Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR' 18), pages 1199-1208, 2018
[59] Ahmed Taha, Abhinav Shrivastava, and Larry Davis. Knowledge evolution in neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR'2I), pages 12843-12852, 2021.
[60] Akbar Telikani, Amirhessam Tahmassebi, Wolfgang Banzhaf, and Amir H Gandomi. Evolu- tionary machine learning: A survey. ACM Computing Surveys, 54(8):1-34 2021.
[61] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini urXiv:2307.099: 09288,, 2023.
[62] Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning a uni- versal template for few-shot dataset generalization. In Proceedings of International Conference on Machine Learning (ICML 21), pages 10424-10433, 2021.
[63] Hamit Taner Unal and Fatih Bazçiftçi. Evolutionary design of neural network architectures: a review of three decades of research. Artificial Intelligence Review, 55(3):1723-1803 2022.
[64] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS 16), pages 3630 3638, 2016.
14

---

[65] Conrad H Waddington. Canalization of development and the inheritance of acquired characters. Nature, 150(3811):563- 1942.
[66] C. Wah, S. Branson, P. Welinder, Perona, and Belongie. The caltech-uesd birds-200-2011 dataset. Technical Report 11S-TRR-011-001 California Institute of Technology. 2011.
[67] Can Wang, Defang Chen, Jian-Ping Mei, Yuan Zhang, Yan Feng, and Chun Chen. Semekd: Semantic calibration for cross-layer knowledge distillation. IEEE Transactions on Knowledge and Data Engineering, 35(6):6305-631, 2023.
[68] Qiu-Feng Wang, Xin Geng, Shu-Xia Lin, Sui-Yu Xia, Lei Qi, and Ning Xu. Learngene: From open-world to your learning task. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'22), 22), pages 3557-8565, 2022.
[69] Qiufeng Wang, Xu Yang, Fu Feng. Jing Wang, and Xin Geng. Cluster-learngene: Inheriting adaptive clusters for vision transformers. Proceedings of the Annual Conference Neural Information Processing Systems (NeurIPS 24), 17:26308-226334 2024.
[70] Qiufeng Wang. Xu Yang, Shuxia Lin, Jing Wang. and Xin Geng. Learngene: Inherit- ing condensed knowledge from the ancestry model descendant models. arXiv preprini arXiv:2305.02279, 2023.
[71] Xudong Wang and X Yu Stella. Tied block convolution: Leaner and better cnns with shared thinner filters. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'21),
pages 10227-10235, 2021.
[72] Yunhe Wang. Chang Xu, Chunjing Xu, Chao Xu, and Dacheng Tao. Learning versatile filters for efficient convolutional neural networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS' 18), pages 1-11, 2018
[73] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'I9) 19), pages 11933 11302, 2019.
[74] Dongyu Wei, Vaishali Talwar, and Dayu Lin. Neural circuits of social behaviors: innate yet flexible. Neuron, 109(10):1600-112,, 2021.
[75] Shiyu Xia, Miaosen Zhang, Xu Yang, Ruiming Chen, Haokun Chen, and Xin Geng. Trans- former as linear expansion of learngene. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'24), pages 16014-16022, 2024.
[76] Guotian Xie, Kuiyuan Yang. and Jianhuang Lai. Filter-in-filerr low cost enn improvement by sub-filter parameter sharing. Pattern Recognition, 91:391-403, 2019.
[77] Yucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang, and Xin Geng. Fine: Factorizing knowledge for initialization of variable-sized diffusion models. arXiv preprint arXiv:2409. 19289, 2024.
[78] Yucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang. Yong Rui, and Xin Geng. Divcontrol: Knowl- edge diversion for controllable image generation. arXiv preprint arXi::2507.23620 2025.
[79] Yucheng Xie, Fu Feng, Jing Wang, Yong Rui, and Xin Geng. Kind: Knowledge integration and diversion in diffusion models. In Proceedings of the International Conference on Machine Learning (ICML 25), 2025.
[80] Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones. In Proceedings of the International Conference Learning Representations (ICLR'24) 24), pages 1-13, 2024.
[81] Yingzhen Yang, Jiahui Yu, Nebojsa Jojic, Jun Huan, and Thomas S Huang. Fsnet: Compression of deep convolutional neural networks by filter summary. In Proceedings of the International Conference on Learning Representations (ICLR") 20), pages 1-10, 2020.
[82] Anthony M Zador. critique of pure learning and what artificial neural networks can learn from animal brains. Nature Communications, 10(1):3770, 2019.
15

---

[83] Jinnian Zhang, Houwen Peng, Kan Wu, Mengehen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'22), pages 12145- 12154, 2022.
[84] Xun Zhou, A Kai Qin, Maoguo Gong, and Kay Chen Tan. A survey on evolutionary construction of deep neural networks. IEEE Transactions on Evolutionary Computation, 25(5):894-91 2021.
[85] Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang, and Tom Goldstein. Gradinit: Learning initialize neural networks for stable and efficient training. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS'2I), pages 16410- 16422, 2021.
[86] Fuzhen Zhuang. Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong. and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, (09,7))43-76, 2020.
[87] Barret Zoph, Golnaz Ghiasi, Tsung-Y Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS' 20), pages
2020.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the Introduction (Section D which clearly states the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Section and Appendix
C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: The datasets used the paper are all public datasets, which are described in the Section and Appendix C
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NewrIPS code and data submission guidelines (https://nips. cc/
public/ /twpudeeeeooduuunnninnn ssinnssssssoooooooo suliooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https)
anips.cc/publis.
SerodeSubbe. ss assonPoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide experimental setting and hyperparameters in Section 4and Ap- pendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provided variance and 95% confidence intervals in Table and Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research strictly conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Appendix
F
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of models data, hence there associated risks requiring safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Data and experimental setup details, including source citations and licensing compliance, are thoroughly documented in the experiments section and supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, baaasstts.ppprrsseee paperssws thoode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not used as essential, original, or non-standard component in the core methodology of this research, All research methods and analyses were conducted independently of L.LMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy
LLM) for what should or should not be described.
23

---

A Additional Related Work
A.1 Common Knowledge in Neural Networks
Common knowledge refers to transferable representations that can be shared across neural networks. It typically falls into two categories: size-agnostic knowledge, which generalizes across network architectures of different scales, and task-agnostic knowledge, which transfers across tasks.
Size-agnostic Knowledge. The modular nature of neural networks (e.g., convolutional filters and Transformer blocks) enables knowledge sharing across models of varying widths and depths. For example, iilter-i-Fillter (76), versatile filters [72], and TBC (71] show that filters can be reused to detect multiple patterns or reduce redundancy. Other works [36 81 34] compress CNNs through cross-filter weight sharing. Similarly, in Transformer- based models, reusable blocks [75] and paraneter-sharing techniques 33 33 achieve efficient scaling without performance degradation.
agsk-agnostic Knowledge. Task-agnostic knowledge captures general visual features that are transferable across domains. Tape (38] encodes such priors for image restoration, while Park et al.[47) and Polyhistor [3) learn gencralizable representations and reusable adapters for multi-task learning. The idea of universal backbone, as Universal Template [62], improves generalization in few-shot settings. Meta-learning approaches like OML [25 and iTAMML 48 enhance transferability by decoupling task-specific heads from shared representations.
ECO integrates and condenses such size- and task-agnostic knowledge into adaptive neural fragments called learngenes, which encapsulate what we refer to as core knowledge. These learngenes enable flexible adaptation across network scales and task types.
B
Additional Details of Methods
B.1
Learngenes in ResNets
Kara
3xbx1
1500 1e 64N4
320 e 160x1x1
5.0 e 320xIx1
8
64.8
160
Shortcut
9X3k3
64x3
160 e 64xc3x3
160 e 360x
3x3
320 @ 160xx 3x
320x3x3
320x
640 e 320x3x3
640 @ 640x3x3
160 e 160%
64 e
64
320
320
640
Figure 7: The form of the learngenes in ResNet12, where the kernels in skip connection layers are also integrated as components of the learngenes.
ResNets enhance traditional CNN architectures by introducing skip connections, typically realized through 1 xx convolutions. These connections enable feature propagation across non-adjacent layers, promoting representational reuse.
To accommodate this architectural feature, the formulation of learngenes in ResNets should extended beyond standard convolutional layers to explicitly incorporate skip connections. Let L, and L, denote the source and target layers of a skip connection, where L, > L,, and let the corresponding skip layer be denoted as Lge. To ensure structural alignment with the main pathway, the number of kernels and filters in the learngenes at Ly: are defined as:
(ow) Nf(".) N( =N(
K
(8)
Accordingly, the selected kernel and filter index sets of learngenes given by:
K F, =s
This configuration ensures that learngenes encapsulate both primary and residual pathways, forming complete kernel-level circuits essential for effective knowledge transfer, as illustrated in Figure 7
24

---

Algorithm Mutation of the Learngene
Input: Learngene G. Mutation probability Pm, Number of layers Nr, and number of filters N in 1-th
layer
1: # Mutation layer by layer
2: for == 1-to to N₁, do
3: # Whhether mutate in I-th layer
Randomly generate number r ~ (0, 1).
5: while S pm do
6: # Mutate!
7:
Whether increase and decrease filter of in layer I Randomly generate number (0, 1).
if 8 I Pi then
10:
# Random increase a filter
11:
Randomly select a filter index from [1, N
12:
# Update the corresponding sets Pi and Ki
13:
Fi < Fi {f}
14:
K++1 - K1+1 ({)
15:
else
16:
# Random decrease a filter
17:
Randomly select a filter index f from Fi
18:
# Update the corresponding sets Fi and Ka {f}
19:
Fi
20:
X + K {f}
21:
end if
22:
# Whether to continue mutating in layer
23:
Randomly generate UO, 1). end while
25:
end for
Algorithm 2 Genetic Transfer Learning
Input: Training dataset Dusia with n, classes, Validate dataset Daa with n. classes, Population number ny, and total number of generation NG
1: for g 0 NG do
2: Initialize population with learngenes
3: Randomly initialize population P, with np networks N,
4: if g * then
Select ancestry learngenes (G1, Ge,,,,,.,,,,, I.,,) from the Gene Pool for each Ns using Eq. Then initialize each N, by inheriting G;
6: end if
7: # Train population P Drrin
for each network N; do
Sample i classes from Dinin to form task 10: Train N₁ on T;
9:
11:
end for
12:
Learngene extraction and evaluation
each learngene do Mutate G. according to Appendix Algorithm I Initialize critic networks C, with mutated Gi Train C, -wy classification task Dial Calculate learngenc score 8, based the accuracy of C.
14:
15:
16:
17:
18:
end for
19:
Update Gene Pool and Gene Tree
20:
Perform learngene competition and select winners g"
21:
Take winners as leaf nodes of the Gene Tree
22:
Update scores of ancestral learngenes according Eq. 6
23:
Refresh Gene Pool by adding winners and climinating learngenes with lower scores
24:
end for
25

---

Nonmal Kernel
Learngene Kernel
Zero Kernel
Identity Kernel
Initialize
(a) Same Structure
(b) Narrower Structure
(c) Deeper Structure
Figure 8: The learngenes exhibit scalability, enabling the initialization of networks with the same structures and the flexibility to initialize networks with narrower/wider and deeper structures. The normal kernels in networks are randomly initialized, while the zero kernels and identity kernels are initialed by (i.e., zero matrix) and i, respectively.
B.2 Further Details on Learngene Mutation
Learngene mutation serves only to alter structural configurations but also to refine the condensed core knowledge. This process increases structural diversity while maintaining broad reservoir of "raw material" for subsequent selection, thereby facilitating the identification core knowledge most suitable for transfer.
The complete mutation procedure is outlined in Algorithm
B.3 Further Details of Genetic Transfer Learning (GTL)
The complete GTL procedure is presented in Algorithm 2
complementing the illustration Figure 3
A key component of GTL is the recursive update of learngene scores, which begins at selected high-performing learngene (i.e., a leaf node the current generation) and proceeds upward along its lineage to the root node. This process ensures that the performance feedback of descendants reflected in the evaluation of their ancestors. The specific score update strategy is described in Algorithm
Algorithm 3 Update of the Learngene Score
Input: Winner learngenes g* Parental decay
coefficient
Path
length
and
Gene
Tree (GT)
I: for each Gi in G* do
2: Initialize node pointer pl G,¹
path length <- 4: Recursively traverse Gene Tree for ancestral learngenes
5: while pł has parent Gμ do Increase path length
6:
t T
Update pavental learngene score 8p
9:
8p & + 8,
10: # Move pointer
11:
Gp end while
13: end for
26

---

B.4 Further Details on Learngene Inheritance
Learngenes possess high structural scalability, enabling the initialization of target networks with varying widths, depths, and architcctures, as illustrated in Figure S
Identical Model Sizes and Architectures. When the target network shares the same architecture as the source (Figure(a)) initialization achieved by directly replacing the randomly initialized kernels at the indices specified by the learngenes. Filters containing unassigned kernels are padded with
zeros, denoted as white kernels Figure X
Width Adaptation. For target networks that differ in width, index mappings must be adapted to avoid index overflows. Specifically, the filter and kernel indices in the /-th layer are re-indexed
Before
After
as Ji [1, [Fi]] and K [1, [Kil], ensuring compatibility with the narrower wider target network (Figure Bp and Figure 99).
Sort
Depth Extension.
When the number of layers in
(d) exceeds that in the learn- gene N(C additional Partial Identity Mapping (PIM) layers are inserted to preserve feature con- tinuity (FigureKc). Each PIM layer bridges adja- cent layers L and L++1 by introducing an inter- mediate mapping: La + Liim + L1-1, where the filter set defined as:
the target network
Layer
Layer 1
Layer
Layer
Figure 9: Sorting the positions of learngene filters and kernels. To prevent index overflow when ini- tializing a narrower model, the indices of learngene kernels and filters rearranged starting from 1.
Fpim Fi
(10)
For each filter f E Fi, the corresponding kernel in the PIM layer is initialized as:
{
i
kf otherwise
0 1ax3
0
(11)
0.00..
This preneiiy- initialization ensures seamless propagation of core features across expanded network depths.
C Implementation Details
C.1 Hyperparameters for Evolution
Table 6 provide the hyperparameters used for evolving the learngenes using GTL.
Table 6: Hyperparameters in evolution.
Hyperparameter
VGG11
ResNet12
ResNet50
MobileNetV3-Large
Training Class Number n4
64
64
640
640
Validation Class Number The
16
16
160
160
Novel Class Number
20
20
200
200
Init Class kynit
50
Max Class Kmax
15
15
70
110
Initial Percentage of Learngene Filters
0.3
0.3
0.3
0.4
Mutation Probability Pm
0.1
0.1
0.2
0.2
Population Size n2
20
20
Gene Poul Size p
Obsolete Number
2
Generational Decay Coefficient 8
0.9
0.9
0.9
0,9
Parental Decay Coefficient 1)
0.1
0.1
0.1
Networks Number in Competition Generation Number
3
3
3
3
250
250
100
100
27

---

Table 7: Performance of ECO across various model sizes and downstream datasets. "Scratch" refers to training from scratch, while "Ggggi" " and "9re12" represent the initialization using learngenes extracted from VGGI1 and ResNet12, respectively. "-N/W" denotes narrower or wider network width. Note that results are given in terms of Top-3 accuracy.
VGG
ResNet
Datasets
11
11_N
11.
16
19
12
12.x
12.w
18
Scratch
51.41
49.18
53.24
47.32
41.78
56.64
56.16
56.58
56.24
Flower
69.25*
66.94
66.97"
72.00"
74.50"
68.99
66.22
66.21
63.15
Gres12
59.15
56.94
57.65
69.75
60.66
77.33"
79,83'
74.63*
76.96*
Scratch
70.75
66.86
73.28
66.60
69.31
63.96
63.72
64.03
67.57
CUB
Gregt1
79.65*
78.67*
81.57
82.74
84.21'
76.67
75.15
76.70
78.24
9res12
77.89
75.99
79.58
83.53"
81.15
81.55*
82.24'
80.34*
83.62*
Scratch
86.01
81.07
88.24
87.14
89.16
80.13
74.43
80.40
82.51
Cars
Grgll
92.25'
91.36'
93.04*
94.62
95.29*
92.80
91.74
93.02
93.62
Shes12
89.78
88.05
91.33
95.20"
94.68
95.63"
95.92"
95.37'
96.10"
Scratch
80.74
79.73
80.29
79.54
76.23
84.62
84.21
85.28
87.59
Food
Gogell Pres1
85.49*
84.73"
84,79*
87.30"
86.85'
88.19
87.32
88.32
89.00
84.69
83.06
84.44
86.46
84.28
89.54"
89.09
89.74*
90.57"
20 iter
50 iter
100 iter
20 iter
50 iter
100 iter
60
60
60
ResNet12
60
1
60
VGG11
40
40-
40-
40-
40
20.
20
20.
20
GradInit
Heur-LG
Auto-LG
Wt Select
ECO
Figure 10: Performance of ECO and other methods novelty classes with minimal training steps.
"iter" indicates parameter update iterations (one iteration equals one optimizer update).
D
Additional Results and Analysis
D.1 Scalability and Adaptability of Learngenes
To further evaluate the scalability and adaptability of learngenes, we initialize networks with varying widths (e.g.. VGGII-N, VGGII-W, ResNet12-N, ResNet12-W), depths (e.g., VGGI6, VGG19, ResNet18), 8), and even across architectures (e.g., VGG=-ReNeet), The results on four downstream datasets presented in Table 7
Learggenes (i.c., Gyggi1 and 9rss1) successfully initialize networks varying depths (c.g.. VGGII, VGG16. VGG19; ResNet12. ResNet18), consistently outperforming models trained from scratch. They also enhance performance when initializing networks with different widths. For example, Gugell initialize wider VGG11-W achieving better performance CUB (81.57% 79.65%) while Grss2 initialize a narrower ResNetl2-N. surpassing the standard ResNet12 on Flower (79.83% vs. 77.33%).
Moreover, learngenes demonstrate effective cross-architecture transferability, with Gugel1 successfully initializing ResNet18 and outperforming models trained from scratch. Despite potential architectural incompatibilities, learngenes show greater flexibility and scalability compared to pre-trained models, which are often limited by architecture. This underscores their capability transfer core knowledge across varying model sizes and architectures in downstream tasks.
D.2 Instincts Models Brought by Learngenes
To further explore the initialization ability of learngenes, we extend the concept of "instinct" from reinforcement learning [ to supervised learning, referring to the innate ability that model initializa- tion provides to neural networks. Figure 10 illustrates the early training performance (within the first epoch) ECO and other methods on VGG11 and ResNet12, with one iteration represents a single optimizer step.
28

---

ECO exhibits notable advantages during early training, requiring only minimal updates to achieve significant performance improvements. Despite transferring more parameters, other methods often struggle to adapt quickly to new classes due to over-transfer of knowledge, which reduces network flexibility. In contrast, ECO selectively transfers only core knowledge via learngenes, enabling descendant networks to quickly adapt to novel tasks. This early-stage capacity for classification with minimal training is referred to as the "instinct" provided by learngenes.
Limitations and Future Works
ECO demonstrates robust performance convolutional architectures by integrating size- and task- agnostic knowledge through learngenes. While inherently scalable, current evaluations are limited to small and medium-sized models. Extending ECO to larger architectures remains promising, though it may incur additional training time and computational overhead. Beyond convolutional networks, we also aim to extend ECO transformer-basel models, where the modular representation of learngenes may further facilitate cross-architecture knowledge inheritance and generalization.
Impact Statement
ECO offers a unified framework for modular knowledge transfer through learngenes, enabling efficient model initialization across architectures and scales. By reducing the need for extensive retraining, ECO lowers computational costs and improves performance in low-data and resource-constrained scenarios. Its architecture-agnose the reuse pre-trained models with minimal overhead, ECO contributes to more sustainable and accessible machine learning, aligning with the goals of green AI.
design supports broad applicability across domains. By promoting
29


Reviews:
[{"review_id": "dB35hqWjQf", "reviewer": "Reviewer_evzs", "summary": "This paper introduces ECO (Evolving Core Knowledge), a novel framework designed to improve the efficiency and flexibility of knowledge transfer in neural networks. The work draws inspiration from biological evolution, proposing a method to distill essential, reusable knowledge into modular components called \"learngenes\".\n\nThe authors argue that existing transfer learning methods, which often rely on reusing entire sets of parameters, are inefficient and lack adaptability across different model architectures and tasks. To address these limitations, ECO introduces a paradigm called Genetic Transfer Learning (GTL). GTL simulates an evolutionary process where a population of neural networks is trained on a variety of tasks. High-performing networks are selected, and their core knowledge, encapsulated in \"learngenes\", is passed on to the next generation. These learngenes are defined as interconnected neural circuits, such as sets of kernels in CNNs.\n\nThrough iterative cycles of inheritance, mutation, and selection, GTL is designed to refine these learngenes, ensuring they contain compact, task-agnostic, and transferable knowledge. This approach allows for the initialization of diverse neural network architectures, of varying sizes, with this evolved core knowledge, significantly reducing computational and memory costs compared to traditional pre-training and fine-tuning.\n\nThe paper presents extensive experiments across a range of datasets, including CIFAR-FS, miniImageNet, and ImageNet, using various architectures including VGG, ResNet, MobileNetV3, and ViT. The results demonstrate that ECO consistently outperforms existing methods in terms of both accuracy and resource efficiency. Notably, it can achieve comparable or even superior performance to full fine-tuning while drastically reducing the number of transferred parameters.", "questions": "1. Considering the significant upfront investment for the evolutionary process, have you analyzed the break-even point where the benefits of reusing the efficient learngenes outweigh the initial training cost? For example, how many downstream models must be initialized with a learngene to make it more cost-effective than standard pre-training approaches?\n\n2. The CAM visualizations effectively show that learngenes help models focus on semantically meaningful regions. Beyond these visualizations of their effects, have you conducted any preliminary analysis into the intrinsic properties of the learngenes themselves? For instance, do the evolved kernels represent recognizable low-level features like edge detectors or texture filters in early layers?\n\n3. Please also address the weakness comments.", "limitations": "Please see the weakness points above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "S1. The analogy to biological genetics is not merely a superficial inspiration but is thoughtfully integrated into the technical framework of GTL. The concepts of \"learngenes\", mutation, inheritance, and selection are well-defined and operationalized for neural networks. This provides an elegant solution to the problem of identifying and isolating transferable knowledge.\n\nS2. The paper provides a robust and comprehensive evaluation of ECO. The consistent outperformance across a variety of datasets, architectures, and tasks, including few-shot learning and downstream applications, lends strong support to the authors' claims. The demonstrated efficiency in terms of reduced parameter transfer and computational overhead is particularly noteworthy.\n\nW1. While the resulting learngenes are efficient to use, the evolutionary process itself appears computationally intensive. The paper notes that the total cost is \"comparable to training a typical medium-scale model,\" but evolving learngenes for hundreds of generations with a population of networks is a significant upfront investment. Further analysis of the trade-offs between the cost of evolution and the benefits of the resulting learngenes would be beneficial.\n\nW2. The GTL framework introduces a new set of hyperparameters related to the evolutionary process, such as population size, mutation rates, and selection mechanisms. The sensitivity of the final learngene quality to these parameters could be a crucial factor in the practical application of this method. A more detailed ablation study on these evolutionary hyperparameters would strengthen the paper.\n\nW3. While the paper visualizes the effects of learngenes on model attention, the intrinsic meaning of the knowledge encoded within a learngene is still largely a black box. Future work could delve into methods for interpreting what specific features or computational primitives these evolved neural circuits represent.\n\nW4. I guess similar mechanism as been studied in \"Delta: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks\", ICLR 2019. Authors in this paper also proposed a method to select CNN kernels with discriminative powers and conduct knowledge distillation from the visual features generated by these kernels for transfer learning.\n\nW5. While the paper provides a comprehensive evaluation across several foundational models, the selection of architectures could be considered somewhat dated, which may temper the broader claims of scalability and generalizability. The study primarily relies on architectures such as VGG11, ResNet, and MobileNetV3. Although these models are undoubtedly influential and serve as important benchmarks, the field of neural architecture design has evolved significantly. Newer architectures often incorporate more advanced features such as sophisticated attention mechanisms, dynamic convolutions, and more efficient scaling strategies that are not present in these older designs.", "was_revised": true, "final_justification": "Authors' responses and additional results (especially case studies and ablations) provided have address my concerns.", "rebuttal": "Dear Reviewer evzs,\n\nThank you for your careful review of our manuscript. We sincerely appreciate your recognition of the biological inspiration behind our method, as well as your acknowledgment of the comprehensiveness of our evaluation.\nBelow, we provide detailed responses to your comments.\n\n---\n\n> *Q1: While the resulting learngenes are efficient to use, the **evolutionary process itself appears computationally intensive**.*\n\nThank you for the valuable question. To clarify, when we state that \"the total cost is comparable to training a typical medium-scale model,\" we mean that **the overall computational cost of the learngene evolution** is similar to pre-training a model of the same size from scratch.\n\nWhile evolutionary methods are often computationally intensive and rarely used for network optimization, our approach is designed to minimize such costs.\n- Instead of training on the full dataset, **each network solves a lightweight few-shot task by sampling $n$-way classification subsets (typically $n=5$)**, improving efficiency and facilitating the discovery of shared core knowledge. \n\n- Instead of **performing traditional parameter search**, our evolution establishes a genetic mechanism for knowledge transfer. Learngenes are inherited and refined across networks through gradient-based updates, enabling efficient accumulation of transferable knowledge.\n\nTo clarify the computational cost of learngene evolution, we provide a simple comparison. \nFor instance, conventional training of ResNet-12 on ImageNet typically involves 100 epochs.\nGiven that ImageNet contains 1,000 classes, this results in approximately 100,000 class-level updates (1,000 classes × 100 epochs).\n\nIn our evolutionary setup of ResNet12, each generation contains 20 networks, **each trained on a 5-way classification task** for 15 epochs. This results in 1,500 class-level updates per generation (20 network × 15 epoch × 5 classes). \nOver 250 generations, the total becomes 375,000 class-level updates (250 generation × 1,500).\n\nHowever, due to the smaller network size and **the parallel nature of population-based training**, this cost is distributed across devices. For example, on a single 24GB RTX 4090, we can train 4 networks in parallel, effectively reducing the cost to 93,750 class-level updates (375,000 / 4)—comparable to the 100,000 updates in standard training.\n\n||#Class|#Epoch|#Model|#Generation|Total Class-Level Updates| Parallelism|Final Class-Level Updates|\n|-|-|-|-|-|-|-|-|\n|Standard Training|1000|100|1|–|100,000|×1|100,000|\n|Ours|5|15|20|250|375,000|×4|93,750|\n\n> *Q2: The GTL framework introduces a new set of hyperparameters related to the evolutionary process, such as population size. The **sensitivity of the final learngene quality to these parameters** could be a crucial factor in the practical application of this method.*\n\nThank you for your valuable suggestions. As noted in our response to Q1, GTL departs from traditional search-based evolutionary algorithms by focusing on knowledge transfer and refinement rather than explicit evolutionary search. \n**Consequently, GTL is relatively insensitive to hyperparameter choices.**\n\nIn response to your suggestion, we conducted additional experiments on population size, mutation rate, and selection strategy. \n**The results confirm that learngene quality remains robust across these variations, highlighting the stability of our evolution process centered on knowledge transfer.** \n\n|Population Size|2|10|*20**|40\n|-|-|-|-|-\n|VGG11 Acc.|75.9|76.8|77.0|77.1\n|ResNet12 Acc.|80.7|81.5|81.6|81.6\n\n|Mutation Rates|0.1|*0.2**|0.3|0.5\n|-|-|-|-|-\n|VGG11 Acc.|76.4|77.0|76.9|76.5\n|ResNet12 Acc.|80.9|81.6|81.4|80.2\n\n|Selection Mechanisms|Roulette Wheel|Elitism|*Tournament**\n|-|-|-|-\n|VGG11 Acc.|77.1|76.8|77.0\n|ResNet12 Acc.|81.5|81.4|81.6\n\n> *Q3: While the paper visualizes the effects of learngenes on model attention, the intrinsic meaning of the **knowledge encoded within a learngene** is still largely a black box.*\n\nThank you for the insightful question. In response, we conducted a detailed analysis of the internal properties of learngene kernels.\nWhile *the NeurIPS rebuttal policy limits the inclusion of figures*, we provide supporting quantitative results here and will include visualizations in revision.\n\n- **Compact Representation**: We analyzed the weight magnitude distribution of learngene kernels and found that 68.5% of their values fall below 0.01, compared to only 39.7% in standard pre-trained models. This suggests that **learngenes are more compact, likely encoding essential transferable knowledge with reduced redundancy**.\n\n    ||Percentage of Weights < 0.01\n    |-|-\n    |Pre-trained Model|39.7\n    |Learngene|68.5\n\n- **Low-level Features**: We evaluated learngene activations across multiple tasks and observed higher cross-task similarity than in pre-trained models, suggesting that **learngenes consistently respond to shared visual features**. \nThese features exhibit clear low-level characteristics, as reflected by higher similarity in early layers (more visibly demonstrated in the visualizations).\n\n    ||Cross-task Similarity\n    |-|-\n    |Pre-trained Model|24.0\n    |Learngene|45.3\n\n- **Near-Orthogonality**: We computed pairwise similarity between kernel weights and found exhibit significantly lower average off-diagonal similarity, indicating near-orthogonality. This suggests that **each kernel encodes distinct, non-redundant information, enhancing feature diversity and minimizing overlap**.\n\n    ||Off-diagonal Similarity\n    |-|-\n    |Pre-trained Model|8.69e-2\n    |Learngene|1.40e-2\n\n> *Q4: I guess similar mechanism as been studied in \"Delta: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks\".*\n\nThank you for your insightful question. **DELTA focuses on identifying kernels most relevant to a specific target dataset, which contrasts with our task-agnostic, generalizable approach.**\n\nAs a distillation-based method, DELTA performs **task-specific transfer from one source to one target domain**, retaining only features useful for that task. \nHowever, this limits generality: **the transferred knowledge is tailored to a single task and cannot be reused across tasks**. \nAs a result, DELTA must be repeated $n×$ for $n$ downstream datasets, reducing scalability.\n\nIn contrast, learngenes support more general and efficient transfer by **extracting task-agnostic core knowledge applicable across diverse tasks**. \nEven with multiple downstream tasks, **they can be directly used for initialization without incurring additional overhead** (e.g., no repeated evolution).\n\n> *Q5: While the paper provides a comprehensive evaluation across several foundational models, the **selection of architectures could be considered somewhat dated**, which may temper the broader claims of scalability and generalizability.*\n\nThank you for pointing this out. While models like VGG and ResNet may not reflect recent architectural advances, **this does not affect the scalability or generalizability of learngenes.**\n\nIn CNNs, learngenes are constructed at the level of convolutional kernels, which **remain fundamental computational units in modern architectures**.\nAlthough recent models incorporate advanced mechanisms—such as attention, dynamic convolutions—convolutional kernels continue to play a central role in capturing localized visual patterns (e.g., edges, textures).\n**This structural consistency ensures that learngenes retain their validity for knowledge transfer, even as architectural designs evolve.**\n\nTo validate this, we evaluate learngenes on GhostNet-v2 and EfficientNetV2, which incorporate DFC attention and compound scaling.\nAs shown in the table below, despite these enhancements, both architectures retain the core convolutional paradigm, **allowing kernel-level learngenes to remain effective with strong compatibility and transferability**.\n\n|GhostNet|Flower|CUB|Cars|Food\n|-|-|-|-|-\n|He-Init|44.46|50.35|64.79|72.33\n|Auto-LG|60.72|56.47|70.54|73.79\n|ECO|**67.44**|**61.70**|**75.40**|**74.27**\n\n|EfficientNet|Flower|CUB|Cars|Food\n|-|-|-|-|-\n|He-Init|49.47|47.08|65.15|68.74\n|Auto-LG|65.58|59.60|74.83|74.13\n|ECO|**71.08**|**64.58**|**79.72**|**77.97**\n\n> *Q6: Considering the significant upfront investment for evolutionary process, have you **analyzed the break-even point** where the benefits of reusing the efficient learngenes outweigh the initial training cost?*\n\nThank you for your valuable comments. We understand the concern regarding the computational overhead of evolutionary methods.\nHowever, in GTL, **each network is trained on a few-way classification task using a small subset of training data, and the population can be trained in parallel, making the overall cost comparable to standard pre-training** (see analysis in Q1).\n\nAs network size increases, population-level parallelism may be limited by GPU memory. For instance, we can parallelize 4 networks per GPU for VGG11 and ResNet12, but only 2 for larger models like ResNet50 and ViT. \nNevertheless, the additional computational cost remains modest—much less than double the original. \nThus, **even for large networks, initializing just two downstream networks is sufficient to reaches a break-even point, while for medium- and small-scale models, initializing a single downstream model is sufficient.**\n\n> *Q7: The CAM visualizations effectively show that learngenes help models focus on semantically meaningful regions. Have you conducted any preliminary analysis into **the intrinsic properties of the learngenes themselves**?*\n\nThank you for your valuable suggestions. As discussed in our response to Q3, learngenes exhibit compact representations that emphasize essential and transferable knowledge. \nThey consistently capture low-level features shared across tasks—particularly in early layers. In addition, their near-orthogonal structure indicates that each kernel encodes distinct, non-redundant information. Please refer to Q3 for further details."}, {"review_id": "sTL8eGTAFi", "reviewer": "Reviewer_aR1z", "summary": "The authors tackle the task of transfer learning using a novel representation called learngenes. These are defined as subnetworks from optimized neural networks which are extracted to be used as population members of an evolutionary algorithm. At each iteration networks are initialized with the subnetwork defined by the learngene plus other randomly initialized units. Learngenes can be muted by expanding and reducing the units that they cover. The authors test the approach on several transfer learning benchmarks.", "questions": "- Are learngenes fixed to particular positions? Doesn't that limit the ability to use the alleged core knowledge in other parts of the network?\n- Are there no other approaches such as NEAT or Quality-Diversity algorithm that could provide similar benefits and thus serve as useful baselines? \n- What are the possible limitations of the approach? How effectively can it scale to larger networks?", "limitations": "The authors do not mention any limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The problem is well motivated as good algorithms for transfer learning is very much an open question.\n- The idea is similar to approaches such as NEAT, but manages to integrate it well with gradient descent approaches that have proven so successful.\n\nWeaknesses:\n- The representation is not general, and must in fact be defined for each possible architecture one wishes to use.\n- I find some the way learngenes are described to be a bit misleading. For example the authors claim that they are optimized for knowledge transfer, not task performance, but it appears to me that it is indeed the later which they do. This is based on the fact that they are optimized and selected for their performance. Knowledge transfer is not optimized for, but just a consequence of the compressed nature of the learngene.\n- Similarly, core knowledge seems to be a weird way to describe what is found by learngenes. In fact it looks more like finding the lottery ticket hypothesis network within the overall architecture.", "was_revised": true, "final_justification": "I find the idea interesting and the authors response to my concerns sufficient to raise my score.", "rebuttal": "Dear Reviewer aR1z,\n\nWe sincerely thank you for your thoughtful and encouraging review.  We appreciate your recognition of our motivation and the positive assessment of the GTL framework.\nBelow, we provide detailed responses to your comments.\n\n---\n\n> *Q1: The representation is not general, and **must in fact be defined for each possible architecture** one wishes to use.*\n\nThank you for your insightful comment. This work explores how transferable core knowledge can be decoupled from neural networks and encapsulated into reusable learngenes for efficient transfer. \nWe evaluate learngenes for both CNN and Transformer architectures, **with a unified representation that generalizes well across diverse model families.**\n\nSpecifically, for CNNs, learngenes are constructed at the level of convolutional kernels—**the fundamental units in modern architectures** for capturing localized visual patterns.\nAlthough architectures like GhostNet and EfficientNet introduce advanced modules (e.g., Ghost modules and SE-enhanced MBConv blocks), they retain the core convolutional structure, thereby **preserving the effectiveness of kernel-level learngenes for knowledge transfer.**\n\nTo validate the applicability of learngenes in modern CNNs, we extend them on GhostNet-v2 and EfficientNetV2.\nAs shown in table below, learngenes remain compatible and exhibit strong transferability despite architectural innovations.\n|GhostNet|Flower|CUB|Cars|Food\n|-|-|-|-|-\n|He-Init|44.46|50.35|64.79|72.33\n|Auto-LG|60.72|56.47|70.54|73.79\n|ECO|**67.44**|**61.70**|**75.40**|**74.27**\n\n|EfficientNet|Flower|CUB|Cars|Food\n|-|-|-|-|-\n|He-Init|49.47|47.08|65.15|68.74\n|Auto-LG|65.58|59.60|74.83|74.13\n|ECO|**71.08**|**64.58**|**79.72**|**77.97**\n\n> *Q2: The authors claim that they are **optimized for knowledge transfer**, not task performance, but it appears to me that it is indeed the later which they do.*\n\nThank you for the valuable question. The Genetic Transfer Learning (GTL) optimizes knowledge transfer by addressing a key challenge in transfer learning: **identifying what knowledge should be transferred to maximize benefit and efficiency, particularly under large domain shifts**.\n\nTo this end, we focus on isolating **a compact subset of parameters that encapsulates highly transferable, task-agnostic knowledge for efficient transfer**, referred to as learngenes.\nHowever, in conventionally pre-trained models, such knowledge is **typically entangled throughout the parameter space**, hindering its extraction.\n\nTo explicitly decouple and encapsulate transferable knowledge for efficient reuse, we propose the GTL framework, which introduces a genetic mechanism into neural networks.\n**In GTL, learngenes are continuously transferred across networks and tasks, during which the core knowledge they encode is incrementally optimized and accumulated.**\n\nThis process gradually disentangles the most essential transferable components from the full network and encapsulates them within the learngene.\nThus, **GTL optimizes the transferability of knowledge itself encoded in leanrgenes**, rather than directly maximizing task performance within a single training instance.\n\n> *Q3: Core knowledge seems to be a weird way to describe what is found by learngenes. In fact it **looks more like finding the lottery ticket hypothesis network** within the overall architecture.*\n\nWe appreciate the connection to the Lottery Ticket Hypothesis (LTH), which posits that a dense network contains a sparse subnetwork that can match its performance when trained independently with the same initialization.\n**Although both LTH and Learngene involve identifying parameter subsets within networks, they differ fundamentally in motivation, mechanism, and outcome.**\n\nThe key distinctions are summarized below:\n\n- **Fundamental Motivation**:\n  - **LTH**: Identifies **task-specific subnetworks** that are easier to train, but are effective only when retrained with the same initialization and data, and therefore **lack cross-task or cross-domain transferability** (see table below).\n    ||Flower|CUB|Cars|Food\n    |-|-|-|-|-\n    |LTH|54.76|57.28|70.89|72.93\n    |ECO|**77.02**|**65.12**|**82.61**|**76.13**\n\n  - **Learngene**: Encodes **task-agnostic core knowledge** optimized for cross-task and cross-domain transfer, and supports **scalable adaptation** across networks of varying sizes.\n\n- **Optimization Target**:\n\n  - **LTH**: Seeks to **reduce parameter count** while preserving performance on a fixed task, effectively optimizing for **model compression**.\n\n  - **Learngene**: Seeks to **maximize knowledge reuse** across diverse tasks with minimal parameter overhead, focusing on **transfer efficiency** rather than compression.\n\n- **Discovery Mechanism**:\n\n  - **LTH**: Discovers sparse subnetworks **after full training via pruning and rewinding**, relying on static analysis.\n\n  - **Learngene**: Constructed and refined **during training through evolutionary inheritance across diverse networks and tasks**, enabling continual accumulation of core transferable knowledge.\n\n> *Q4: Are learngenes **fixed to particular positions**? Doesn't that limit the ability to **use the alleged core knowledge in other parts of the network?***\n\nThank you for pointing this out. Due to the translational invariance property of CNNs, the ordering of kernels within a layer is not fixed. \n**As demonstrated in Appendix Figure 10, reordering kernels does not affect functionality as long as the corresponding input channels in subsequent layers are adjusted accordingly.** Therefore, learngenes are not tied to specific kernel positions and can be flexibly adapted to networks of varying depth or width.\n\nWe then explain *how non-learngene components benefit from core knowledge*. In CNNs, when the $i$-th filter in layer $l$ includes learngene kernels, the resulting $i$-th channel in the feature map encodes core representations. **This channel is subsequently processed by all filters in the next layer—regardless of whether they belong to a learngene—allowing core knowledge to propagate throughout the network.**\n\n> *Q5: Are there no other approaches such as **NEAT or Quality-Diversity algorithm** that could provide similar benefits?*\n\nWe appreciate your valuable suggestion. As noted in our response to Q2, **our method is not a traditional search-based evolutionary algorithm (e.g., NEAT)**, but departs from conventional evolutionary algorithms (EAs) in several fundamental ways:\n\n- **Optimization Objective:**\n\n  - **EA**: Aims to **search for task-specific optimal solutions** by evolving populations of candidate solutions for a given task.\n\n  - **GTL**: Aims to **decouple task-agnostic core knowledge** to enable efficient adaptation to novel tasks across diverse domains.\n\n- **Gene Form and Function:**\n\n  - **EA**: \"Genes\" refers to **encodings of a candidate solution**—typically real-valued vectors (e.g., in Evolution Strategies) or binary strings (e.g., in Genetic Algorithms).\n\n  - **GTL**: Learngenes are **network fragments that encapsulate transferable knowledge**. \n  Rather than encoding solutions, they serve as knowledge carriers that facilitate rapid adaptation in new networks.\n\n- **Evolutionary Mode:**\n\n  - **EA**: Follows Darwinian principles, evolving populations through mutation and crossover to **explore the solution space**.\n\n  - **GTL**: Adopts a Lamarckian approach, where knowledge acquired through gradient-based learning is retained and passed on via learngenes, enabling **cumulative transfer across tasks**.\n\nBuilding on this, learngene fundamentally differs from traditional search-based methods such as NEAT or Quality-Diversity (QD). While these methods can search over network structures or parameters, they primarily target task-specific optimization and are often computationally intensive, especially for deep networks.\n\n**Instead, we include a more relevant baseline tailored to transfer learning: P-Transfer, which employs evolutionary strategies to identify the most suitable layers for fine-tuning.**\nAs shown in the table below, despite transferring all parameters, P-Transfer performs poorly under large domain shifts, supporting our central claim that full-model transfer is suboptimal under substantial distributional differences.\n\n|VGG11|Param.|Flower|CUB|Cars|Food\n|-|-|-|-|-|-\n|He-Init|0M|34.22|49.81|60.94|78.67\n|P-Transfer|28.11M|**72.32**|52.21|72.37|61.54\n|ECO|1.94M|64.42|**60.20**|**78.12**|**79.47**\n\n|ResNet12|Param.|Flower|CUB|Cars|Food\n|-|-|-|-|-|-\n|He-Init|0M|50.74|46.89|61.27|81.10\n|P-Transfer|12.43M|81.01|63.12|83.27|79.56\n|ECO|2.47M|**81.48**|**64.14**|**84.28**|**82.41**\n\n> *Q6: What are the possible limitations of the approach? How effectively can it **scale to larger networks?***\n\nThanks for your question.\nOur main contribution is showing that transferring task-agnostic core knowledge, rather than the full model, yields better transferability and efficiency.\nWe also acknowledge the following limitations, as noted in Appendix E.\n\nDue to computational constraints, our experiments focus on small- to medium-scale networks, and we have not yet extended GTL to large architectures such as GPT-2.\nNevertheless, as discussed in the paper, **the overall cost of the evolutionary process is comparable to standard full-model training.**\nFurthermore, **GTL supports scalability via population-level parallelism and the few-shot nature of individual tasks, enabling application to larger models without significant additional overhead.**\n\nThat said, we acknowledge that increasing model depth and width—particularly in CNNs—results in exponential growth in kernel count, introducing greater structural uncertainty during learngene optimization. \n**To address this, future work may explore coarser learngene granularity, such as using entire filters instead of individual kernels, thereby reducing search space complexity and improving scalability.**\n\nWe will elaborate on potential extensions to large-scale networks in the revised manuscript."}, {"review_id": "wZwIIiAd5l", "reviewer": "Reviewer_fHjf", "summary": "This paper proposes an evolutionary method, ECO, that operates by selection of subnetworks called learngenes. For CNNs a learngene is defined by a subset of filters at each layer and the corresponding learned kernels between them. Evolution is both Darwinian in that successful learngenes are passed on with mutations, and Lamarckian in that learned adaptations during training are inherited.", "questions": "When mutation adds a filter to a learngene how are the weights of that filter within the learngene initialized? Does it inherit the learned weights of the host network?\n\nEq 5 implies larger learngenes are more likely to grow, making the dynamics unstable. Is this as intended?\n\nWhat about a steady downward pressure on learngene size, meaning the probability of mutation by removal is always greater than the probability of mutation of addition? This might act as regularization to encourage more compact learngenes.\n\nAre the critic networks trained on the full validation set, i.e. including all n_v classes?\n\nAre the weights within the learngene adaptable during training of the critic network?\n\nCan filters that are part of the learngene take input from prior-layer filters that are not part of the learngene, or are those kernels fixed to zero? In the former case are those learned weights simply dropped during inheritance?\n\nIs it possible for a learngene’s filter set to be empty at some layer? Maybe this would be useful since it could allow a learngene to be agnostic about how information is processed in the first few and last few layers, which are probably more task-specific than middle layers.\n\nI think it would be useful to compare ECO to a single model with the same total training time, or to a conventional ensemble. For example VGG11 and ResNet12 received 250 x 20 x 15 = 75,000 total epochs of training (plus critic training on the validation set). With such a large compute budget it's important to verify that simpler methods would not perform as well.\n\nCan you give more support for the claim that “total computational cost [is] comparable to training a typical medium-scale model”? A bit of research suggests training a single ResNet12 on ImageNet would take on the order of 100 epochs.\n\nI’m not familiar with most of the baseline methods so I’m unclear on whether they also require training hundreds of separate networks. Do the comparisons of FLOPs and parameter counts account for these factors?\n\nHave you experimented with multiple learngenes per network? This seems feasible and could unlock the parallel and conjunctive search that makes genetic algorithms so powerful.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nNovel and clever evolutionary training method. \n\nExtensive testing on multiple training and transfer datasets and diverse architectures shows significant gains over baselines in both improved performance and fewer transfer parameters.\n\nStrong transfer across downstream tasks and changes to network architecture.\n\n(Note: I don't follow the evolutionary computation literature closely so am not confident about the novelty or the choice of baselines. My evaluation is primarily based on the conceptual and mathematical coherence of the proposal and the strong performance relative to the reported baselines.)\n\nWeaknesses:\n\nHigh compute cost. It's not clear from the experiment results that the large number of parallel models and the large number of generations are paying off relative to non-evolutionary methods.", "was_revised": true, "final_justification": "Novel evolutionary training method, strong experimental results on multiple training and transfer datasets and diverse architectures, and compute cost comparable to baselines.", "rebuttal": "Dear Reviewer fHjf,\n\nThank you for your kind and thorough review, and for recognizing the novelty and contributions of our work. \nYour thoughtful evaluation and constructive feedback have been invaluable in helping us refine and improve our approach. Please find our detailed responses below.\n\n---\n\n> *Q1: When mutation adds a filter to a learngene **how are the weights of that filter within the learngene initialized**? Does it inherit the learned weights of the host network?*\n\nThank you for the insightful question. In the evolutionary process, **structural mutation is employed to introduce discrete changes into the learngene**, fostering diversity and preventing over-reliance on gradient updates, which could lead to local optima.\n\nSpecifically, when a new filter is added via mutation, the goal is to explore transferable knowledge that may not be accessible through gradient descent alone. \nAs you correctly noted, **the newly added filter is initialized directly from the corresponding position in the full network to ensure its relevance and facilitate smooth integration into the learngene.**\n\n> *Q2-1: Eq (5) implies **larger learngenes are more likely to grow**, making the dynamics unstable. Is this as intended?*\n\n> *Q2-2: What about a steady downward pressure on learngene size, meaning the probability of mutation by removal is always greater than the probability of mutation of addition?*\n\nThank you for pointing this out, and we sincerely apologize for the confusion. **This was indeed a typo—we mistakenly wrote $p^+$ instead of $p^-$.**\nWe understand that this could have caused confusion regarding the intended steady downward pressure on learngene size and the relative probabilities of mutation through removal versus addition. \n\nThe correct equation should be:\n$p^{\\scriptscriptstyle -}_l = \\alpha \\cdot \\frac{|\\mathcal{F}_l|}{N_F^{(l)} - |\\mathcal{F}_l|}$\nThis design aims to **balance the probabilities of adding and removing kernels, avoiding uncontrolled growth of learngenes and promoting a focus on discovering the most transferable core knowledge.** \nBy maintaining this balance, learngenes can efficiently explore and adapt to useful representations within a controlled parameter budget, thereby optimizing both transferability and efficiency.\nWe will correct this in revision and provide further clarification.\n\n> *Q3: Are the critic networks trained on the full validation set, i.e. including all $n_v$ classes?*\n\nThank you for your insightful question. As you rightly pointed out, the critic network evaluates learngenes using the full set of validation classes, rather than sampling subsets of classes as in the training phase.\n\n**During evaluation, utilizing the full set of validation classes ensures consistency and fairness by eliminating task difficulty variations**, which could introduce noise or bias and potentially cause instability in the evolution process.\n\n> *Q4: Are the **weights within the learngene adaptable** during training of the critic network?* \n\nWe are grateful for your valuable feedback. **During evaluation on validation classes, the weights within the learngene are updated together with the non-learngene components of the critic network**.\nIn practice, it is also technically challenging to freeze specific kernels while updating others within the same filter or layer.\n\nThis procedure does not compromise the validity of the evaluation. Our objective is not to measure the static performance of a fixed learngene, but to **evaluate how effectively its core knowledge supports rapid adaptation**. An optimal learngene should allow its core knowledge to adapt quickly to new tasks, thereby enabling faster convergence on unseen classes.\n\nMoreover, this process also does not affect the evolution of learngenes, as **all updates to learngenes within the critic networks are discarded after each evaluation**. Therefore, no knowledge from the validation classes is incorporated into the learngenes during evolution.\n\n> *Q5: Can filters that are part of the learngene **take input from prior-layer filters that are not part of the learngene**, or are those kernels fixed to zero? In the former case are those learned weights simply dropped during inheritance?*\n\nThank you for pointing this out.\nDuring inheritance, **only the parameters corresponding to learngenes are inherited by the target network, while the non-learngene parts are randomly initialized to provide flexibility during training.** \nSpecifically, to preserve core knowledge, **we initialize the non-learngene kernels in learngene filters to zero.** \nThis ensures that the feature map channels corresponding to learngene filters retain only the core knowledge, preventing interference from features extracted by non-learngene filters during the early stages of training. \nAfter inheritance, all parameters are updated normally during training. \nFurther details of learngene inheritance are provided in Appendix Figure 9.\n\n> *Q6: Is it possible for **a learngene’s filter set to be empty at some layer?***\n\nWe sincerely appreciate your valuable suggestion. We agree that making learngenes agnostic to task-specific lower and higher layers is an interesting and valuable perspective. While our algorithm was not explicitly designed with this goal, it naturally allows this behavior to emerge. **Through successive kernel-removal mutations, the number of kernels in a layer can indeed be reduced to zero.**\n\nUpon examining the evolved learngenes, we observed that some indeed follow this pattern. Specifically, at the end of the evolution process, **learngenes allocate more kernels to mid-level layers, with fewer in the early and late layers.**\nWe appreciate your observation and will include this in the revised manuscript.\n\n> *Q7: Can you **give more support** for the claim that “total computational cost is comparable to training a typical medium-scale model”?*\n\nThank you for pointing out this important consideration. We provide a concrete comparison to support our claim.\nAs you mentioned, training a ResNet-12 on ImageNet typically requires around 100 epochs. Given that ImageNet contains 1,000 classes, this results in approximately 100,000 class-level updates (1,000 classes × 100 epochs).\n\nIn our evolutionary setup of ResNet12, each generation contains 20 networks, **each trained on a 5-way classification task** for 15 epochs. This results in 1,500 class-level updates per generation (20 network × 15 epoch × 5 classes). \nOver 250 generations, the total becomes 375,000 class-level updates (250 generation × 1,500).\n\nHowever, due to the smaller network size and **the parallel nature of population-based training**, this cost is distributed across devices. For example, on a single 24GB RTX 4090, we can train 4 networks in parallel, effectively reducing the cost to 93,750 class-level updates (375,000 / 4)—comparable to the 100,000 updates in standard training.\n\n||#Class|#Epoch|#Model|#Generation|Total Class-Level Updates| Parallelism|Final Class-Level Updates|\n|-|-|-|-|-|-|-|-|\n|Standard Training|1000|100|1|–|100,000|×1|100,000|\n|Ours|5|15|20|250|375,000|×4|93,750|\n\n> *Q8: I think it would be useful to **compare ECO to a single model with the same total training time**.*\n\nThank you for your valuable suggestion. As discussed in Q7, the overall training cost of our method is comparable to that of training a medium-scale model, due to the few-way setting used for training individual networks and the parallel nature of population-based evolution.\n\nIn fact, once a model has converged under standard training, extending the training time yields diminishing returns, particularly for small and medium-sized architectures. \nTo demonstrate this, we trained a ResNet-12 for 5× longer (~500 epochs), and the performance gain was marginal compared to the results presented in our paper. This further highlights the advantage of learngenes in efficiently transferring core knowledge across tasks.\n\n||VGG11|ResNet12|\n|-|-|-|\n|1× Train|75.3|78.2|\n|5× Train|75.9|78.7|\n|ECO|**78.5**|**80.2**|\n\n> *Q9: Whether the baseline models require training hundreds of separate networks. Do the **comparisons of FLOPs and parameter counts account for these factors**?*\n\nThank you for your constructive suggestion. Our method efficiently transfers core knowledge across tasks and model scales. We compare it to baselines focused on efficient initialization using pre-trained parameters, with detailed descriptions provided in Appendix C.2.\n\nMost baselines incur additional training costs, except for heuristic approaches like Heur-LG. Generally, **higher training costs correlate with better transfer performance**. To assess transfer efficiency, we report the number of transferred parameters in Tables 1 and 2. \n**FLOPs are comparable across methods, as they share the same backbone architecture**, with the notable exception of knowledge distillation, which introduces a teacher model and results in higher computational costs, as clearly indicated in Table 2.\n\n> *Q10: Have you experimented with **multiple learngenes per network***?\n\nThanks for your constructive suggestion. **Combining multiple learngenes is feasible, especially in CNNs, where kernels naturally function as modular units.** In our original manuscript, we focused on evolving a single learngene per network, without exploring multi-learngene combinations. Following your suggestion, we conducted preliminary experiments by combining learngenes from the final generation. The results showed modest performance gains compared to using a single learngene.\n\n||VGG11|ResNet12|\n|-|-|-|\n|Single Learngene|78.5|80.2|\n|Multiple Learngenes|79.0|80.6|\n\nHowever, since our initial evolution strategy does not explicitly encourage combining multiple learngenes, redundant core knowledge may be present across different learngenes. \nTo address this, **we plan to extend the GTL framework to support multi-learngene combinations, similar to genetic \"crossover\" in biological systems.**"}, {"review_id": "UzdS5uKlSC", "reviewer": "Reviewer_X8DF", "summary": "The authors introduce a new method for transfer learning in CNNs, ResNets, and vision transformers based on learngenes, neural circuits that are mutated and propagated through generations. In CNNs, learngenes are defined as collections of kernels that extend from network input to output, preserving information flow and facilitating possible transfer to other networks. In vision transformers, learngenes are realized as continuous circuits that extend from input to output. learngenes are inserted into networks and then trained on k-way image classification. They are then extracted, mutated, and inserted into critic networks, which evaluate learngenes on their ability to transfer core knowledge by performing classification on held out categories. Through tournament-style competition, the best learngenes are selected and passed on to the next generation. This method outperforms others based on gradient heuristics and meta-learning and is more efficient in terms of parameter transfer than fine-turning while retaining similar performance. ECO appears to shine in few-shot contexts. The authors claim their method leads vision transformers to generate focused attention maps when classifying unseen classes of objects.", "questions": "Major:\n\nWhen is supervised learning performed in the training loop? Ideally this would be included in figure 3. \n\nWhat was the rationale behind the decision to constrain the attention matrices as in eq. 12?\n\nWhy are learngenes structurally mutated before initializing the critic? The rationale for this design choice is unclear to me.\n\nMinor:\n\nDo the authors know what is being learned at the circuit level? A follow-up paper that determines exactly what ECO extracts would be of great interest.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis is a thorough and complete study that proves the utility and generalizability of the ECO method by testing across a range of architectures and datasets. Transfer learning as the inheritance of useful neural circuits appears to be effective and also comports heavily with what we know from biology. The authors demonstrate ECO can identify reusable motifs that can be inserted into architectures of varying sizes, retaining performance while transferring fewer parameters than other methods. The method appears to be particularly strong in the few-shot context. The paper is clear, demonstrates substantial knowledge of the relevant literature, and is easy to follow.\n\nWeaknesses:\n\nOverall, I think the paper had very few weaknesses.\n\nMy primary concern is that the validation classes are actually used to train the learngenes via the described method; therefore, scores on this division of the data are analogous to training and not test performance. This should be made clear to the reader.\n\nSome details of the training regimen are a bit unclear, e.g. when typical backpropagation is used and for how many training iterations.\n\nAlso, it would be nice to show these results hold in the event that equation 12 in the appendix is relaxed, i.e. that attention heads do not have matching key, query, and value matrices.\n\nThe following papers should probably be cited:\nTransfer Learning Layer Selection Using Genetic Algorithm", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns. I trust that the final version of the paper will mention the follow up analysis on removing the structural constraint from learngene architecture and also mention that the validation classes are technically part of the training data. I will maintain my score.", "rebuttal": "Dear Reviewer X8DF,\n\nWe sincerely appreciate your valuable feedback and the recognition of the innovation and performance demonstrated in our work. Your thoughtful and detailed review has been highly beneficial, and we are grateful for the time and effort you dedicated to evaluating our manuscript. Below, we provide our detailed responses to your comments and suggestions.\n\n---\n\n> *Q1: **The validation classes are actually used to train the learngenes** via the described method; therefore, scores on this division of the data are analogous to training and not test performance*\n\nThank you for your thoughtful feedback. To clarify, **validation classes are not used for training learngenes but rather for evaluating their generality during evolution**. This process is facilitated by a critic network, which inherits the learngene and is trained on the validation classes. However, **the updates to the critic network are discarded after each evaluation and do not impact the parameters of the learngenes being assessed**.\n\nThus, **no knowledge from the validation classes is incorporated into the learngenes during evolution.** \nThe results on validation classes presented in Table 1 are intended to show that learngenes perform equally well on both novel and validation classes, proving they don’t overfit to the training data.\nWe will revise the title of Table 1 to clarify this distinction and prevent any potential confusion. Thank you again for your valuable suggestion.\n\n> *Q2: Some details of the training regimen are a bit unclear, e.g. **when typical backpropagation is used** and for how many training iterations.*\n\nThank you for your careful review. We agree that some training details were too concise and may have led to confusion. \n**Backpropagation is employed both to update the learngenes, enabling them to acquire knowledge from the training classes, and to train critic networks for evaluating the generality of learngenes.** \nThis process is illustrated by the arrows between Figure 3(b)(c) and (a)(e). We will revise the text and figure annotations to clarify this.\n\nIn Section 4, we briefly mention the required number of training epochs, where **for medium-sized networks, only 15 epochs are needed for a 5-way classification task**. Detailed hyperparameters for training and evolving learngenes in each network are provided in Appendix Tables 6 and 7. \nWe will also review and include any missing hyperparameters to further improve clarity. Thank you again for your meticulous feedback.\n\n> *Q3 It would be nice to show these results hold in the event that equation 12 in the appendix is relaxed, i.e. that **attention heads do not have matching key, query, and value matrices**.*\n\nThank you for the insightful comment. Learngenes are constructed as parameter subsets that encode core, transferable knowledge within neural networks. \n**Equation (12) imposes a structural prior to ensure continuity of feature propagation—i.e., maintaining valid neural circuits across layers.** \nThis constraint is crucial for isolating core representations from non-learngene components, thereby preserving their task-agnostic nature and compositional integrity.\n\nIn response to your suggestion, we explored a relaxed setting without the structural constraint. \n**We hypothesize that tournament-based evolutionary selection may implicitly favor learngenes that maintain inter-layer connectivity, thus enforcing the constraint in a self-organizing manner.**\nAs shown in the table below, even without an explicit prior, the evolution process consistently produces high-quality learngenes, underscoring the robustness and implicit regularization effects of our framework.\n\n||Flower|CUB|Cars|Food|*Aver.*|\n|-|-|-|-|-|-|\n|He-Init|40.90|18.29|8.56|55.49|*30.81*| \n|Auto-LG|61.80|43.44|46.11|68.92|*55.07*|\n|ECO-w/o constrait|74.23*|51.13*|51.20*|69.05*|*61.40**|\n|ECO|**74.91**|**51.55**|**51.95**|**69.41**|***61.88***|\n\n> *Q4: The following papers should probably be cited: Transfer Learning Layer Selection Using Genetic Algorithm*\n\nThank you for pointing this out. This is an important work on using evolutionary algorithms to select transferable layers in transfer learning, and we will include it in the Related Work section.\n\n> *Q5: When is **supervised learning performed** in the training loop?*\n\nThank you for your question. As stated in our response to Q2, supervised learning is employed to train learngenes on the training classes and evaluate them on the validation classes. This process is depicted by the arrows between Figure 3(b)(c) and (a)(e), and we will clarify this further in the text and figure.\n\n> *Q6: What was the rationale behind the decision to **constrain the attention matrices** as in eq. 12?*\n\nThank you for your insightful question. We apologize for any confusion caused by our wording. \nOur intention was not to enforce identical parameters across all attention heads but to **ensure that the corresponding learngene parameters in the query ($Q$), key ($K$), and value ($V$) matrices share the same position**. \nIn other words, Eq. (12) should be more accurately written as $\\mathcal{W}^q = \\mathcal{W}^k = \\mathcal{W}^v$, where $\\mathcal{W}^q$ denotes **the set of indices** representing the learngene neurons in $Q$.\n\n**This ensures that during the continuous mapping of core features, the learngene parameters remain separate from the non-learngene components, thus preserving the integrity of the learned core knowledge.** Specifically, in the attention computation  (i.e., $A_i = \\text{softmax}\\left(\\frac{Q_i K_i^\\top}{\\sqrt{d}}\\right) V_i$), this constraint ensures that **the learngene parameters in $Q$, $K$, and $V$ are not multiplied with the non-learngene parameters**, preserving their distinct functionality in the attention mechanism.\nWe hope this clarification addresses your concerns, and we will provide further details in the revised manuscript.\n\nAdditionally, as suggested, we relaxed this constraint (as detailed in our response to Q3) and found that our evolutionary framework can implicitly preserve it through tournament-based selection. **Even without explicitly enforcing positional alignment across $Q$, $K$, and $V$, the evolution process naturally retains learngenes that satisfy this structure.** This highlights the robustness and implicit regularization capabilities of our framework.\n\n> *Q7: Why are learngenes **structurally mutated** before initializing the critic?*\n\nThank you for your constructive question. The critic network is used to evaluate the generality of the knowledge encoded in learngenes. \nIn GTL, the learngene parameters are updated through two complementary mechanisms:\n- Continuous gradient-based updating: This mechanism is applied to training classes and **is more efficient than traditional search-based approaches**. However, it may limit diversity and **risk getting trapped in local minima**, hindering exploration of a broader range of knowledge representations.\n\n- Structural mutation: To address these limitations, we introduce structural mutations, such as adding or removing kernels. **This discrete mutation introduces diversity into the learning process, allowing the learngenes to escape local minima** and explore a wider set of potentially more generalizable representations.\n\nThe combination of these two mechanisms—gradient-based updating for efficiency and structural mutation for diversity—facilitates a more robust and dynamic evolution of the core knowledge encoded in learngenes, promoting improved generalization across tasks.\n\n> *Q8: Do the authors know **what is being learned at the circuit level?** A follow-up paper that determines exactly what ECO extracts would be of great interest.*\n\nThank you for the insightful question. In response, we conducted a detailed analysis of the internal properties of learngene kernels.\nWhile *the NeurIPS rebuttal policy limits the inclusion of figures*, we provide supporting quantitative results here and will include visualizations in revision.\n\n- **Compact Representation**: We analyzed the weight magnitude distribution of learngene kernels and found that 68.5% of their values fall below 0.01, compared to only 39.7% in standard pre-trained models. This suggests that **learngenes are more compact, likely encoding essential transferable knowledge with reduced redundancy**.\n\n    ||Percentage of Weights < 0.01\n    |-|-\n    |Pre-trained Model|39.7\n    |Learngene|68.5\n\n- **Low-level Features**: We evaluated learngene activations across multiple tasks and observed higher cross-task similarity than in pre-trained models, suggesting that **learngenes consistently respond to shared visual features**. \nThese features exhibit clear low-level characteristics, as reflected by higher similarity in early layers (more visibly demonstrated in the visualizations).\n\n    ||Cross-task Similarity\n    |-|-\n    |Pre-trained Model|24.0\n    |Learngene|45.3\n\n- **Near-Orthogonality**: We computed pairwise similarity between kernel weights and found exhibit significantly lower average off-diagonal similarity, indicating near-orthogonality. This suggests that **each kernel encodes distinct, non-redundant information, enhancing feature diversity and minimizing overlap**.\n\n    ||Off-diagonal Similarity\n    |-|-\n    |Pre-trained Model|8.69e-2\n    |Learngene|1.40e-2"}]

Author rebuttal:

Meta-review: This paper introduces ECO, a novel, bio-inspired framework that evolves modular neural circuits called "learngenes" to achieve efficient and robust knowledge transfer, and evaluates the method on a variety of datasets. The proposed method presents an intriguing means of combining gradient-based training with evolutionary methods to produce qualitatively different features than what might be expected via pure gradient descent. While reviewers expressed concerns about the method’s computational cost and generality, the additional evaluations provided during the rebuttal period allayed these concerns. Overall, the paper presents an interesting and creative approach to training which, while unlikely to usurp gradient-based pretraining methods in the near future, provides a refreshing change of perspective on neural network training which will benefit the NeurIPS community.
Decision: Accept (poster)