Title: ECO: Evolving Core Knowledge for Efficient Transfer
Venue: NeurIPS 2025
Accepted: yes
Keywords: Knowledge Transfer

Population-Based Adaptation

Biologically Inspired Computation
Abstract: Knowledge in modern neural networks is often entangled and structurally opaque, making current transfer methods—typically based on reusing entire parameter sets—inefficient and inflexible. Efforts to improve flexibility by reusing partial parameters frequently depend on handcrafted heuristics or rigid structural assumptions, which constrain generalization. In contrast, biological evolution enables efficient knowledge transfer by encoding only essential information into genes through iterative refinement under environmental pressure. Inspired by this principle, we propose **ECO**, a framework that **E**volves **CO**re knowledge into modular, reusable neural components—termed *learngenes*—through similar evolutionary dynamics. To this end, we redefine learngenes as neural circuits and introduce Genetic Transfer Learning (GTL), a biologically inspired paradigm that establishes a genetic mechanism within neural networks in the context of supervised learning. GTL simulates evolutionary processes by generating diverse network populations, selecting high-performing individuals, and transferring their learngenes to subsequent generations. Through iterative refinement, GTL enables learngenes to accumulate transferable common knowledge. Extensive experiments show that ECO achieves efficient initialization and strong generalization across diverse models and tasks, while significantly reducing computational and memory costs compared to conventional methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a new method for transfer learning in CNNs, ResNets, and vision transformers based on learngenes, neural circuits that are mutated and propagated through generations. In CNNs, learngenes are defined as collections of kernels that extend from network input to output, preserving information flow and facilitating possible transfer to other networks. In vision transformers, learngenes are realized as continuous circuits that extend from input to output. learngenes are inserted into networks and then trained on k-way image classification. They are then extracted, mutated, and inserted into critic networks, which evaluate learngenes on their ability to transfer core knowledge by performing classification on held out categories. Through tournament-style competition, the best learngenes are selected and passed on to the next generation. This method outperforms others based on gradient heuristics and meta-learning and is more efficient in terms of parameter transfer than fine-turning while retaining similar performance. ECO appears to shine in few-shot contexts. The authors claim their method leads vision transformers to generate focused attention maps when classifying unseen classes of objects.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis is a thorough and complete study that proves the utility and generalizability of the ECO method by testing across a range of architectures and datasets. Transfer learning as the inheritance of useful neural circuits appears to be effective and also comports heavily with what we know from biology. The authors demonstrate ECO can identify reusable motifs that can be inserted into architectures of varying sizes, retaining performance while transferring fewer parameters than other methods. The method appears to be particularly strong in the few-shot context. The paper is clear, demonstrates substantial knowledge of the relevant literature, and is easy to follow.\n\nWeaknesses:\n\nOverall, I think the paper had very few weaknesses.\n\nMy primary concern is that the validation classes are actually used to train the learngenes via the described method; therefore, scores on this division of the data are analogous to training and not test performance. This should be made clear to the reader.\n\nSome details of the training regimen are a bit unclear, e.g. when typical backpropagation is used and for how many training iterations.\n\nAlso, it would be nice to show these results hold in the event that equation 12 in the appendix is relaxed, i.e. that attention heads do not have matching key, query, and value matrices.\n\nThe following papers should probably be cited:\nTransfer Learning Layer Selection Using Genetic Algorithm\n\nQUESTIONS:\nMajor:\n\nWhen is supervised learning performed in the training loop? Ideally this would be included in figure 3. \n\nWhat was the rationale behind the decision to constrain the attention matrices as in eq. 12?\n\nWhy are learngenes structurally mutated before initializing the critic? The rationale for this design choice is unclear to me.\n\nMinor:\n\nDo the authors know what is being learned at the circuit level? A follow-up paper that determines exactly what ECO extracts would be of great interest.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an evolutionary method, ECO, that operates by selection of subnetworks called learngenes. For CNNs a learngene is defined by a subset of filters at each layer and the corresponding learned kernels between them. Evolution is both Darwinian in that successful learngenes are passed on with mutations, and Lamarckian in that learned adaptations during training are inherited.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nNovel and clever evolutionary training method. \n\nExtensive testing on multiple training and transfer datasets and diverse architectures shows significant gains over baselines in both improved performance and fewer transfer parameters.\n\nStrong transfer across downstream tasks and changes to network architecture.\n\n(Note: I don't follow the evolutionary computation literature closely so am not confident about the novelty or the choice of baselines. My evaluation is primarily based on the conceptual and mathematical coherence of the proposal and the strong performance relative to the reported baselines.)\n\nWeaknesses:\n\nHigh compute cost. It's not clear from the experiment results that the large number of parallel models and the large number of generations are paying off relative to non-evolutionary methods.\n\nQUESTIONS:\nWhen mutation adds a filter to a learngene how are the weights of that filter within the learngene initialized? Does it inherit the learned weights of the host network?\n\nEq 5 implies larger learngenes are more likely to grow, making the dynamics unstable. Is this as intended?\n\nWhat about a steady downward pressure on learngene size, meaning the probability of mutation by removal is always greater than the probability of mutation of addition? This might act as regularization to encourage more compact learngenes.\n\nAre the critic networks trained on the full validation set, i.e. including all n_v classes?\n\nAre the weights within the learngene adaptable during training of the critic network?\n\nCan filters that are part of the learngene take input from prior-layer filters that are not part of the learngene, or are those kernels fixed to zero? In the former case are those learned weights simply dropped during inheritance?\n\nIs it possible for a learngene’s filter set to be empty at some layer? Maybe this would be useful since it could allow a learngene to be agnostic about how information is processed in the first few and last few layers, which are probably more task-specific than middle layers.\n\nI think it would be useful to compare ECO to a single model with the same total training time, or to a conventional ensemble. For example VGG11 and ResNet12 received 250 x 20 x 15 = 75,000 total epochs of training (plus critic training on the validation set). With such a large compute budget it's important to verify that simpler methods would not perform as well.\n\nCan you give more support for the claim that “total computational cost [is] comparable to training a typical medium-scale model”? A bit of research suggests training a single ResNet12 on ImageNet would take on the order of 100 epochs.\n\nI’m not familiar with most of the baseline methods so I’m unclear on whether they also require training hundreds of separate networks. Do the comparisons of FLOPs and parameter counts account for these factors?\n\nHave you experimented with multiple learngenes per network? This seems feasible and could unlock the parallel and conjunctive search that makes genetic algorithms so powerful.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the task of transfer learning using a novel representation called learngenes. These are defined as subnetworks from optimized neural networks which are extracted to be used as population members of an evolutionary algorithm. At each iteration networks are initialized with the subnetwork defined by the learngene plus other randomly initialized units. Learngenes can be muted by expanding and reducing the units that they cover. The authors test the approach on several transfer learning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The problem is well motivated as good algorithms for transfer learning is very much an open question.\n- The idea is similar to approaches such as NEAT, but manages to integrate it well with gradient descent approaches that have proven so successful.\n\nWeaknesses:\n- The representation is not general, and must in fact be defined for each possible architecture one wishes to use.\n- I find some the way learngenes are described to be a bit misleading. For example the authors claim that they are optimized for knowledge transfer, not task performance, but it appears to me that it is indeed the later which they do. This is based on the fact that they are optimized and selected for their performance. Knowledge transfer is not optimized for, but just a consequence of the compressed nature of the learngene.\n- Similarly, core knowledge seems to be a weird way to describe what is found by learngenes. In fact it looks more like finding the lottery ticket hypothesis network within the overall architecture.\n\nQUESTIONS:\n- Are learngenes fixed to particular positions? Doesn't that limit the ability to use the alleged core knowledge in other parts of the network?\n- Are there no other approaches such as NEAT or Quality-Diversity algorithm that could provide similar benefits and thus serve as useful baselines? \n- What are the possible limitations of the approach? How effectively can it scale to larger networks?\n\nLIMITATIONS:\nThe authors do not mention any limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ECO (Evolving Core Knowledge), a novel framework designed to improve the efficiency and flexibility of knowledge transfer in neural networks. The work draws inspiration from biological evolution, proposing a method to distill essential, reusable knowledge into modular components called \"learngenes\".\n\nThe authors argue that existing transfer learning methods, which often rely on reusing entire sets of parameters, are inefficient and lack adaptability across different model architectures and tasks. To address these limitations, ECO introduces a paradigm called Genetic Transfer Learning (GTL). GTL simulates an evolutionary process where a population of neural networks is trained on a variety of tasks. High-performing networks are selected, and their core knowledge, encapsulated in \"learngenes\", is passed on to the next generation. These learngenes are defined as interconnected neural circuits, such as sets of kernels in CNNs.\n\nThrough iterative cycles of inheritance, mutation, and selection, GTL is designed to refine these learngenes, ensuring they contain compact, task-agnostic, and transferable knowledge. This approach allows for the initialization of diverse neural network architectures, of varying sizes, with this evolved core knowledge, significantly reducing computational and memory costs compared to traditional pre-training and fine-tuning.\n\nThe paper presents extensive experiments across a range of datasets, including CIFAR-FS, miniImageNet, and ImageNet, using various architectures including VGG, ResNet, MobileNetV3, and ViT. The results demonstrate that ECO consistently outperforms existing methods in terms of both accuracy and resource efficiency. Notably, it can achieve comparable or even superior performance to full fine-tuning while drastically reducing the number of transferred parameters.\n\nSTRENGTHS AND WEAKNESSES:\nS1. The analogy to biological genetics is not merely a superficial inspiration but is thoughtfully integrated into the technical framework of GTL. The concepts of \"learngenes\", mutation, inheritance, and selection are well-defined and operationalized for neural networks. This provides an elegant solution to the problem of identifying and isolating transferable knowledge.\n\nS2. The paper provides a robust and comprehensive evaluation of ECO. The consistent outperformance across a variety of datasets, architectures, and tasks, including few-shot learning and downstream applications, lends strong support to the authors' claims. The demonstrated efficiency in terms of reduced parameter transfer and computational overhead is particularly noteworthy.\n\nW1. While the resulting learngenes are efficient to use, the evolutionary process itself appears computationally intensive. The paper notes that the total cost is \"comparable to training a typical medium-scale model,\" but evolving learngenes for hundreds of generations with a population of networks is a significant upfront investment. Further analysis of the trade-offs between the cost of evolution and the benefits of the resulting learngenes would be beneficial.\n\nW2. The GTL framework introduces a new set of hyperparameters related to the evolutionary process, such as population size, mutation rates, and selection mechanisms. The sensitivity of the final learngene quality to these parameters could be a crucial factor in the practical application of this method. A more detailed ablation study on these evolutionary hyperparameters would strengthen the paper.\n\nW3. While the paper visualizes the effects of learngenes on model attention, the intrinsic meaning of the knowledge encoded within a learngene is still largely a black box. Future work could delve into methods for interpreting what specific features or computational primitives these evolved neural circuits represent.\n\nW4. I guess similar mechanism as been studied in \"Delta: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks\", ICLR 2019. Authors in this paper also proposed a method to select CNN kernels with discriminative powers and conduct knowledge distillation from the visual features generated by these kernels for transfer learning.\n\nW5. While the paper provides a comprehensive evaluation across several foundational models, the selection of architectures could be considered somewhat dated, which may temper the broader claims of scalability and generalizability. The study primarily relies on architectures such as VGG11, ResNet, and MobileNetV3. Although these models are undoubtedly influential and serve as important benchmarks, the field of neural architecture design has evolved significantly. Newer architectures often incorporate more advanced features such as sophisticated attention mechanisms, dynamic convolutions, and more efficient scaling strategies that are not present in these older designs.\n\nQUESTIONS:\n1. Considering the significant upfront investment for the evolutionary process, have you analyzed the break-even point where the benefits of reusing the efficient learngenes outweigh the initial training cost? For example, how many downstream models must be initialized with a learngene to make it more cost-effective than standard pre-training approaches?\n\n2. The CAM visualizations effectively show that learngenes help models focus on semantically meaningful regions. Beyond these visualizations of their effects, have you conducted any preliminary analysis into the intrinsic properties of the learngenes themselves? For instance, do the evolved kernels represent recognizable low-level features like edge detectors or texture filters in early layers?\n\n3. Please also address the weakness comments.\n\nLIMITATIONS:\nPlease see the weakness points above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}