Dynamic Neural Regeneration: Enhancing Deep Learning Generalization on Small Datasets
Authors: Vijaya Raghavan T Ramkumar
Elahe Arani
Bahram Zonooz
Venue: NeurIPS 2024 poster (2024)
Keywords: Small datasets Generalization
Overfitting
Iterative training
Neurogenesis
Primary area: machine_vision
TL;DR: A novel iterative learning paradigm with data-aware dynamic masking removes redundant connections, increases DNNs' capacity for learning, and improves generalization on small datasets.

Abstract: The efficacy of deep learning techniques is contingent upon access to large volumes of data (labeled or unlabeled). However, in practical domains such as medical applications, data availability is often limited. This presents a significant challenge: How can we effectively train deep neural networks on relatively small datasets while improving generalization? Recent works have explored evolutionary or iterative training paradigms, which reinitialize a subset of parameters to enhance generalization performance for small datasets. However, these methods typically rely on randomly selected parameter subsets and maintain fixed masks throughout training, potentially leading to suboptimal outcomes. Inspired by neurogenesis in the brain, we propose a novel iterative training framework, Dynamic Neural Regeneration (DNR), that employs a data-aware dynamic masking scheme to eliminate redundant connections by estimating their significance. This approach increases the model's capacity for further learning through random weight reinitialization. Experimental results demonstrate that our approach outperforms existing methods in accuracy and robustness, highlighting its potential for real-world applications where data collection is challenging.

Dynamic Neural Regeneration: Enhancing Deep Learning Generalization on Small Datasets
Vijaya Raghavan T Ramkumar's Elahe Arani! Bahram Zonooz!
Eindhoven University of Technology Wayve raghavijay95@gmai.. com, e. arani@tue nnl, b.zonooz@tue. .nl
Abstract
The efficacy deep learning techniques contingent upon access large volumes of data (labeled or unlabeled). However, in practical domains such as medical applications, data availability is often limited. This presents a significant challenge: How can we effectively train deep neural networks on relatively small datasets while improving generalization? Recent works have explored evolutionary or iterative training paradigms, which reinitialize a subset of parameters to enhance generalization performance for small datasets. However, these methods typically rely randomly selected parameter subsets and maintain fixed masks throughout training, potentially leading to suboptimal outcomes. Inspired by neurogenesis the brain, we propose novel iterative training framework, Dynamic Neural Regen- eration (DNR), that employs data-aware dynamic masking scheme to climinate redundant connections by estimating their significance. This approach increases the model's capacity for further learning through random weight reinitialization. Experimental results demonstrate that approach outperforms existing methods in accuracy and robustness, highlighting its potential for real-world applications where data collection is challenging.
Introduction
Deep neural networks (DNNs) have become essential for solving complex problems in various fields, such as image and speech recognition, natural language processing, and robotics (I). With the increasing availability of data, DNNs have achieved unprecedented performance, surpassing human-level performance some applications
(2). However, success of DNNs limited when dealing with small datasets, where the model tends to overfit and fails to generalize to new data. For example, often diffieult to obtain a large amount data in medical diagnosis due to complexity and high cost of the procedures involved. In such cases, lack of generalization can be dangerous, which can lead to incorrect diagnosis and treatment.
Recently, several studies based on weight reinitialization methods 3 4 have been proposed in the literature to improve generalization by iteratively refining the learned solution through partial weight reinitialization. These methods select and retain a subset of parameters while randomly reinitializing the rest of the network during iterative/eoolutionary training schemes. For example, a sfate-of-the-ar method named Knowledge Evolution (KE) (5) improves generalization by randomly splitting the network into fit and reset subnetworks and constantly reinitializing the reset subnetwork after each iteration. However, the KE approach is limited by its reliance a predetermined mask creation, where random subset of parameters selected and kept constant throughout the iterative training process. This constraint may impede the model's ability to learn effectively from small datasets, ultimately limiting its generalization capabilities. These limitations raise two important questions:
"Equal contribution.
Codee available at http::://gggttub..mom//rttttaaa
Neerniii--eeeaeeee -eeural-Regeneration
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1) Can we leverage an evolutionary training paradigm to evolve adapt the mask over generations, instead of using fixed mask, in order to enhance the generalization performance of deep learning models trained on small datasets? 2) Can we utilize the available data and the internal state of the model to dynamically determine the important parameters for each generation, rather than randomly presetting them?
In our quest address these questions. we draw inspiration from the phenomenon of neurogenesis in the brain. Neurogenesis, the process of dynamically generating or eliminating neurons in response to environmental demands, has been found to play a crucial role in learning and memory consolidation (7) This intricate process enables the brain to adapt to new experiences and stimuli, enhancing generalizability. Recent advances in neuroscience have shed light on the non-random integration of new neurons within the brain (9). For instance, in rodents, synaptic connections has been observed in the hippocampus, where the integration of new neurons leads to the elimination of less active synaptic connections (IO 11). Selective neurogenesis is critical to improving generalization ability by providing a diverse pool of neurons with distinct properties that can integrate into existing neural networks and contribute to adaptive learning (12). Although the precise mechanisms that govern selective neurogenesis are not fully understood, these findings suggest that selective neurogenesis in the human brain enhances generalization capabilities through its dynamic and selective nature. Thus, by emulating the characteristics of selective neurogenesis, we unlock its potential to improve generalization in deep neural networks.
refinement of
Therefore, we present a novel iterative training approach called Dynamic Neural Regeneration (DNR), which distinguishes itself from the conventional Knowledge Evolution (KF) method through its mask computation. Unlike predetermined fixed mask, DNR utilizes a data-aware dynamic masking criterion that evolves and adapts the mask over generations. Through extensive experiments on multiple datasets, we demonstrate that our proposed training paradigm greatly improves the performance and generalization of the models. Furthermore, DNR effectively addresses overfitting on relatively small datasets, alleviating the need for extensive data collection. The main contributions of the paper are as follows.
Dynamic Neural Regeneration (DNR) is an evolutionary training paradigm that incorporates data-aware dynamic masking to selectively transfer knowledge across generations.
Our proposed training paradigm facilitates the learning of generalizable features and in- creases the overall performance of DNNs across small datasets.
DNR exhibits robustness in solving more common challenges in real-world problems, including class imbalance, natural corruption, and adversarial attacks.
2
Related work
Iterative training and weight reinitialization for DNNs is a prominent area of research 5 that focuses mainly on improving generalization performance by partially refining the learned solution or fully iterating the learned solution. Sense-Sparsc-Dentee (DSD) (3) propose a three-phase approach where weights with small magnitudes are pruned after initial training to induce sparsity and retrain the network by reinitializing the pruned weights to zero. Zaidi et al. (15) conducted extensive investigation into the conditions under which reinitialization proves beneficial. BANs (Born Again -edgeediitiilltiinn- based method that follows a similar iterative training
Neural Networks) (4) is paradigm. However, the critical difference between our work and BANs is that it employs the class- logits distribution instead of the network weights transfer knowledge between successive networks. Recently, Zhou et al. (16) (LLFF) propose the forget and relearn hypothesis, which aims to harmonize various existing iterative algorithms by framing them through the lens of forgetting. This approach operates on the premise that initial layers capture generalized features, while subsequent layers tend to memorize specific details. Accordingly, they advocate for the repeated reinitialization and retraining of later layers, effectively erasing information related to challenging instances. Similarly, the LW (17) approach progressively reinitializes all layers. However, these weight reinitialization methods rely on architecture specific assumptions that do not take the data into account. They are based on presumed properties inherent to the model and its learning process. Consequently, these methods lack prior knowledge of which features, layers, or parameters should be reinitialized in a general context. Moreover, research indicates that memorization in neural networks not confined to final layers but involves neurons distributed throughout the model (18). Knowledge Evolution (KE) (5) splits model

---

Evolutionary training paradigm trained for N generations
Data dynamic masking
(c)
Trained Parameters
Task-spocific neurons
Neurons removed
Randomly initialized
Figure 1: Schematics of proposed Dynamic Neural Regeneration (DNR) framework. Our framework
utilizes a data-aware dynamic masking scheme to remove redundant connections and increase the network's capacity for further learning by incorporating random weight reinitialization. Thus, effectively improving the performance and generalization of deep neural networks on small datasets.
weights into fit and reset parts randomly and iteratively reinitializes the reset part during training. The splitting method can be arbitrary (weight-leve splitting (WELS)) or structured (Kemnl-level convolutional-aware splitting (KELS)). This approach involves perturbing the reset hypothesis to evolve the knowledge within the fit hypothesis over multiple generations. Our framework (DNR) distinguishes itself from the conventional Knowledge Evolution (KE) method through its mask computation. DNR utilizes data-aware dynamic masking that adapts the mask over generations and transfers selective knowledge.
Moreover, we differentiate our study from current literature on neural architecture search (NAS) (19 and growing neural networks (20). Our emphasis is consistent network architecture, maintaining fixed connections and parameter count throughout our analysis. Notably, our work sets itself apart from the dynamic sparse training literature (21 (2), as our objective is not achieve sparsity but rather to enhance generalization on small datasets.
Methodology
3.1 Evolutionary training paradigm
We first introduce the evolutionary/ieeraive training paradigm as envisioned in KE (5). Evolutionary training paradigms allow neural networks be trained for many generations, where each generation focuses on optimizing the model to converge towards local minimum while progressively improving generalization. Each generation within the training process is denoted as g. where g ranges from to the total number of generations, N.
We define deep neural network with L layers and is characterized by the set of parameters θ We assume a dataset D consisting of n input-output pairs, denoted { B1)), For a classification task, we define the cross entropy loss for training the network as:
Lee Σ y log(sottmax(f(;;e)) (1 y:) log(1 softmax(f(z,; ());
(I)
where & = fxx; ) is the network's 's predicted output for input x, We initialize the weights and biases of the network randomly.
KE starts by introducing a binary mask M, which partitions the weights of the neural network into two hypotheses before starting training: the fit hypothesis Hm and the reset hypothesis Hiset- This partitioning expressed as follows:
Ha = M and
Hiese = (1 - 11-Man e
(2)
Here, the element-wise multiplication operator @ applied to the mask M and the parameter to obtain the fit hypothesis Hn. Similarly, the reset hypothesis Haset is obtained by clement-wise

---

multiplying the complement of the mask (1 - M) with the parameter set Q These parameters are chosen random before the start of the first generation. This binary mask M is kept constant throughout the evolutionary training; i.e., the parameters belonging to the fit and reset hypotheses remain that category across all generations.
We use the stochastic gradient descent (SGD) algorithm to train the network with a learning rate a. We run SGD for e epochs on the dataset D. The beginning of every new generation is characterized by introducing perturbations applied to the network weights to induce a high loss. This is done by reinitializing the parameters the reset hypothesis while transferring or retaining the parameters belonging to the fit hypothesis. This dynamic process triggers subsequent round of optimization, guiding the neural network toward the search for a new minimum in the parameter space. The initialization of the network f for the next generation fg is as follows:
9 - M 0 + (1 1 M) @ Reiniit
where 0 and O are the parameters of the network f belonging to the previous generation and current generation, respectively. Reinit corresponds to the randomly initialized tensor sampled from a uniform distribution. We then train the next generation of the network /g using SGD with the same hyperparameters and epochs as the first generation.
3.2 Dynamic Neural Regeneration (DNR) with Data-aware Dynamic Masking
Unlike KE, we propose a methodology that offers a distinct advantage regarding binary mask compu- tation and parameter reinitialization. Motivated by the symbiotic link between generalization and selective neurogenesis in biological neural networks (), we introduce a Data-aware Dynamic Mask- ing (DDM) that emulates the process of selective neurogenesis in evolutionary training. The benefits of DDM's way of reinitialization are two-fold. 1) takes advantage of the evolutionary training paradigm and adapts the mask dynamically in each generation rather than using a predetermined mask. This introduces flexibility in the network and improves the generalization performance of deep learning models trained on small datasets. 2) Our masking scheme leverages a model's data and internal state to dynamically determine the important parameters for given task, rather than relying on random pre-setting, to enhance the performance of deep learning models on small datasets. Our way of masking offers a priori knowledge of where and what parameters and layers should be reinitialized in the general case.
The mask M is calculated at the beginning of each generation a data-dependent manner. We assess the importance or sensitivity of each connection in the network to the specific task by employing the SNIP method (23). SNIP decouples the connection weight from the loss function to identify relevant connections. We randomly sample a small subset of data (π) from the current dataset to evaluate connection sensitivity. We define a connection sensitivity mask M E {0, 1).)) where |e] denotes the number of parameters in the network. The mask designed to maintain a sparsity constraint which specifies the percentage of parameters to retain. The computation of connection sensitivity is performed as follows:
Lee (M
Lee ((M 8c,) O ①
O T) lim
5 5-50
8
M=1
where j corresponds to the parameter index and ej is the mask vector of the index j, where the magnitude of the derivatives is then used to calculate the saliency criteria (sy);
(g
8)
-k=1 9 [T)))
(5)
After calculating the saliency values, we apply the sparsity constraint k to the connection sensitivity mask, which ensures that only the top-k task-specific connections are retained. The sparsity constraint k is defined as follows:
M2 = 1 [s,
20
Vje {1..... m1.
where &g is the Ath largest element in the saliency vector 8, 1[-] is the indicator function and m represents the total number of parameters in the neural network. Subsequently, using the saliency values obtained from the connection sensitivity analysis, we select and preserve the top-k important connections. The parameters associated with the connections deemed less important for the current

---

Table 1: Compares the results of method with the other weight reinitialization methods ResNet18. in fg indicates the number of generations the model is trained.
Small Datasets
Methods
CUB
Aircraft
Dog
Flower
MIT
Mean
CE (f1)
53.57 #0.20
51.28 +0.65
63.83
48.48
#0.28
54.49
DSD
53,00 #0.32
47.24
63.58
51.39
53,21 +0.37
55.68
BAN (f10)
53.71 I +0.35
53.19400.22
64,16+0.11
48.53++0.1
55.65 0.028
55.05
KE (f10)
58.11 +0.25
53.2 4:0.4
64.56+
56,15
58.33 +
58.07
DNR (f10)
59.72
55.87
65.76+t013
58.10
61.78
60.25
Smth (f1)
58.92 #0.24
57.16 #0.91
63.64
10.16
51.02 =0.09
57.74
57.70
Smth + LW (fs)
70.50 20.26
67.10
1:032
65.76
1:036
66.92
20.20
61.67
20030
66.39
Smth + LLF (fs)
71.30 #0.14
68.87
#0.12
66.35
#0.22
67.20
=0.24
63. 14.144 #0.18
67.37
Smth + DNR (fs))
70.95 #0.16
66.10
+0.25
66.56
+0.18
68.50
=0.27
63.94
#0.21
67.21
Smth + LB (f10)
69.80 10
65.29+0.51
+6.19
66.89 +0.23
61.29 40.49
65.89
Smth + KE (f10)
66,51 1 +0.070
63.32
63,86++0.23
62.56
59.58 60.62
63.17
Smth + DNR (f1o)
71.37 #0.22
66.63 +0.37
66.81 +-020
68.36
64.10 00..8
67.45
generation are then reinitialized. This process effectively induces selective neurogenesis, allowing the network to adapt and free up its capacity for learning more generalized representations in subsequent generations. Finally, the network for subsequent generation is initialized as shown in Equation 3
Intuitively, incorporated selective neurogenesis as replacement mechanism, reinitializing the input and output synaptic weights of specific subsets of network parameters dynamically during the evolutionary training process (24). Due to the challenges associated with where, how, and when to create neurons (20), we explore data-aware dynamic masking to drive neuron creation and removal, which could improve learning. We first select the erucial parameters based on the computed saliency mask. Ideally, we would like the mask to keep the knowledge learned from the previous generation much possible and to have enough learning capacity accommodate the learning happening in the new generation. The additional learning capacity facilitates the fast adoption of generalized knowledge and reinforces the knowledge retained from the previous generation. In this way, selective neurogenesis is achieved that inherently adapts the network connectivity patterns in data-dependent way learn generalized representations without altering overall network size.
The network with the new initialization undergoes next-generation training with the same data for the epochs, where is kept the same for each generation. The network is trained with the loss function shown in Equation] I Thus, we favor the preservation of the task-specific connections more precisely than the mask criteria used in KE that can guide the network towards those desirable traits that efficiently improve the performance and generalization of DNNs in small datasets.
Experimental Setup
Here, we provide the details on the experimental setup, implementation details, and datasets used in our empirical evaluation.
Datasets: We evaluate the proposed method using five datasets: 102 (25). CUB-200-201
26),, MIT64 (27). Stanford Dogs (28), FGVC-Aireraft (29). The summaries of the statistics of the data set are mentioned in Appendix, Table 10
Since our framework is direct extension of the KE, we follow the same experimental setup. The efficacy of our framework is demonstrated in two widely used architectures: R and ResNet50 (30). We randomly initialize the networks and optimize them with stochastic gradient descent (SGD) with momentum 0.9 and weight decay 1e - 4. We use the cosine learning rate decay with an initial learning rate Ir 10. 0.256) on specific datasets. The networks are trained iteratively for N generations ( (N=11) with batch size b=32 for c=200 epochs without early stopping. The standard data augmentation techniques, such as flipping and random cropping, are used. We employ SNIP 23 with network sparsity & to find the critical subset of parameters at the end of each
Implementation Details:
5

---

generation. For the importance estimation, we use 20% of the whole dataset as subset (T). For all our experiments, we reinitialize a fixed 20% parameters of the network globally. All training settings (Ir, b, e) are constant throughout generations.
Baselines: To evaluate and benchmark the effectiveness of our proposed approach, we conduct comprehensive evaluation by comparing it against several existing methods that involve iterative retraining and reinitialization. Specifically, we benchmark our method against the following tech- niques: 1) Dense-Sparse-Denss Networks (DSD) (3: 2) Born Again Networks (BANs) (); and 3) Knowledge Evolution (KE) with its variant, KELS (5). We also compare our method against non non-iterative approach known as the Long Baseline (LB), which undergoes training for the same number of epochs as the corresponding iterative methods. Since our framework is built on top of KE, we follow the same procedure in all our experiments unless specified.
5 Results
In this section, we conduct comprehensive experimental evaluations of our method across multiple datasets and ablation studies. For an in-depth analysis, extended robustness experiments, including robustness to 15 types of natural corruptions at varying severity levels, adversarial attacks, and class imbalance, are provided in Appendix.
5.1 Evaluation on Small Datasets
Table presents the quantitative classification evaluation results using ResNet18. fg denotes the result at the end of gt generation. We compare DNR with two different configurations: (1) using naive cross-entropy loss (CE), and (2) incorporating label smoothing (Smth) regularizer with
hyperparameter 0.1 31
The Dynamic Neural Regeneration (DNR) framework demonstrates flexibility and consistently im- proves performance over the considered baselines across datasets. Interestingly, KE underperforms in terms of performance compared to long baseline (LB) with equal computation cost. This discrepancy may be attributed to the use of fixed masking criteria throughout evolutionary training, limiting the model's 's adaptability. In contrast, DNR outperforms both longer baselines and KE, consistently improving generalization performance across all datasets.
Similarly, we compare the performance of our method with the label smoothing regularizer (Smth) applied to the baselines. Table [1 shows that method consistently outperforms LB and KE on all datasets. DNR showcases slightly better performance compared to LW and LLF, with a key advantage being its independence from architetture-specffic assumptions. Unlike LLF and LW, which lack prior knowledge of which features, layers, or parameters to reinitialize, DNR uses data- aware dynamic masking selectively remove redundant connections. This reduces complexity and improves scalability as the model grows. Moreover, research indicates that memorization in neural networks is not confined final layers but involves neurons distributed throughout the model (18). highlighting the limitations of architecture-specffi assumptions and underscoring the effectiveness of our approach. Additionally, real-world settings where data arrives in batches (32), DNR can dynamically iterate over each batch, optimizing performance. These results demonstrate the efficacy of the data-aware dynamic masking and selective reinitialization employed by DNR. By adapting task- susk-specific parameters in each generation, DNR achieves superior performance and enhances the model's generalization.
5.2 Evaluation on Large Datasets
Our work is a direct extension of KE (5), which focuses explicitly on improving generalization in the low data regime. However, we also thoroughly evaluate our method on large datasets such as -mageNn (33). CIFARIO, and CIFAR 100 compares the effectiveness of our method (DNR) with Knowledge Evolution (KE) and longer baseline (LB) in larger data sets. For each model, we trained it on top of the baseline for a specific number of generations (f1o), where indicates the number generations. The proposed approach exhibits promising performance and generalization across various large-scale datasets, such as TinylmageNet, when compared to KE and longer baselines. Furthermore, while the performance of KE and longer basclines (LB) falls below the normal standard training (fi). the DNR framework demonstrates
(34) using ResNet50 to assess its scalability. Table 2

---

Block-1
Block-2
Block-3
Block-4
1-2
86.5
92.7
89.4
83.5
78.7
92.2
88.6
83.5
79.1
89.4
83366
84.2
82.2
89.4
78.0
79.0
77.5
93.7
2-3
68.2
3-4
99
96
91.1
89.0
78.0
95
92
94.8
93.4
96.8
95.0
94.
93
93.4
64.2
75.9
65.8
95.6
Generations
83.3
S0.7
94.7
93.3
5-6
98.6
96.3
90.6
92.9
82.0
96.8
94.1
96,9
95.6
98.0
96.9
93.2
94.1
94.4
86.2
17.3
67.3
96.2
6-7
99.6
94.1
94.
77.3
7-8
97.4
97.5
91.8
87.5
97.9
95.4
97.4
96.4
97.5
94.
94.9
78.1
69.0
96.7
8-9
98.4
97.5
93.9
95.5
91.6
98.5
95 9
97.9
97.2
98.7
97.8
96.5
94.9
95.0
88.4
8.0.5
71.1
97.1
9-10
96.9
95.3
1
4
5
6
7
8
10
11
12
13
14
15
16
17
18
Layers in ResNet18
Figure 2: Layer-wise percentage overlap of the retained parameters in consecutive generations.
Table Compares the results of the DNR framework with the KE and longer baselines for ResNet50 on large datasets. in fg indicates the number of generations the model is trained.
Large datasets
Methods
CIFARIO
CIFAR100
ImnglntgeNet
Mean
Smth (fi)
94.3220.38
73.8310.23
54.15
74.10
Smth + LB (f1o)
93.60
40.21
51.16
40.21
72.99
Smth + KE (f10)
93.50 o..22
73.92
52.56
$0.17
73.33
Smth + DNR (f10)
94.61 +0.30
75.05
54.50
1:026
74.72
comparable or slightly improved performance this scenario. This suggests that a selective way of reinitializing benefits iterative training and can effectively handle the complexities and challenges associated with larger datasets and architectures.
5.3 Robustness of Connection Selection across Training Steps
Unlike KE, which employs a randomly predetermined and fixed masking strategy. DNR provides a notable advantage through the utilization of Data-aware Dynamic Masking (DDM) for parameter reinitialization. Therefore, it crucial to investigate whether DNR fully leverages the benefits the evolutionary training paradigm by dynamically adapting the mask in each generation.
The proposed DNR framework employs SNIP 23 as a masking criterion to selectively regulate the parameters that have the least impact performance in each generation of training. To examine this, we analyze the CUB200 dataset using the ResNet18 architecture. We save the mask generated by SNIP after the end of every generation. Visualizing the mask generated by the DNR framework can be challenging due to the large number of parameters in each layer of the backbone. To assess the consistency of connections across generations, of retained parameters between the masks created in consecutive generations. This metric provides a quantitative analysis of the degree of flexibility induced by DNR in the evolutionary training process.
adopt metric based the percentage of overlap
Figure 2 illustrrates the layer-wise percentage overlap retained parameters between consecutive generations in the DNR framework. The results reveal that the earlier layers consistently exhibit high overlap percentage across all generations, indicating a consistent selection connections.
The overlap percentage decreases in the later layers (specifically, layer 4 in ResNet) as the model learns class-specific information. This observation suggests that the mask adapts capture task- specific features while maintaining stability in the earlier layers. Interestingly, we observe that the overlap percentage of the mask progressively increases as the evolutionary training progresses. Specifically, the overlap between the 9th and 10th generations is higher compared to the overlap between the and 2nd generations. This observation suggests that the mask becomes more saturated and stable as the model state converges to lower-loss landscape. This flexible nature of the DNR framework, allowing for the regulation of connections in both early and later layers, contributes its effectiveness in improving generalization performance.
7

---

Table 3: Comparative analysis of DNR and transfer learning across diverse datasets.
Baselines
CUB
Aircraft
Dog
Flower
MIT
Smth transfer learning (13) Smth + DNR (13)
65.63 1.021
61. 02.02.2 10.23
63.8
10.17
57.62 10.19
58.04 10.31
68.56 10024
64.37 19.19
65.72 10.15
62.13 20.23
62.62 10.51
5.4 Comparison with Transfer Learning
Our approach using DNR, indeed differs widely from the domain of transfer learning. Unlike transfer learning, which primarily focuses on leveraging pre-trained models trained on large datasets from different domains to boost task performance on downstream tasks, DNR is intricately designed to tackle the intricate challenge of enhancing generalization in the presence of inherently limited or small datasets. A key issue with transfer learning arises when the pre-trained model's source domain vastly differs from the target domain of interest. This discrepancy between domains often leads to domain shifts, where the knowledge transferred from the pre-trained model fails to adapt well to the specificities of the target domain, thereby resulting in suboptimal performance.
In particular, in scenarios like medical applications, obtaining sufficient labeled data that closely aligns with the task at hand is exceptionally challenging. Though transfer learning is predominantly used in this field, the need for domain expertise, privacy concerns, and the uniqueness of each applica- tion domain make it exceedingly difficult to find pre-trained model that seamlessly fits. Furthermore, the presence of domain shift between the source and target might lead to compromised perfor- mance, affecting the accuracy and generalization of the model on specific task with limited data. DNR, on the other hand, offers novel solution these intricate challenges. By employing data- aware dynamic masking and selective reinitial- ization, DNR fosters the gradual evolution of the network, enabling it to adapt more effectively to the characteristics of the specific dataset. This process circumvents the problems of domain shifts that often plague transfer learning meth- ods. Thus, while transfer learning remains valu- able in contexts with abundant and well-alin alignad data, DNR stands out as a specialized approach to address the unique hurdles faced in scenarios of limited data availability, where the domain- shift problem can severely hinder model perfor- mance and generalization.
70
68
Accuracy
68
64-
Transter Leariing (Vanilla Fine-tuning)
60
DNR
6
10
Generation
Figure 3: Convergence behavior: evaluating per- formance across generations in DNR and trans- fer learning with ResNet18 architecture trained on CUB dataset.
Furthermore, we have included comparative analysis in Table that involves an instance of transfer learning within the iterative training pro- cess, a process we refer to as vanilla fine-tuning.
In this particular case, weights are directly transferred from one generation to the next without undergoing reinitialization.
This comparison serves highlight the unique effectiveness of the DNR method. Our results distinctly demonstrate that DNR enhances the process of generalization, showcasing superior performance in comparison to the approach of directly transferring the complete network's weights across generations. This outcome further underscores the distinct advantage of DNR in evolving the network's 's capacity for better adaptation and learning in the evolutionary training paradigm.
5.5 Analyzing Convergence Patterns: A Comparative Study between DNR and Transfer Learning
In Figure 3 we present the convergence behavior of our proposed DNR algorithm juxtaposed with vanilla fine-tuning. The x-axis delineates different generations during the training process, while the y-axis represents the performance at the end of each generation. We observe that the convergence of vanilla fine-tuning unfolds at more gradual pace. In contrast, DNR demonstrates faster
8

---

convergence rate. Across generations, DNR consistently surpasses vanilla fine-tuning, delivering a heightened performance level within shorter training duration. The utilization of data-aware dynamic masking through SNIP in DNR amplifies this efficiency, enabling the model to concentrate on the most pertinent information for effective generalization.
5.6 Effect of Importance Estimation Method
We conduct an investigation into the effective- ness of different methods to estimate the impor- tance of parameters within our proposed train- ing paradigm. Specifically, we explore Fisher Importance (FIM), weight magnitude, random selection, and SNIP (23) criteria. In Table 4 present the performance and generalization results of the model trained with these various selection methods the CUB200 dataset using the ResNet18 8 architecture.
Table 4: Evaluating the performance of DNR with different importance estimation.
Importance Criteria
CUB200
Flower
LB Random (KE)
69.80 #0.13
66.89 40.23
66.51 #0.07
62.56 #0.17
FIM
67.73 #0.28
65.96 #0.20
Weight Magnitude SNIP
64.18 +0.19
66.90 #0.11
71.87 10.22
68.36 10.14
Our findings demonstrate that the use of SNIP as data-aware dynamic masking yields superior performance compared to all other baseline methods. Surprisingly, the importance criteria based on weight magnitude exhibited inferior performance compared random selection. However, the lottery ticket hypothesis (35) suggests the existence of sparse subnets within neural networks. Remarkably, when these subnets are trained in isolation, they can achieve a final performance accuracy comparable to that of the entire network in the same even fewer training epochs. In particular, neurons within these winning subnets demonstrate higher rates of weight changes relative to other neurons. This observation raises the possibility of selectively reinitializing neurons that undergo minimal weight changes during training, as they contribute the least to loss function. Merely relying on the l norm, which fails capture the rate of weight changes, as described by the lottery ticket hypothesis, may not adequately capture the notion of importance. Therefore, our findings suggest that the utilization of SNIP for data-aware dynamic masking proves to be more effective, as considers the rate of weight changes in determining the importance of parameters. This approach aligns better with the lottery ticket hypothesis and leads improved performance and enhanced generalization capabilities in our experimental evaluations.
Conclusion
We presented Dynamic Neural Regeneration (DNR), an iterative/evolutionry training paradigm designed to improve the generalization of deep networks on small datasets. Our framework incor- porates selective reinitialization at the end of each generation, employing a data-aware dynamic masking scheme remove redundant connections based on their importance. This enables the model to increase its capacity for further learning, emphasizing the acquisition of generalizable features. Empirical results demonstrate that the proposed framework substantially enhances performance and generalization across small datasets compared to other reinitializing techniques. Moreover, DNR exhibits improved robustness in challenging real-world scenarios, including adversarial attacks and learning with class imbalances, while enhancing generalization on natural corruption data.
As direction for future research, delving into the possibilities offered by growing networks (36) presents an intriguing path worth exploring. Our primary aim was to demonstrate the practical effectiveness of DNR, laying the foundation for theoretical exploration in this domain. We envision future studies to delve deeper into elucidating the theoretical underpinnings of DNR's success and exploring its application in diverse domains beyond those examined in this study. One possible limitation that emerged from our empirical evaluation is the sensitivity of our method to the choice of parameter selection. Addressing this limitation in future work could involve developing a robust parameter selection or importance estimation technique. This enhancement would improve our method's 's overall performance and deepen our understanding of overfitting dynamics by disentangling the contributions of different parameters. Finally, we believe the development of techniques like DNR holds the potential to advance the capabilities of deep learning models, paving the way for robust, adaptable, and generalizable artificial intelligence systems.

---

References
[1] Y. LeCun, Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 4p..436-446 436-444, 2015.
[2] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, et al., "Mastering the game of go with deep neural networks and tree search," nature, vol. 529, no. 7587, pp. 484 489, 2016.
[3] S. Han, J. Pool, S. Narang, H. Mao, E. Gong. S. Tang, E. Elsen, P. Vajda, M. Paluri, J. Tran, et al., "Dsd: Dense-sparse-dense training for deep neural networks," arXiv preprint arXiv:1607.044 00331,, 2016.
[4] Furlanello, Z. Lipton, M. Tschannen, Itti, and Anandkumar, "Born again neural networks," in International Conference on Machine Learning, pp. 1607-1616, PMLR, 2018.
[5] A. Taha, A. Shrivastava, and L.. Davis, "Knowledge evolution neural networks," in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12843- 12852, 2021.
[6] T. J. Shors, G. Miesegaes, A. Beylin, M. Zhao, T. Rydel, and E. Gould, "Neurogenesis in the adult is involved the formation of trace memories," Nature, vol. 410, no. 6826, pp. 372-376, 2001.
[7] A. Garthe, Roeder, and G. Kempermann, "Mice in an enriched environment learn more flexibly because of adult hippocampal neurogenesis," Hippocampus, vol. 26, no. 2, pp. 261-271, 2016.
[8] G. Kempermann, H. Song, and H. Gage, "Neurogenesis in the adult hippocampus," Cold Spring Harbor perspectives in biology, vol. 7, no. 9, p. a018812, 2015.
[9] M. Yasuda, E. M. Johnson-Vennatesh, H. Zhang, J. M. Parent, M. A. Sutton, and H. Umemori, "Multiple forms of activity competition refine hippocampal circuits in vivo," Neuron, vol. 70, no. 6, pp. 1 1128-1 1442, 2011.
[10] J. B. Aimone, Y. Li. S. W. Lee, G. D. Clemenson, W. Deng, and F. H. Gage, "Regulation and function of adult neurogenesis: from genes to cognition," Physiological reviews, 2014.
[11] K. C. Vadodaria and F. H. Gage, "Snapshot: adult hippocampal neurogenesis," Cell, vol. 156, no. 5, pp. 1114-1114, 2014.
[12) N. Toni, D. A. Laplagne, C. Zhao, G. Lombardi, C. E. Ribak, F. H. Gage, and A. F. Schinder, "Neurons born in the adult dentate gyrus form functional synapses with target cells," " Nature neuroscience, vol. 11, no. 8, pp. 901-907, 2008.
[13] Oh, S. Kim, N. Ho, J.-H. Kim, H. Song, and S.-Y. Yun, "Refine: Re-randomization before fine-tuning for cross-domain few-shot learning," in Proceedings of the s1 ACM International Conference on Information Knowledge Management, pp. 4359 4363, 2022.
[14] S. Zaidi, T. Berariu, H. Kim, J. Bornschein, C. Clopath, Y. W. Tch, and R. Pascanu, "When does re-initialization work?," in Proceedings on, pp. 12-26, PMLR, 2023.
[15] S. Zaidi, T. Berariu, H. Kim, J. Bornschein, C. Clopath, Y. W. Teh, and R. Pascanu, "When does re-initialization work?," arXiv preprint arXiv:2206. 10011 2022.
[16] H. Zhou, A. Vani, H. Larochelle, and A. Courville, "Fortuitous forgetting in connectionist networks." arXiv preprint arXiv:2202.00/55 2022.
[17] I. Alabdulmohsin, Maennel, and D. Keysers, "The impact reinitialization on generalization in convolutional neural networks." arXiv preprint arXiv:2109.0020 2021.
[18] Maini, M. C. Mozer, H. Sedghi, C. Lipton, J. 7. Kolter, and Zhang, "Can neural network memorization be localized?," arXiv preprint 20X::2307700942. 2023.
10

---

[19] A. Gordon, E. Eban, O. Nachum, B. Chen, H. Wu, T.-J. Yang, and E. Choi, "Morphnet: Fast & simple sessource-consraiiig structure learning of deep networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1586-1595, 2018.
[20] U. Evci, B. van Merrienboer, T. Unterthiner. M. Vladymyrov, and F. Pedregosa. "Gradmax: Growing neural networks using gradient information," " arXiv preprint arXiv:2201.05125, 2022.
[21] U. Evei, T. Gale, J. Menick, P. S. Castro, and E. Elsen, "Rigging the lottery: Making all tickets winners," in International Conference on Machine Learning, pp. 2943-2952, PMLR, 2020.
[22] S. Liu, T. Chen, X. Chen, Z. Atashgahi, L. Yin, H. Kou, L. Shen, M. Pechenizkiy, Z. Wang, and D. C. Mocanu, "Sparse training via boosting pruning plasticity with neuroregenerattonn Advances in Neural Information Processing Systems, vol. 34, pp. 9908-9922, 2021.
[23] N. Lee, T. Ajamthan, and H. Torr, "Snip: Single-shot network pruning based on connection sensitivity," arXiv preprint arXiv: 181000344 2018.
[24] L. M. Tran, Santoro, L. Liu, A. Josselyn, B. A. Richards, and P. W. Frankland, "Can neurogenesis act as neural regularizer?," bioRxiv, pp. 2022-022-22222 04, 2022.
[25] M.-E. Nilsback and A. Zisserman, "Automated flower classification over large number of classes," in 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729, IEEE, 2008
[26] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, "The caltech- cusd hirds-200-200 dataset," 2011.
[27] A. Quattoni and A. Torralba, "Recognizing indoor scenes," in 2009 IEEE conference on computer vision and pattern recognition, pp. 413-420. IEEE, 2009.
[28] A. Khosla, N. Jayadevaprakash, B. Yao, and F.-F. Li, "Novel dataset for fine-grained image categorization: Stanford dogs," " in Proc. CVPR workshop on fine-grained visual calegorization (FGVC), vol. 2, Citeseer, 2011.
[29] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, "Fine- Frine-grained visual classification of aircraftt" arXiv preprint 1306::333..11/// 2013.
[30] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[31] R. Müller, S. Kornblith, and Hinton, "When does label smoothing help?," Advances in neural information processing systems, vol. 32, 2019.
[32] L. Caccia, J. Xu, M. Out, M. Ranzato, and L. Denoyer, "On anytime learning at macroscale," in Conference Lifelong Learning Agents, pp. 165-182, PMLR, 2022.
[33] Y. Le and X. Yang, "Tiny imagenet visual recognition challenge," " CS 23/N, vol. 7, no. 7, p. 2015.
[34] A. Krizhevsky, G. Hinton, al., "Learning multiple layers of features from tiny images," " 2009.
[35] Frankle and M. Carbin, "The lottery ticket hypothesis: Finding sparse, trainable neural networks." arXiv preprint arXiv:1803.0m 03665, 2018.
[36] X. Dai, H. Yin, and N. K. Jha, "Nest: A neural network synthesis tool based on a grow-and- paradigm," IEEE Transactions on Computers, vol. 68, no. 10, pp. 1487-1497, 2019.
[37] D. Hendrycks and T. Dietterich, "Benchmarking neural network robustness to common corrup- tions and perturbations," arXiv preprint 120. 2019.
[38] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, Goodfellow, and R. Fergus, "Intriguing properties of neural networks," arXiv preprint arXiv: 2013.
11

---

[39] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," arXiv preprint arXiv:1706.0600 06083, 2017.
[40] A. Chrysakis and M.-F. Moens, "Online continual learning from imbalanced data," in Interna- tional Conference on Machine Learning, pp. 1952-1961, PMLR, 2020.
[41] O. Kovalyk, J. Morales-Sánchez, R. Verdό-Monedere I. Sellés-Nvvrrro, A. Palazón-Cabanes and J.-L. Sancho- -aebz, "Papila: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment," Scientific Data, vol. no. 1, p. 291, 2022.
12

---

A Appendix/ Supplemental Material
A.1 Broader Impact
We believe that the research work presented herein has the potential to significantly impact the field of deep learning by facilitating the development models on small datasets, thus reducing the necessity for extensive data collection. This contribution is of particular significance in specific domains like autonomous navigation and medical imaging, as data acquisition can be cost-prohibitive, even in cases where labeling is unnecessary. By mitigating overfitting small datasets, the proposed approach can improve the accuracy and reliability deep learning models in these areas, ultimately leading to improved patient outcomes and safer autonomous systems. Moreover, this work has the potential to usher in a more efficient and sustainable use of resources diverse range of domains where data collection is bottleneck.
A.2 Evolution of Mask across Generations
In this section, we present the evolutionary process of mask over multiple generations and its impact on the performance and generalization capabilities of the DNNs. We evaluate the effectiveness of DNR in dynamically adapting and evolving the mask throughout the training process. The ResNet18 architecture with CUB200 is used for this evaluation.
Figure illustrates the evolution of the mask between generations. training progresses, the mask undergoes iterative updates based on the data-aware dynamic masking criteria employed by DNR. The mask becomes more refined and selective with each generation, preserving important connections while pruning less relevant ones.
Block-1
Block-2
Block-3
1-2
56.5
92.7
69.4
63.5
78.7
92.2
88.6
83.5
79.1
89.4
63.6
84.2
82.2
89.4
76,0
79.0
77.5
93.7
99.8
913
87.1
83.6
81.7
1-4
98.8
92.9
87.0
81.3
73.4
90.7
86.5
81.5
75.7
88.8
82.4
83.4
81.5
89.3
77.9
78.5
77.4
93.7
Generations
1-5
99.8
92.4
85.2
80.6
70.8
90.6
85.9
81.3
75.0
88.8
82.3
83.4
81.5
89.3
77.8
78.6
77.4
93.7
1-6
1-7
99.8
92.3
84.9
81.0
69.2
90.3
85.6
81.3
74.7
88.8
82.
83.2
81.4
89.3
77.9
78.6
77.3
93.7
1-8
99.8
91.9
84.6
80.8
68.6
90.2
85.3
81.3
88.8
82.2
83.1
81.3
89.3
77.8
78.7
77.4
93.8
1-9
99.8
92.0
84.2
9
68.7
90.2
85.2
91.3
74.7
88.8
82
83.2
81.4
89.3
78.4
77.1
1-10
99.8
92.1
84.1
80.5
68.3
90.3
85.0
81.6
75.0
88.8
82.2
83.2
81.3
89.3
77.9
78.6
77.3
93.7
1
2
4
5
9
10
11
12
13
14
15
16
17
18
Layers ResNet18
Figure 4: Layer-wise percentage overlap of the retained parameters between first and corresponding generations.
To quantify the evolution of the mask, we measure the percentage of overlap of parameters retained between the first and the corresponding generations. We observe a gradual decrease in overlap from the initial generation to subsequent generations, indicating the emergence of masks in evolutionary training scenario. This progressive mask evolution contributes to the network's enhanced capacity for learning and generalization, evident from the test accuracy.
In conclusion, our results highlight the evolutionary nature of the mask throughout generations in the DNR framework. The dynamic adaptation and refinement of the mask lead to effective masking and improved performance and generalization of the DNN. These findings support the effectiveness of our approach in leveraging the evolutionary training paradigm to enhance the learning and generalization capabilities of deep neural networks compared to KE.
A.3 Robustness to Natural Corruptions
In practical applications, deep neural networks often operate in dynamic environments characterized by variations such as lighting and weather conditions. Consequently, is crucial to assess the robust- ness of DNNs to data distributions that undergo natural corruption. We investigate the robustness
13

---

Blur
Weather
Digital
LB
22.9
21.8
20.5 18.8 18.9
21
28.5
24.5
20.1
27.7
18.1
21.3
26.2
21
32.9
KE
16.0
19.9 19.8
19.1 19.2
12.4 12.5
11.8
10.7
10.3
21.3
21.0
15.7
11.8
21.0
10.8
12.1
16.8
15.7
22.5
DNR
27.7 23.5 27.6 31.8 28.3
22.4 21.1 20.2 18.0 17.6
49.6
35.3
29.2
19.5 41.0
16.0
20.9
25.3
31.0 48.4
mCA
Gaussian
Shot
impluse
Speckle
Defocus
Glass
Gaussian
Mation
Zoom
Brightness
Snow
Frost
Fog
Spatter
Contrast
Elastic
JPEG
Pixelate
Saturate
Figure 5: Robustness natural corruptions on CIFARIO-C 37). DNR is more robust against the
majority corruptions compared to the baselines.
of DNNS 15 common types of corruptions using the CIFAR-10-C dataset (37). Our models
trained on clean images of the CUB dataset and evaluated on CIFAR- 1--C (37). To quantify the
performance under natural corruption, use the Mean Corruption Accuracy (mCA) metric.
N
N
mCA
Ac,a
No X N
where N₂ and N, represent the number of corruptions (in this case, 19) and the number of severity levels (in this case, 5), respectively. Figure F iillustrate the average accuracy of the models across 19 different corrpttions at five severity levels. In particular, our proposed method (DNR) achieves a higher mCE (27.7%) compared to the longer bascline (23 (3) and KE (16%), demonstrating its effectiveness in improving the robustness to various types of corruption. These findings highlight the benefits of selectively reinitializing network parameters using data-aware masking approach, resulting in enhanced robustness natural corruptions.
A.4 Computational Cost
In our experiments, we consistently maintained a fixed training duration of 200 epochs for each generation, with the number of generations set 10 to ensure a fair comparison. The computational cost of evolutionary training methods, including KE, LLF and DNR, scales linearly with the number of generations (T). For instance, if KE is trained for generations, the total computational cost becomes 5T times that of training a single generation. To ensure fair comparison, we trained a long baseline for the same number of epochs. The additional computational cost incurred by DNR for computing data-aware dynamic masking with SNIP is minimal. For example, on the CUB dataset with 20% subset, this process takes approximately 20.3 seconds per generation. This calculation is performed once at the end of each generation and can be further optimized by using just 128 samples estimate the importance without compromising performance. Appendix Table demonstrates that DNR's performance is minimally sensitive to changes in the subset size.
A.5 Robustness to Adversarial Attacks
DNNs are vulnerable adversarial attacks, where imperceptible perturbations are added the input during inference to deceive the network and induce false predictions (38). Therefore, we investigate the robustness of DNNs trained against adversarial attacks using the PGD-10 attack (39) on models trained on the CIFARIO dataset. We vary the intensity of the PGD attack and evaluate the models' performance. shown Figure our proposed framework (DNR) exhibits greater robustness to adversarial attacks at different attack strengths compared to KE and the long baseline. This highlights the efficacy of our framework in training models that can learn high-level abstractions robust small perturbations in the input data.
14

---

Adversarial perlurbation
Robustness to class imbalance
85
70
Class balanosd Accuracy
Adversanal Accuracy
60
83
60
82
80
20
79
10
78
0
6
LB
KE
DNR
Attack Strength (Epsilon)
Figure 7: Robustness to class imbalance
Figure 6: Robustness to adversarial attacks
A.6 Robustness to Class Imbalance Dataset
In real-world applications, class imbalance is common characteristic of the input distribution, where certain classes are more prevalent than others.
This inherent class imbalance can affect the training of DNNs, they tend be biased towards the majority classes, thereby neglecting the minority classes (40). To address this issue, we explore the contribution of reimitialization
model training with class imbalance. We incorporate class imbalance using the power-law model on CIFAR10. The number of training samples for class determined by n. a/(b + c)². where y represents the imbalance ratio, and and are offset parameters specifying the largest and smallest class sizes. We set fixed gamma value of 1 in our experiments maintain a power law class distribution. The offset parameters a and b are chosen such that the maximum and minimum class counts are 5000 and 250, respectively. We used balanced accuracy as a metric to measure the robustness of the model under class imbalance scenario. Our findings in Figure 7 demonstrate that the DNR framework consistently outperforms KE and longer baselines in scenarios with class imbalance. This highlights the effectiveness DNR addressing the challenges posed by imbalanced class distributions and underscores its potential for practical applications.
Table 5: Performance evaluation with varying the percentage of reinitialized parameters during training using ResNet18. Test accuracy at the end of 10 generations reported on Aircraft and CUB datasets.
Reinit. Params (%)
Aircraft
CUB
65.34
69.95
10
66.10
70.15
20
66.63
71.37
30
64.13
68.42
40
62.79
66.87
A.7 Effect of Adjusting Reinitialized Parameter Ratios
Table 5 shows the effect of varying the number of reinitialized parameters on the performance and generalization of the model. We train the model in evolutionary settings using the DNR framework by varying different percentages of reinitialized parameters (5%, 10%, 20%, 30%, and 40%). The experiments were carried out with ResNet18. The results show that the reinitialization percentage of parameters has no impact performance, while the reinitialization of more than 30% has less impact on test accuracy. We find that reinitialization 20% of the parameters results in the best performance.
5%
15

---

Table 6: Evaluating the effectiveness of the sparse model.
Method
Full model
Sparse model
KE (fio)
66.51 71.37
66.21 70.08
DNR (f10)
A.8 Evaluating the Effectiveness of the Sparse Model
In this section, assess the effectiveness of the sparse model (containing 20% fewer parameters than the full/dense model) obtained through the selective neurogenesis process during the inference phase. We examine the sparse and dense models' test performance compared the original KE framework. For this, we measure the performance of the ResNetl 8 model trained on CUB200. Table presentss the accuracy results obtained by the sparse model compared the dense model. Surprisingly, despite the considerable reduction in the number of parameters, the sparse model achieves comparable accuracy compared the dense model the DNR framework. Furthermore, DNR demonstrates superior performance in both the full and sparse model scenarios compared to the KE This indicates that the selective neurogenesis process successfully retains the critical connections necessary for accurate predictions while eliminating redundant or less informative connections. Our evaluation demonstrates that the sparse model obtained through the selective neurogenesis process offers several benefits during inference. It maintains high accuracy while achieving improved computational efficiency compared to the KE. These results highlight the practicality and efficacy of leveraging selective neurogenesis create efficient and compact deep learning models that can be readily deployed in real-world scenarios.
A.9 Varying the quantity of data used for Importance estimation
In our experiments, we randomly sampled 20% of the dataset to estimate the importance of the parameters after the end of each generation. Here, we analyze the impact of the number data used to determine the important estimation the final performance. Similar to Lee et al. (23), we used as few as 128 samples to estimate the important parameters using SNIP. Table that DNR not sensitive to the variation in the input data used to estimate the importance as the final performance remains unchanged.
Table Evaluation with varying the quantity of data for importance estimation. Test accuracy the end of 10 generations is shown on Aircraft and CUB datasets.
# samples
Aircraft
CUB
0.2 [D]
66.63
71.37
DNR
128
66.45
71.26
A.10 Experiments on Medical dataset
To explore the performance of deep learning low-data medical settings, we conducted experiments using ResNet-50 on the Papila dataset(41). This dataset, with its limited size and complex features, is representative of the challenges in medical imaging. The Papila dataset is a medical imaging dataset focused on analyzing and identifying ocular structures, particularly the optic disc and retinal layers. We conducted this experiment using ResNet-50 with a learning rate 0.001, batch size of 32, and trained for 3 generations (1 generation 200 epochs). For DNR 20% of the parameter is reinitialized using SNIP. Our preliminary results suggest that Dynamic Network Reconfiguration (DNR) shows promise in the medical domain when compared to baseline training. However, further hyperparameter tuning is necessary to fully optimize performance.
While initial findings are encouraging, we highlight the need for more detailed studies in reinitialization-basee methods within the medical domain on different datasets. Future work should address these challenges to improve generalizability across low-data medical imaging datasets.
16

---

Table 8: Performance of ResNet-50 on Papila Dataset: Comparison of DNR and Baseline in terms of Sensitivity, Specificity, and Accuracy.
Sensitivity
Specificity
Accuracy
Baseline (fa)
56.11 57.03
58.68 57.83
56.90 58.20
DNR (/a)
Table 9: Performance of DNR across generations on the Flower dataset
Generations
Accuracy (%)
10
68.36
20
72.10
30
73.56
A.11 Performance of DNR Across Generations on Flower Dataset
As the number of training generations increases, the performance of DNR begins saturate. This indicates diminishing returns in accuracy improvement with extended training. This saturation effect corroborated by analysis of the mask evolution across generations, illustrated in Figure The overlap percentage of the mask progressively increases, with a higher overlap observed between the 9th and 10th generations compared to the 1st and 2nd generations. This trend suggests that the mask becomes more saturated and stable, aligning with the model's convergence to a lower-loss landscape.
A.12 Summary of Datasets and Implementation Details
Taha et al. (5) employs various image resizing techniques for different datasets; however, they do not provide specific details about the resizing parameters their paper. To ensure consistency across our experiments, we resize all datasets to a fixed size of (256, 256). Moreover, to tine-tune the hyperparameters, we utilize a validation split, and the reported results are based on the test set whenever it is available.
Table 10: Details of the five used classification datasets.
Datasets
Classes
Train
Validation
Test
Total
CUB-200 26
200
5994
N/A
5794
11788
Flower- 102 25
102
1020
1020
6149
8189
MIT67 27
67
5360
N/A
1340
6700
Aircraft 29
100
3334
3333
3333
10000
Standford- Dogs (28
120
12000
N/A
8580
20580
For experiments on large datasets, we used the following settings. The experiments were conducted on three different datasets: CIFAR-10/100, Tiny-ImageNet For CIFAR- 100100, the training was performed for 160 epochs. batch size of 64 was used, along with a step-based learning rate scheduler. The learning rate decay was applied between epochs 80 and 120, with decay factor of 10. The momentum was set to 0,9, and regularization was applied with coefficient of 5c-4. The initial learning rate used was 0.1. There were warmup epochs in this case.
For the Kiny-ImageNer dataset, the training was also conducted for 160 epochs. The batch size was reduced to 32, and step-based learning rate scheduler was used. Similar to 1LFAR-10/100 the learning rate decay occurred between epochs 80 and 120, with decay factor of 10. The momentum and 12 regularization were set to 0.9 and 5e-4, respectively. Additionally, 20 warmup epochs were applied. Throughout experiments, resetting ratio of 20% used for all generations. All the training and evaluation are done the NVIDIA RTX-2080 Ti GPU. The time required to approximately train generations of DNR on CUB200 with ResNet18 is approximately 1.68 hours. It is worth mentioning that to compare method with other baselines, utilized the results
17

---

presented in the KE paper (5) as a point of reference. For the hyperparameters used in training small datasets, please refer to Section
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and/or introduction should clearly state the claims made, including the contributions made the paper
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
is mentioned as part of conclusion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
This is an experimental study.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the experimental details to reproduce the results are mentioned in the Experimental setup section.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: Public datasets are used for this paper. The code will be publically available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
mublite/giiaes//odeeabnnisss//s iooooloo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//uuui guwdes/ddee/eee Coc//ooooeeesiseeeeeee hwtpibbmsssiioooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have mentioned all the training details and evaluation settings in the Experimental setup section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experimental results are done for 3 runs across 8 different set of datasets to ensure statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The type of compute workers, memory, time of execution is mentioned in appendix section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /ttp::::///eurip... hwwpar/pp.../pplllcc/ccccccollla
Answer: [Yes]
Justification: Yes, code of ethics follwed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The potential impact of the method is discussed in the broder impact section in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not any which we forsee
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No experiments using human subjects is performed.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "n2Fklwr4dP", "reviewer": "Reviewer_DwbC", "summary": "This manuscript introduces the Dynamic Neural Regeneration (DNR) framework, which improves the generalization of deep neural networks on small datasets. DNR uses the SNIP method to reinitialize less important neural connections for the current generation selectively. Experimental results show that DNR outperforms the existing original Knowledge Evolution (KE) method and long baseline (LB) in 5 small datasets and 3 large datasets.", "questions": "### 1. The performance of DNR does not surpass the STOA.\n\n1.a Despite the superior performance of DNR compared to KE and LB, it does not surpass the existing SOTA, such as Smth+LLF. For the reported results in this manuscript, Smth+LLF either showed better or marginally lower performance compared to Smth+DNR. Additionally, the results reported in the original LLF study [1] were higher than the author-reported Smth+LLF as well as the proposed Smth+DNR for all 5 small datasets (Table 1 in [1] vs Table 1 in the manuscript) and on the TinyImageNet dataset (Table A7 in [1] vs Table 2 in the manuscript). The authors did not address this discrepancy, making it hard to interpret the reported results.\n\n1.b Following the comment above, statistical analysis should be provided to better demonstrate the difference between different methods.\n\n1.c DNR requires calculating SNIP to determine the connections needed to be reinitialized. However, the time complexity related to data size and model size is not provided. Therefore, it is still questionable whether DNR is practically useful, especially considering that its performance may not surpass the SOTA (comment 1.a).\n\n### 2. The experiments lack details, making it hard to interpret the results.\n\n2.a The experimental setting seems arbitrary. For example, why do some methods, such as KE and Smth+KE, use 10 generations while Smth+LLF and others use 8 generations? It seems the authors followed the experimental setting of prior work, such as Zhou et al. [1], but it was not mentioned in the manuscript.\n\n2.b The authors reported mean and std for each experiment, but the number of runs for each experiment is not mentioned.\n\n2.c It is very confusing what kind of \"transfer learning\" is used in Section 5.4. What is the difference between \"transfer learning\" and \"long baseline\"?\n\n2.d Many reported results in Table 1 are identical to the results in [1] (Table 1), such as Smth+KE with CUB, Aircraft, and MIT datasets, while others are not. Please clarify whether the results are reproductions or simply referrals from [1]. If they are reproduced results, please explain the discrepancy between the reported and reproduced results.\n\n### 3. Minor comments\n\n3.a Both LLF [1] and KE [2] demonstrated that they performed the best with CS-KD [3]. It would be interesting to see how DNR performs with the same setting.\n\n3.b The letter \"g\" was used to denote the generation number (line 107) and connection sensitivity (eq. (4)). Please consider using different letters.\n\n3.c What \"m\" stands for in eq. (5) and (6).\n\n\n\n###Reference\n\n[1] Zhou, Hattie, et al. \"Fortuitous forgetting in connectionist networks.\" International Conference on Learning Representations. 2021.\n\n[2] Taha, Ahmed, Abhinav Shrivastava, and Larry S. Davis. \"Knowledge evolution in neural networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[3] Yun, Sukmin, et al. \"Regularizing class-wise predictions via self-knowledge distillation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.", "limitations": "None.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed Dynamic Neural Regeneration (DNR) is interesting, intuitive, and well explained.\n- The DNR demonstrated superior performance compared to knowledge evolution (KE) and long baseline (LB).\n- The DNR demonstrated better robustness to natural corruptions, adversarial attacks, and class imbalance compared to KE and LB.", "weaknesses": "- The performance of DNR does not surpass the STOA.\n- The experiments lack details, making it hard to interpret the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Empirical Validation\n\nThank you for highlighting the concern. We have performed the empirical validation of our method by including results on five small datasets and three large datasets—CIFAR-10, CIFAR-100, and Tiny ImageNet. **Each experiment is conducted three times, and the mean and standard deviation are reported.** Notably, in the majority of these datasets, DNR consistently outperforms LLF, Knowledge Evolution (KE), and the longer baseline. These results affirm that DNR brings discernible benefits in terms of improving generalization.\n\n> Dataset Choice and Result Reporting\n\nThe choice of datasets aligns with the baselines established in the original paper, ensuring a fair comparison and leveraging the availability of their results. The results for DSD, and BAN in Table 1 were referenced from the KE paper, while our reproduced results included label smoothing for KE, LLF, and LW. The results reported are reproduced using the best possible hyperparameters mentioned in the original paper. While some discrepancies may arise due to slight variations in implementation or experimental conditions, we have ensured that our setup closely follows the original methodology to the best of our ability. \n\n> Computational Cost Comparison\n\nIn our experiments, we consistently maintained a fixed training duration of 200 epochs for each generation, with the number of generations set at 10 to ensure a fair comparison. The computational cost of evolutionary training methods, including KE, LLF and DNR, scales linearly with the number of generations (T). For instance, if KE is trained for 5 generations, the total computational cost becomes 5T times that of training a single generation. To ensure a fair comparison, we trained a long baseline for the same number of epochs. The additional computational cost incurred by DNR for computing data-aware dynamic masking with SNIP is minimal. For example, on the CUB dataset with a 20% subset, this process takes approximately 20.3 seconds per generation. This calculation is performed once at the end of each generation and can be further optimized by using just 128 samples to estimate the importance without compromising performance. Appendix Table 8 demonstrates that DNR’s performance is minimally sensitive to changes in the subset size.\n\n> Advantage of DNR vs LLF\n\nWhile DNR shows superior performance compared to the SOTA on most datasets, it’s important to highlight the limitations of methods like LLF and LW, which rely on specific architectural assumptions. LLF assumes later layers focus on memorization, but studies like \"Can Neural Network Memorization Be Localized?\" indicate:\n\nMemorization occurs throughout the network, not just in later layers, making fixed layer-by-layer reinitialization suboptimal. Crucial parameters can reside at different network depths depending on the task and dataset. DNR's data-aware dynamic masking offers significant advantages by analyzing connection sensitivity and identifying influential parameters regardless of their location. This approach aligns better with findings that critical parameters can be distributed throughout the network.\n\nDNR’s flexibility allows it to adapt to various datasets and tasks without rigid architectural assumptions, making it suitable for real-world scenarios with varied datasets, despite a slight increase in computational cost.\n\n> 2a  Rationale behind the choice of generation\n\nWe based our experimental setup on the configuration used in Zhou et al. [1], which aligns with standard practices in the field. Specifically, KE and Smth+KE utilized 10 generations to ensure a thorough exploration of the parameter space and to conform to the established practices for iterative reinitialization. For other methods such as LW (Layer-wise Forgetting), the choice of generation is limited by the architecture-specific assumptions since it operates on a layer-wise basis. For instance, LW proposes a layer-wise reinitialization scheme, which proceeds from bottom to top, reinitializing one fewer layer each generation. Consequently, the number of generations is limited to 8 to match the number of layers. This setup facilitates a direct comparison with methods that have similar structural constraints.\n\n> 2c Clarification regarding the Transfer learning experiment in section 5.4:\n\nWe present an instance of transfer learning where the weights at the end of each generation are transferred to the next generation without reinitialization, a process we refer to as vanilla fine-tuning. In contrast, the long baseline method involves training the model for a prolonged, uninterrupted period, typically 2000 epochs, as a single continuous generation. This method does not involve any intermediate weight transfer or reinitialization steps, and the model continuously learns from the data throughout the entire training period.\n\nWe will update the manuscript to clearly explain the rationale behind the choice of generation numbers for each method and ensure that all this information provided in the rebuttal is included in the revised experimental section.\n\n> 3a Results with CS-KD:\n\n| Method | Flower (Acc)(%)| CUB200 (Acc) (%) |\n|---|---|---|\n| CS-KD | 68.68 ± 0.28| 69.59 ± 0.40|\n| KE | 67.29 ± 0.74| 69.54 ± 0.60|\n| LLF| 74.68 ± 0.19| 73.51 ± 0.35|\n| DNR| **75.23 ± 0.21**| **74.18 ± 0.16** |\n\n**The table above presents the accuracy results of different methods combined with Class-wise Knowledge Distillation (CS-KD) on the Flower and CUB200 datasets over 10 generations.** The results demonstrate that DNR, when combined with CS-KD, outperforms both KE and LLF methods in terms of accuracy on both datasets. These results are sourced from the LLF paper, providing a robust comparison and validation of our approach.\n\n> 3b, 3c\n\nThank you for pointing this out. We will use different letters to represent generation number and connection sensitivity. Regarding m in Equations (5) and (6), it represents the total number of parameters in the neural network.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v7Oujl9Qfh", "reviewer": "Reviewer_cy5M", "summary": "This submission investigates efficient training and generalization of deep neural networks in the low-data regime. Drawing inspiration from neurogenesis in the brain, authors propose an iterative training framework termed Dynamic Neural Regeneration (DNR). The authors further investigate the efficacy of the proposed approach through experiments on five datasets (Flower102, CUB-200-2011, MIT64, Stanford Dogs and FGVC-Aircraft).", "questions": "1. Please refer to Weakness #2.\n2. For Table 1, would it be helpful to include the notations “CE” in methods that are based on cross-entropy loss, such as “CE + DNR$(f_{10})$”? Or did I misunderstand?\n3. For Table 1-3, would it be helpful to include an additional column that reports the mean over all datasets, or ranking across all datasets?\n4. I have not seen discussions on Figure 2. Could you point me to the relevant lines in case I missed it?", "limitations": "Yes, the authors adequately addressed the limitations and, if applicable, potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method adapting Knowledge Evolution by incorporating data-aware dynamic masking instead of randomly pre-selected masks is intuitive. Furthermore, the authors draw an analogy between the proposed method and the phenomenon of neurogenesis in the brain which adds more intuition to the proposal. \n2. Empirical results comparing against other evolutionary or iterative training methods look quite promising. The proposed method shows improvements by a decent margin in various experiments.\n3. It is thoughtful to include the comparison with transfer learning, and the results look very good too. It would be better to include error bars, though I believe the improvement is significant enough.", "weaknesses": "1. Since the authors repeatedly claim that data shortage is often seen in medical diagnosis (in Abstract line 2-3, Introduction line 23-25, and Results line 285-286) it would be great if they also include experiments in such scenarios.\n2. I find the schematics (Figure 1) confusing instead of enlightening. What I read from it is that, (1) We have a dataset and a neural network. (2) We train the neural network on the dataset. (3) We apply data-aware masking where some neurons (minority, shown in blue) are kept the same and the others (majority, shown in green) are somehow changed --- but it is unclear how they are changed. (4) Then, we remove the blue neurons (???) --- so that nothing is kept from the current iteration? (5) We randomly initialize the removed neurons --- if so, why do we remove them from the first place? Why don’t we skip neuron deletion and directly perform neuron initialization? I am not sure how other people would process this figure, but at least to me it needs some re-editing for it to be a helpful illustration. Another minor tip: if the weights of certain neurons are kept unchanged between two steps, I would recommend either using the same color to color code them, or using a small symbol to indicate weight freezing (such as a lock or a snowflake).\n3. This submission is among the works where the empirical results are critical, and therefore I would be concerned that the authors have not submitted their code as the supplementary material, despite stating that they would release the code upon acceptance. At least on my portal no supplementary material is uploaded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  Experiments on Medical Datasets:\n\nWe have chosen to focus on widely-used benchmark datasets that are representative of various low-data regimes. We believe these benchmarks provide a robust and fair comparison of our method’s performance. While we understand the importance of validating our approach in the medical domain, conducting experiments on additional datasets such as Harvard-GF3300 would require re-running all baseline methods, which is beyond our current resource capacity. We recognize the value of including medical datasets and plan to incorporate these experiments in a revised version of the paper to demonstrate the versatility of our approach across different domains.\n\n> Clarification of Figure 1:\n\nWe apologize for any confusion caused by Figure 1. To improve clarity, we will revise the figure to provide a more detailed, step-by-step explanation:\n\n- **Initialization**: We start with a dataset and a randomly initialized neural network (shown in black).\n- **Training**: The neural network is trained on the dataset, and the parameters after training are shown in blue.\n- **Data-Aware Masking**: We apply SNIP to select important neurons, which are highlighted in green.\n- **Neuron Modification**: Least important neurons (blue) are randomly reinitialized, while important ones (green) are retained from the current generation.\n- **Iterative Training**: The network is retrained on the dataset for the next generation.\n\n> Submission of Code \n\nPlease let us know if there is a way to submit the code during the review process, and we will provide it promptly. Otherwise, as stated in the paper, we assure you that the code will be made available upon acceptance. Thank you for your understanding.\n\n> Notations in Table 1:\n\nThank you for the suggestion. We will update Table 1 to include notations such as “CE” for methods based on cross-entropy loss. This notations should make it clearer.\n\n> Adding mean as metric in Tables 1-3:\n\nWe appreciate your insightful suggestion. Although including a column that reports the mean performance across all datasets could offer a convenient summary, we initially chose not to include it due to the significant variability in dataset complexity, size, and other characteristics, which might make such an average potentially misleading. Nevertheless, we recognize the value of this addition and will incorporate it in the revised version. To ensure clarity and prevent misinterpretation, we will also include a discussion on the limitations of interpreting this average.\n\n> Discussion on Figure 2:\n\nFigure 2 illustrates the layer-wise percentage overlap of retained parameters between consecutive generations, demonstrating how DNR dynamically adapts its mask during training. For a detailed discussion, please refer to Section 5.3 of the manuscript. This section explains how DNR uses SNIP to selectively reinitialize parameters, thereby enhancing the model’s generalization performance. It maintains stability in earlier layers while adapting to task-specific features in later layers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vPIrvdFCuO", "reviewer": "Reviewer_ZzXp", "summary": "This paper presents a novel iterative training framework called Dynamic Neural Regeneration (DNR) designed to enhance the generalization of deep learning models on small datasets. The DNR approach utilizes a data-aware dynamic masking scheme inspired by neurogenesis to eliminate redundant connections, thereby increasing the model's capacity for further learning. Through extensive experiments, the authors demonstrate that DNR outperforms existing methods in both accuracy and robustness, making it a promising technique for applications with limited data availability.", "questions": "How does the computational cost of DNR compare to other state-of-the-art methods? Is there a significant increase in training time due to the dynamic masking process?\nCan the authors provide more detailed guidelines on how to implement the DNR framework in practice, including hyperparameter settings and potential pitfalls to avoid?", "limitations": "Could the authors provide a careful and detailed discussion of the limitations of the work?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The technical claims of the paper are well-supported by thorough experimental results. The methodology is clearly described, and the use of data-aware dynamic masking is both innovative and effective. The experiments are comprehensive, covering multiple datasets and including robustness tests against common challenges such as class imbalance and adversarial attacks. Overall, the research methodology is sound, and the central claims are convincingly supported by the evidence provided.", "weaknesses": "***Complexity of Implementation***: The DNR framework may be complex to implement for practitioners without a strong background in iterative training paradigms and dynamic masking techniques. However, including more detailed implementation guidelines or open-sourcing the code could mitigate this issue.\n***Scalability***: While the approach is effective for small datasets, its scalability to very large datasets or more complex models is not fully explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Complexity of Implementation:\n\nWe appreciate the reviewer's concern regarding the complexity of implementing the DNR framework. To address this, we have expanded the Appendix section in the revised manuscript that includes detailed implementation guidelines. This section provides step-by-step instructions, hyperparameter settings, and potential pitfalls to avoid. Additionally, we plan to open-source our code upon acceptance, which will make it more accessible to practitioners and researchers.\n\n> Scalability:\n\nThe primary focus of our paper is to enhance generalization in small datasets, where the DNR framework demonstrates substantial effectiveness. In Section 5.2, we have conducted experiments on larger datasets such as CIFAR-10, CIFAR-100, and TinyImageNet. The results show that DNR, KE, and LB methods offer limited advantages over vanilla training on large datasets due to factors like dataset complexity and model capacity, which can lead to performance saturation. The core strength of DNR is its ability to improve generalization in scenarios with inherently limited data. We believe that exploring scalability in real-world situations where data comes in the form of stream will be an intriguing avenue for future research. \n\n> Computational Cost Comparison:\n\nIn our experiments, we consistently maintained a fixed training duration of 200 epochs for each generation, with the number of generations set at 10 to ensure a fair comparison. The computational cost of evolutionary training methods, including KE, LLF and DNR, scales linearly with the number of generations (T). For instance, if KE is trained for 5 generations, the total computational cost becomes 5T times that of training a single generation. To ensure a fair comparison, we trained a long baseline for the same number of epochs.\n\nThe additional computational cost incurred by DNR for computing data-aware dynamic masking with SNIP is minimal. **For example, on the CUB dataset with a 20% subset, this process takes approximately 20.3 seconds per generation.** This calculation is performed once at the end of each generation and can be further optimized by using just 128 samples to estimate the importance without compromising performance. Appendix Table 8 demonstrates that DNR’s performance is minimally sensitive to changes in the subset size.\n\nWe will add a dedicated section in the revised manuscript to discuss the computational cost of DNR in more detail. This section will provide a comprehensive analysis of the computational demands of DNR compared to long baselines and KE, highlighting the efficiency of DNR relative to the substantial improvements in generalization performance it offers.\n\n> Limitations: \n\nDue to page limitations, a detailed discussion of the limitations has been added to the conclusion section. Additionally, we will include a more comprehensive analysis in the appendix to ensure thorough coverage of the potential limitations of our approach. Thank you for your suggestion, and we hope this addresses your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eSvbOquBYu", "reviewer": "Reviewer_Vrf1", "summary": "This paper proposes Dynamic Neural Regeneration (DNR), a framework to enhance the generalization of deep neural networks on small datasets. The method is inspired by neurogenesis and offers more flexibility in defining a parameter mask as compared to previous approaches such as Knowledge Evolution (KE). The results show strong performance on small datasets with sufficient ablation studies.", "questions": "1. The DNR method evaluates connection sensitivity from a subset of data. However, different subsets can capture different distributions of the data and hence, lead to different connection sensitivity possibly affecting the final performance of DNR. Can the authors provide a justification or some results on this?\n2. How is the mask defined for the DNR framework? Is it over each model parameter (in this case, the mask length would be of a similar magnitude as the model parameter count) or each ResNet layer? Details on this should be included in the paper.\n3. Hypothetically, if the DNR framework is allowed to run for a sufficiently long number of generations (g=100, for example), we should be able to observe a saturation in the mask at a point (say g=55) beyond which the mask does not change at all. It would be beneficial to include a similar experiment.", "limitations": "Please see the Weaknesses section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The motivation of the paper is clear and sound. The authors have provided sufficient justifications about how their method differentiates from NAS and dynamic sparse training.\n3. The results show strong performance of the proposed method against several baselines \n4. The ablation studies are sound, especially the one on studying the effect of importance estimation methods.", "weaknesses": "1. Even though the paper begins by emphasising limited data availability in the medical domain, no experiments on medical datasets have been presented. It would be beneficial to include results on a few medical datasets such as Papila [1], Harvard-GF3300 [2] that represent working in a low-data regime.\n2. No analysis of computational cost especially for larger datasets has been presented. \n3. The transfer learning experiment performed in section 5.4 is not clear. What does generation mean in vanilla fine-tuning? More details are required here.\n4. In theory, DNR could be well incorporated with transfer learning i.e. instead of starting from scratch, use the weights of a pre-trained model and follow the iterative approach of selecting and retaining parameters after each generation. Is there a specific reason why transfer learning experiments were not included?\nFurthermore, even though there is a large domain gap between natural and medical images, transfer learning (full vanilla fine-tuning or parameter-efficient fine-tuning [3]) still remains the de facto practice in medical image analysis.\n5. The results on large datasets (Table 2) show that DNR (and other baselines) are not very effective in the high-data regime. In fact, standard training seems sufficient when the dataset size is large enough.\n\nReferences\n\n1. Kovalyk, Oleksandr, et al. \"PAPILA: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment.\" Scientific Data 9.1 (2022): 291.\n2. Luo, Yan, et al. \"Harvard glaucoma fairness: a retinal nerve disease dataset for fairness learning and fair identity normalization.\" IEEE Transactions on Medical Imaging (2024).\n3. Dutt, Raman, et al. \"Parameter-efficient fine-tuning for medical image analysis: The missed opportunity.\" arXiv preprint arXiv:2305.08252 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Experiments on Medical Datasets:\n\nWe have chosen to focus on widely-used benchmark datasets that are representative of various low-data regimes. We believe these benchmarks provide a robust and fair comparison of our method’s performance. While we understand the importance of validating our approach in the medical domain, conducting experiments on additional datasets such as Harvard-GF3300 would require re-running all baseline methods, which is beyond our current resource capacity. We recognize the value of including medical datasets and plan to incorporate these experiments in a revised version of the paper to demonstrate the versatility of our approach across different domains.\n\n> Computational Cost Comparison:\n\nIn our experiments, we consistently maintained a fixed training duration of 200 epochs for each generation, with the number of generations set at 10 to ensure a fair comparison. The computational cost of evolutionary training methods, including KE, LLF and DNR, scales linearly with the number of generations (T). For instance, if KE is trained for 5 generations, the total computational cost becomes 5T times that of training a single generation. To ensure a fair comparison, we trained a long baseline for the same number of epochs.\nThe additional computational cost incurred by DNR for computing data-aware dynamic masking with SNIP is minimal. For example, on the CUB dataset with a 20% subset, this process takes approximately 20.3 seconds per generation. This calculation is performed once at the end of each generation and can be further optimized by using just 128 samples to estimate the importance without compromising performance. Appendix Table 8 demonstrates that DNR’s performance is minimally sensitive to changes in the subset size.\nWe will add a dedicated section in the revised manuscript to discuss the computational cost of DNR in more detail. This section will provide a comprehensive analysis of the computational demands of DNR compared to long baselines and KE, highlighting the efficiency of DNR relative to the substantial improvements in generalization performance it offers.\n\n> Clarification regarding the Transfer learning experiment in section 5.4:\n\n In our study, we present an instance of transfer learning where the weights at the end of each generation are transferred to the next generation without reinitialization, a process we refer to as vanilla fine-tuning. In contrast, the long baseline method involves training the model for a prolonged, uninterrupted period, typically 2000 epochs, as a single continuous generation. This method does not involve any intermediate weight transfer or reinitialization steps, and the model continuously learns from the data throughout the entire training period.\n\n\n> Integration with Transfer Learning\n\nThank you for your insightful comments. Here’s our rationale for not including transfer learning experiments in this study:\n- *Domain Shift Challenges:* Transfer learning often struggles with domain shifts when the source and target domains differ significantly, leading to suboptimal performance. DNR addresses these challenges directly within the context of small datasets.\n- *Domain-Specific Applications:* In fields like finance, obtaining sufficient labeled data is difficult, and transfer learning can introduce biases from source datasets. Additionally, privacy concerns and the uniqueness of each application domain make it challenging to find suitable pre-trained models. DNR, with its data-aware dynamic reinitialization, better utilizes limited data without relying on potentially unsuitable pre-trained models.\n\n> Effectiveness in High-Data Regimes:\n\nThe primary focus of our paper is to enhance generalization in small datasets, where the DNR framework demonstrates substantial effectiveness. In Section 5.2, we have conducted experiments on larger datasets such as CIFAR-10, CIFAR-100, and TinyImageNet. The results show that DNR, KE, and LB methods offer limited advantages over vanilla training on large datasets due to factors like dataset complexity and model capacity, which can lead to performance saturation. The core strength of DNR is its ability to improve generalization in scenarios with inherently limited data.\n\n> Connection Sensitivity from Subsets of Data:\n\nWe acknowledge the reviewer's concern regarding potential variability in connection sensitivity due to different subsets. In our experiments, we sampled 20% of the dataset to estimate parameter importance after each generation. To assess the impact of varying sample sizes, we conducted additional experiments using as few as 128 samples. As detailed in Section A.8 and Table 7 of the appendix, our results show that DNR's performance remains consistent across different sample sizes, demonstrating the robustness of our method.\n\n> Definition of the Mask in the DNR Framework:\n\nThe mask in the DNR framework is defined over each model parameter, making its length comparable to the model parameter count. So when we say 20% parameters is removed it is removed globally.\n\n> Saturation of the Mask Over Generations:\nWe conducted an experiment where DNR was run for 30 generations.\n\n#### Performance of DNR Across Generations on Flower Dataset \n\n| Generations | Accuracy (%) | \n|-------------|--------------| \n| **10**      | 68.36 | \n| **20**      | 72.10 | \n| **30**      | 73.56 |\n\n As the number of training generations increases, the performance of DNR begins to saturate. This indicates diminishing returns in accuracy improvement with extended training. This saturation effect is corroborated by our analysis of the mask evolution across generations, as illustrated in Figure 2. The overlap percentage of the mask progressively increases, with a higher overlap observed between the 9th and 10th generations compared to the 1st and 2nd generations. This trend suggests that the mask becomes more saturated and stable, aligning with the model’s convergence to a lower-loss landscape.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers raised several concerns in their reviews largely focusing on computation cost comparisons, clarifications on transfer learning, and other bits and pieces. The rebuttal addressed most of these concerns, and additional points were clarified through author-reviewer discussions.

Most importantly though the authors have presented a robust method based on iterative/evolutionary training paradigm that performs well on small datasets, improving generalisation through incorporating generalisable features during further learning. The experiments and additional ablations shown in the appendix provide extra confidence in this method's advantages in the settings presented in the paper. 

I would like to ask the authors to incorporate the additional results presented below and all the other elements they promised to do in the camera ready version, including this "Additional Experiments on Medical Domain Datasets: While we couldn't include medical datasets due to resource constraints, we committed to performing these experiments in a revised version of the paper.".

Congratulations on the acceptance of your work.
Decision: Accept (poster)