Dynamic Neural Regeneration: Enhancing Deep Learning Generalization on Small Datasets

Authors: vijaya  raghavan
t  ramkumar
elahe  arani
bahram  zonooz
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:14:41

Abstract: The efficacy of deep learning techniques is contingent upon access to large volumes of data (labeled or unlabeled). However, in practical domains such as medical applications, data availability is often limited. This presents a significant challenge: How can we effectively train deep neural networks on relatively small datasets while improving generalization? Recent works have explored evolutionary or iterative training paradigms, which reinitialize a subset of parameters to enhance generalization performance for small datasets. However, these methods typically rely on randomly selected parameter subsets and maintain fixed masks throughout training, potentially leading to suboptimal outcomes. Inspired by neurogenesis in the brain, we propose a novel iterative training framework, Dynamic Neural Regeneration (DNR), that employs a data-aware dynamic masking scheme to eliminate redundant connections by estimating their significance. This approach increases the model's capacity for further learning through random weight reinitialization. Experimental results demonstrate that our approach outperforms existing methods in accuracy and robustness, highlighting its potential for real-world applications where data collection is challenging.

Main paper: 
 introduction: 
 Deep neural networks (DNNs) have become essential for solving complex problems in various fields, such as image and speech recognition, natural language processing, and robotics (1) . With the increasing availability of data, DNNs have achieved unprecedented performance, surpassing human-level performance in some applications (2) . However, the success of DNNs is limited when dealing with small datasets, where the model tends to overfit and fails to generalize to new data. For example, it is often difficult to obtain a large amount of data in medical diagnosis due to the complexity and high cost of the procedures involved. In such cases, lack of generalization can be dangerous, which can lead to incorrect diagnosis and treatment.Recently, several studies based on weight reinitialization methods (3; 4) have been proposed in the literature to improve generalization by iteratively refining the learned solution through partial weight reinitialization. These methods select and retain a subset of parameters while randomly reinitializing the rest of the network during iterative/evolutionary training schemes. For example, a state-of-the-art method named Knowledge Evolution (KE) (5) improves generalization by randomly splitting the network into fit and reset subnetworks and constantly reinitializing the reset subnetwork after each iteration. However, the KE approach is limited by its reliance on a predetermined mask creation, where a random subset of parameters is selected and kept constant throughout the iterative training process. This constraint may impede the model's ability to learn effectively from small datasets, ultimately limiting its generalization capabilities. These limitations raise two important questions:1) Can we leverage an evolutionary training paradigm to evolve or adapt the mask over generations, instead of using a fixed mask, in order to enhance the generalization performance of deep learning models trained on small datasets? 2) Can we utilize the available data and the internal state of the model to dynamically determine the important parameters for each generation, rather than randomly presetting them?In our quest to address these questions, we draw inspiration from the phenomenon of neurogenesis in the brain. Neurogenesis, the process of dynamically generating or eliminating neurons in response to environmental demands, has been found to play a crucial role in learning and memory consolidation (6; 7; 8) . This intricate process enables the brain to adapt to new experiences and stimuli, enhancing generalizability. Recent advances in neuroscience have shed light on the non-random integration of new neurons within the brain (9) . For instance, in rodents, neurogenesis-dependent refinement of synaptic connections has been observed in the hippocampus, where the integration of new neurons leads to the elimination of less active synaptic connections (10; 11) . Selective neurogenesis is critical to improving generalization ability by providing a diverse pool of neurons with distinct properties that can integrate into existing neural networks and contribute to adaptive learning (12) . Although the precise mechanisms that govern selective neurogenesis are not fully understood, these findings suggest that selective neurogenesis in the human brain enhances generalization capabilities through its dynamic and selective nature. Thus, by emulating the characteristics of selective neurogenesis, we unlock its potential to improve generalization in deep neural networks.Therefore, we present a novel iterative training approach called Dynamic Neural Regeneration (DNR), which distinguishes itself from the conventional Knowledge Evolution (KE) method through its mask computation. Unlike a predetermined fixed mask, DNR utilizes a data-aware dynamic masking criterion that evolves and adapts the mask over generations. Through extensive experiments on multiple datasets, we demonstrate that our proposed training paradigm greatly improves the performance and generalization of the models. Furthermore, DNR effectively addresses overfitting on relatively small datasets, alleviating the need for extensive data collection. The main contributions of the paper are as follows.• Dynamic Neural Regeneration (DNR) is an evolutionary training paradigm that incorporates data-aware dynamic masking to selectively transfer knowledge across generations.• Our proposed training paradigm facilitates the learning of generalizable features and increases the overall performance of DNNs across small datasets.• DNR exhibits robustness in solving more common challenges in real-world problems, including class imbalance, natural corruption, and adversarial attacks. 
related work: 
 Iterative training and weight reinitialization for DNNs is a prominent area of research (5; 4; 13; 14) that focuses mainly on improving generalization performance by partially refining the learned solution or fully iterating the learned solution. Dense-Sparse-Dense (DSD) (3) propose a three-phase approach where weights with small magnitudes are pruned after initial training to induce sparsity and retrain the network by reinitializing the pruned weights to zero. Zaidi et al. (15) conducted an extensive investigation into the conditions under which reinitialization proves beneficial. BANs (Born Again Neural Networks) ( 4) is a knowledge-distillation-based method that follows a similar iterative training paradigm. However, the critical difference between our work and BANs is that it employs the classlogits distribution instead of the network weights to transfer knowledge between successive networks. Recently, Zhou et al. (16) (LLF) propose the forget and relearn hypothesis, which aims to harmonize various existing iterative algorithms by framing them through the lens of forgetting. This approach operates on the premise that initial layers capture generalized features, while subsequent layers tend to memorize specific details. Accordingly, they advocate for the repeated reinitialization and retraining of later layers, effectively erasing information related to challenging instances. Similarly, the LW (17) approach progressively reinitializes all layers. However, these weight reinitialization methods rely on architecture-specific assumptions that do not take the data into account. They are based on presumed properties inherent to the model and its learning process. Consequently, these methods lack prior knowledge of which features, layers, or parameters should be reinitialized in a general context. Moreover, research indicates that memorization in neural networks is not confined to final layers but involves neurons distributed throughout the model (18) . Knowledge Evolution (KE) (5) splits model The splitting method can be arbitrary (weight-level splitting (WELS)) or structured (Kernel-level convolutional-aware splitting (KELS)). This approach involves perturbing the reset hypothesis to evolve the knowledge within the fit hypothesis over multiple generations. Our framework (DNR) distinguishes itself from the conventional Knowledge Evolution (KE) method through its mask computation. DNR utilizes data-aware dynamic masking that adapts the mask over generations and transfers selective knowledge.Moreover, we differentiate our study from current literature on neural architecture search (NAS) (19) and growing neural networks (20) . Our emphasis is on a consistent network architecture, maintaining fixed connections and parameter count throughout our analysis. Notably, our work sets itself apart from the dynamic sparse training literature (21; 22) , as our objective is not to achieve sparsity but rather to enhance generalization on small datasets. 
evolutionary training paradigm: 
 We first introduce the evolutionary/iterative training paradigm as envisioned in KE (5) . Evolutionary training paradigms allow neural networks to be trained for many generations, where each generation focuses on optimizing the model to converge towards a local minimum while progressively improving generalization. Each generation within the training process is denoted as g, where g ranges from 1 to the total number of generations, N .We define a deep neural network f with L layers and is characterized by the set of parameters Θ. We assume a dataset D consisting of n input-output pairs, denoted {(x i , y i )} n i=1 . For a classification task, we define the cross entropy loss for training the network as:EQUATIONwhere ŷi = f (x i ) is the network's predicted output for input x i . We initialize the weights and biases of the network randomly.KE starts by introducing a binary mask M , which partitions the weights of the neural network into two hypotheses before starting training: the fit hypothesis H fit and the reset hypothesis H reset . This partitioning is expressed as follows:EQUATIONHere, the element-wise multiplication operator ⊙ is applied to the mask M and the parameter set Θ to obtain the fit hypothesis H fit . Similarly, the reset hypothesis H reset is obtained by element-wise multiplying the complement of the mask (1 -M ) with the parameter set Θ. These parameters are chosen at random before the start of the first generation. This binary mask M is kept constant throughout the evolutionary training; i.e., the parameters belonging to the fit and reset hypotheses remain in that category across all generations.We use the stochastic gradient descent (SGD) algorithm to train the network with a learning rate α.We run SGD for e epochs on the dataset D. The beginning of every new generation is characterized by introducing perturbations applied to the network weights to induce a high loss. This is done by reinitializing the parameters in the reset hypothesis while transferring or retaining the parameters belonging to the fit hypothesis. This dynamic process triggers a subsequent round of optimization, guiding the neural network toward the search for a new minimum in the parameter space. The initialization of the network f for the next generation f g is as follows:EQUATIONwhere Θ g-1 and Θ g are the parameters of the network f belonging to the previous generation and current generation, respectively. Θ Reinit corresponds to the randomly initialized tensor sampled from a uniform distribution. We then train the next generation of the network f g using SGD with the same hyperparameters and epochs as the first generation.Unlike KE, we propose a methodology that offers a distinct advantage regarding binary mask computation and parameter reinitialization. Motivated by the symbiotic link between generalization and selective neurogenesis in biological neural networks (9) , we introduce a Data-aware Dynamic Masking (DDM) that emulates the process of selective neurogenesis in evolutionary training. The benefits of DDM's way of reinitialization are two-fold. 1) It takes advantage of the evolutionary training paradigm and adapts the mask dynamically in each generation rather than using a predetermined mask. This introduces flexibility in the network and improves the generalization performance of deep learning models trained on small datasets. 2) Our masking scheme leverages a model's data and internal state to dynamically determine the important parameters for a given task, rather than relying on random pre-setting, to enhance the performance of deep learning models on small datasets. Our way of masking offers a priori knowledge of where and what parameters and layers should be reinitialized in the general case.The mask M is calculated at the beginning of each generation in a data-dependent manner. We assess the importance or sensitivity of each connection in the network to the specific task by employing the SNIP method (23) . SNIP decouples the connection weight from the loss function to identify relevant connections. We randomly sample a small subset of data (π) from the current dataset to evaluate connection sensitivity. We define a connection sensitivity mask M ∈ {0, 1} |Θ| , where |Θ| denotes the number of parameters in the network. The mask is designed to maintain a sparsity constraint k, which specifies the percentage of parameters to retain. The computation of connection sensitivity is performed as follows:EQUATIONwhere j corresponds to the parameter index and e j is the mask vector of the index j, where the magnitude of the derivatives is then used to calculate the saliency criteria (s j ):EQUATION)After calculating the saliency values, we apply the sparsity constraint k to the connection sensitivity mask, which ensures that only the top-k task-specific connections are retained. The sparsity constraint k is defined as follows:EQUATIONwhere sk is the k th largest element in the saliency vector s, 1[.] is the indicator function and m represents the total number of parameters in the neural network. Subsequently, using the saliency values obtained from the connection sensitivity analysis, we select and preserve the top-k important connections. The parameters associated with the connections deemed less important for the current generation are then reinitialized. This process effectively induces selective neurogenesis, allowing the network to adapt and free up its capacity for learning more generalized representations in subsequent generations. Finally, the network for subsequent generation is initialized as shown in Equation 3.Intuitively, we incorporated selective neurogenesis as a replacement mechanism, reinitializing the input and output synaptic weights of specific subsets of network parameters dynamically during the evolutionary training process (24) . Due to the challenges associated with where, how, and when to create neurons (20) , we explore data-aware dynamic masking to drive neuron creation and removal, which could improve learning. We first select the crucial parameters based on the computed saliency mask. Ideally, we would like the mask to keep the knowledge learned from the previous generation as much as possible and to have enough learning capacity to accommodate the learning happening in the new generation. The additional learning capacity facilitates the fast adoption of generalized knowledge and reinforces the knowledge retained from the previous generation. In this way, selective neurogenesis is achieved that inherently adapts the network connectivity patterns in a data-dependent way to learn generalized representations without altering overall network size.The network with the new initialization undergoes next-generation training with the same data for the e epochs, where e is kept the same for each generation. The network is trained with the loss function shown in Equation 1. Thus, we favor the preservation of the task-specific connections more precisely than the mask criteria used in KE that can guide the network towards those desirable traits that efficiently improve the performance and generalization of DNNs in small datasets. 
experimental setup: 
 Here, we provide the details on the experimental setup, implementation details, and datasets used in our empirical evaluation.We evaluate the proposed method using five datasets: Flower102 (25), CUB-200-2011 (26), MIT64 (27) , Stanford Dogs (28), FGVC-Aircraft (29) . The summaries of the statistics of the data set are mentioned in Appendix, Table 10 .Implementation Details: Since our framework is a direct extension of the KE, we follow the same experimental setup. The efficacy of our framework is demonstrated in two widely used architectures: ResNet18 and ResNet50 (30) . We randomly initialize the networks and optimize them with stochastic gradient descent (SGD) with momentum 0.9 and weight decay 1e -4. We use the cosine learning rate decay with an initial learning rate lr = {0.1, 0.256} on specific datasets. The networks are trained iteratively for N generations (N =11) with a batch size b=32 for e=200 epochs without early stopping. The standard data augmentation techniques, such as flipping and random cropping, are used. We employ SNIP (23) with network sparsity k to find the critical subset of parameters at the end of each generation. For the importance estimation, we use 20% of the whole dataset as a subset (π). For all our experiments, we reinitialize a fixed 20% parameters of the network globally. All training settings (lr, b, e) are constant throughout generations.Baselines: To evaluate and benchmark the effectiveness of our proposed approach, we conduct a comprehensive evaluation by comparing it against several existing methods that involve iterative retraining and reinitialization. Specifically, we benchmark our method against the following techniques: 1) Dense-Sparse-Dense Networks (DSD) (3); 2) Born Again Networks (BANs) (4); and 3) Knowledge Evolution (KE) with its variant, KELS (5) . We also compare our method against a non-iterative approach known as the Long Baseline (LB), which undergoes training for the same number of epochs as the corresponding iterative methods. Since our framework is built on top of KE, we follow the same procedure in all our experiments unless specified. 
results: 
 In this section, we conduct comprehensive experimental evaluations of our method across multiple datasets and ablation studies. For an in-depth analysis, extended robustness experiments, including robustness to 15 types of natural corruptions at varying severity levels, adversarial attacks, and class imbalance, are provided in Appendix.Table 1 presents the quantitative classification evaluation results using ResNet18. f g denotes the result at the end of g th generation. We compare DNR with two different configurations: (1) using naive cross-entropy loss (CE), and (2) incorporating label smoothing (Smth) regularizer with a hyperparameter α = 0.1 (31) .The Dynamic Neural Regeneration (DNR) framework demonstrates flexibility and consistently improves performance over the considered baselines across datasets. Interestingly, KE underperforms in terms of performance compared to long baseline (LB) with equal computation cost. This discrepancy may be attributed to the use of fixed masking criteria throughout evolutionary training, limiting the model's adaptability. In contrast, DNR outperforms both longer baselines and KE, consistently improving generalization performance across all datasets.Similarly, we compare the performance of our method with the label smoothing regularizer (Smth) applied to the baselines. Table 1 shows that our method consistently outperforms LB and KE on all datasets. DNR showcases slightly better performance compared to LW and LLF, with a key advantage being its independence from architecture-specific assumptions. Unlike LLF and LW, which lack prior knowledge of which features, layers, or parameters to reinitialize, DNR uses dataaware dynamic masking to selectively remove redundant connections. This reduces complexity and improves scalability as the model grows. Moreover, research indicates that memorization in neural networks is not confined to final layers but involves neurons distributed throughout the model ( 18), highlighting the limitations of architecture-specific assumptions and underscoring the effectiveness of our approach. Additionally, in real-world settings where data arrives in batches (32) , DNR can dynamically iterate over each batch, optimizing performance. These results demonstrate the efficacy of the data-aware dynamic masking and selective reinitialization employed by DNR. By adapting task-specific parameters in each generation, DNR achieves superior performance and enhances the model's generalization.Our work is a direct extension of KE (5) , which focuses explicitly on improving generalization in the low data regime. However, we also thoroughly evaluate our method on large datasets such as Tiny-ImageNet (33), CIFAR10, and CIFAR100 (34) using ResNet50 to assess its scalability. comparable or slightly improved performance in this scenario. This suggests that a selective way of reinitializing benefits iterative training and can effectively handle the complexities and challenges associated with larger datasets and architectures.Unlike KE, which employs a randomly predetermined and fixed masking strategy, DNR provides a notable advantage through the utilization of Data-aware Dynamic Masking (DDM) for parameter reinitialization. Therefore, it is crucial to investigate whether DNR fully leverages the benefits of the evolutionary training paradigm by dynamically adapting the mask in each generation.The proposed DNR framework employs SNIP (23) as a masking criterion to selectively regulate the parameters that have the least impact on performance in each generation of training. To examine this, we analyze the CUB200 dataset using the ResNet18 architecture. We save the mask generated by SNIP after the end of every generation. Visualizing the mask generated by the DNR framework can be challenging due to the large number of parameters in each layer of the backbone. To assess the consistency of connections across generations, we adopt a metric based on the percentage of overlap of retained parameters between the masks created in consecutive generations. This metric provides a quantitative analysis of the degree of flexibility induced by DNR in the evolutionary training process.Figure 2 illustrates the layer-wise percentage overlap of retained parameters between consecutive generations in the DNR framework. The results reveal that the earlier layers consistently exhibit a high overlap percentage across all generations, indicating a consistent selection of connections.The overlap percentage decreases in the later layers (specifically, layer 4 in ResNet) as the model learns class-specific information. This observation suggests that the mask adapts to capture taskspecific features while maintaining stability in the earlier layers. Interestingly, we observe that the overlap percentage of the mask progressively increases as the evolutionary training progresses. Specifically, the overlap between the 9th and 10th generations is higher compared to the overlap between the 1st and 2nd generations. This observation suggests that the mask becomes more saturated and stable as the model state converges to a lower-loss landscape. This flexible nature of the DNR framework, allowing for the regulation of connections in both early and later layers, contributes to its effectiveness in improving generalization performance. Our approach using DNR, indeed differs widely from the domain of transfer learning. Unlike transfer learning, which primarily focuses on leveraging pre-trained models trained on large datasets from different domains to boost task performance on downstream tasks, DNR is intricately designed to tackle the intricate challenge of enhancing generalization in the presence of inherently limited or small datasets. A key issue with transfer learning arises when the pre-trained model's source domain vastly differs from the target domain of interest. This discrepancy between domains often leads to domain shifts, where the knowledge transferred from the pre-trained model fails to adapt well to the specificities of the target domain, thereby resulting in suboptimal performance.In particular, in scenarios like medical applications, obtaining sufficient labeled data that closely aligns with the task at hand is exceptionally challenging. Though transfer learning is predominantly used in this field, the need for domain expertise, privacy concerns, and the uniqueness of each application domain make it exceedingly difficult to find a pre-trained model that seamlessly fits. Furthermore, the presence of domain shift between the source and target might lead to compromised performance, affecting the accuracy and generalization of the model on a specific task with limited data. DNR, on the other hand, offers a novel solution to these intricate challenges. By employing dataaware dynamic masking and selective reinitialization, DNR fosters the gradual evolution of the network, enabling it to adapt more effectively to the characteristics of the specific dataset. This process circumvents the problems of domain shifts that often plague transfer learning methods. Thus, while transfer learning remains valuable in contexts with abundant and well-aligned data, DNR stands out as a specialized approach to address the unique hurdles faced in scenarios of limited data availability, where the domainshift problem can severely hinder model performance and generalization.Furthermore, we have included a comparative analysis in Table 3 that involves an instance of transfer learning within the iterative training process, a process we refer to as vanilla fine-tuning. In this particular case, weights are directly transferred from one generation to the next without undergoing reinitialization.This comparison serves to highlight the unique effectiveness of the DNR method. Our results distinctly demonstrate that DNR enhances the process of generalization, showcasing superior performance in comparison to the approach of directly transferring the complete network's weights across generations. This outcome further underscores the distinct advantage of DNR in evolving the network's capacity for better adaptation and learning in the evolutionary training paradigm.In Figure 3 , we present the convergence behavior of our proposed DNR algorithm juxtaposed with vanilla fine-tuning. The x-axis delineates different generations during the training process, while the y-axis represents the performance at the end of each generation. We observe that the convergence of vanilla fine-tuning unfolds at a more gradual pace. In contrast, DNR demonstrates a faster convergence rate. Across generations, DNR consistently surpasses vanilla fine-tuning, delivering a heightened performance level within a shorter training duration. The utilization of data-aware dynamic masking through SNIP in DNR amplifies this efficiency, enabling the model to concentrate on the most pertinent information for effective generalization. We conduct an investigation into the effectiveness of different methods to estimate the importance of parameters within our proposed training paradigm. Specifically, we explore Fisher Importance (FIM), weight magnitude, random selection, and SNIP ( 23) criteria. In Table 4 , we present the performance and generalization results of the model trained with these various selection methods on the CUB200 dataset using the ResNet18 architecture.Our findings demonstrate that the use of SNIP as data-aware dynamic masking yields superior performance compared to all other baseline methods. Surprisingly, the importance criteria based on weight magnitude exhibited inferior performance compared to random selection. However, the lottery ticket hypothesis (35) suggests the existence of sparse subnets within neural networks. Remarkably, when these subnets are trained in isolation, they can achieve a final performance accuracy comparable to that of the entire network in the same or even fewer training epochs. In particular, neurons within these winning subnets demonstrate higher rates of weight changes relative to other neurons. This observation raises the possibility of selectively reinitializing neurons that undergo minimal weight changes during training, as they contribute the least to loss function. Merely relying on the ℓ 1 norm, which fails to capture the rate of weight changes, as described by the lottery ticket hypothesis, may not adequately capture the notion of importance. Therefore, our findings suggest that the utilization of SNIP for data-aware dynamic masking proves to be more effective, as it considers the rate of weight changes in determining the importance of parameters. This approach aligns better with the lottery ticket hypothesis and leads to improved performance and enhanced generalization capabilities in our experimental evaluations. 
conclusion: 
 We presented Dynamic Neural Regeneration (DNR), an iterative/evolutionary training paradigm designed to improve the generalization of deep networks on small datasets. Our framework incorporates selective reinitialization at the end of each generation, employing a data-aware dynamic masking scheme to remove redundant connections based on their importance. This enables the model to increase its capacity for further learning, emphasizing the acquisition of generalizable features. Empirical results demonstrate that the proposed framework substantially enhances performance and generalization across small datasets compared to other reinitializing techniques. Moreover, DNR exhibits improved robustness in challenging real-world scenarios, including adversarial attacks and learning with class imbalances, while enhancing generalization on natural corruption data.As a direction for future research, delving into the possibilities offered by growing networks (36) presents an intriguing path worth exploring. Our primary aim was to demonstrate the practical effectiveness of DNR, laying the foundation for theoretical exploration in this domain. We envision future studies to delve deeper into elucidating the theoretical underpinnings of DNR's success and exploring its application in diverse domains beyond those examined in this study. One possible limitation that emerged from our empirical evaluation is the sensitivity of our method to the choice of parameter selection. Addressing this limitation in future work could involve developing a robust parameter selection or importance estimation technique. This enhancement would improve our method's overall performance and deepen our understanding of overfitting dynamics by disentangling the contributions of different parameters. Finally, we believe the development of techniques like DNR holds the potential to advance the capabilities of deep learning models, paving the way for robust, adaptable, and generalizable artificial intelligence systems. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Not any which we forsee Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We believe that the research work presented herein has the potential to significantly impact the field of deep learning by facilitating the development of models on small datasets, thus reducing the necessity for extensive data collection. This contribution is of particular significance in specific domains like autonomous navigation and medical imaging, as data acquisition can be cost-prohibitive, even in cases where labeling is unnecessary. By mitigating overfitting on small datasets, the proposed approach can improve the accuracy and reliability of deep learning models in these areas, ultimately leading to improved patient outcomes and safer autonomous systems. Moreover, this work has the potential to usher in a more efficient and sustainable use of resources in a diverse range of domains where data collection is a bottleneck.In this section, we present the evolutionary process of the mask over multiple generations and its impact on the performance and generalization capabilities of the DNNs. We evaluate the effectiveness of DNR in dynamically adapting and evolving the mask throughout the training process. The ResNet18 architecture with CUB200 is used for this evaluation.Figure 4 illustrates the evolution of the mask between generations. As training progresses, the mask undergoes iterative updates based on the data-aware dynamic masking criteria employed by DNR. The mask becomes more refined and selective with each generation, preserving important connections while pruning less relevant ones. To quantify the evolution of the mask, we measure the percentage of overlap of parameters retained between the first and the corresponding generations. We observe a gradual decrease in overlap from the initial generation to subsequent generations, indicating the emergence of masks in an evolutionary training scenario. This progressive mask evolution contributes to the network's enhanced capacity for learning and generalization, evident from the test accuracy.1 2 3 4 5 1-2 1-3 1-4 1-In conclusion, our results highlight the evolutionary nature of the mask throughout generations in the DNR framework. The dynamic adaptation and refinement of the mask lead to effective masking and improved performance and generalization of the DNN. These findings support the effectiveness of our approach in leveraging the evolutionary training paradigm to enhance the learning and generalization capabilities of deep neural networks compared to KE.In practical applications, deep neural networks often operate in dynamic environments characterized by variations such as lighting and weather conditions. Consequently, it is crucial to assess the robustness of DNNs to data distributions that undergo natural corruption. We investigate the robustness of DNNs to 15 common types of corruptions using the CIFAR-10-C dataset (37) . Our models are trained on clean images of the CUB dataset and evaluated on CIFAR-10-C (37). To quantify the performance under natural corruption, we use the Mean Corruption Accuracy (mCA) metric.EQUATIONwhere N c and N s represent the number of corruptions (in this case, 19 ) and the number of severity levels (in this case, 5), respectively. Figure 5 illustrates the average accuracy of the models across 19 different corruptions at five severity levels. In particular, our proposed method (DNR) achieves a higher mCE (27.7%) compared to the longer baseline (23.4%) and KE (16%), demonstrating its effectiveness in improving the robustness to various types of corruption. These findings highlight the benefits of selectively reinitializing network parameters using a data-aware masking approach, resulting in enhanced robustness to natural corruptions.In our experiments, we consistently maintained a fixed training duration of 200 epochs for each generation, with the number of generations set at 10 to ensure a fair comparison. The computational cost of evolutionary training methods, including KE, LLF and DNR, scales linearly with the number of generations (T). For instance, if KE is trained for 5 generations, the total computational cost becomes 5T times that of training a single generation. To ensure a fair comparison, we trained a long baseline for the same number of epochs. The additional computational cost incurred by DNR for computing data-aware dynamic masking with SNIP is minimal. For example, on the CUB dataset with a 20% subset, this process takes approximately 20.3 seconds per generation. This calculation is performed once at the end of each generation and can be further optimized by using just 128 samples to estimate the importance without compromising performance. Appendix Table 7 demonstrates that DNR's performance is minimally sensitive to changes in the subset size.DNNs are vulnerable to adversarial attacks, where imperceptible perturbations are added to the input during inference to deceive the network and induce false predictions (38) . Therefore, we investigate the robustness of DNNs trained against adversarial attacks using the PGD-10 attack (39) on models trained on the CIFAR10 dataset. We vary the intensity of the PGD attack and evaluate the models' performance. As shown in Figure 6 , our proposed framework (DNR) exhibits greater robustness to adversarial attacks at different attack strengths compared to KE and the long baseline. This highlights the efficacy of our framework in training models that can learn high-level abstractions robust to small perturbations in the input data. In real-world applications, class imbalance is a common characteristic of the input distribution, where certain classes are more prevalent than others.This inherent class imbalance can affect the training of DNNs, as they tend to be biased towards the majority classes, thereby neglecting the minority classes (40) . To address this issue, we explore the contribution of reinitialization to model training with class imbalance. We incorporate class imbalance using the power-law model on CIFAR10. The number of training samples for a class c is determined by n c = a/(b + c) γ , where γ represents the imbalance ratio, and a and b are offset parameters specifying the largest and smallest class sizes. We set a fixed gamma value of 1 in our experiments to maintain a power law class distribution. The offset parameters a and b are chosen such that the maximum and minimum class counts are 5000 and 250, respectively. We used balanced accuracy as a metric to measure the robustness of the model under the class imbalance scenario. Our findings in Figure 7 demonstrate that the DNR framework consistently outperforms KE and longer baselines in scenarios with class imbalance. This highlights the effectiveness of DNR in addressing the challenges posed by imbalanced class distributions and underscores its potential for practical applications. Table 5 shows the effect of varying the number of reinitialized parameters on the performance and generalization of the model. We train the model in evolutionary settings using the DNR framework by varying different percentages of reinitialized parameters (5%, 10%, 20%, 30%, and 40%). The experiments were carried out with ResNet18. The results show that the reinitialization of a 5% percentage of parameters has no impact on performance, while the reinitialization of more than 30% has less impact on test accuracy. We find that reinitialization 20% of the parameters results in the best performance. In this section, we assess the effectiveness of the sparse model (containing 20% fewer parameters than the full/dense model) obtained through the selective neurogenesis process during the inference phase. We examine the sparse and dense models' test performance compared to the original KE framework.For this, we measure the performance of the ResNet18 model trained on CUB200. Table 6 presents the accuracy results obtained by the sparse model compared to the dense model. Surprisingly, despite the considerable reduction in the number of parameters, the sparse model achieves comparable accuracy compared to the dense model in the DNR framework. Furthermore, DNR demonstrates superior performance in both the full and sparse model scenarios compared to the KE. This indicates that the selective neurogenesis process successfully retains the critical connections necessary for accurate predictions while eliminating redundant or less informative connections. Our evaluation demonstrates that the sparse model obtained through the selective neurogenesis process offers several benefits during inference. It maintains high accuracy while achieving improved computational efficiency compared to the KE. These results highlight the practicality and efficacy of leveraging selective neurogenesis to create efficient and compact deep learning models that can be readily deployed in real-world scenarios.A.9 Varying the quantity of data used for Importance estimationIn our experiments, we randomly sampled 20% of the dataset to estimate the importance of the parameters after the end of each generation. Here, we analyze the impact of the number of data used to determine the important estimation on the final performance. Similar to Lee et al. (23) , we used as few as 128 samples to estimate the important parameters using SNIP. Table 7 shows that DNR is not sensitive to the variation in the input data used to estimate the importance as the final performance remains unchanged. To explore the performance of deep learning in low-data medical settings, we conducted experiments using ResNet-50 on the Papila dataset (41) . This dataset, with its limited size and complex features, is representative of the challenges in medical imaging. The Papila dataset is a medical imaging dataset focused on analyzing and identifying ocular structures, particularly the optic disc and retinal layers. We conducted this experiment using ResNet-50 with a learning rate of 0.001, batch size of 32, and trained for 3 generations (1 generation = 200 epochs). For DNR 20% of the parameter is reinitialized using SNIP. Our preliminary results suggest that Dynamic Network Reconfiguration (DNR) shows promise in the medical domain when compared to baseline training. However, further hyperparameter tuning is necessary to fully optimize performance.While initial findings are encouraging, we highlight the need for more detailed studies in reinitialization-based methods within the medical domain on different datasets. Future work should address these challenges to improve generalizability across low-data medical imaging datasets. As the number of training generations increases, the performance of DNR begins to saturate. This indicates diminishing returns in accuracy improvement with extended training. This saturation effect is corroborated by our analysis of the mask evolution across generations, as illustrated in Figure 2 .The overlap percentage of the mask progressively increases, with a higher overlap observed between the 9th and 10th generations compared to the 1st and 2nd generations. This trend suggests that the mask becomes more saturated and stable, aligning with the model's convergence to a lower-loss landscape.A.12 Summary of Datasets and Implementation Details Taha et al. (5) employs various image resizing techniques for different datasets; however, they do not provide specific details about the resizing parameters in their paper. To ensure consistency across our experiments, we resize all datasets to a fixed size of (256, 256). Moreover, to fine-tune the hyperparameters, we utilize a validation split, and the reported results are based on the test set whenever it is available. For experiments on large datasets, we used the following settings. The experiments were conducted on three different datasets: CIFAR-10/100, Tiny-ImageNet. For CIFAR-10/100, the training was performed for 160 epochs. A batch size of 64 was used, along with a step-based learning rate scheduler. The learning rate decay was applied between epochs 80 and 120, with a decay factor of 10. The momentum was set to 0.9, and l2 regularization was applied with a coefficient of 5e-4. The initial learning rate used was 0.1. There were no warmup epochs in this case.For the Tiny-ImageNet dataset, the training was also conducted for 160 epochs. The batch size was reduced to 32, and a step-based learning rate scheduler was used. Similar to CIFAR-10/100, the learning rate decay occurred between epochs 80 and 120, with a decay factor of 10. The momentum and l2 regularization were set to 0.9 and 5e- • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The potential impact of the method is discussed in the broder impact section in Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The reviewers raised several concerns in their reviews largely focusing on computation cost comparisons, clarifications on transfer learning, and other bits and pieces. The rebuttal addressed most of these concerns, and additional points were clarified through author-reviewer discussions.

Most importantly though the authors have presented a robust method based on iterative/evolutionary training paradigm that performs well on small datasets, improving generalisation through incorporating generalisable features during further learning. The experiments and additional ablations shown in the appendix provide extra confidence in this method's advantages in the settings presented in the paper. 

I would like to ask the authors to incorporate the additional results presented below and all the other elements they promised to do in the camera ready version, including this "Additional Experiments on Medical Domain Datasets: While we couldn't include medical datasets due to resource constraints, we committed to performing these experiments in a revised version of the paper.".

Congratulations on the acceptance of your work.

Reviews: review: paper_summary: paper_summary: This manuscript introduces the Dynamic Neural Regeneration (DNR) framework, which improves the generalization of deep neural networks on small datasets. DNR uses the SNIP method to reinitialize less important neural connections for the current generation selectively. Experimental results show that DNR outperforms the existing original Knowledge Evolution (KE) method and long baseline (LB) in 5 small datasets and 3 large datasets.
strength_weakness: strength_weakness: - The performance of DNR does not surpass the STOA.
- The experiments lack details, making it hard to interpret the results.
questions: questions: ### 1. The performance of DNR does not surpass the STOA.

1.a Despite the superior performance of DNR compared to KE and LB, it does not surpass the existing SOTA, such as Smth+LLF. For the reported results in this manuscript, Smth+LLF either showed better or marginally lower performance compared to Smth+DNR. Additionally, the results reported in the original LLF study [1] were higher than the author-reported Smth+LLF as well as the proposed Smth+DNR for all 5 small datasets (Table 1 in [1] vs Table 1 in the manuscript) and on the TinyImageNet dataset (Table A7 in [1] vs Table 2 in the manuscript). The authors did not address this discrepancy, making it hard to interpret the reported results.

1.b Following the comment above, statistical analysis should be provided to better demonstrate the difference between different methods.

1.c DNR requires calculating SNIP to determine the connections needed to be reinitialized. However, the time complexity related to data size and model size is not provided. Therefore, it is still questionable whether DNR is practically useful, especially considering that its performance may not surpass the SOTA (comment 1.a).

### 2. The experiments lack details, making it hard to interpret the results.

2.a The experimental setting seems arbitrary. For example, why do some methods, such as KE and Smth+KE, use 10 generations while Smth+LLF and others use 8 generations? It seems the authors followed the experimental setting of prior work, such as Zhou et al. [1], but it was not mentioned in the manuscript.

2.b The authors reported mean and std for each experiment, but the number of runs for each experiment is not mentioned.

2.c It is very confusing what kind of "transfer learning" is used in Section 5.4. What is the difference between "transfer learning" and "long baseline"?

2.d Many reported results in Table 1 are identical to the results in [1] (Table 1), such as Smth+KE with CUB, Aircraft, and MIT datasets, while others are not. Please clarify whether the results are reproductions or simply referrals from [1]. If they are reproduced results, please explain the discrepancy between the reported and reproduced results.

### 3. Minor comments

3.a Both LLF [1] and KE [2] demonstrated that they performed the best with CS-KD [3]. It would be interesting to see how DNR performs with the same setting.

3.b The letter "g" was used to denote the generation number (line 107) and connection sensitivity (eq. (4)). Please consider using different letters.

3.c What "m" stands for in eq. (5) and (6).



###Reference

[1] Zhou, Hattie, et al. "Fortuitous forgetting in connectionist networks." International Conference on Learning Representations. 2021.

[2] Taha, Ahmed, Abhinav Shrivastava, and Larry S. Davis. "Knowledge evolution in neural networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[3] Yun, Sukmin, et al. "Regularizing class-wise predictions via self-knowledge distillation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
limitations: limitations: None.
paper_summary: paper_summary: This submission investigates efficient training and generalization of deep neural networks in the low-data regime. Drawing inspiration from neurogenesis in the brain, authors propose an iterative training framework termed Dynamic Neural Regeneration (DNR). The authors further investigate the efficacy of the proposed approach through experiments on five datasets (Flower102, CUB-200-2011, MIT64, Stanford Dogs and FGVC-Aircraft).
strength_weakness: strength_weakness: 1. Since the authors repeatedly claim that data shortage is often seen in medical diagnosis (in Abstract line 2-3, Introduction line 23-25, and Results line 285-286) it would be great if they also include experiments in such scenarios.
2. I find the schematics (Figure 1) confusing instead of enlightening. What I read from it is that, (1) We have a dataset and a neural network. (2) We train the neural network on the dataset. (3) We apply data-aware masking where some neurons (minority, shown in blue) are kept the same and the others (majority, shown in green) are somehow changed --- but it is unclear how they are changed. (4) Then, we remove the blue neurons (???) --- so that nothing is kept from the current iteration? (5) We randomly initialize the removed neurons --- if so, why do we remove them from the first place? Why don’t we skip neuron deletion and directly perform neuron initialization? I am not sure how other people would process this figure, but at least to me it needs some re-editing for it to be a helpful illustration. Another minor tip: if the weights of certain neurons are kept unchanged between two steps, I would recommend either using the same color to color code them, or using a small symbol to indicate weight freezing (such as a lock or a snowflake).
3. This submission is among the works where the empirical results are critical, and therefore I would be concerned that the authors have not submitted their code as the supplementary material, despite stating that they would release the code upon acceptance. At least on my portal no supplementary material is uploaded.
questions: questions: 1. Please refer to Weakness #2.
2. For Table 1, would it be helpful to include the notations “CE” in methods that are based on cross-entropy loss, such as “CE + DNR$(f_{10})$”? Or did I misunderstand?
3. For Table 1-3, would it be helpful to include an additional column that reports the mean over all datasets, or ranking across all datasets?
4. I have not seen discussions on Figure 2. Could you point me to the relevant lines in case I missed it?
limitations: limitations: Yes, the authors adequately addressed the limitations and, if applicable, potential negative societal impact of their work.
paper_summary: paper_summary: This paper presents a novel iterative training framework called Dynamic Neural Regeneration (DNR) designed to enhance the generalization of deep learning models on small datasets. The DNR approach utilizes a data-aware dynamic masking scheme inspired by neurogenesis to eliminate redundant connections, thereby increasing the model's capacity for further learning. Through extensive experiments, the authors demonstrate that DNR outperforms existing methods in both accuracy and robustness, making it a promising technique for applications with limited data availability.
strength_weakness: strength_weakness: ***Complexity of Implementation***: The DNR framework may be complex to implement for practitioners without a strong background in iterative training paradigms and dynamic masking techniques. However, including more detailed implementation guidelines or open-sourcing the code could mitigate this issue.
***Scalability***: While the approach is effective for small datasets, its scalability to very large datasets or more complex models is not fully explored.
questions: questions: How does the computational cost of DNR compare to other state-of-the-art methods? Is there a significant increase in training time due to the dynamic masking process?
Can the authors provide more detailed guidelines on how to implement the DNR framework in practice, including hyperparameter settings and potential pitfalls to avoid?
limitations: limitations: Could the authors provide a careful and detailed discussion of the limitations of the work?
paper_summary: paper_summary: This paper proposes Dynamic Neural Regeneration (DNR), a framework to enhance the generalization of deep neural networks on small datasets. The method is inspired by neurogenesis and offers more flexibility in defining a parameter mask as compared to previous approaches such as Knowledge Evolution (KE). The results show strong performance on small datasets with sufficient ablation studies.
strength_weakness: strength_weakness: 1. Even though the paper begins by emphasising limited data availability in the medical domain, no experiments on medical datasets have been presented. It would be beneficial to include results on a few medical datasets such as Papila [1], Harvard-GF3300 [2] that represent working in a low-data regime.
2. No analysis of computational cost especially for larger datasets has been presented. 
3. The transfer learning experiment performed in section 5.4 is not clear. What does generation mean in vanilla fine-tuning? More details are required here.
4. In theory, DNR could be well incorporated with transfer learning i.e. instead of starting from scratch, use the weights of a pre-trained model and follow the iterative approach of selecting and retaining parameters after each generation. Is there a specific reason why transfer learning experiments were not included?
Furthermore, even though there is a large domain gap between natural and medical images, transfer learning (full vanilla fine-tuning or parameter-efficient fine-tuning [3]) still remains the de facto practice in medical image analysis.
5. The results on large datasets (Table 2) show that DNR (and other baselines) are not very effective in the high-data regime. In fact, standard training seems sufficient when the dataset size is large enough.

References

1. Kovalyk, Oleksandr, et al. "PAPILA: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment." Scientific Data 9.1 (2022): 291.
2. Luo, Yan, et al. "Harvard glaucoma fairness: a retinal nerve disease dataset for fairness learning and fair identity normalization." IEEE Transactions on Medical Imaging (2024).
3. Dutt, Raman, et al. "Parameter-efficient fine-tuning for medical image analysis: The missed opportunity." arXiv preprint arXiv:2305.08252 (2023).
questions: questions: 1. The DNR method evaluates connection sensitivity from a subset of data. However, different subsets can capture different distributions of the data and hence, lead to different connection sensitivity possibly affecting the final performance of DNR. Can the authors provide a justification or some results on this?
2. How is the mask defined for the DNR framework? Is it over each model parameter (in this case, the mask length would be of a similar magnitude as the model parameter count) or each ResNet layer? Details on this should be included in the paper.
3. Hypothetically, if the DNR framework is allowed to run for a sufficiently long number of generations (g=100, for example), we should be able to observe a saturation in the mask at a point (say g=55) beyond which the mask does not change at all. It would be beneficial to include a similar experiment.
limitations: limitations: Please see the Weaknesses section.

Comments: title: Follow-up on Rebuttal
Response to Reviewer DwbC
Response to rebuttal
Thanks for the responses
Rebuttals not visible yet
Clarification for Manuscript Changes
Authors Reply - Clarification for Manuscript Changes
Rebuttal Reply - Clarification for Manuscript Changes
General Comment
comment: We are following up on our rebuttal. We have addressed all the points raised in your feedback and would greatly appreciate any additional questions or comments you might have.
To enhance the context and relevance of our study, we'll include references to the suggested works on dynamic masking. Could you please review the rebuttal and let us know if it meets your expectations for a score adjustment?
Thank you for your positive feedback and for reconsidering your rating. We appreciate your thoughtful review and are pleased that our updates have addressed your concerns.
Thank you for your feedback. 

To assess the performance of our proposed DNR method, we employed paired t-tests across all datasets.  DNR demonstrates statistically significant improvements over both the LB and KE methods, with p-values of 0.011 and 0.0012, respectively. These results support the claim that DNR offers meaningful enhancements in generalization performance over these established baselines. In the comparison with Smth+LLF, the p-value was 0.0670, which, while not meeting the conventional threshold  (0.05) for statistical significance, suggests that the performance of DNR is comparable to LLF. It is important to note, however, that DNR's unique features—such as **dynamic reinitialization** and **data-aware masking**—provide additional benefits that may not be fully captured by statistical tests alone. DNR's ability to analyze connection sensitivity and identify influential parameters throughout the network, regardless of their rigid architectural assumptions, **offers a flexible and adaptive approach** that can be particularly advantageous in a wide range of scenarios.

As per your suggestion, we will also include a comparison with Smth+LLF for larger datasets in Table 2 of the revised manuscript. 
 
We hope this response satisfactorily addresses your concerns.
Thanks for the authors for the patient updates and addressing the comments. I do not have major concerns, and I am willing to increase the rating.
Thank you for your detailed rebuttal. The responses have been useful in addressing most of my concerns, and I appreciate the efforts and would like to change my initial rating. However, there are still some concerns that have not been addressed.

First, the reproduced results for Smth+LLF are lower than those reported in the original study, which might be attributed to stochastic effects such as initialization. I recommend conducting some statistical analysis, as mentioned in my previous comment 1.b, to robustly determine whether DNR significantly outperforms other baselines. This is essential to support the claims made in the authors' response (the “Empirical Validation” section), but is still missing.

Additionally, as outlined in my comment 1.a, Smth+LLF has shown higher performance on the TinyImageNet dataset compared to the DNR results presented in the manuscript. I would suggest the authors include a comparison with Smth+LLF for larger datasets in Table 2, which has not been provided in the response.
Dear Reviewers and Authors,

Rebuttals have not been made visible to reviewers yet, whereas the comments are made available as submitted.
Therefore some of the comments assume that the reviewers have access to the rebuttals.

May I kindly ask you to wait for a few hours until rebuttals have been made visible to everyone?

Thank you!

AC
Thank you for your reply. Could you point clarify the same in the comments. For instance, where can we find Computational Cost Comparison, Hyperparameter Settings, etc in the paper? It would be great if you could point out the sections and the sub-sections.
Yes, we agree. Unfortunately, the conference guidelines do not allow for the submission of revised manuscripts at this stage of the review process. We are committed to addressing all reviewer comments and suggestions, and we will ensure that the necessary revisions and improvements are made in the final version of the paper along  with the references to works on dynamic masking to enhance the context and relevance of our study. Thank you for your understanding.
Thank you for taking out the time to reply to the reviews. I want to highlight a few points:

1. It would be extremely helpful to the reviewers if you could highlight all the changes (both in the manuscript and in reply) that you made after the review. In the current state, it is impossible to locate what changes were made and where in the paper.

2. It would be good to include citations for a few other works that have explored dynamic masking [1,2] for both general and medical image analysis.

These changes can be incorporated very quickly but would greatly enhance the reviewing process.

References:

1. Zhang, Dinghuai, et al. "Can subnetwork structure be the key to out-of-distribution generalization?." International Conference on Machine Learning. PMLR, 2021.
2. Dutt, Raman, et al. "Fairtune: Optimizing parameter efficient fine tuning for fairness in medical image analysis." arXiv preprint arXiv:2310.05055 (2023).
We thank all the reviewers for their thoughtful feedback. We are pleased that you found our motivation and idea to be strong, clear, and novel, and our extensive experiments insightful. Based on the valuable feedback, we will make the following modifications to the manuscript:

- **Computational Cost Comparison:** Added a comparison of the computational costs associated with DNR versus other methods to provide a clearer understanding of its practical implications.
- **Additional Experiments on Medical Domain Datasets:** While we couldn't include medical datasets due to resource constraints, we committed to performing these experiments in a revised version of the paper.
- **Transfer Learning:** Clarified the implementation and difference between vanilla fine-tuning and the long baseline in the context of transfer learning.
- **Experimental Details:**
  - **Number of Runs:** Clarified that each experiment was conducted three times with different random seeds, and the mean and standard deviation are reported.
  - **Generations for Different Methods:** Explained the rationale for the number of generations used in different methods based on standard practices and architectural constraints.
- **Dataset Choice and Result Reporting:** Explained the rationale behind the empirical validation and the results.
- **Hyperparameter Settings:** Provided detailed guidelines on the hyperparameters used in our experiments and identified potential pitfalls to avoid in practical implementations.
- **Added New Experiments with CS-KD:** Included experiments with Class-Wise Knowledge Distillation (CS-KD) to evaluate the performance of DNR in this setting.
- **Saturation of Mask Experiments:** Conducted and reported experiments showing the saturation behavior of masks over generations to further validate the stability and convergence of DNR.
- **Subset Sensitivity Analysis:** Provided results and justification for the effect of different subsets on connection sensitivity and the final performance of DNR, demonstrating its robustness.
- **Clarity Improvements:**
  - **Figure 1:** Revised for better clarity. Updated the figure and its explanation to more clearly illustrate the process of data-aware masking and neuron reinitialization in the DNR framework.

In conclusion, we have meticulously addressed all the specific questions raised by the reviewers. We believe these revisions significantly strengthen the paper and enhance its overall quality. We kindly ask the reviewers to consider these substantial improvements and reconsider their scores accordingly.