Improving Transfer Learning by means of Ensemble Learning and Swarm Intelligence-based Neuroevolution

Authors: a  gómez
m  abella
m  desco
Venue: automl_cc_AutoML_2024_Conference
Published: 2024-02-25 11:06:49

Abstract: Neural Architecture Search (NAS) methods, when applied to very small but complex datasets, tend to overfit on the validation partitions and underperform compared to Transfer Learning models. In order to reduce the bias and variance of their predictions, Deep Ensemble Learning (DEL) can be used. The combination of NAS and DEL has only been employed on large datasets in the literature, but these scenarios do not present the overfitting in validation we typically experience, for instance, on medical imaging applications. In this work, we empirically assess the feasibility of NAS, DEL and the combination of the two on both small and large dataset scenarios. We find that the performance of the ensembles highly depend on the degree of overfitting of the standalone models, but always will outperform the worst generalizing models in the population.

Main paper: 
 introduction: 
 Most advances on Computer Vision are designed for and tested on large natural image datasets such as IMAGENET or CIFAR. Pretrained, highly performing models are readily available for the Deep Learning community, making Transfer Learning the standard approach for many research fields, such as medical image processing. However, despite their widespread popularity, the advantage of using these pretrained models on medical images has been questioned [Raghu et al. (2019) , Neyshabur et al. (2020) ]. The suitability of any given pretrained model has been shown to heavily rely on the inductive biases inherent to its architecture [Wang et al. (2019) , Goyal and Bengio (2022) ].Selecting an adequate model for any specific application is not trivial, but several methods have been proposed to reduce bias and variance of sub-optimal models. One such method is Deep Ensemble Learning (DEL), which can achieve a high degree of complexity by combining the outputs of several relatively simple learners [Ganaie et al. (2022) ]. Although powerful, DEL presents its own set of challenges. On the one hand, training several models requires considerably more time than fine-tuning a single architecture. On the other, it is difficult to obtain a high diversity on the features learnt by the individual learners if their architectures are identical and thus present the very same inductive biases.Modifying the architecture of a pretrained Convolutional Neural Network is a complex, subjective and time-consuming endeavor because the optimal model hyperparameters are problemdependent. Neural Architecture Search (NAS) techniques are used to explore the space of feasible models in an automatic way and have achieved state of the art performance on several computer vision benchmarks [White et al. (2023) ]. However, balancing the exploration-exploitation trade-off is not trivial, especially when very deep networks are required: more layers imply more combinations between their hyperparameters, generating massive spaces to be explored. Moreover, considering a large number of architectures comes at the risk of overfitting on the validation partition of the datasets, which are used to select or discard the models. This problem is particularly relevant for complex datasets where few samples are available as it may be very difficult to construct training and validation partitions that are representative of the whole task.The problems faced by NAS can be mitigated with the use of DEL and vice versa. Ensemble methods could allow to limit the search to shallower models, considerably reducing the required computational expenses while maintaining the complexity of the prediction. Moreover, the feature sharing and weighting enhances the generalization capabilities of the models, overcoming the instances of overfitting on the validation partitions. Conversely, NAS methods already work with populations of solutions with distinct hyperparameters, which inherently present the high diversity that traditional DEL lacks. Combining the resulting models instead of selecting the best performing one might yield considerable benefits with negligible extra computational cost.In this work, we have assessed the feasibility of DEL, NAS and a combination of the two to deal with the challenges of Transfer Learning in two completely different scenarios. The main goal is to assess the robustness of the methods against different degrees of overfitting. First, the different approaches were validated on the CIFAR-10 classification task. Several data and model weighting strategies were compared for the application of DEL. The NAS method employed was an in-house optimizer called the Chimera Algorithm. The combination of the Chimera Algorithm with DEL methods gave rise to a new, enhanced optimizer, referred to as the Gaggle Algorithm. All these methods were then employed to tackle a problem of biomedical interest to test their performance on a task where Transfer Learning would be the standard approach.2 Related works 
deep ensemble learning: 
 Ensemble Learning consists on training several individual learners on subsets of the original dataset and combine their predictions to leverage the influence of the most useful features learnt by each one of them [Breiman (2000) ]. Deep Ensemble Learning (DEL) uses this strategy to enhance the predictive capabilities of Deep Learning models [Ganaie et al. (2022) ]. There are many strategies to train Deep Learning models and combine them into ensembles, such as Bagging, Boosting, Stacking or Negative correlation Ensemble Learning. Particularly, Boosting Ensemble Learning methods train the different learners sequentially, based on the performance of the growing population. For each new learner, the dataset is either weighted or resampled in order to pay more attention to misclassified data. Each learner is also weighted based on its standalone performance, so that the final prediction is mainly driven by the most accurate ones. This approach allows for the use of shallower models that focus on specific parts of the task at hand rather than a single deep model that is more powerful on its own, but more prone to overfitting. However, some of the models present in the ensemble might not provide any valuable contribution, and even worsen the predictions. In order to limit this effect, ensemble pruning -also known as selective ensemble or ensemble selection-methods based on the validation error, kappa measure, complementary measure, margin, and diversity are commonly used [Margineantu and Dietterich (1997) , Martínez-Muñoz and Suárez (2004) , Guo et al. (2018) ].Neural Architecture Search (NAS) is a sub-field of metaheuristic optimization that strives to automatize the creation and optimization of Artificial Neural Networks. It spans a wide range of optimization algorithms, from the earliest works in the field employing Genetic Algorithms [Miller et al. (1989) ] to modern approaches such as supernet optimization [Cha et al. (2023) ], Multi-Fidelity MetaLearning [Zimmer et al. (2021) ] or training Controller Recurrent Neural Networks using Reinforcement Learning to generate the architectures [Zoph and Le (2017) ]. Each method presents their own strengths and drawbacks, but most of them are prone to overfit on the dataset partition used to select or discard the models, if it is not representative enough of the whole task. This Weighting each model's contribution for each output based on its validation confusion matrix or average loss for each one overfit is tangential to most methods, as it mainly depends on the distribution of our data and the model selection criterion employed [Cawley and Talbot (2010) ]. Unfortunately, using rigorous model selection criteria within a NAS pipeline is oftentimes unfeasible due to computational costs. Combining the models generated through NAS into an ensemble has been shown to surpass the performance of the individual learners and is much less computationally expensive. However, the very few works that assess this approach in the literature [e.g. Herron et al. (2020) , Chen et al. (2021) ] only do so on large, widely studied datasets such as CIFAR-10, MNIST, IMAGENET or COCO. The behavior of ensembles attained through NAS in small and complex datasets, for which overfitting is a prevalent issue, is yet to be analyzed. 
algorithms: 
 3.1.1 Deep Ensemble Learning. As previously stated, Boosting Ensemble methods allow for the use of shallower models. This is especially relevant in the context of NAS, as deeper models are much more computationally expensive to find -not just because of the higher training times, but because the possible hyperparameter combinations grow exponentially with the number of layers, generating a wider search space-. A modular Boosting Ensemble creator was developed, which sequentially trains copies of a given base model, weights their contributions to the growing ensemble, and modifies the training dataset according to the strategies specified by the user. In this work, we explored four strategies to modify the training dataset to assign a heavier weight to the samples that the ensemble is unable to properly characterize, as well as three strategies to calculate the contribution of each individual model to the complete ensemble. These are shown in Table 1 . The sample weights were the ones employed by the AdaBoost classifier [Breiman (2000) ]. The algorithm's pseudocode, default hyperparameters and explanation of the weights employed are depicted in Algorithm 1 in Appendix A.3.1.2 Neural Architecture Search. The NAS method employed in this work is the Chimera Algorithm, a metaheuristic based on the Artificial Bee Colony Algorithm [Karaboga and Basturk (2007) ] that was developed in-house. This algorithm deploys two kinds of optimizer agents -referred to as Employed and Onlooker Bees-on a population of candidate solutions. These agents select models from said population based on their individual performances, and explore similar architectures by using the flexible mutation operators employed in traditional Genetic and Evolutionary NAS methods. Unlike Genetic and Evolutionary approaches, though, the Chimera Algorithm is able to maintain a much higher diversity within the population -which is useful to obtain good performing ensembles-as the worst models are not discarded on each iteration, but rather simply given less attention. Moreover, its local nature facilitates the exploration of similar architectures to that of any pretrained model provided by the user, exploring outwards from already promising positions within the search space. Its pseudocode is depicted in Algorithm 2 in Appendix A, along with a detailed algorithm description and a definition of its default hyperparameters.3.1.3 Combination of DEL and NAS. Both DEL and NAS approaches have been combined into the Gaggle Algorithm, which operates similarly to the Boosting Ensemble creator but calls the Chimera Algorithm to generate batches of optimized models instead of simply training a base architecture. Its pseudocode is depicted in Algorithm 3 in Appendix A.The performance of DEL, NAS, and their combination were compared with that of predesigned architectures in two scenarios fully described in Sections 3.2.1 and 3.2.2. In the first scenario, these methods were evaluated on the classification problem of the natural images in the CIFAR-10 [ Krizhevsky and Hinton (2009) ] dataset. In the second scenario, they were tested on a problem of biomedical interest: the estimation of the horizontal misalignment of the detector in a Computed Tomography system by analyzing the artifacts present in the volumes reconstructed from a set of projections spanning an angle of 180º. In both scenarios, the Leslie Smith's learning rate test [Smith (2018) ] was used to approximate the optimal learning rate for each model, as it proved itself much faster and reliable than adding an extra dimension to the hyperparameter space. Crossvalidation has not been employed to compare individual models but, rather, to assess the validity of each method as a whole, generating a different split for each run to avoid biases due to the data partition used. The code developed can be accessed through this link: https://github.com/HGGM-LIM/Chimera. The seeds employed for randomization are specified in the code itself. An Intel® Core™ i7-7700 CPU and a NVIDIA® GeForce® RTX 2060 Super™ GPU were used for all tests performed.3.2.1 Scenario I: natural image classification on CIFAR-10. The following tests have been performed to evaluate the different approaches on the CIFAR-10 dataset, which consists on 60000 32x32 colour images divided into 10 diferent classes. 10000 images are reserved for testing and, in our setup, we used 90% of the remaining images for training, using the leftover ones for validation. No data augmentation was employed. All models were trained until the validation loss converged, with a patience of 8 epochs.• One shallow and one deep feed-forward architectures were trained on the dataset to check the performance of standalone models. The shallow architecture selected was LeNet-5 [Lecun et al. (1998) ] -which has around 60 thousand weights to optimize-and the deep architecture selected was VGG11 -over 133 million parameters, as reported in [Simonyan and Zisserman (2014)]-. In the case of VGG11, the last fully connected layer was modified to reduce the number of output classes from 1000 to 10. The weights of LeNet-5 were initialized at random, while VGG11 was tested both with random initialization and inheriting the weights from pretraining in ImageNet. The three cases were tested 32 times, generating a different training and validation partition of the dataset in each iteration. The AdamW optimizer was used. The learning rates employed were 10 -3 and 10 -4 for the shallow and deep models respectively, as suggested by the Leslie Smith test.• LeNet-5 was used as a template architecture to create ensembles with. For this test, we employed all the data and model weighting strategies depicted in Table 1 . Each new model was randomly initialized before training on the auxiliary dataset, which was obtained according to the data weighting scheme and the ensemble accuracies on the complete, unweighted dataset. Population pruning was performed after the whole ensembles were created. For simplicity, the models were selected simply based on their standalone validation error. Each data and model weighting combination was tested 10 times, reshuffling the training and validation partitions. • The performance obtained with a simple NAS method was evaluated, using the Chimera Algorithm to create models from scratch. The models generated consisted of a feature extractor with convolutional and pooling layers and a single fully connected layer with 10 output classes. Two tests were performed. Firstly, it was run with a population size of 4 models and a search length of 16 iterations to check the capabilities of the algorithm to find good performing models in relatively shallow searches. Then, it was run with a population size of 8 models, a search length of 32 iterations and an exhaustion limit of 16 iterations to measure the possible improvement that a deeper search would yield. Each test was performed 4 times, reshuffling the training and validation partitions.• The Gaggle Algorithm was employed to create ensembles with models generated from scratch.Once again, two distinct tests were performed. Firstly, the Gaggle Algorithm was run with 4 batches, a population size of 4 models, a search length of 16 iterations, accuracy-based model weighting and failed prediction data resampling in between batches. Then, the Gaggle Algorithm was run with a single batch, a population size of 8 models, a search length of 32 iterations and accuracy-based model weighting. Each test was performed 4 times, reshuffling the training and validation partitions.Scenario II: misalignment regression on CT slices with artifacts. Horizontal misalignment of the detector in a Computed Tomography system leads to artifacts in the reconstructed volumes, which can render the studies unusable for proper diagnosis. When the volumes are reconstructed with an angular span of 180º, these artifacts appear in the axial slices as arcs, as can be seen in Fig. 1 . Their thickness depends on the misalignment value, and their orientation is given by the misalignment direction. In order to correct for these artifacts, a calibration file needs to be acquired manually by a technician. Theoretically, the geometrical misalignments of any acquisition could be automatically characterized based on the artifacts that appear on the pre-reconstructed volumes. Doing so would eliminate the need for manual calibration, and allow for the reconstruction of any study for which the callibration file is not available.We generated a small dataset to train models able to predict the misalignment corresponding to any given individual slice. We first acquired six properly calibrated rodent cranial studies using a SEDECAL micro-CT system. Then, 25 sets of miscalibrated projections were simulated for each volume. Simulated horizontal misalignments values were taken from a random uniform distribution with a range of ±1mm. An FDK-based algorithm was used for reconstruction, generating volumes of 256×256×200 pixels, which were then normalized to ImageNet's mean and standard deviation -0.485 and 0.225, respectively-and separated into 2D axial slices. The tolerance, defined as the smallest misalignment that produces artifacts noticeable to the naked eye, was set to ±0.1 millimeters [Abella et al. (2021) ]. The database was split into a training partition with five rodents, a validation partition with one rodent, and a test partition with one rodent as well -25000, 5000 and 5000 images, respectively-.All models tested in this scenario were designed to work on individual slices, but the misalignment prediction for each volume was defined as the median of the predicted misalignments for all its slices. The use of the median reduced the influence of outliers coming from slices with less information. The Huber Loss [Huber (1964) ] was used as the loss function for the prediction of the detector horizontal misalignment for every slice during training. The 𝛿 value in the Huber Loss, which controls the slackness for outliers, was selected via grid-search within a range [0, 0.5] in 0.1 intervals. The validation performance of VGG11 yielded 0.1 as the best 𝛿 value, which was consistent with the tolerance value defined and was thus employed for training all models. The following tests have been performed:• Firstly, the shallowest and deepest pretrained models from the VGG Transfer Learning family proposed in [Simonyan and Zisserman (2014)], VGG11 and VGG19, were trained on the dataset defined. These were selected as they are some of the highest performing and most widely studied feedforward neural networks in image processing. The last fully connected layer of each classifier was modified to obtain a single regression value instead of the original 1000 class probabilities. The grayscale images were transformed into 3-channeled ones in order to match the expected input size. The AdamW optimizer was used with a cyclical learning rate whose bounds were selected by means of the Leslie Smith test with a cycle length of 4 epochs.• Then, models were generated with the Chimera Algorithm in three different searches: (1) starting from scratch with a population size of 6 models and a search length of 16 iterations, (2) starting from 4 mutated copies of the VGG11 model and a search length of 12 iterations, and (3) starting from 4 mutated copies of the VGG19 model and a search length of 8 iterations. The population sizes and search lengths were selected such that all runs would take around 48 hours to complete on our setup. No exhaustion limit was specified in any of the three cases. In all cases, only the feature extractors were mutated, keeping the classifiers constant as three fully connected layers of 4096, 4096, and 1 output features respectively with ReLU and dropout layers with a probability of 0.5 in between. The AdamW optimizer was used with a cyclical learning rate with a cycle length of 4 epochs as well. Suitable bounds for the learning rates had to be calculated before training every model using the Leslie Smith test, as in this case it did not produce stable learning rates for differently sized randomly generated models. To reduce the time required to obtain such learning rates, the search was restricted to an order of magnitude above and below those of the parent model to the one to be trained, relying on the assumption that small changes in the architecture would not yield great fluctuations in the optimum learning rate bounds. The Chimera Algorithm's output models are referred to as Scratch NAS , VGG11 NAS and VGG19 NAS .• Each of the three populations generated with the Chimera Algorithm were combined into ensembles by means of the Gaggle Algorithm. These were pruned based on the models' Mean Absolute Error (MAE) in the median prediction of the validation volumes, generating an ensemble out of each population. These ensembles are referred to as Scratch DEL , VGG11 DEL and VGG19 DEL . Moreover, in order to palliate the effect of the simple ensemble pruning strategy employed, all model combinations within each population have been tested to check the theoretical performance of an optimal ensemble generated from each population. The models were weighted based on their validation MAE, but the best ensembles were selected based on their test MAE. These optimum ensembles are referred to as Scratch Best , VGG11 Best and VGG19 Best .All methods -Transfer Learning, the Chimera and the Gaggle Algorithms-have been tested twice, selecting a different validation rodent for each of the two cross-validation partitions.  
results: 
 The main results obtained by the previously discussed approaches are presented for the CIFAR-10 image classification task and the CT geometrical misalignment characterization task in Sections 4.1 and 4.2, respectively. The complete behavior of all approaches is laid out in Appendix B.The accuracies attained in the CIFAR-10 dataset with all methods proposed are shown in Table 2 .Regarding the standalone predesigned architectures, VGG11 attained better accuracies in training, validation, and testing throughout cross-validation. The behavior of an example ensemble employing accuracy-based data resampling and model weighting is shown on Fig. 2 . The ensemble accuracy increased as more models were added, although the rate of improvement decayed rapidly. The final ensemble was pruned, further improving the ensemble performance up until fundamental models started to get removed and the accuracy lowered back to that of the best standalone one. In all cases, the pruned populations produced better results than the original ones, using less models. The maximum training, validation and test accuracies were obtained when using accuracy-based data resampling and accuracy-based model weighting, which are shown in Table 2 . For the complete results of all data and model weighting strategies, refer to Table 4 in Appendix B.The best models in the NAS setting were always found in the deep searches. However, some low performing models also appeared in deep searches due to the late exhaustion and reinitialization of some solutions, which lowered their mean population accuracy. The time required and the number Combining NAS with DEL produced better results than any of the two methods on their own, but the ensembles were not able to surpass the accuracy of the pretrained VGG11. Using the Gaggle Algorithm with failed prediction data resampling in between several shallow Chimera searches produced slightly more consistent results than simply combining the results of a single long search.The most representative results for each method in the CT artifacts scenario are shown in Table 3 for both cross-validation partitions. The performance of the models and ensembles was measured by the MAE of their volume-wise predictions. The best and worst generalizing models generated from each search are shown, as well as the ensembles -composed of at least two models-generated by pruning and the best possible model combination. The complete results are reported in Appendix B. VGG19 performed better than VGG11, and starting the search from either of the two resulted in considerably better performance than starting from scratch. All the ensembles generated with the Gaggle Algorithm outperformed the worst model of their population in test, but none outperformed the best one. 
discussion and conclusions: 
 Each method considered on the CIFAR-10 dataset attained a very similar accuracy in the validation and test partitions, as shown in Table 2 . The validation partition was representative enough of the whole task, and thus the validation accuracy could be used both for early stopping and as a model selection criterion, with no signs of overfitting. The importance of pretraining the models is highlighted by the significant difference in the accuracies attained by VGG11 depending on the weight initialization strategy employed.The accuracy of the LeNet-5 ensembles matched that of the randomly initialized VGG11, and the NAS searches commencing from scratch surpassed both of them. The combination of DEL and NAS performed consistently better than either standalone method, which is consistent with the literature [Herron et al. (2020) , Chen et al. (2021) ]. Combining the models required negligible computational requirements, the resampling of the training dataset made each epoch slightly shorter and, especially, allowed to limit the search to very shallow models which, when combined, almost rivaled the accuracy of the pretrained VGG11. This was no trivial achievement, considering that the models were created from scratch, with no data augmentation, learning rate scheduler, nor a priori guidance, using a simple NAS algorithm and in under two days of computing.Pruning was always necessary to achieve the best performing ensembles, which makes sense particularly when the learners were created via NAS because they presented a much higher diversity -and performance-within the population. Thus, further research onto how the models are combined -to only extract valuable contributions from each model, for instance by employing a Mixture of Experts approach [Chen et al. (2022) ]-and trained -to avoid the creation of uninformative models whatsoever-is crucial to further improve the performance of our systems while at the same time reducing the time and resources needed to create them.On the other hand, the models trained on the CT misalignments dataset presented a much greater tendency to overfit on the validation partition, as evidenced by the order of magnitude difference between validation and test performances in most cases throughout Table 3 . This occurred due to the relatively small dataset size compared to the problem complexity. The models trained on this dataset presented a great variance in their predictions because the validation partition failed to characterize the whole problem [Cawley and Talbot (2010) ]. In most cases, all the models generated within a NAS run performed similarly, attaining much better validation than test MAE. In these cases, their combination into ensembles, even with the theoretical optimal pruning, attained similar or even worse results. In the second cross-validation partition, there was a significant difference between the overfitting presented by some of the VGG19 NAS . This led to the VGG19 DEL generated to perform considerably worse than the best VGG19 NAS on its own. In all cases, the models that overfit the most are introduced first and their contributions are given more weight due to their higher validation accuracy.From all this, we derive that the combination of the models generated into ensembles based on their validation performance seems to be beneficial when there is a low risk of overfitting. If that is not the case, there are two possible outcomes. In the best case scenario, all models present a similar degree of overfitting, and their combination performs similarly. Otherwise, we might add models to the ensemble that smear the contribution of better generalizing ones, giving more weight to their contributions due to their higher performance in validation. One option to perform a well-founded ensemble pruning is to consider the uncertainty of the models predictions, as quantified through methods such as those explored in the literature [Abdar et al. (2021) ]. Moreover, the ensembles could be pruned based on the performance of the models on meta-validation dataset partitions, given that enough data is available. Despite not always outperforming the best individual learners, all ensembles generated generalize considerably better than the worst models on their own. Reducing the variance of the models via DEL might still prove itself a sensible approach for small dataset scenarios, considering that the test partition can be as biased as the validation one.All of these observations are carried out in two very specific scenarios, though. A single NAS method is employed, modifying only the feature extractors of feedforward Convolutional Neural Networks. No data augmentation approaches are used, and the analysis is performed on just two datasets that, although representative of the kind of problems they present, are not sufficient to characterize the wide NAS use case. The interaction with the many techniques employed in the Deep Learning workflow might vary the behavior of the ensembles and the optimal way of combining and weighting the models. Moreover, there is very limited knowledge about how to select the best NAS, DEL and ensemble pruning method for any given problem, requiring significant expertise from the final user. We call for further research on use cases such as the one presented herein, for which the Computer Vision tools at our disposal are not yet applicable.On the bright side, given that the data available is either representative of the whole task or abundant enough as to make independent model selection and ensemble pruning partitions, the combination of DEL and NAS appears to be a promising strategy. DEL can be integrated almost effortlessly to any population-based NAS framework, with negligible extra computation costs to the final user. Moreover, these methods require minimal human intervention, which makes them perfect for settings where Deep Learning experts are rarely available -like biomedical laboratories-. Coincidentally, the kind of data produced and processed in these settings tend to differ greatly from that of large natural images datasets for which powerful Transfer Learning models are readily available.In conclusion, we found that the combination of populations generated through NAS methods into ensembles only generalize better than the standalone models provided that they all present a similar degree of overfitting. Such is not always the case for models generated via NAS using very small datasets, for which building representative validation and test partitions is not trivial. In these scenarios, DEL can mitigate the influence of the worst models generated, but it does not ensure the best generalization to the test datasets. Sensible ensemble pruning must be carried out to identify and discard the heavily overfitting models in order to achieve the most out of the generated populations. 
broader impact statement: 
 We have shown that the combination of the resulting models from a Neural Architecture Search into Deep Ensembles can significantly increase their generalization capabilities, even in scenarios where the low amount of data yields an overfitting on the validation partitions. This allows the use of AutoML in scenarios where they were previously unfeasible. However, automation bias might lead non-experts in Deep Learning to fall for the luring generalization of the resulting models to the test datasets, i.e. trusting their predictions with no regards for their interpretability. Said interpretability, which is crucial in high-stakes scenarios such as medical imaging, is generally lost upon the combination of the individual models into ensembles [Kook et al. (2022) ]. Advancements on the AutoML field must be taken with caution until we are able to reliably distinguish sensible architectures from fortuitous nonsense.On the other hand, AutoML methods benefit greatly from higher computational resources, leading to a monopolization of AutoML by those with access to high computing power. Although this holds true for our Deep Ensembles scenario, the methods proposed herein allow to obtain feasible results while limiting the search to shallower models. This could reduce the computational resources required which, in turn, can both mitigate the environmental impact of AutoML and help democratize the application of Deep Learning in low-budget scenarios. 
 
 Appendix: 1. For all authors. . . (a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The scope is defined in the introduction, and the contributions are summarized in the abstract, with the problems described in both being reflected on the experiments performed.(b) Did you describe the limitations of your work? [Yes] The limitations of the work are described Section 5.(c) Did you discuss any potential negative societal impacts of your work?[Yes] The main potential negative societal impacts are assessed in Section 6.(d) Did you read the ethics review guidelines and ensure that your paper conforms to them? https://2022.automl.cc/ethics-accessibility/ [Yes] As mentioned, the main potential negative societal impacts are assessed in Section 6. The datasets did not contain any human data, thus there is no harm that could be done to non-consenting people. The figures are designed to be colorblind barrier-free. Non-gendered language has been used throughout and author names have been checked to be kept up to date.2. If you ran experiments. . .(a) Did you use the same evaluation protocol for all methods being compared (e.g., same benchmarks, data (sub)sets, available resources)? [Yes] All methods proposed have been tested in both datasets mentioned in the paper. The exact same setup has been used throughout.(b) Did you specify all the necessary details of your evaluation (e.g., data splits, pre-processing, search spaces, hyperparameter tuning)? [Yes] The CIFAR-10 dataset is used as is, while the creation and normalization of the CT dataset is explained in sufficient detail in Section 3.2.2. All the default hyperparameters for the algorithms are mentioned in the Appendix A.(c) Did you repeat your experiments (e.g., across multiple random seeds or splits) to account for the impact of randomness in your methods or data? [Yes] Cross-validation was used in bot scenarios. In the CT application, the methods were only cross-validated twice, but the results were consistent in all cases.(d) Did you report the uncertainty of your results (e.g., the variance across random seeds or splits)? [Yes] The variance is explicitly mentioned in all results.(e) Did you report the statistical significance of your results? [No] This work focuses on the empirical testing of DEL and NAS to a particular scenario, rather than the statistical differences between the methods or models tested. Moreover, the differences in the metrics attained are significant enough and the results are consistent with those reported in previous literature.(f) Did you use tabular or surrogate benchmarks for in-depth evaluations? [N/A] Ensembles of models cannot be validated on surrogate benchmarks for standalone models. Moreover, there are very few surrogate benchmarks for datasets with a low number of complex images to depict the medical imaging scenario, which is the main focus of this work.(g) Did you compare performance over time and describe how you selected the maximum duration? [N/A] The performance for shallower and deeper searches has been assessed. The maximum duration was arbitrarily selected based on computational resources, but the main point of this work is not to propose a specific, optimized method but to assess the interaction between Ensemble Learning and Neural Architecture Search in the context of the aforementioned small dataset regime.(h) Did you include the total amount of compute and the type of resources used (e.g., type of gpus, internal cluster, or cloud provider)? [Yes] setup used has been specified in the introduction of Section 3. The compute times were reported along with every result.(i) Did you run ablation studies to assess the impact of different components of your approach?[Yes] The DEL and NAS methods have been validated separately and in conjunction in different settings to assess their individual contribution to the complete approach.3. With respect to the code used to obtain your results. . .(a) Did you include the code, data, and instructions needed to reproduce the main experimental results, including all requirements (e.g., requirements.txt with explicit versions), random seeds, an instructive README with installation, and execution commands (either in the supplemental material or as a url)? [Yes] The code developed and the environment.yml/requirements.txt required, as well as the data generated for the CT misalignments scenario, can be accessed through this link: https://github.com/HGGM-LIM/Chimera. The code is given as jupyter notebooks with instructions and comments throughout. The random seeds are given within the code.(b) Did you include a minimal example to replicate results on a small subset of the experiments or on toy data? [No] The whole results can be replicated with the code provided. To produce a minimal example, it suffices to eliminate samples from the dataloaders to suit the computational requirements of the reader.(c) Did you ensure sufficient code quality and documentation so that someone else can execute and understand your code? [Yes] The code is provided in the form of jupyter notebooks with instructions and comments throughout. (c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] No human data was employed. The models are trained until convergence in order to properly compare their performance on the validation partition. Each model generated is then assigned a score based on the Artificial Bee Colony fitness value, given by (1 + 𝑙𝑜𝑠𝑠 𝑣𝑎𝑙𝑑𝑎𝑡𝑖𝑜𝑛 ) -1 . This score is used to compare the original and mutated models and discard the worst performing one. The 𝐵𝑒𝑒𝑠 𝑂𝑛𝑙𝑜𝑜𝑘𝑒𝑟 will also use this score to select models.The hyperparameters of the Chimera Algorithm are: the condition to stop the search 𝑠𝑡𝑜𝑝_𝑐𝑟𝑖𝑡𝑒𝑟𝑖𝑜𝑛 -either a maximum number of iterations or a threshold for the objective loss function-, the population size 𝑁 𝑝, the model exhaustion limit 𝑒𝑥 𝑚𝑎𝑥 , the arguments for the initialize function 𝑖𝑛𝑖𝑡_𝑎𝑟𝑔𝑠, and the arguments for the mutate function 𝑚𝑢𝑡_𝑎𝑟𝑔𝑠. The latter comprises the number of mutations per mutate call, the probability of performing each mutation type -that is, adding, subtracting or modifying each kind of layer-, and, optionally, the bounds to the model hyperparameters' space -the maximum number and types of layers, kernel sizes or strides, or the range of learning rates-.The default hyperparameters are defined as follows. The number of mutations performed on each exploration attempt is defined as |𝑁 (1, 3 √ 1 + 𝑒𝑥ℎ𝑎𝑢𝑠𝑡𝑖𝑜𝑛 𝑖 )| rounded upwards, to ensure that most of the time we perform very few mutation steps. Bigger steps to overcome local minima are only allowed if the solution is close to exhaustion. In this way, we make sure that the close neighborhood of a given solution is properly exploited before trying to reach further away. The probabilities of adding, deleting, or mutating a layer on each exploration step are set to 30% each, while there is a 10% probability of simply resetting the weights of some layers while leaving the architecture intact. When adding or mutating a layer, its probability of being a convolutional layer was set as 5𝑛 𝑝 /(5𝑛 𝑝 + 𝑛 𝑐 ), where 𝑛 𝑝 is the number of pooling layers and 𝑛 𝑐 that of convolutional ones in the model to mutate. Otherwise, the new layer is a pooling one, with equal probability of being either max or average pooling. This ensures that our models will tend to present 5 times as many convolutional layers as pooling ones. Convolutional and pooling kernel sizes are drawn from a uniform distribution from 1 to 7, and the model length is only limited by the available GPU memory. When adding a convolutional layer, the input and output channels are given by those of the surrounding layers. By default, when mutating a convolutional layer, the number of output channels -and thus input channels of the next convolutional layer-can vary from half to twice the original number of channels. Model exhaustion is deactivated by default, with 𝑒𝑥 𝑚𝑎𝑥 being infinity. If no base architecture is provided, the models will be initialized at random within the search bounds. There are no default 𝑠𝑡𝑜𝑝_𝑐𝑟𝑖𝑡𝑒𝑟𝑖𝑜𝑛 as this is specific to every problem and available resources.The stopping criteria and the search bounds are the hyperparameters that most significantly affect the quality of the models produced. The initialization arguments, probability of each mutation type, population size and model exhaustion limit simply define the starting point, preferred direction and average search speed throughout the hyperspace. For instance, 1) keeping a higher probability of removing layers rather than adding them yields a search focused on decreasing model complexity, or 2) using a small population size with a high exhaustion limit favors a thorough exploitation of a few regions rather than widespread exploration. The optimal hyperparameters will be problemdependent, and could be either fixed or adaptive, leveraging a-priori knowledge provided with the quality of the models found throughout the search. AutoML 2024© 2024 the authors, released under CC BY 4.0 


Decision: The authors did a good job responding to the reviews. While there's still some disagreement among the reviewers, the positives now outweight the negatives.

Reviews: review: paper_summary: paper_summary: The paper presents a comprehensive study on enhancing transfer learning models using Deep Ensemble Learning (DEL) and Neural Architecture Search (NAS) driven by swarm intelligence. It addresses the challenges faced by NAS in very small or complex datasets, such as overfitting during validation, by integrating DEL to mitigate bias and variance in predictions. The study evaluates the combined approach of NAS and DEL on two distinct scenarios: CIFAR-10 image classification and misalignment regression in CT slices with artifacts. It demonstrates that ensembles, particularly when combined with NAS, significantly outperform individual models and pre-designed architectures, especially in handling small or complex datasets typical in medical imaging applications. This is achieved by promoting model diversity and focusing on specific parts of the task through ensemble methods, effectively reducing the computational expenses while maintaining prediction complexity.
main_review: main_review: The paper "Improving Transfer Learning by means of Ensemble Learning and Swarm Intelligence-based Neuroevolution" presents an intriguing approach to enhance transfer learning by integrating Deep Ensemble Learning (DEL) and Neural Architecture Search (NAS), particularly focusing on the context of small and complex datasets. The paper is heavily empirical, it builds on established theories of ensemble learning, transfer learning, and neural architecture search. That said many sections require additional work. 

If the weakness identified are resolved, this can become an accept!
review_summary: review_summary: More clarity for the experiments conducted are required to provide a comprehensive review.
paper_summary: paper_summary: The paper  proposes a combination of Deep Ensemble Learning (DEL) and Neural Architecture Search (NAS) for small datasets, where NAS typically overfits. On two types of tasks the combination of DEL and an in-house NAS optimizer improves results compared to  stand-alone models.
main_review: main_review: The paper is primarily a case study on how one specific NAS method produces questionable results through presumed overfitting, and how DEL in conjunction with the specific NAS method yields improvement over the two in isolation. Thus, the learnings are thus very niche. Unfortunately, I do not see there being a substantial technical contribution (e.g. the development of a novel algorithm or fundamental learnings related to NAS or DEL). 

The paper lacks clarity in multiple areas, such as the technical description of the method, and the empirical results.
review_summary: review_summary: See above.
paper_summary: paper_summary: The paper examines the combination of neural architecture search and ensembling. It does so on two applications - a standard data set, as well as a rodent imaging data set. It uses an in-house NAS method, combining it with ensembling for more robust results.
main_review: main_review: The authors wrote a thoughtful ethics statement, which is a big plus. They also reflected about the limitations of the paper (second to last paragraph).
review_summary: review_summary: The paper includes a thoughful consideration of its limitations and ethical implications, and brings in a less-studied and important application. However, I have concerns about the reproducibility, as well as finding some of the paper hard to follow (see clarity).

Edit: I have updated my rating from 'Weak Reject' to 'Borderline Leaning Accept' following the author responses.
paper_summary: paper_summary: This work proposes evolutionary NAS and NAS+DEL algorithms and analyzes them in the context of a simple standard baseline (CIFAR10) and a biomedical transfer learning task.
main_review: main_review: The introduction is currently very long. If it is split into a shorter introduction and a related works and/or background section, your readers can navigate it more easily.

As all algorithms are in the appendix, please describe each one (at a high level) more in your methods section. What the algorithms are doing is much more important to include in the main paper than, for example, the hyperparameters and data preprocessing (which are details usually put in the appendix).

Minor comments (not counted negatively):
* The citation style seems odd. Use `\citep` for parenthetical citations, rather than manually adding square brackets. 
* The writing switches between past tense ("was", "were") and present tense ("is", "are"). Try to be consistent with the tense.
* When linking to a URL, please put the full URL rather than hiding it with text.
* Line 59: "...differ the greatest..." -> "...differ greatly..." 
* Line 71: "...with the use DEL..." -> "...with the use of DEL..."
* Line 110: "The performance of DEL, NAS, and its combination was..." -> "The performances of DEL, NAS, and their combination were..."
* Line 151: "...tests we performed..." -> "...tests were performed..." 
* Figure 1: there is no red indication as mentioned
* Table 2: "Comuting time" -> "Computing time"
review_summary: review_summary: This work proposes some algorithms that may be useful and impactful, but the experimental flaws hinder the validity of the claims made.