Bractivate: Dendritic Branching in Medical Image Segmentation Neural Architecture Search
Authors: Leila Abdelrahman
Venue:  (2021)
Keywords: Neural Architecture Search
Segmentation
Computer Vision

Abstract: Researchers manually compose most neural networks through painstaking experimentation.
This process is taxing and explores only a limited subset of possible
architecture. Researchers design architectures to address objectives ranging from low
space complexity to high accuracy through hours of experimentation. Neural architecture
search (NAS) is a thriving field for automatically discovering architectures
achieving these same objectives. Addressing these ever-increasing challenges in
computing, we take inspiration from the brain because it has the most efficient
neuronal wiring of any complex structure; its physiology inspires us to propose
Bractivate, a NAS algorithm inspired by neural dendritic branching. An evolutionary algorithm that adds new skip connection combinations to the most active blocks in the network, propagating salient
information through the network. We apply our methods to lung x-ray, cell nuclei
microscopy, and electron microscopy segmentation tasks to highlight Bractivate's robustness.
Moreover, our ablation studies emphasize dendritic branching's necessity: ablating
these connections leads to significantly lower model performance. We finally
compare our discovered architecture with other state-of-the-art UNet models,
highlighting how efficient skip connections allow Bractivate to achieve comparable
results with substantially lower space and time complexity, proving how
Bractivate balances efficiency with performance. We invite you to work with our
code here: \href{https://tinyurl.com/bractivate}{\textcolor{violet}{https://tinyurl.com/bractivate}}.

Under review as a conference paper at ICLR 2021
BRACTIVATE: DENDRITIC BRANCHING IN MEDI- CAL IMAGE SEGMENTATION NEURAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Researchers manually compose most neural networks through painstaking exper- imentation. This process is taxing and explores only a limited subset of possible architecture. Researchers design architectures to address objectives ranging from low space complexity to high accuracy through hours of experimentation. Neural architecture search (NAS) is a thriving field for automatically discovering archi- tectures achieving these same objectives. Addressing these ever-increasing chal- lenges computing, we take inspiration from the brain because has the most efficient neuronal wiring of any complex structure; its physiology inspires propose Bractivate, a NAS algorithm inspired by neural dendritic branching. An evolutionary algorithm that adds new skip connection combinations to the most active blocks in the network, propagating salient information through the network. We apply our methods to lung x-ray, cell nuclei microscopy, and electron mi- croscopy segmentation tasks to highlight Bractivate's robustness. Moreover, our ablation studies emphasize dendritic branching's 's necessity: ablating these con- nections leads to significantly lower model performance. We finally compare our discovered architecture with other state-of-the-art UNet models, highlight- ing how efficient skip connections allow Bractivate to achieve comparable results with substantially lower space and time complexity, proving how Bractivate bal- ances efficiency with performance. We invite you work with our code here: https:/tinyuufl.combbrriiaall
INTRODUCTION
Researchers manually composing neural networks must juggle multiple goals for their architectures. Archi- tectures must make good decisions; they must be fast, and they should work even with limited computa- tional resources. These goals are challenging achieve manually, and researchers often spend months attempting discover the perfect architecture. To overcome these challenges, we turn to the human brain's efficient neural wiring for automated architecture discovery. Neuroscience already underlies core
Figure 1: Through Bractivate, we discover UNet architecture with high spatio-temporal efficiency by mimicking the brain's 's dendritic branching.
neural network concepts: The perceptron |Rosenblat| 1958) is directly analogous to human neuron. One of the brain's fundamental learning mechanisms is dendritic branching (Greenough & Volkmar 1973) whereby active neurons send out signals for other neurons to form connections, strengthening signals through that neural pathway. This neuroscience concept inspires us to devise Bractivate, a Neural Architecture Search (NAS) algorithm for learning new efficient UNet architectures networks, capable of being trained twice fast as the traditional UNet, and often one

---

Under review as a conference paper at ICLR 2021
to two orders of magnitude lighter in terms of trainable parameters. We apply Bractivate three medical imaging segmentation problems: cell nuclei, electron microscopy, and chest X-ray lung segmentation.
Medical image segmentation is a growing field in Deep Learning Computer Assisted Detection (CAD): it is powerful component in clinical decision support tools and has applications in retinal fundus image, lung sean, and mammography analysis. Most papers now approach medical image segmentation with the UNet (Ronneberger et al. 2015); the model architecture is straightforward: it has symmetric, hierarchical convolutional blocks, which are components of an initial contracting path and final expanding path. with an apex bottleneck layer. Between parallel contracting and expanding blocks. the traditional UNet contains skip connections that pass information through concatenation (Ronneberger et al., 2015). Traditional UNet skip connections involve feature map aggregation with same-scale convolutional blocks, but recent advances have yielded more complex connections ranging from the UNet++ (Zhou et al. 20 to the NasUNet (Weng et al. 2019)). While the UNet is powerful tool, it does have many limitations:
1. The depth necessary for many segmentation tasks is initially unknown, and traditional neu- ral architecture search (NAS) struggles to identify the optimal UNet depth.
2. Researchers often manually choose skip connection locations, leading to potentially missed optimal connections.
3. Scientists need NAS algorithm addressing many implementation objectives, including computational time. number of model parameters, and robust segmentation performance.
On broader level, discovering efficient UNet architectures is crucial because it can generate simpler models for applications on mobile devices, which need low latency for online learning. In the Télemedicine age, many medical applications rely mobile Deep Learning to segment medical images and process raw patient data (Xu et al.] [017]] Vaze et al.) (020). We address the Medical and Engineering fields' need for efficiency with Bractivate, NAS algorithm discover lightweight UNet architectures for medical image segmentation tasks. We present the following three primary contributions:
An evolutionary algorithm that non-randomly samples from a distribution of various UNet Model depths and skip connection configurations, with both tensor concatenation and ad- dition operators.
2. "Dendritic Branching" iinspired mutations that, just as in the brain, cause salient UNet blocks to branch to other blocks in the network through dendritic skip connections, cre- ating efficient networks that preserve information signals through the network.
3. Bractivate generates high-performing models with lower space complexity than the current state-of-theeart.
The remainder of the paper structured as follows: Section 2 we discuss prior works, and what gaps in the literature inspire us to propose Bractivate. Then, in Section [ we discuss the search algorithm and the dendritic branching mutation. Later, Section 4 we implement our algorithm with various experiments ranging from changing the search space depth to an ablation study. We report our quantitative and qualitative results, along with baseline comparisons in Section before concluding in Section
RELATED WORKS
Deep learning algorithms are often restricted to manual model design (Simonyan & Zisserman 2014 He et al. 2016;; Oktay et al., 2018 Ronneberger et al.) 2015). To automate model schemes, NAS is the process of selecting candidate architectures through various search strategies to achieve optimal performance (Elsken et al.) 2019). Advances in NAS have branched into different areas, including evolutionary algorithms Miiler et al. 1989 de Garis 1990; Yao 1993 Fogel et al. 1990. Angeline al. 1994, Real et al.) 2018; Yao, 1999) and automatic pattern recognition (Cai et al. 2018 Radosavovic et al. (0220). While both approaches are merited, the tasks address image clas- sification problems, and although some focus on skip connections, they lack deeper investigation
2

---

Under review as a conference paper at ICLR 2021
of their optimal configurations. Recent advances in the UNet have led to alternative skip connec- tion implementations, including addition (Ghamdi et al. 2020), max out operations (Estrada et al.) 2019 Goodfellow et al. 2013) and multiplication by gating function (Oktay et al. 2018). Ghamdi et al. (2200) reports these connections' improved efficacy over traditional concatenation, as they overcome vanishing gradients and preserve salient features.
Auto-DeepL.ab, which Liu et al. 2019) present for semantic segmentation, is a graph-based NAS algorithm that addresses changing model depth and connection locations in hierarchical models. Building off this work. Zhou et al. (2020) propose a similar graph-search algorithm, termed UNet++, for improved NAS; the final model incorporates dense skip connections achieve multi-scale fea- ture aggregation. Although UNet++ successfully addresses the model depth problem, it ignores choosing the skip connection operator and relies on pretraining and pruning to generate skip con- nection configurations.
The Differential Architecture Search (DARTs) algorithm by Liu et al. (2018) continuously relaxes the architecture representation to enable gradient-based optimization. Advancing this algorithm, Chen et (2019) proposes the Progressive Differentiable Architecture Search Algorithm (PDARTs) to allow the searched model's depth to grow during the search; when applied to ImageNet (Deng ct al. 2009). CIFAR- 10 (Krizhevsky et al. 2009), or CIFAR- 100 (Krizhevsky et al. 2009), the total training time is approximately seven hours.
Although the DARTS and PDARTs algorithms are specific to image classification and sequential model architecture, they lack applications for segmentation models. [Weng al. (2019) suggest NASUNet method with modified DART' search for medical imaging segmentation; their approach addresses searching for model parameters in the convolutional blocks to reduce the space complex- ity found in attention-based (Oktay et al., 2018 Hy 2018) and recurrent (Alom et al.] 2018 Hy
2018) UNets. yet NASUNet still preserves same scale concatenation skip connections, overlooking alternate skip connection possibilities across network blocks.
Many existing NAS algorithms use modified objective functions for evaluating the searched model performance e.g. NAS-Bench-101 101 (Ying et al., 2019) uses the cross-entropy loss, Stochastic Neural Architecture Search (SNAS) (Xie al., 2019) devises a cost function deemed Memory Access Cost (MAC) that incorporates the floating ppintt operations (FLOPs) and number of parameters, and PDARTs (Chen et al. 2019) employs auxiliary loss (Szegedy et al.) (014)). To target gaps in the literature related to skip connection search for efficient models, we propose Bractivate, NAS algorithm inspired by the brain's 's dendritic branching to facilitate optimal architecture discovery.
3 THE BRACTIVATE NAS ALGORITHM
3.1 DENDRITIC ARBORIZATION
Table translates neuroscience into computational terms we use throughout the paper. In the neuro- science field, dendritic branching occurs when stim- ulating environments cause neurons to form new connections (Greenough & Volkmar 1973 Gree- nough et al.] 1985). These neural connections are associated with learning, and even learning-impaired children with fetal alcohol syndrome display lower dendritic branching levels (Hamilton et al.] 2010) compared to their healthy peers. This branching phe- nomenon parallels Deep Neural Networks: in the brain, dendrites form new connections to the hyper- active soma: the perceptron's activation function to the biological soma the incoming connections to dendrites. Perceptrons be stacked together to form multi-layer perceptrons (Rumelhart et al.) 1986), with parallel architecture similar to the brain, and this structure underlies convolutional neural networks (LeCun et al. 1995).
Table 1: Defining from neuroscience we use and their parallels in Deep Learning.
Neuroscience
Deep Learning
Soma Dendrite
Layer in block Input Conncction
Axon
Sender output Connec- tion Connection to first layer in active receiving block
Dendritic Branch
For the UNet, we consider layer in the network's blocks to be a neural soma, then we can think about block's "activity" as the mean-absolute value of its layers' activations, as shown by Equa-
3

---

Under review as a conference paper at ICLR 2021
Frodunic Performmne
licntity Adtivity
C) Resnching
M
Bas M
But 200 2M
Addition
M Laye: Dimensoon
F- Filters
Low Activasion
Activatioo
Figure During each search iteration, Bractivate chooses model from the randomly initialized search main. After we train on the dataset, we evaluate its performance. To mutate, we identify the most active block in the "best model" architecture, as per Equation [ to that active block, we initialize new skip connections (branches) pointing from other blocks in the network to the highest-activation block, propagating their signals through randomly chosen branches.
tion |
A I t=
A
(1)
where A, represents block activation, b E B, Ar is the weight of each layer in the block, L, and L is the total number layers in the block. Knowing the block's location, b, with max(As), surrounding blocks then form skip connections around this active node, process analogous dendritic branch- ing. We apply this method to target conv and deconv layers, excluding batch normalization layers they contain static weights and high values that overwhelm the mutation's layer selection. When new connections are formed, across blocks with various tensor dimensions, we overcome spatial di- mensional mismatch by resizing the incoming connection tensors the receiving tensor by bilinear interpolation.
3.2 NAS WITH DENDRITIC BRANCHING MUTATIONS
We implement our dendritic branching mutations to our evolu- tionary algorithm based adding dendritic branching mutations to randomized search domain, as
displayed in Figure 3 This muta- tion applies mutation operator to form new skip connection combi- nations the active block in the network. This NAS algorithm initializes random domain with n model genotype. Each genotype codes for the model's depth, It also encodes the number of filters per conv and deconv layer, the degree of skip connections for each block, and the skip connection operator type (concatenation or addition). A detailed discussion on the genotype initialization and its micro-architecttuee found in Appendix A. When
Sample
Desaltic:
Randonly
Iriraline Dorais
Figure 3: Bractivate samples from randomly initialized domain, constrained by model depth, The efficiency evaluator compares a selected model with the current "hest model" genome. If the "best model" outperforms the current model, mutate the "best model" (Figure [ and replace the chosen model with the mutated version in the search space.

---

Under review as a conference paper at ICLR 2021
initializing the genotype, constrain the number of feature maps grow by 1.5 in the encoder and decrease it by the same factor for each block in the decoder.
3.3 EFFICIENT LOSS
NAS methods often focus on accuracy as the main performance metric (Real et al. 2018 Ra-
dosavovic et al. 22200), but often lack consideration for discovered model space and time complexity. To address this, we proposc an efficient loss function for the NAS evaluation step. Traditional binary cross-entropy is given by Equation 2 During the NAS mutation and selections, the search process accelerates as "better" models have faster training steps.
1 y log(ý;) + y;) log(1 yr))
m
BCL
m
where m is the number of samples, y, is the sample image's true segmentation mask tensor, and y; the model's prediction tensor. We propose alternate efficient loss function equation, Efficient Loss Scaling (ELs). It uses the number of model parameters, P. and the training time per epoch, T.
EFFICIENCY LOSS SCALING
We also explore Efficiency penalty scaling where log(P) and log(T') scale the overall loss function through multiplication, hence:
ELg = 7 x log( P) x log(T) x BCL
In our experiments we set 7 = 0.01. We use Equation in Section 4.4 during model search. A detailed ablation study how this equation favors efficient networks and outperforms standard BCL can be found in Appendix A.33
4
EXPERIMENTS
4.1
IMPLEMENTATION
For the following experiments, we execute our programs using TensorFlow 2.3.0 and Keras 2.4.3, using Python 3.6.9. For training the models and inference, we use virtual machine with 25 GB of RAM, 150 GB of disk space, and one Tesla v-100 GPU.
4.2 THE DATA
To validate Bractivate NAS, we work with three datasets:
1. The Montgomery Pulmonary Chest X-Ray (Jaeger et al. 2014) dataset (N = 138)
2. The EPFL CVlab Electron Microscopy (Lucchi et al. 2011) dataset (N=330)
3. The Kaggle Data Science Bowl Cell Nuclei Segmentation Challenge CCacceoo et al. 2019 dataset (N=670)
detailed discussion of the dataset follows Section
4.3 PREPROCESSING
We resize all images in the three datasets to 128x 128 pixels with bilinear interpolation, and later min-max normalize the pixels, that pixel intensities are [0, 1]. We apply the preprocessing to the image masks. We use a 0.1 validation split percentage on all three datasets to evaluate the optimized model loss function and the Dice coefficient during the search.
5

---

Under review as a conference paper at ICLR 2021
4.4 PENALIZING TIME AND SPACE COMPLEXITY
We experiment with using the standard binary BCL loss function in Equation and the ELs function in Equation We record our results both with the Dice Score, representing the precision and recall harmonic mean. We note that these loss functions used only during the search step- During full model training, we use the BCT loss on the "best model" and use that for our later experiments in Sections 4.7 and 4.8
4.5 DISCOVERING ARCHITECTURE
For all three datasets, use the same Bractivate search algorithm based dendritic branching. We initialize our search domain as queue with 20 randomly generate model genotypes and initialized generated models from the genotypes; we then train the models for 50 epochs on the datasets with Early Stopping. The Early Stopping has patience of 10 and min A of 0.05. In a first-in-firstouut (FIFO) fashion, we evaluate each genotype in the queue: if the generated model has E then it becomes the "Best Model," and we place this genotype back in the queue. Suppose it has EL > Ebmin- that case, search algorithm mutates the "Best Model" genotype with dendritic branching method described in Figure 2 before replacing the mutated genotype in the search queue.
4.6 GOING DEEPER
We notice that the original Bractivate search algorithm with a minimum depth of two yields mainly shallow networks, usually with two three expanding and contracting blocks. To explore how model depth affects the search algorithm's 's results, we constrain the search space such that depth is [5, 10] and later [7, 10). and observe how the Dice coefficient and Dice:time ratio change while using the ELg (Equation 3
4.7 ABLATING BRANCHING CONNECTIONS
We hypothesize that active block branching increases signal propagation, reducing time complexity, and improving overall model performance. Thus, must prove these new branches necessary for the model's success by ablating them and measuring how their absence affects the model perfor- mance. We perform these experiments by training the selected model architecture for 200 epochs on the dataset, and use the Keract (Remy) 2018) library to measure layer activations active layer (the second convolutional layer in the decoder of we calculate the layer's average activation from Equation 1 and then ablate all dendritic (input) con- nections of the most active block. We record the results quantitatively with the Dice coefficient and visualize them by analyzing changes in the activation heat maps.
the most deep UNet. For each layer,
4.8 BASELINE COMPARISON
We compare Bractivate models to other state-of-the-art methods, including the standard UNet model Ronneherger al.] 2015), Wide-UNet (Zhou al.) (008). UNet++ (Zhou et al.] 2020), and attention-based based models. We obtain all model architectures from GitHub repositories made by these models' authors or by contributors who create implementations Keras and use them with Xavier weight initialization the three datasets for comparison.
RESULTS
We observe clear patterns pointing to how dendritic branching allows for efficient neural network selection through experimentation. Figure presents example of the discovered architecture with the ELS function.
5.1 DISCOVERED ARCHITECTURES

---

Under review as a conference paper at ICLR 2021
Because the mutations add more con- nections to salient blocks, the most ac- tive node has the highest input connec- tions. Bractivate randomly assigns addi- tion concatenation connections the most active block, optimizing the best connecting operator combination.
Figure E shows the discovered archi- tecture when the search space is con- strained such that D E [5. 10). Al Ghamdi el al. (Ghamdi et al. 2020) re- port that the addition operator can per- form as well or better than the concate- nation operator,
Addition
Concatenation
Figure 4: A sample of the discovered architecture through the efficiency loss scaling in Equation [] The green block is the most active block, having the highest number of incoming den- dritic connections.
we incorporate both operators as modules into the NAS, with each mutation yielding new random connecting operator combination to the salient block.
5.2 DEPTH AND TIME-SPACE COMPLEXITY
In Figure we observe negative correla- tion between the training time per epoch and the Dice coefficient, indicating that Bractivate favors shallower over deeper networks, as the skip connections branch- ing towards active blocks carry the nec- essary information for the segeentation task. This slope varies between datasets; a simple problem like segmenting two- component lungs confers an overall larger Dice: time ratio than the cell nuclei and electron microscopy tasks. These prob- lems are challenging because they have more than two connected components, yielding lower Dice: time ratios. More importantly, our methods determine that given the early stopping constraint, shal- lower models (D = 2) with fewer pa- rameters have comparable performance to deeper models. When UNet models are too deep (D [0]), the input signal may be lost in transformations, leading to lower performance; shallower models pre- serve this signal.
DV
Nadel
segmentation
Search
100
300
100
Paruza
3.8
26.5
63
Time Spock
Thme par
Spedi
Fi Comparing the effects of model depth on the corre- lations between the Dice coefficient performance metric and different depth constraint over three trials. Top: Dice time per epoch; Middle: Dice vs. number of parameters (spatial complexity) Bottom: Dice vs the number of model params.
5.3 SKIP CONNECTION ABLATION STUDY
Our ablation study most strongly confirms our hypothesis that dendritic branching to active blocks significantly improves segmentation performance: Figure 6 examines the saliency maps produced by the model architecture in Figure before and after ablating connections to the most active block. Before ablation, the most active block's 's saliency maps show encoded information that significantly influences the decoded feature maps during deconvolution.
After ablation, the saliency maps for the EM and nuclei segmentation tasks lack accurate late-stage saliency maps. When the salient block has configured dendritic branches from neighhoring blocks,
7

---

Under review as a conference paper at ICLR 2021
Part Chonn
Faast Cance
Part Coont
Fast
Denth
Flst
Derth
0:550
0L000
FM
Ablacca
Searchel
0,859
Nucle
Figure Activation map comparison for the second deconv block's second convolutional layer for the archi- tecture described in Figure Blue represents low saliency, and red represents high saliency. We ablate most salient block (The green block containing Decode2 Conv2) in the network and display activation differences after ablation.
the output signal highly accurate. However, when these vital encodings in the Decode block lack input from neighboring blocks, the output signal is degraded. This degradation is especially true for the EM and nuclei segmentation tasks.
The EM and nuclei segmentation tasks contain more than two connected components; removing dendrites to salient blocks prevents valuable information from neighboring blocks to travel to the most salient block, degrading the signal through the last four blocks in the network. The model's Dice score is significantly lower in the ablated architecture than in the intact practivate-selectee model. The added information from these dendritic skip connections, explicitly targeting salient block in the model, generates more accurate saliency maps, helping the model learn faster. Before ablation, activations concentrates in the encoder. This observation may be because removing connections towards an active block forces surrounding layers compensate by increasing their activations.
more salient during the decoding phase than post-ablation, where saliency
5.4 BASELINE COMPARISON
Table 2:
Comparing the spacial complexity of Bractivate with various state-of-the-an UNet architectures. The top group represents manually -esigled models. The middle row comprises differentiable search. The bottom is ours, based dendritic branching. We report the results "Dice (# Params)."
Model
Lung
EM
Nuc
UNet |Ronncherger et
2015
0.925 (7.7c6)
0.811 (7.7c6)
0.833 (7.7e6)
R2-UNet Alom ct al.
2018
0.596 (9.5c7)
0.464 (9.5c7)
0.049 (9.5c7)
Atin-UNet (Oklay
2018
0.954 (3.2e7)
0.937 (3.2e7)
0.721 (3.2e7)
UNet++ Zhou al.
2018)
0.903 (9.0e6)
0.846 (9.0e6)
0.841 (9.0e6)
WideUNet (Zhou al.
2018
0.888 (9.3e6)
0.811 (9.3e6)
0.828 (9.3e6)
NasUNet Weng al.
2019)
0.934 (1.2e5)
0.729 (4.8e5)
0.774 (1.2e5)
Bractivate
0.942 (3.1e4) 0.929 (4.8e5)
0.878 (4.8e5)
Figure highlights how Bractivate achieves comparable performance larger models when ini- tialized with Xavier initialization (Glorot & Bengio 2010). Table 2 highlights how Bractivate is significantly smaller than many of the other state-of-the-art models: it exchanges high spatial com-
8

---

Under review as a conference paper at ICLR 2021
plexity for more skip connections, these branches allow information to propagate through salient blocks in the network. For domain-speciffr tasks, high parameters reduce the signal: noise ratio in the network; simpler models like Bractivate rely powerful skip connections, analogous to den- drites, to carry most of the signal. Because these connections consist of simple concatenation or addition operators, they greatly reduce the number of trainable parameters, preventing overfitting; this speaks to Bractivate's s comparable-or better-Dice scores as compared to the baseline models.
Lungs
Nuckei
UNet
R2-UNet
AtnuNeet
0.6
WideUNet
0.2
0.2
6.2
NosUNet
Bructivate
0.0
0.0
6.0
6.5
7.0
7.5
0.0
3.0
5.5
6.0
6.5
7.0
7.5
8.0
Log
Log Porams
Log
Figure 7: Comparing model performance measured by the Dice score as function of different model archi- tectures. Note that these models are all Xavier initialized.
6 CONCLUSION
Throughout this paper, highlight how dendritic branching in the brain inspires efficient skip connections in Deep Learning models. With our focus on segmentation, we present Bractivate as method for identifying skip connection configurations search, Bractivate mutates the architecture so that the most salient blocks in the network branch out their "dendrites" other network blocks. By replacing the oldest model in the search space with the new mutated architecture, we accelerate the search rate.
elevate the traditional UNet. During the
The ablation study strongly supports our hypothesis that dendritic branching is necessary for efficient model discovery; when we ablate dendritic connections to the most salient block, the Dice Score decreases. Before and after ablation, the saliency maps reveal stark contrasts, with the ablated activation maps lacking apparent features for the final segmentation layer in the UNet's decoder. We finally weigh our methods with other baselines, highlighting how smaller networks can perform segmentation tasks well given limited pretraining data.
Overall, we present how optimally configured skip connections, inspired by the brain, yield robust signal streaming paths through lightweight network. Our algorithm is an asset to many mobile medical computing technologies that rely on low latency and high computational efficiency.
REFERENCES
Md Zahangir Alom, Mahmudul Hasan, Chris Yakopcie, Tarek Taha, and Vijayan Asari. Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation. arXiv preprint (i0i:/800206953 2018.
Peter J Angeline, Gregory Saunders, Jordan Pollack. An evolutionary algorithm that constructs recurrent neural networks. IEEE transactions on Neural Nenworks, 5(1):54-65, 65, 1994.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. AAAI, 2018.
Juan C Caicedo, Allen Goodman, Kyle W Karhohs, Beth A Cimini, Jeanelle Ackerman, Marzieh Haghighi, CherKeng Heng. Tim Becker, Minh Doan, Claire McQuin, al. Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nature methods, 16(12): 1247-123,, 2019.
X. Chen, L. Xie. J. Wu, and Q. Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In 2019 TEEF/CVF International Conference Computer Vision (ICCV), pp. 1294-1303, 2019.

---

Under review as a conference paper at ICLR 2021
Hugo de Garis, Genetic programming: Modular evolution for darwin machines. In Proceedings of the 1990 International Joint Conference Neural Nenworks, pp. 194-197, 1990.
J. Deng, W. Dong, Socher, I.. Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image datahase. 2009 IEFE Conference on Computer Vision and Pottern Recognition, pp. 248-255, 2009.
Thomas Elsken, Jan Hendrik Metzen, Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21,200 2019.
Santiago Estrada, Ran Lu, Sailesh Conjeti, Ximena Orozco-Ruie Ruiz. Joana Panos Whils-Willuhn, Monique M. B. Breteler, and Martin Reuter. Fatsegnet A fully automated deep learning pipeline for adipose tissue seg- mentation ahdominal dixon MRI. CoRR, ahs/1904.02082, 2019. URL http:://axvvv.o///aa//
1904. 02082
David B Fogel, Lawrence J Fogel, and VW Porto. Evolving neural networks. Biological cybernetics, 63(6):
487 493, 1990.
M. A. Ghamdi, M. Abdel-Mottalch, and F, Collado-Mesa. Du-net: arterial calcifications in mammograms. IEEE Transactions on Medical Imaging, pp. 1-1, 2020.
Convolutional network for the detection of
Xavier Glorot Yoshua Bengio. Understanding difficulty training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249- 256, 2010.
Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Y. Bengio. Maxout networks. 30th International Conference on Machine Learning, /CML 2013, 1302, 02 2013.
William T Greenough and Fred R Volkmar. Pattern of dendritic branching in occipital cortex of rats reared in complex environments. Experimental neurology, 40(2):91-5504 1973.
William Greenough, John R Larson, and Ginger Withers. Effects of unilateral and bilateral training in reaching task dendritic branching of neurons in the rat motor-sensory forelimb cortex. Behavioral and neural biology, 44(2):301-314 1985.
Gillian Hamilton, Lee Whitcher, Anna Klintsova. Postnatal binge-like alcohol exposure decreases dendritic complexity while increasing the density of mature spines in mpfc layer ti/iii pyramidal neurons. Synapse, 64(2):127-135, 2010.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770 778, 2016
Lee Jun Hy. eegguunyunnimmngeessemeeeeee attention r2u-net. on 09/06/2020).
Pytorch implementation of u-net, r2u-net, attention u-net, and on 2018. (Accessed
com leemm///leeeuuuuu..ee
Stefan Jaeger, Sema Candemir, Sameer Antani, Yi-Xiáng Wäng, Pu-Xuan Lu, George Thoma. Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery, 4(6):455, 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar: huml, 6,1, 2009.
Yann LeCun, Yoshua Bengio, al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, (16) 1995.
C. Liu, Chen, F. Schroff. H. Adam, W. Hua. A. L. Yuille, and Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 82 82-92, 2019.
Hanxiao Liu, Karen Simonyan,
Viming Yang. DARTS: differentiable architecture search. CoRR, abs/1800 his/1806.09055, 2018. URI htw.lhtt:://ee /wt./v..vvv..////a/ wttpii/orr////////// 1806. 800.09055
Aurélien Lucchi, Kevin Smith, Radhakrishna Achanta, Graham Knott, and Pascal Fua. Supervoxel- segmentation of mitochondria in em image stacks with learned shape features. IEEE transactions on medical imaging, 31(2):4744444 486, 2011.
Geoffrey F Miller, Peter M Todd, and Shailesh U Hegde. Designing neural networks using genetic algorithms. In ICGA, volume 89, pp. 374-384, 1989.
10

---

Under review as a conference paper at ICLR 2021
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bemhard Kainz, al. Attention u-net: Learning where to look for the pancreas. arXiv preprint 1804,3990 2018.
Iija Radosavovic, Raj Prateek Kosaraju. Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces, 2020.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Lc. Regularized evolution for image classifier architecture search, 2018.
Philippe Remy. philipperemm/kkraet Activation maps (layers outputs) and gradients in keras. bttps::/ wwtphw..cc//palipppippipeeee http:////aallpppppeeeeeeeeeeeeee 2018. (Accessed on 08/31/2020).
Olaf Ronneberger. Philipp Fischer, Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Lecture Notes in Computer Science, pp. 234-241, Springer International Publishing, 2015.
Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization the brain. Psychological review, 65(6):386, 1958.
David E. Rumelhart. James L. McClelland, and CORPORATE PDP Research Group (eds.). Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations. MIT Press, Cambridge, MA, USA, 1986. ISBN 026268053X.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, 2014.
Sagar Vaze, Weidi Xie, and Ana IL Namburete. Low-memory enns enabling real-timo ultrasound segmentation towards mobile deployment. IEEE Journal Biomedical and Health Informatics, 24(4):1059 1069, 2020.
Yu Weng, Tianbao Zhou, Yujie Li, and Xiaoyu Qiu. Nas-unet: Neural architecture search for medical image segmentation. IEEE Access, 7:44247 44257, 2019.
Sirui Xie, Hchui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In Inter- national Conference on Learning Representations. 2019.
Xiaowei Xu, Qing Lu, Tianchen Wang, Jinglan Liu, Cheng Zhuo, Xiaobo Sharon Hu, and Yiyu Shi. Edge seg- mentation: Empowering mobile telemedicine with compressed cellular neural networks. In 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 880-887. IEEE, 2017.
Xin Yao. Evolutionary artificial neural networks, International journal of neural systems, 4(03):203 222, 1993.
Xin Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423-144 1447, 1999.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-bench- 101: Towards reproducible neural architecture search. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7105 71 7114, Long Beach, Califomia, USA, 09 15 Jun 2019. PMLR.
Z. Zhou, M. M. R. Siddiquee. N. Tajbakhsh, and J. Liang. Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE Transactions Medical Imaging, 3966)1856-18666 2020.
Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture medical image segmentation. Deep Learning in Medical Image Analysis and Mul- timodal Learning for Clinical Decision Support. pp. 3-11. Springer, 2018
A APPENDIX
A.
GENOME MICRO-ARCHITECTTRRE
When designing search space, formulate genotypes that code for model architectures. Fol- lowing common patters in convolutional networks, and the UNet [Ronneberger et al. (2015). we first impose the following constraints search space:

---

Under review as a conference paper at ICLR 2021
The network blocks must be symmetrical. This means that the number of blocks both in the network encoder and decoder are identical, with mirror internal layer configurations (types of layers, numbers of filters, and number of layers in the block)
The network must be hierarchical. When designing models for medical image segmenta- tion, we rely hierarchical backbones for both the encoder and decoder, reflected in Figure
We constrain skip connection directionality. In the network, skip connections only oceur from earlier to later layers in the background.
Figure shows the standard micro-architecture for contracting and expanding parts of net- work. We also note that while the layer arrangements are constant, the number of filters, for each block initially random. However, each integer value of filter numbers is scaled by a factor of 1.5 for each subsequent block, as Figure 8 highlights.
Chanclema
Nato
Chaneels
Bust
Choctt Tomppor
Montendeg
CorttD
Figure 8: Each prism represents layer in the overall repeating block motifs in the network. A) The con- tracting black micro-architecture for bloe. Note that this motif repeats throughout the contracting phase network. n, the number channels, is factored by 1.5 for each subsequent contracting block. B) The Expanding block of the micro-architecture n, the number of channels, is factored by .75 for each subsequent expanding block.
A.2 GPU RUN-TIME
Overall, the search algorithm had GPU run-times, shown in Table We note that these results are reported after running the search algorithm on a Tesla-v100. The reported values are an average of three independent trials. Oftentimes, the time dependent the dataset size. Because the Cell Nuclei dataset had the highest number of sample images, it took longer to train on as compared the smaller Lung dataset.
Table 3: GPU Run-time for all three datasets with time measured in hours averaged over three trials with image dimensions of 128 x 128.
Dataset
Run Time (hrs)
Lungs EM Cell Nuclei
0.483 + 0.067 0.878 0.164
1.31 +. 0.170
12

---

Under review as a conference paper at ICLR 2021
A.3 ABLATING EFFICIENT LOss
Ablating Lffficiency Loss Function
0.95
0.85
0.8
BCL
Dice
ELS
0.65
0.5
100000
1000000
Log Parmss
Figure 9:
The efficiency loss scaling (EL.) loss function selects a smaller model (orange) that perform the level as selected simply by BCL alone (blue).
We also examine the effect of ablating the efficiency loss' parameter and time penalty terms on the overall model selection. Through our investigation, model select smaller models, that can perform at the level of larger models selected by the BCL loss function. Figure 9 highlights this trend for the Lung Dataset. The results averaged over three trial runs.
find that the efficiency loss does help the
We that removing the penalty terms for high space and time complexity still yields high- performing models. However, these models are larger, and computationally costly.
13


Reviews:
[{"review_id": "zX76m061gA1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a neural architecture search (NAS) algorithm inspired by brain physiology. In particular, they propose a NAS algorithm based on neural dendritic branching, and apply it to three different segmentation tasks (namely cell nuclei, electron microscopy, and chest X-ray lung segmentation). The authors share their codes with the scientific community, which is highly appreciated. \n\n- I have a concern about one of the main motivations of the paper. The authors say that they \"take inspiration from the brain because it has the most efficient neuronal wiring of any complex structure\" (lines 7 and 8 in the Abstract). My question is: What is the problem with using biologically implausible NAS methods or with the fact that some approaches are incompatible with current understandings in neuro-biology? I consider that all proposals to improve neural networks (NNs) training and design are welcome, but I do not see the relevance of explicitly searching for biologically realistic algorithms in NNs. As Yann LeCun would say (http://matt.colorado.edu/compcogworkshop/talks/lecun.pdf): \"Let's be inspired by nature, but not too much\". It is indeed nice to imitate nature, but we mainly need to understand what is actually relevant for our practical purposes. For instance, as pointed out by LeCun, in the case of airplanes, we developed aerodynamics and compressible fluid dynamics, and we figured out that feathers and wing flapping were not crucial. In this sense, I'd like to see a better justification and contextualization of this dendritic branching approach.\n\n- The paper, in my humble opinion, is a bit confusing, because the different contributions and overall structure of the paper dilute the main focus of the paper. The paper first starts with the presentation of a novel bioinspired NAS method. Then, we move towards medical imaging segmentation and the limitations of U-Net, and later we discover that there are also evolutionary computation algorithms involved (as a primary contributions that is not even cited in the Abstract). I would encourage the authors to rewrite the paper in a clearer way, identifying the core contribution and challenges, as well as the motivation and the rationale behind the approach they use. In particular, I'd like to better understand how the submitted paper compares (quantitatively and qualitatively) to prior art, and what are the conceptual or empirical advantages of the proposed approach.\n\n- Why other NAS algorithms were not included in the experimental comparison (apart from Weng et al. (2019))? Regarding the competitor methods, why not to introduce in the comparison methods like Neuron-Evolution for Augmenting Topology (NEAT) [1], HyperNEAT [2], ENAS [3] or [4], that represent popular methods in neural architecture search? Without a more extensive experimental comparison with prior methods and different strategies is difficult to elucidate the actual empirical contribution of the proposed method. \n[1] Kenneth O. Stanley, and Risto Miikkulainen. \"Evolving neural networks through augmenting topologies.\" Evolutionary computation 10(2): 99-127, 2002.\n[2] Kenneth O. Stanley, David B D’Ambrosio, and Jason Gauci. \"A hypercube-based encoding for evolving large-scale neural networks\". Artificial life, 15(2):185–212, 2009.\n[3] Pham, Hieu, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. \"Efficient neural architecture search via parameter sharing.\" arXiv preprint arXiv:1802.03268 (2018).\n[4] Zoph, Barret, and Quoc V. Le. \"Neural architecture search with reinforcement learning.\" arXiv preprint arXiv:1611.01578 (2016).\n\n- Evolutionary computation techniques have been employed many times for both training neural networks and or designing their architecture and building blocks (e.g., activation functions), with papers on the subject already published in the 80's and 90's. From this point of view, statements like \"Within the last five years, advances in NAS have branched into different areas, including evolutionary algorithms (Real et al., 2018)\" look insufficient. The related works section is, in my opinion, not complete and clear enough. See the following papers as example:  \n[5] Miller, Geoffrey F., Peter M. Todd, and Shailesh U. Hegde. \"Designing Neural Networks using Genetic Algorithms.\" ICGA. Vol. 89. 1989.\n[6] de Garis, H. (1990). \"Genetic programming: Modular evolution for Darwin machines\". In Proceedings of the 1990 International Joint Conference on Neural Networks (pp. 194-197).\n[7] Fogel, David B., Lawrence J. Fogel, and V. W. Porto. \"Evolving neural networks.\" Biological cybernetics 63.6 (1990): 487-493.\n[8] Yao, X. (1993). \"Evolutionary artificial neural networks\". International journal of neural systems, 4(03), 203-222.\n[9] Angeline, Peter J., Gregory M. Saunders, and Jordan B. Pollack. \"An evolutionary algorithm that constructs recurrent neural networks.\" IEEE transactions on Neural Networks 5.1 (1994): 54-65.\n[10] Yao, X. (1999). \"Evolving artificial neural networks\". Proceedings of the IEEE, 87(9), 1423-1447.\n\n- In relation to the proposed model itself (called Bractivate), there are many aspects that remain unclear:\n(1) What do the authors mean by semi-random evolutionary algorithm?\n(2) What evolutionary algorithm do they use? It appears that they just apply a mutation operator on the most active block in the best architecture found so far. \n(3) The authors propose a loss that includes the number of model parameters and the training time per epoch. In this sense, what do the authors mean by BCE in Equation 3? I guess they actually refer to BCL, right?\n(4) If I understood correctly, Bractivate does not allow to create or insert new blocks but it only branches already existing blocks (using two skip connection operator types: concatenation or addition). This seems a severe limitation in the neural architectures it can generate. Am I correct? \n(5) The authors state that \"Our ablation study most strongly confirms our hypothesis that dendritic branching to active blocks significantly improves segmentation performance\". Did the authors employ several independent runs, some experimental validation protocol (like Cross-validation) and statistical tests to verify the existence of statistically significant differences between the methods under comparison?  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "vG4RWv-7FLt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1.\tBorrow neuroscience concepts into NAS problem is novel. \n2.\tThe paper claims that healthy brain has more branches, thus adding new skip-connections can improve the performance. This idea is of great interest. \n\nCons:\n1.\tYour model is too simple so that the technical contribution is too limited. You claim that the healthy have more dendritic branches in their brain, but the differences between their somas are not discussed. Thus, it may enough to only add skip connections on a network to improve its performance. The layers may also need to be tuned. \n2.\tIt is not clear that how many skip-connections should be added to the network. Do more edges lead to higher performance? If so, why do not we directly use densely connected network?\n3.\tThe experiment results show your model cannot reach SOTA in 2/3 datasets. \n4.\tNo ablation study of the proposed efficient loss.\n5.\tI suggest you improve your set type. It is not easy to detect the main text and the caption of Figure 3 and Figure 5. \n6.\tIn Figure 6. You should explain the meaning of the color. In caption, you should explain the effectiveness with more detail. Such as which part of the image is highlight and what does it mean.\n7.\tYou need to improve you writing. There are too many strange expressions in your paper. In Sec 3.3 you can say “NAS methods often focus on accuracy, which is regarded as the main performance metric”. In Sec 4.4 you can say “We experiment with the standard binary BCL loss function in Equation 2, and the ELs function in Equation 3.” You may need lots of efforts working on this. \n\nOverall Review:\nThis paper borrows neuroscience concepts into NAS problems, which is a good story. However, this paper has too many problems to be solved, as mentioned above. You may need lots of effort improving your paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IBFdRqFRuH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of the proposed Bractivate is to search an architecture from the predefined search space which performs well and efficient, as the proposed efficiency loss simultaneously constrains the parameter and training cost of candidate. However, the proposed method misses many crucial details, which may make the empirical results not convincing to me. \n\nMy major concerns are as follows:\n1) The paper did not provide the micro-architecture of model genotypes, so the initialized search domain is ambiguous. As I understand, the micro-architecture of U-Net’s block is fixed and only searching for the connections between blocks.\n2) The article did not give experimental results or explanations to prove the rationality of the formulation in Eq. 1. Why the block with max (Ab) should be chosen? And what the difference if the algorithm chooses another block?\n3) The GPU hours it takes to finally search an optimal architecture. Evolutionary algorithms are usually time-consuming so decreasing the necessary of searching compared with other manual-tuned baselines.\n4) How to deal with the miss-matching of spatial dimension and channels when aggerating the activations of different blocks?\n5) What about applying the proposed Bractivate into other popular segmentation benchmarks and see if the proposed method can still achieve SOTA performance, e.g., COCO, Pascal etc.\n\nBesides, there’s a strict upper limit of **8** pages for the main text of the initial submission this year. However, this submission breaks this rule and the conclusion part exceeds 8 pages.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper would greatly benefit from some reorganization/rewriting since, as pointed out by some of the reviewers, it’s hard to follow in its current form. While a biologically inspired NAS algorithm could be an interesting direction to explore, the current paper falls short in providing evidence that the approach is well-motivated or empirically strong.  In terms of empirics, too many details are missing on the search space/architecture, ablations and comparison with existing methods. For future submissions, it would be particularly useful for the authors to explicitly discuss why they don’t find competing methods applicable to their setting.
Decision: Reject