BioNAS: Incorporating Bio-inspired Learning Rules to Neural Architecture Search

Authors: Imane Hamzaoui, Riyadh Baghdadi
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Feedback Alignment, Neural Architecture Search, Learning Rules, Biologically Plausible

Abstract:
Bio-inspired neural networks have gained traction due to their adversarial robustness, energy efficiency, and for being biologically plausible. While these bio-inspired networks have shown significant progress, they still fall short in terms of accuracy and are hard to scale to complex tasks. In this paper, we propose to use neural architecture search to further improve state-of-the-art bio-inspired neural networks. We
achieve this thanks to BioNAS, a framework for neural architecture search that explores different bio-inspired neural network architectures and learning rules. The novelty of BioNAS lies in exploring the use of different bio-inspired learning rules for the different layers of the model. The motivation for this choice comes from recent work in the field suggesting that different learning mechanisms might be used in different
regions of the human brain. Using BioNAS, we get state-of-the-art bio-inspired neural network performance achieving an accuracy of 94.86 on CIFAR10, 76.48 on CIFAR-100 and 43.42 on ImageNet16-120, surpassing state-of-the-art bio-inspired neural networks. We show that a part of this improvement comes from the use of different learning rules instead of using a single algorithm for all the layers. We release BioNAS to the community and make the code available via this link (https://anonymous.4open.science/r/LR-NAS-DFE1)

Decision: reject
Reviewer scores: 5, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730359956174

thread_type:
official_review

<thread_object>
creator: reviewer_x5ja

<content>
summary:
This paper uses the NAS method to search for suitable edge and learning rules of the network cell, constructing a biologically inspired neural network that achieves the comparable performance of the network trained by end-to-end backpropagation algorithm. Simultaneously, this paper demonstrates that the mixed biologically inspired learning rules can reduce gradient variance and enhance the network’s white-box adversarial robustness.

strengths:
This paper introduces the different learning rules into the network search method for the first time. Experiment results show that mixed learning rules can effectively improve the performance of the network and its white-box adversarial robustness. I think the integration of NAS with various learning methods represents a promising approach.

weaknesses:
1. This paper does not detail how to simultaneously search for cell architecture and the learning rules. And the search space is much larger than the original DARTS and EGNAS; the search time will increase about threefold. Therefore, if DARTS and EGNAS adopt the same search time as this paper, will their performance be improved?

2. Different rows (training methods) in Table 2 use different network structures, but they need to use the same network structure for comparison.

3. This paper lacks the theoretical analysis that adopting the mixed learning rule can enhance the white-box adversarial robustness.

questions:
1. Are the attack image gradients for the white-box attack method calculated by end-to-end backpropagation or the searched learning algorithm?

2. Methods such as FA use a new matrix to replace the weight matrix in the back propagation process, which also needs end-to-end learning. Why are these training methods related to bio-inspired?

3. After the search is completed, does the subnet undergo random weight retraining or inherit the supernet's weight for fine-tuning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730387231489

thread_type:
official_review

<thread_object>
creator: reviewer_yidi

<content>
summary:
This paper presents BioNas, a neural architecture search algorithm tailored for bio-inspired neural networks. BioNas focuses on biologically plausible learning algorithms and simultaneously optimizes both the network architecture and the learning rule. The proposed framework achieves state-of-the-art performance on the CIFAR-10 and CIFAR-100 datasets, demonstrating its effectiveness in enhancing bio-inspired model design.

strengths:
The authors utilize a more bio-plausible learning rule instead of the commonly used BP algorithm, which is interesting.

The authors provide strong empirical support for their approach by conducting extensive experiments across multiple datasets and varied settings.

weaknesses:
The motivation of combining FA approaches and NAS is not clear. While the concept of applying different learning rules to train SNNs is intriguing, the combination of NAS and FAs in this context raises questions. NAS traditionally involves exploring a vast space of candidate models to find an optimal architecture with significant computational cost. Its primary aim is to achieve high performance without regard to training efficiency, which contrasts with the efficient and light weight bio-inspired learning rules. 

The novelty is limited, as the BioNAS framework closely resembles DARTS and EG-NAS. The primary difference introduced here is the addition of various existing learning rules to the search space.

questions:
One important concern is whether the authors have verified if there is a gradient obfuscation issue within their model. The observation that the multi-step PGD attack performs worse than the single-step FGSM attack may indicate potential gradient masking or unsuccessful attacks [1]. This issue can often lead to misleading conclusions about a model’s robustness.

[1] Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." ICML 2018.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714247861

thread_type:
official_review

<thread_object>
creator: reviewer_hruh

<content>
summary:
This paper proposes a neural architecture search framework, BioNAS, that supports certain types of biologically plausible learning algorithms. The framework is built upon existing NAS framworks DARTS and EG-NAS, and incorporates different feedback alignment techniques from BioTorch. The authors compare the performance of their BioNAS generated networks against some previous works and claim they can get state-of-the-art bio-inspired network performance on several benchmarks.

strengths:
Adversarial attacks on generated bio-inspired networks are introduced to illustrate the advantage of their framework.

weaknesses:
Although the direction of this work is interesting, my major concern is that this work appears to be prelimilary, and some of the key conclusions are *overclaimed*.  

Major Weakness
1. The authors claim that their proposed BioNAS is designed for bio-inspired networks, but currently only several variants of FA algorithms are supported, making this work less significant, and their claim should be toned down.
2. The proposed BioNAS is implemented by adding operations backended by BioTorch to existing NAS frameworks, which does not show much novelty nor effort.
3. The authors claim that they can get state-of-the-art bio-inspired network performance on several benchmarks, but for CIFAR-10 they only compare against two previous Hebbian-based results, and for CIFAR-100 and ImageNet16-120 no previous results of bio-inspired algorithms are shown.

questions:
Questions:
1. How might other forms of bio-inspired learning algorithms (e.g., Hebbian, predictive coding, target propagation) be integrated in the proposed framework?
2. Does the distribution of different learning algorithms in the generated networks show interesting patterns? And if so, what might be the (intuitive) explaination for generating such patterns, what might be the advantages for such patterning, and can such patterns be generalized to different network architectures to produce competitive performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717729856

thread_type:
official_review

<thread_object>
creator: reviewer_ugag

<content>
summary:
The paper explores a method that combines bio-inspired learning rules with Neural Architecture Search (NAS) to optimize the performance of neural networks. The authors propose BioNAS, a framework capable of automatically searching for the best architecture and learning rules. Through a series of experiments, particularly on the CIFAR-10 and CIFAR-100 datasets, the paper demonstrates the advantages of BioNAS in terms of accuracy and adversarial robustness, comparing it with existing backpropagation training methods.

strengths:
The integration of bio-inspired learning rules with NAS is a fresh perspective that could significantly contribute to the field of neural network optimization. The paper is well-organized and clearly written, making it accessible for readers with varying levels of expertise in NAS and bio-inspired techniques.

weaknesses:
Is there an analysis of energy consumption.

questions:
Whether it is applicable to larger datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732554156883

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to reviewers:

comment:
We sincerely thank the reviewers for their thoughtful and insightful comments, which have greatly improved the quality of our work. Below, we address the main concerns raised, including the generalizability of our method to larger datasets, the extension to other bio-inspired learning rules, additional baseline comparisons, and the motivation behind our approach.

**Generalizability to Larger Datasets:**

The reviewers expressed concerns regarding the generalizability of our method on larger datasets. To address this, we have evaluated our framework on the full ImageNet dataset. The results demonstrate that mixing learning rules continues to perform well, even on large-scale datasets, further validating the scalability of our approach. 

**Extension to Other Bio-Inspired Learning Rules:**

The reviewers also suggested exploring extensions of our framework to other bio-inspired learning rules. To this end, we have incorporated a Hebbian convolution operation into the search space, which serves as an example of integrating bio-inspired rules that rely solely on a forward pass. Details of this addition, including its implementation, can be found in the appendix and in the supplementary code.
However, we emphasize that while the Hebbian operation was successfully integrated, it did not improve the accuracy during evaluation. This aligns with our initial focus on learning rules that aim to perform as closely as possible to backpropagation in terms of accuracy. We highlight that the framework remains flexible and can accommodate a wide range of bio-inspired learning rules, whether they rely on a forward pass only or both forward and backward passes. Methods requiring two forward passes, while theoretically possible, pose additional complexity in integration.

**Comparisons with Baselines:**

In response to the reviewers' request for additional comparisons, we have included results for other open-sourced baselines. These results show that our framework consistently outperforms the newly added baselines.

**Motivation for the Work:**

The reviewers requested clarification on the motivation behind this work. Our primary inspiration comes from neuroscience, which highlights the brain's use of diverse learning rules, mechanisms, and cost functions to optimize performance. This insight drove us to investigate the potential of mixed learning rules in artificial neural networks, aiming to enhance their adaptability and robustness. Recognizing that Neural Architecture Search (NAS) can be computationally expensive, we focus on identifying architectures that are performant and efficient during inference, which is the reason why bio-inspired learning rules are interesting in the first place. 


We believe these updates address the reviewers' concerns and provide stronger evidence for the contributions and significance of our work. Thank you again for your valuable feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies NAS for bio-inspired learning rules that are not based on backpropagation. This is achieved by combining existing NAS methods with elements from BioTorch. The reviewers found bio-inspired methods without backpropagation intriguing, including the angles of adversarial robustness and of different learning rules in different layers. Criticisms included a lack of novelty (directly combining NAS and BioTorch), some elements of the empirical evaluation, a lack of theoretical justifcation, and a preliminary nature along with a certain amount of overclaiming results. This preliminary nature, along with concerns about the empirical evaluation were my main reasons for the decision to reject.