# Dynamic Architectural Optimization of Artificial Neural Networks

Kaitlin Maile

<figure><img src="image_1.png" /></figure>

## To cite this version:

Kaitlin Maile. Dynamic Architectural Optimization of Artificial Neural Networks. Computer Science [cs]. Université Toulouse 1 Capitole, 2023. English. NNT: tel-04568380

HAL Id: tel-04568380

https://hal.science/tel-04568380

Submitted on 4 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
<figure><img src="image_2.png" /></figure>

Université
de Toulouse

THÈSE

En vue de l'obtention du

DOCTORAT DE L'UNIVERSITÉ DE TOULOUSE

Délivré par : l'Université Toulouse 1 Capitole (UT1 Capitole)

Présentée et soutenue le 04/10/2023 par :

Kaitlin MAILE

Dynamic Architectural Optimization of Artificial Neural Networks

JURY

CAROLA DOERR
DECEBAL C. MOCANU
MARC SCHOENAUER
RUFIN VAN RULLEN
FRANK HUTTER
HERVÉ LUGA
DENNIS G. WILSON

Rapportrice
Rapporteur
Examinateur
Examinateur
Examinateur
Directeur de Thèse
Co-encadrant de Thèse

École doctorale et spécialité :

MITT : Domaine STIC : Intelligence Artificielle

Unité de Recherche :

IRIT : Institut de Recherche en Informatique de Toulouse

Directeur(s) de Thèse :

Hervé LUGA et Dennis G. WILSON

Rapporteurs :

Carola DOERR et Decebal C. MOCANU
Abstract

Despite the significant and exponential advancements in artificial neural networks (ANNs), their
current capabilities still fall short of human-like intelligence. Many assumptions of current artificial
learning approaches diverge from the observed characteristics of the brain, such as conventionally
static and hand-designed ANN architectures opposed to dynamic and self-modifying biological con-
nectivity. This thesis aims to further progress by exploring structural learning as a brain-inspired
tool for augmenting the power of ANNs, taking initial steps of expanding architectural search spaces
and learning paradigms to real practical applications. Towards this end, the proposed structural
learning framework unifies multiple subfields of artificial intelligence research and identifies key
foundational challenges that are investigated in the subsequent work. Towards architectural search
space generalization, equivariance-aware neural architecture search (NAS) optimizes the architectural
constraints imposed by partial equivariance to symmetry groups, enhancing the performance and
generalization of ANNs for tasks exhibiting symmetries. Algorithmic improvements to differentiable
NAS, focusing on dynamic scheduling and regularization, enhance the efficiency and reliability of the
search process. Neurogenesis in ANNs, a relatively unstudied problem, is decomposed into scheduling
and initialization decisions towards a suite of neurogenesis strategies that enable the dynamic growth
of performant networks. Finally, aspects of all of the preceding findings are synthesized towards
architectural optimization in dynamic learning environments such as transfer learning. The findings
and methodologies presented have the potential to significantly impact the standard pipeline of ANNs,
reducing engineering, training, and deployment costs while increasing their efficacy and power for
practitioners.
# Résumé

Malgré les progrès considérables et exponentiels des réseaux neuronaux artificiels (ANNs), leurs capacités actuelles sont encore loin de l'intelligence humaine. De nombreuses hypothèses des approches actuelles de l'apprentissage artificiel divergent des caractéristiques observées du cerveau, telles que les architectures ANN conventionnellement statiques et conçues à la main, opposées à la connectivité biologique dynamique et auto-modifiante. Cette thèse vise à faire avancer les choses en explorant l'apprentissage structurel en tant qu'outil inspiré du cerveau pour augmenter la puissance des ANNs, en prenant des mesures initiales pour étendre les espaces de recherche architecturaux et les paradigmes d'apprentissage à des applications pratiques réelles. À cette fin, le cadre d'apprentissage structurel proposé unifie plusieurs sous-domaines de la recherche en intelligence artificielle et identifie les principaux défis fondamentaux qui sont étudiés dans les travaux ultérieurs. En ce qui concerne la généralisation de l'espace de recherche architecturale, la recherche d'architecture neuronale (NAS) tenant compte de l'équivalence optimise les contraintes architecturales imposées par l'équivalence partielle aux groupes de symétrie, améliorant ainsi les performances et la généralisation des ANNs pour les tâches présentant des symétries. Les améliorations algorithmiques apportées à la recherche d'architecture neuronale différentiable, axées sur la planification dynamique et la régularisation, renforcent l'efficacité et la fiabilité du processus de recherche. La neurogenèse dans les ANNs, un problème peu étudié, est décomposée en décisions d'ordonnancement et d'initialisation vers une suite de stratégies de neurogenèse qui permettent la construction automatique et dynamique de réseaux performants. Enfin, les aspects de tous les résultats précédents sont synthétisés en vue d'une optimisation architecturale dans des environnements d'apprentissage dynamiques tels que l'apprentissage par transfert. Les résultats et les méthodologies présentés ont le potentiel d'avoir un impact significatif sur le pipeline standard des RNA, en réduisant les coûts d'ingénierie, de formation et de déploiement tout en augmentant leur efficacité et leur puissance pour les praticiens.
Acknowledgements

I would like to thank my advisors, Dennis and Hervé. Their help was imperative from our humble beginnings as a remote internship to test the waters before committing to the full PhD, then continuously navigating french bureaucracy to successfully relocate to France and secure funding for the thesis and associated travels. Their advisement style adapted to my growth as a researcher, dynamically providing just the right amount of constructive guidance.

I would like to thank my parents, Keith and Sandi. Their persistent support and inspiration has made my entire academic path possible, evidenced by their willingness to chauffeur me to a generous number of academic and extracurricular activities throughout my childhood as well as their unfaltering support at each life path decision.

I would like to thank the rest of my jury members: Carola, Decebal, Marc, Rufin, and Frank, for their role in the final stages of the doctorate and all of the intriguing scientific discussions with each one leading up to the defense.

I would like to thank the SuReLI lab at ISAE-SUPAERO and the AMLab at the University of Amsterdam for providing research environments that both catalyzed insightful discussions as well as offered a cordial social opportunities to relax in-between.

I am grateful for all of my research experiences leading up to my doctoral studies. David helped me navigate the world of research and then of graduate school applications to transition from biomedical engineering to computer science. Risto and Manish then supported my growth as a graduate researcher at UT Austin as well as my decision to begin this thesis.

I would like to thank the remaining co-authors, collaborators, and connections made at conferences, internships, and beyond that provided inspirational discussions and useful advice.

I would like to thank all of the old and new friends I have made along the way, in addition to connections via sports such as climbing partners and ultimate frisbee teammates, for keeping me sane and providing so many unforgettable experiences.
# Contents

## 1 Introduction 9

1.1 Current Challenges and Goals 10

1.1.1 Improving the Automation of Neural Networks 10

1.1.2 Adaptive Learning 10

1.1.3 Expanding the Domains and Search Spaces of Structural Learning 10

1.2 Contributions 10

1.2.1 Sections 2.2 and 3 - Structural Learning in Artificial Neural Networks: A Neural Operator Perspective 11

1.2.2 Section 4 - Equivariance-aware Architectural Optimization for Neural Networks 11

1.2.3 Section 5 - DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS 11

1.2.4 Section 6 - When, Where, and How to Add New Neurons to ANNs 11

1.2.5 Section 7 - Neural Growth and Pruning in Dynamic Learning Environments 12

1.3 Manuscript Outline 12

## 2 A Background on the Current State of Neural Networks 13

2.1 Artificial Neural Networks 13

2.1.1 Basic Functionality 13

2.1.2 Optimization of Weights 13

2.1.3 Convolutional Neural Networks 14

2.1.4 Architecture Design 15

2.2 Biological Neural Networks 15

2.2.1 The Basic Units of Biological Neural Networks 16

2.2.2 Neurogenesis and Neural Migration 16

2.2.3 Axon Migration and Synaptogenesis 17

2.2.4 Synaptic Pruning 18

2.2.5 Neural Cell Death 19

2.2.6 From Biological to Artificial 20
3 Analysis of Structural Learning 21

3.1 Framework Definitions 21

3.2 Characterizing Structural Learning in ANNs 22

3.2.1 Scope of Study 22

3.2.2 Characterization of the Collected Corpus 24

3.3 Synaptic Pruning and Neural Pruning 26

3.3.1 Pruning Generalities 27

3.3.2 Pruning Connections 28

3.3.3 Pruning Units 30

3.3.4 Pruning Conclusions 34

3.4 Neuron creation and synaptogenesis 34

3.4.1 Creation Generalities 35

3.4.2 Growing Connections 36

3.4.3 Growing Units 37

3.4.4 Creation Conclusions 39

3.5 Perspectives 40

3.5.1 Implementations and Trends 40

3.5.2 Adjacent Domains 42

3.5.3 Current Challenges and Future Directions for Development 43

4 Equivariance-aware Architectural Optimization 48

4.1 Related Works 48

4.2 Background 49

4.2.1 Symmetries and Group Theory 49

4.2.2 Group Convolutional Neural Networks 50

4.3 Towards Architectural Optimization over Subgroups 50

4.3.1 Equivariance Relaxation Morphism 51

4.3.2 [G]-Mixed Equivariant Layer 52

4.4 Equivariance-Aware Neural Architecture Algorithms 52

4.4.1 Evolutionary Equivariance-aware NAS 53

4.4.2 Differentiable Equivariance-aware NAS 53

4.5 Experiments 54

4.5.1 Evolutionary Equivariance-aware NAS 56

4.5.2 Differentiable Equivariance-aware NAS 56

4.6 Discussion 58
<table>
  <tbody>
    <tr>
      <td><b>5 Improving Constrained NAS</b></td>
      <td><b>65</b></td>
    </tr>
    <tr>
      <td>5.1 Background and Related Work</td>
      <td>65</td>
    </tr>
    <tr>
      <td>5.2 Towards Constrained Optimization</td>
      <td>68</td>
    </tr>
    <tr>
      <td>5.2.1 Dynamic FIMT Scheduling</td>
      <td>68</td>
    </tr>
    <tr>
      <td>5.2.2 Regularization</td>
      <td>69</td>
    </tr>
    <tr>
      <td>5.2.3 DARTS-PRIME</td>
      <td>70</td>
    </tr>
    <tr>
      <td>5.2.4 ADMM Regularization</td>
      <td>70</td>
    </tr>
    <tr>
      <td>5.2.5 CRB Activation</td>
      <td>71</td>
    </tr>
    <tr>
      <td>5.3 Experiments</td>
      <td>71</td>
    </tr>
    <tr>
      <td>5.3.1 Datasets and Tasks</td>
      <td>71</td>
    </tr>
    <tr>
      <td>5.3.2 Hyperparameters</td>
      <td>72</td>
    </tr>
    <tr>
      <td>5.3.3 Data and Training Configuration</td>
      <td>73</td>
    </tr>
    <tr>
      <td>5.4 Results and Discussion</td>
      <td>74</td>
    </tr>
    <tr>
      <td>5.4.1 Benefits of Scheduling</td>
      <td>74</td>
    </tr>
    <tr>
      <td>5.4.2 Regularization Improves Reliability</td>
      <td>74</td>
    </tr>
    <tr>
      <td>5.4.3 CRB Activation Produces Smaller Networks</td>
      <td>78</td>
    </tr>
    <tr>
      <td>5.4.4 DARTS-PRIME Performance Beyond CIFAR-10</td>
      <td>78</td>
    </tr>
    <tr>
      <td>5.5 Conclusion</td>
      <td>78</td>
    </tr>
    <tr>
      <td><b>6 Neurogenesis</b></td>
      <td><b>81</b></td>
    </tr>
    <tr>
      <td>6.1 Background</td>
      <td>81</td>
    </tr>
    <tr>
      <td>6.1.1 Problem Statement and Notations</td>
      <td>81</td>
    </tr>
    <tr>
      <td>6.1.2 Neurogenesis</td>
      <td>82</td>
    </tr>
    <tr>
      <td>6.2 A Framework for Studying Neurogenesis</td>
      <td>83</td>
    </tr>
    <tr>
      <td>6.2.1 Triggers</td>
      <td>83</td>
    </tr>
    <tr>
      <td>6.2.2 Initializations</td>
      <td>85</td>
    </tr>
    <tr>
      <td>6.2.3 Neurogenesis for Convolutions</td>
      <td>86</td>
    </tr>
    <tr>
      <td>6.3 Experiments</td>
      <td>87</td>
    </tr>
    <tr>
      <td>6.3.1 Experiment Details</td>
      <td>87</td>
    </tr>
    <tr>
      <td>6.3.2 Generated Data</td>
      <td>88</td>
    </tr>
    <tr>
      <td>6.3.3 MLP MNIST</td>
      <td>89</td>
    </tr>
    <tr>
      <td>6.3.4 Deep CNNs on CIFAR10/CIFAR100</td>
      <td>90</td>
    </tr>
    <tr>
      <td>6.3.5 Hyperparameter studies on MLPs for MNIST</td>
      <td>93</td>
    </tr>
    <tr>
      <td>6.4 Discussion</td>
      <td>93</td>
    </tr>
    <tr>
      <td>6.5 Conclusion</td>
      <td>94</td>
    </tr>
  </tbody>
</table>
<table><tr><td>7 Growing and Pruning ANNs</td><td>97</td></tr><tr><td>7.1 Problem Definition</td><td>97</td></tr><tr><td>7.2 Grow and Prune for Transfer Learning</td><td>98</td></tr><tr><td>7.3 Experiments</td><td>99</td></tr><tr><td>7.4 Discussion</td><td>101</td></tr><tr><td>7.5 Additional Related Works</td><td>101</td></tr><tr><td>7.6 Conclusion</td><td>102</td></tr><tr><td>8 Discussion</td><td>103</td></tr><tr><td>8.1 Future Work</td><td>103</td></tr><tr><td>8.1.1 Structural Plasticity in Reinforcement Learning</td><td>104</td></tr><tr><td>8.1.2 Dynamic Equivariance Search</td><td>104</td></tr><tr><td>8.1.3 Structural Learning for Transformers</td><td>104</td></tr><tr><td>8.1.4 Woke LLMs: Equivariance Towards Socially Unbiased Models</td><td>104</td></tr><tr><td>8.2 Conclusion</td><td>104</td></tr></table>
# 1 Introduction

Artificial intelligence algorithms, and specifically artificial neural networks (ANNs), have been able to grow exponentially in power and complexity since their roots almost a century ago [1]. This is due to innovative automations such as backpropagation that replace some of what has been previously hand designed, as well as breakthroughs in computational power [2]. However, the currently realizable potential of ANNs in human-like intelligence is still nowhere near that of the brain, their original inspiration. Additionally, many assumptions of current learning approaches such as structurally static learning and isolated training from scratch have diverged from what is observed in the brain. This history has produced ANNs with a notably different intelligence from humans, excelling at computational and quantitative tasks yet failing at analytical and abstract aspects. While such assumptions have thus far helped push the frontier of artificial intelligence to its current state by adapting to the characteristics of artificial hardware and software, they may become a bottleneck preventing more powerful and adaptive ANNs that don't require hand-designed architectures.

The architecture of an ANN determines what operations are performed on the input data and how they are ordered to compute the output. An ANN structured specifically for a given domain can approximate a function more practically and efficiently in storage and training time than a generic wide shallow network [3, 4], even though the latter are theoretically capable of approximating any function [5, 6]. Even dense ANNs, or multi-layer perceptrons (MLPs), use structural priors and are not completely dense, as each neuron is only connected to the neurons in the immediately preceding and following layers, rather than all neurons in a network. This imposes a basal amount of structure on the network. Convolutional networks are even sparser than this, limiting neurons to have only spatially local connections and imposing weight-sharing on these connections to create translational equivariance for each layer. Skip-connections are a further structure that can be represented within the broader representation of layers connected in series. Although each of these structures is simply a special case of more generic functions, their constraints yield valuable benefits during the learning process.

The brain also shows structural specialization with hierarchical organization in task-specific regions, such as visual cortex [7], suggesting the benefit of structure for guiding learning. Convolutional layers were originally inspired by the visual cortex [8, 9]. The functional form conventionally used in modern ANNs is provably part of the most general and expressive class of linear functions with translational equivariance in the image domain [10], only adding a constraint of kernel size to enforce locality. This concurrence supports the utility of looking to the brain for inspiration, even though the clear divergence of ANNs from their biological roots must be appropriately considered.

Deeps ANNs are usually hand-designed, structurally static, and used generally across many related tasks within a domain. Even with specialized structures, they are known to be over-parameterized [11, 12], using more parameters than necessary for their performance on a specific task after training. Over-parameterization is conjectured to help gradient descent of parameters of an ANN converge to the global optima for a given training dataset, task, and objective function [13], particularly given that the architectures are reused for many tasks due to prohibitive engineering costs for further specialization. However, this over-parameterization also comes at a cost of space and time efficiency, during training and deployment. Using even more specialized architectures, beyond what is feasible through manual design, that themselves support parameter optimization could evade this cost and increase the efficacy of the ANN in a specific task.

The brain structures itself dynamically using both genetic and environmental information. Our DNA encodes processes that support the formation of the visual cortex, but only in the presence of typical visual input during development [14]. Otherwise, the brain learns to repurpose the unused regions for other structures and functions [15]. This emphasizes the important interplay of the learning algorithm
in addition to the data towards adaptability, such that specialized function-permissive structure may
be learned dynamically.

Automated and dynamic specialized design of ANN architectures is achievable through *structural learning*: a type of architectural optimization where both the architecture and the parameters of that architecture are cooperatively learned in a single process. This manuscript investigates structural learning as a brain-inspired tool for enhancing the learning process and augmenting the power and adaptability of ANNs.

**1.1 Current Challenges and Goals**

The overarching challenges presented subsequently have been identified in the current state of lit-
erature and will be detailed throughout this manuscript, reflected in the accomplished and future
work.

**1.1.1 Improving the Automation of Neural Networks**

Furthering the automation of ANNs beyond just weight optimization would expand their applicability, adaptability, and potential. Structural learning is one of the more brain-inspired approaches, compared to others such as hyperparameter optimization, for tackling this challenge. As computational capacities continue to improve, research into how to implement structural learning in a manner suited to artificial optimization could maximize the utilization of ANNs without the bottleneck of manual architecture design for further innovation in artificial intelligence. Even within existing structural learning approaches, the schedule of applying structural changes must be hand-designed, prompting the desire for automated scheduling: what structural changes to perform and when, where, how, and why to perform them.

**1.1.2 Adaptive Learning**

Architectural optimization permits dynamic architectural adaptations over learning, adjusting not only to the specific learning environment itself but also how it may be changing. Structural learning for ANNs is often studied in *silica*, using many assumptions to isolate and simplify this difficult process. The brain performs structural learning *in vivo*, notably with self-learned weak supervision and internal rewards. Exploring structural learning in more complex learning environments with fewer simplifying assumptions, such as continual learning and reinforcement learning, can support more adaptable models towards more complex and realistic applications. Automated scheduling is also crucial in this challenge, as the structural learning dynamics should adapt to those of both the learning environment and the model itself during training.

**1.1.3 Expanding the Domains and Search Spaces of Structural Learning**

Structural learning has thus far mostly been studied within the same image classification benchmark tasks and structural search spaces akin to existing convolutional architectures. While this allows a deeper understanding, it is not permissive towards applying it to the vast breadth of potential domains beyond simple image classification. Enabling application in more powerful search spaces, such as search through equivariant operations beyond translation to further capitalize on symmetries and structure present in the domain, could further both the study of structural learning as well as of the respective domains of interest.

**1.2 Contributions**

This manuscript combines the work of my doctorate into a cohesive thread and presents my future plans. My contributions to the field of deep learning are presented in the following sections.
1.2.1 Sections 2.2 and 3 - Structural Learning in Artificial Neural Networks: A Neural Operator Perspective [16]

Over the history of ANNs, only a minority of algorithms integrate structural changes of the network
architecture into the learning process. Modern neuroscience has demonstrated that structural change
is an important part of biological learning, with mechanisms such as synaptogenesis and neurogenesis
present even in adult brains and considered important for learning. Despite this history of artificial
methods and biological inspiration, and furthermore the recent resurgence of neural methods in deep
learning, there are relatively few current ANN methods which include structural changes in learning
compared to those that only adjust synaptic weights during the training process. We aim to draw
connections between different approaches of structural learning that have similar abstractions in
order to encourage collaboration and development. In this review, we provide a survey on structural
learning methods in deep ANNs, including a new neural operator framework from a cellular neuro-
science context and perspective aimed at motivating research on this challenging topic. We first give
a background on biological developmental processes in the brain. We then provide an overview of
ANN methods which include structural changes within the neural operator framework in the learning
process, characterizing each neural operator in detail and drawing connections to their biological
counterparts. Finally, we present overarching trends in how these operators are implemented and
discuss the open foundational challenges in structural learning in ANNs towards the overarching
problems presented in Section 1.1 and their respective solutions presented in this manuscript.

**1.2.2 Section 4 - Architectural Optimization over Subgroups for Equivariant Neural Networks [17]**

Incorporating equivariance to symmetry groups as a constraint during neural network training can improve performance and generalization for tasks exhibiting those symmetries, but such symmetries are often not perfectly nor explicitly present. This motivates algorithmically optimizing the architec- tural constraints imposed by equivariance. We propose the equivariance relaxation morphism, which preserves functionality while reparameterizing a group equivariant layer to operate with equivariance constraints on a subgroup, as well as the [G]-mixed equivariant layer, which mixes layers constrained to different groups to enable within-layer equivariance optimization. We further present evolutionary and differentiable neural architecture search (NAS) algorithms that utilize these mechanisms respec- tively for equivariance-aware architectural optimization. Experiments across a variety of datasets show the benefit of dynamically constrained equivariance to find effective architectures with approximate equivariance. This work directly tackles the challenge of expanding the domain and search space of structural learning, utilizing optimizable partial equivariance as structural bias.

1.2.3 Section 5 - DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS [18]

Differentiable Architecture Search (DARTS) is a recent neural architecture search (NAS) method based on a differentiable relaxation of the discrete architectural search space [19]. Due to its success, numerous variants analyzing and improving parts of the DARTS framework have recently been proposed. By considering the problem as a constrained bilevel optimization, we present and analyze DARTS-PRIME, a variant including improvements to architectural weight update scheduling and regularization towards discretization. We propose a dynamic schedule based on per-minibatch network information to make architecture updates more informed, as well as proximity regularization to promote well-separated discretization. Our results in multiple domains show that DARTS-PRIME improves both performance and reliability, comparable to state-of-the-art in differentiable NAS. These proposed contributions work together towards the challenges of automation and adaptive learning, investigating informed scheduling for structural learning.

**1.2.4 Section 6 - When, Where, and How to Add New Neurons to ANNs [20]**

Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of
structural learning like pruning. By decomposing it into triggers and initializations, we introduce a
framework for studying the various facets of neurogenesis: when, where, and how to add neurons
during the learning process. We present the Neural Orthogonality (NORTH*) suite of neurogenesis
strategies, combining layer-wise triggers and initializations based on the orthogonality of activations
or weights to dynamically grow performant networks that converge to an efficient size. We evaluate
our contributions against other recent neurogenesis works across a variety of supervised learning tasks.
This work expands the search space of structural learning by furthering our understanding of neural
creation and empowering its use in architectural optimization algorithms, as explored in the following
work.

1.2.5 **Section 7 - Neural Growth and Pruning in Dynamic Learning Environments [21]**

Training ANNs under the shifting distributions of dynamic learning environments can be augmented
by dynamic architectural adjustments in addition to the standard parameter tuning. Towards effective
yet efficient models, we study neural grow-and-prune in the basic shifting distribution case of transfer
learning: adapting a generically pre-trained model to a target dataset. We propose the Dynamic Neural
Optimization (DYNO) grow-and-prune algorithm, which dynamically performs neural pruning to
remove neurons that are dormant or redundant and neural growth to add orthogonal features during
fine-tuning. Our experiments across a variety of transfer tasks show that DYNO yields an efficient yet
performant network with dynamically shaped layers. This work further tackles the challenge towards
dynamic structural learning, adapting models during the distribution shift of changing tasks.

**1.3 Manuscript Outline**

This manuscript begins by summarizing the overall challenges identified in the structural learning
literature in Section 1.1 and presenting the contributions towards those goals in Section 1.2. Next, a
background on both artificial and biological neural networks, particularly from a structural learning
perspective, is given in Section 2. A review on structural learning is presented in Section 3 to set the
stage for the following experimental works. Section 4 proposes structural tools that permit architectural
optimization of partially equivariant neural networks, while Section 5 proposes algorithmic improve-
ments to differentiable neural architecture search, towards dynamic scheduling. A foundational study
on when, where, and how to perform neurogenesis is presented in Section 6, followed by an extension
towards structural grow-and-prune algorithms for dynamic learning environments in Section 7. Finally,
the overall conclusions and future work to follow these projects are presented in Section 8.
## 2 A Background on the Current State of Neural Networks

The following section serves as a brief background for artificial and biological networks, particularly in the context of structural learning.

### 2.1 Artificial Neural Networks

Artificial neural networks (ANNs) are a family of machine learning methods inspired by the biological brain. Their origins trace back to computational models of neurons [1], inspiring the first artificial neural networks known as perceptrons [22]: simple, hand-designed diagrams of nodes and weighted connections. Over decades, these evolved into the multi-layer perceptrons, a simple type of ANN, still used today [23].

#### 2.1.1 Basic Functionality

The basic unit of a neural network is a neuron, which is a simplified abstraction of it's biological counterpart. A neuron is characterized by the incoming connections with synaptic weights and bias weight **w**, activation function **f**, and outgoing connections. The output of this neuron will be the synaptically weighted sum of the inputs **x** and bias **b**, all passed through the activation function, portrayed in Figure 1a. For a given input, this value passed onto outgoing connections from a neuron is called its activation.

In simple fully-connected feed-forward ANNs, neurons are organized in layers, taking input from all neurons from the previous layer and outputting to all neurons in the subsequent layer, shown in Figure 1b. The inputs to the ANN are treated like a layer, such that the subsequent layer takes their value as inputs with their corresponding weights. Likewise, the output neurons of the ANN receive weighted connections from the previous layer and compute their activation as the output of the ANN.

The activation function serves multiple purposes. In hidden layers, or the layers not including the input and output, it creates nonlinearity, which increases the expressive power added by each layer. Without nonlinearity, the neural network would only be capable of linear functions with no added benefit from depth. Common choices include the Rectified Linear Unit (ReLU) [24] and Sigmoid activation functions. In the output layer, the activation function choice depends on the task. For classification networks, a Softmax function is often used to convert the output activations, with dimensionality equal to the number of classes, to probabilities.

The ANNs discussed thus far are time-invariant, one of the key differences with their biological inspiration. Another type of ANNs are spiking ANNs, which model the time-dependence of biological neurons and output discrete spikes rather than continuous values. However, these are less-suited for standard hardware and have been less developed and less popular than time-invariant ANNs.

#### 2.1.2 Optimization of Weights

The goal of neural networks are to be able to perform well at a given task. In supervised learning, labeled training data is available, allowing the performance of a network to be quantified within the loss function of the network, measuring how far the model deviates from ground truth over the training dataset. Common loss functions include cross-entropy for discrete tasks like classification [25] or mean-squared error for continuous tasks like regression.

The synaptic weights can be optimized automatically through gradient descent of the loss function after random initialization of the weights [26]. Gradient descent has shaped many of the design decisions in ANNs, tending towards properties that are permissive to gradient descent such as differentiability. Other optimization approaches exist, but gradient descent has remained the most popular. Gradient
<figure><img src="image_3.png" /><figcaption>**Figure 1:** Basic standard representations of a neuron and an ANN. The output of neuron *j* of the layer *i* + 1 layer is computed by multiplying the activations of the *i*th layer, **x**<sub>*i*</sub>, by **w**<sub>*i+1,j*</sub>, summing the products together with the bias, and passing through the activation function, *f*.</figcaption></figure>

descent tries to find the global minimum of a convex function, in this case the loss function of the
ANN which measures the error of the network over the training data with respect to its parameters. In
the space of this high-dimensional function, the gradient measures the change of the function value
with respect to change of the parameters as a partial derivative. By subtracting the current gradient
scaled by a learning rate parameter from the current parameter values, the next parameter values are
generated for the gradient to be reevaluated at. This process continues repeatedly until convergence
to an optimum, where the gradient will be zero.

Computing the gradient is accomplished via back-propagation through the layers of the ANN in reverse order [27]. While gradient descent is biologically plausible, back-propagation is much less so [28]. Although feedback connections exist in the brain, back-propagation requires precisely symmetric feedback for error communication and an explicitly known target at the final output of the network, neither of which have any biological evidence. How the brain solves this credit assignment is an ongoing research topic.

The most common basic variant of gradient descent is mini-batch stochastic gradient descent [26], where cycling mini-batches of data are used for each gradient calculation, as modern datasets are too large to efficiently compute the gradient across the entire dataset. Further improvements include momentum terms and adaptive learning rates to increase the speed of optimization and avoid problems such as local optima.

**2.1.3 Convolutional Neural Networks**

The simple ANNs discussed thus far are general networks that do not assume any symmetries in
the data. However, more powerful and specialized ANN structures have been designed to improve
performance on specific tasks. The most popular example is convolutional neural networks (CNNs)
for computer vision tasks [9]. Images are stored in $H \times W \times 3$ format, with a height of *H* pixels, a
width of *W* pixels, and a channel depth of 3 color channels. A convolutional layer is parametrized by filters that are convolved over the image, multiplied element-wise over a local patch and summed for each translation of the filter. Each channel in a convolutional layer can be abstractly equated to a neuron in a standard feed-forward layer. Each channel in each layer is connected to all channels in the layer before and all channels in the layer after. Each connection is parametrized by a filter, which is a small *k* × *k* matrix of values, rather than a single value. CNNs are a special case of ANNs that could be equivalently represented with shared weights for relative local connections and null weights for all other connections. Convolutional layers are equivariant to translation, allowing CNNs to recognize entities in an image invariant of position.

CNN architectures often begin with convolutional layers that gradually increase the channel count while decreasing the spatial dimensions. For classification and similar tasks, the tail of the network will be feed-forward layers to make the final classification decision. Additional layer types often employed throughout the architecture are normalization layers and pooling layers. Normalization layers normalize activations across batches, with the intuitive benefit of avoiding overfitting to the training data distribution. Pooling layers perform downsampling by reducing the spatial dimensions and keeping a summary statistic of the original values such as the maximum or the average.

Further innovations in CNN architectures have potentiated their progress. One of the simplest yet most effective is skip-connections, or connecting the activations of one layer directly to the layer after the next one, thus creating a sum of the outputs of the two prior layers [29]. This is theorized to stabilise gradient descent, avoiding the problem of exploding and vanishing gradients, allowing CNNs to become incredibly deep.

### 2.1.4 Architecture Design

Until the last several years, architectures of deep ANNs were always hand-designed and generally static during the learning process. This provides a consistent basis for gradient descent: computers are better than humans at numerical tasks like optimizing parameter values of a given function such as a neural network, while humans are better at more abstract tasks such as designing the structure of the network. However, the desire to automate network design is increasing as computers become more powerful and human engineering is bottle-necking the potential of ANNs. This leaves formulating ANN design as a more computationally-suited problem as an open challenge.

Applicable to this challenge, there are many related sub-communities tackling various flavors of neural network optimization beyond parameter tuning. Meta-learning, or learning how to learn, aims to improve the learning process itself of an ANN [30]. AutoML tries to automate as much of the full machine learning pipeline as possible, including but not limited to hyperparameter optimization, learning algorithm selection, data preparation, and architecture selection [31].

Structural learning directly tackles this challenge of automated structural design. We focus on one-shot structural learning algorithms, analogous to the development of an individual brain over its lifetime. Thus, the optimization of parameters is performed in conjunction with structural optimization with the final goal of an employable ANN that is structured and parameterized specifically for a given task.

Next, we detail original inspirations particularly of the structural learning processes in the brain to provide a biological context for the rest of this report.

## 2.2 Biological Neural Networks

Although ANNs were originally inspired by the brain, their characteristics have diverged. The basics of neuron anatomy and function are briefly described here before focusing on the structural dynamics of biological neural networks.
### 2.2.1 The Basic Units of Biological Neural Networks

The quintessential unit in the brain is the neuron [32]. This cell is characterized by its dendrites receiving incoming information and the axon sending information to other cells via their dendrites. A single physical connection between two neurons is called a synapse. Mutable properties that affect how two neurons communicate include synapse connection strength, number of synapses, synapse location, and neuromodulators within a synapse. Neurons propagate signals via action potentials: when the cell membrane is activated enough via the dendrites, an action potential fires from the dendrites through the cell body and down the axon to outgoing synapses. Thus, neurons communicate in a time-dependent, spiking regime.

Although neurons are the most well-known cell in the brain, numerous other cell types support and potentiate their function. Glial cells alone outnumber the number of neurons in the human brain by tenfold [33]. They serve many purposes: oligodendrocytes provide myelination that increase speed and reduce loss of action potential transmission over axons, astrocytes modulate synaptic connections and support nutritional needs of neurons, and microglia prune synaptic connections. Neurons and their supporting non-neuronal cells dynamically cooperate to form the biological neural network, capable of the highest form of intelligence we know of today.

Biological networks have much more organic architectures that ANNs, but still show some structure and regularity on various scales [34, 35]. For example, much of cortex is organized into six layers with distinct physical and functional characteristics on each layer [36]. However, a key difference is that the brain structures itself over the lifetime of an individual [37], while ANNs are hand-designed and often have static architectures through the learning process.

Many structural learning algorithms for ANNs are inspired by biological neural development, from synaptic pruning [38] to neurogenesis [39]. To better understand the role of and possibilities for structural learning in ANNs, we investigate the roles of neuron creation, neuron removal, synaptogenesis, and synaptic pruning in the brain. Neural development in biological nervous systems and the brain is a complex process and a comparison with artificial representations necessarily requires simplification, but even a simplified view of biological structural learning offers insight for the artificial process.

We focus on the epigenetic timescale of neural development as it occurs in individual organisms in two distinct stages: early development and structural plasticity in the mature brain. Both have been used as inspiration for structural learning; for example, the self-organizing ANNs in [40] are based on early development while the synapse modulation method proposed in [41] is inspired by learning over the human lifespan. Both periods are relevant for structural learning in ANNs, offering different views on the role of neurogenesis, synaptogenesis, synaptic pruning, and neural cell death.

### 2.2.2 Neurogenesis and Neural Migration

Neurogenesis is abundant and rapid during early development. Starting from around 5 weeks of embryonic development, the human brain will develop almost all of the roughly 86 billion neurons present in the adult brain [42]. After approximately 28 weeks of gestation, this vast neurogenesis slows down considerably, with only a few regions continuing neurogenesis after birth [43, 44].

As this massive neurogenesis is taking place, cells also begin to move to specific locations, organizing into distinct regions and specific structures [45]. Chemical and sensory cues guide this neural migration, which is essential for the functioning of the brain. Neurons which started in distinct germinal zones may travel through specific complex routes to reach their final destination, resulting in different composition of neural types in different sections of the brain. By 18 months of age, much of the structure of the brain will be set by early neurogenesis and neural migration [46]. Structures like the cortical columns, which inspired convolutional neural networks, result from this neurogenesis and neural migration of early development [7].
Neurogenesis becomes much less common once early development is over; indeed, it was previously believed that there were no new neurons added to the adult human brain. There is growing evidence that neurogenesis occurs in the adult brain [47, 48], but has only been identified in specific regions. While the mechanisms behind adult neurogenesis are not yet fully understood, nearby neural activity has been identified as a factor, supporting the idea that the brain can create new neurons in response to needs which are not fulfilled by existing neural circuitry [49].

Beyond their movement and connection to other neurons, neurons are also capable of change during their lifetime. Learning can produce nonsynaptic changes of neural membranes, such as membrane conductances which modulate excitability [50]. While artificial neurons already modulate their bias during conventional, non-structural learning, these changes to the neural cell body could be seen as a more drastic change to the neural activation function.

From an artificial perspective, neurogenesis and neural migration can be considered as a single event in modern artificial neural networks, which rarely consider the physical placement of neurons. The neurogenesis of early development is therefore akin to neural architecture search space definition; the fundamental components of learning are in place but not yet properly wired. Adult neurogenesis in the biological brain, on the other hand, takes place in specific sections where existing structures and well-defined activity patterns abound. An artificial analogy of adult neurogenesis could be an expansion or modification of the possible synaptic weight space, such as the addition of a new layer, convolutional filter, or individual neuron.

Early development sets the stage for cognition and learning in the mature brain. Neurogenesis on a massive, rapid scale, with new neurons being created, differentiating, and migrating, is a hallmark of early development and is relatively unique to this period. Synaptogenesis next takes this stage in early development, with axons branching through complex routes to create the immense connectome of the brain and nervous system.

### 2.2.3 Axon Migration and Synaptogenesis

During early development, as new neurons are created and migrate to specific sections, connections between these new neurons begin to establish the neural circuitry of the brain. Shortly after neurons differentiate, genes which encode proteins necessary for synaptogenesis are activated [51]. The creation of connections between neurons begins during development but continues well into early postnatal life and is thought to contribute significantly to learning and memory in adults [51].

After differentiation, neurons project axons which then also migrate, following paths predetermined by genes or dynamically shaped by the chemical gradient trail of netrins released by other cells [52]. For almost all types of neurons, their singular axon will serve as the output line of communication, carrying an electrical discharge to other neurons. Axon development and migration happen throughout early development, with some axons traversing long and complex routes; the longest in humans is the sciatic nerve, reaching over a meter from the base of the spinal column to the foot.

In the central nervous system, innumerable connections form between neurons in intricate patterns during early development. As a neuron extends its axon, it will also extend many other branches around its cell body: dendrites, which receive electrical stimulation from other neurons. Once a searching axon of one neuron reaches the dendrites of another, synapses, individual connections, may be created. The formation of an individual connection, synaptogenesis, is regulated by chemical signals, activity of both the presynaptic and postsynaptic neurons, and genetic factors [46].

In early development, as in the mature brain, nearby axons and dendrites will exchange a number of signaling molecules which, in tandem with other cues, make neurons more receptive to synaptogenesis. The fate of newly formed synapses will be decided by the activity of the connected neurons, determining if they will be stabilized or eliminated [51]. A single neuron will form many similar connections which compete for stability as failed synapses close and the axon or dendrites retract.
While genetic cues inform many of the initial paths of axon migration and branch, neural activity sculpts neuronal arbor growth and synapse formation. Disruptions to neural activity have been shown to drastically change connectome organization, from the level of individual synapse formation to entire axon branch placement [53]. In developmental synaptogenesis, neural activity is a contributing factor amongst others for synaptogenesis, but as the brain matures, activity becomes an important trigger for further synaptogenesis.

In the mature brain, synaptogenesis plays a central role in associative learning and memory. Pharmaceutical treatments such as antidepressants have been shown to produce significant increases in the number of synapses in the adult brain [54]. New neurons in the mature brain may develop connections in a different way than neonatal neurons, first developing input synapses in their proximal dendritic domain before firing action potentials [55]. In this way, the connections formed by new neurons in the mature brain may be informed by existing neural circuitry before adding new connections and functionality.

From an artificial perspective, the synaptogenesis of early development can be considered as similar to developmental neurogenesis; the preliminary neural circuitry is defined largely by genetic cues which create the possible space for future learning through synaptic tuning. While the synaptogenesis of early development can depend on activity, it has also been demonstrated in the absence of activity, pointing to circuitry designs predetermined by genes [51]. In the mature brain, however, activity is an integral part of the integration of new neurons and synapses into the neural circuitry. New neurons will listen to nearby activity before activating, forming downstream synapses based on observed activity patterns. As with neurogenesis, synaptogenesis sets the stage for learning to take place, but it is with the stabilization and elimination of parts of this neural circuitry where specific learned patterns in the brain are shaped.

### 2.2.4 Synaptic Pruning

Connectivity patterns in the mature brain are learned through a process of correlation detection, synaptic competition, and eventual synapse elimination, i.e. pruning. The pruning process has been identified as a crucial component of learning and of typical neural functionality. It is widespread in the central nervous system but has also been observed in the peripheral nervous system [56]. Pruning occurs at a massive scale during development but continues throughout lifetime.

The prevalence of pruning in the developmental and adult brain indicates the presence of numerous redundant or excess connections formed during development. The reasons for these exuberant connections are not fully clear, but there are multiple possibilities, one being physiological or structural roles of these excess connections during early development [56]. More importantly from an artificial perspective, however, is the information available for neural circuit determination. During neurogenesis and synaptogenesis, possible communication between cells is limited to chemical signals and a limited number of connections. Excess connections offer a rich information pipeline for neural circuits to refine into specific functional patterns.

The information pipeline of highly-connected neural circuits offers a way to detect correlation between connected neurons. Experimental evidence has demonstrated that neural activity regulates synapse elimination [51], showing that correlated synaptic activity informs about the relevance of synaptic connections and influences synapse stability and elimination as postulated by Hebbian theory [57]. Neurons with high levels of correlated activity will retain connections while others are pruned, forming specific circuits based on activity patterns.

The development of the visual system demonstrates the importance of activity for eventual circuit formation. Highly correlated patterns of spontaneous activity in the form of retinal waves occur throughout development, driving activity-dependent pruning in the visual system [58]. Experiments in ocular deprivation have shown that a deprivation of activity in one eye can lead to structural changes
in the visual cortex which result in a lack of response to new activity in the covered eye [59]. The brain undergoes significant rewiring in response to specific novel activity patterns, in the visual cortex and elsewhere.

Beyond the local information of presynaptic and postsynaptic neural activity, other signaling mechanisms influence synapse elimination. Microglia in particular have been identified as an important mediators which promote and constrain synaptic elimination [60]. The microglia can tag unwanted synapses with specific proteins leading to their elimination [61]. While the full internal mechanisms of the glial cell tagging decision are not yet completely understood, the molecular mechanisms involved in synapse elimination are spatially and temporally dependent, and disruption to these mechanisms has been linked to brain dysfunction [56]. From an artificial perspective, this motivates the design of pruning decisions which are made not only based on internal factors such as synaptic weight and activity information, but through complex and dynamic policies which account for various information over entire regions of the brain.

Active neurons can have a large number of synapses pruned with little effect on the neural cell body. However, in the event of pruning along an entire axon branch, the branch will retract or be eliminated from the cell [62]. Retrograde signaling mechanisms pass information from distant axon heads back to a neuron cell body to inform of pruning events and to initiate an appropriate response [63]. This may involve restructuring of the axon or dendritic tree of the neuron, but can also in some cases elicit neural cell death. In this way, sufficient synaptic pruning and appropriate signaling can lead to complete neuron removal, as we examine in the next section.

### 2.2.5 Neural Cell Death

Neural cell death can be distinguished into two main types based on timing: a common early cell death occurring in neural progenitors and immature neurons, and a late cell death which occurs in postmitotic neurons [64]. There are many mechanisms and underlying causes for neural cell death, with some mechanisms shared between the two types, but in many ways, they are distinct processes. We consider both types, aiming to clarify the decisions leading to cell death in both cases.

Early cell death is a necessary part of neural development and occurs either before full differentiation of the neuron and its projections or during synaptogenesis [65]. This process is a mechanism for cell number and quality control, removing unwanted cells and those damaged by proliferation. While this contributes to the final neural circuit through cell number control, experiments on the prevention of neuronal cell death during development have demonstrated that phenotypically similar circuitry can arise despite the presence of excess neurons [66]. However, neural cell death does regulate mature functional circuitry through controlling competition over factors necessary for cell survival such as chemical signals from glial cells [65]. Notably, early neurons require a constant signal to prevent cell death; in this way, elimination can be seen as the default end of new neurons with persistence as the exception.

Early neural cell death is abundant in early development, in tandem with the massive amounts of neurogenesis, and it also accompanies adult neurogenesis. Adult neurogenesis has been observed in the olfactory bulb but from the new neurons generated, only 50% will survive for extended periods of time and be integrated into the existing circuitry. Interesting, in this case, neural activity was demonstrated to be important factor in determining cell survival [55].

From an artificial perspective, inspiration from biological early neural cell death can inform the decision of elimination of new or untrained operators such as layers. As the parameters of these operators are not yet informed, evaluating their potential for removal can be difficult yet rewarding. The biological process of creating a large excess of cells and eliminating quickly can serve as motivation for artificial structural learning.
Programmed cell death in mature and connected neurons is a much more rare event in healthy brains. The main causes of postmitotic neural cell death are the death of other connected neurons (transneuronal degeneration), axon degeneration (Wallerian degeneration), or the result of various signals from other neurons and glial cells [67]. In the adult brain, these events typically occur in response to an injury or as part of a neurogenerative disease. Transneuronal degeneration occurs relatively frequently in young animals where new neurons are still being formed, but is much less common in adults. Wallerian degeneration mostly occurs in response to axon injury, inflammation, or genetic impairment. The signals which can elicit mature cell death are numerous but are most common in neurogenerative diseases such as Alzheimer's disease [67].

As the death of highly-connected, postmitotic neurons is such a rare event in healthy brains, the main contribution of neural cell death to the learning process occurs early on, during or even before synaptogenesis. This can result in a chain of cell death, transneuronal degeneration, but this event becomes rarer as neural circuitry matures. This indicates that, as opposed to individual synapses which are often pruned throughout learning, neurons are costly to remove. While this principle correlates well to highly parameterized ANN operators, it is much less costly to remove an artificial neuron than a real one, as the costs of cell removal and replacement neurogenesis are greatly reduced.

### 2.2.6 From Biological to Artificial

It is a drastic simplification to summarize these complex processes in the brain, which are still not fully understood, with the four previously defined structural learning operators. However, artificial neural networks have different information and energy requirements than biological neurons. Artificial neurons which are represented as distant can be connected just as easily as neurons which are represented as adjacent. Information is transferred between artificial neurons without loss; synapses don't have to be protected from possible interference. There is no bandwidth limitation for individual synapses nor additional information gained from synapse location, so the relationship between presynaptic and postsynaptic neurons can be summarized by a singular continuously-valued synaptic efficacy measure. The similarities and differences between biological neurons and the artificial models inspired by them are constantly evolving and being better understood, but it is clear that their fundamental differences require different learning operator definitions.

The result of these simplifications is that many decisions made by cells in the biological brain are aggregated into the structural learning operators. The variety of information sources for cellular decisions are streamlined in ANN operators; chemical signaling is mostly ignored in ANNs as is cellular contact. However, recent trends in ANN literature have begun to explicitly reintroduce the sort of information pathways used in the biological brain, such as the role of non-neuronal cells [42], Hebbian synaptogenesis [68], and activity-dependent neuron removal [69]. Understanding the complexity of these structural changes in the biological brain can bring new perspectives on the formulation of structural learning operators in ANNs.
# 3 Analysis of Structural Learning

Structural learning is the optimization of both the architecture of an ANN and the parameters of that architecture in a single process. As shown in the following work, most of this form of structural learning used in practice has taken the forms of pruning, or the removal of parameters to reduce network size while maintaining performance, and Neural Architecture Search (NAS), or the automated search of architectures within a search space. Relatively few papers have pursued developmental structural learning, which involves creation of new structures or connections within the network.

In this review, we take a biologically-inspired view of structural learning in ANNs. The latest analysis of structural learning in both biological neural networks and ANNs was completed more than two decades ago [70]. Since then, there have been many advances in both artificial intelligence and neuroscience research, prompting the need for a modern analysis. These links have already been well-defined for the neural and biological modeling communities, but there is still a disconnection between these communities and deep-learning with ANNs.

In an effort to surpass the structure design bottleneck, we look to the parallels of ANNs in biological nervous systems in order to define a framework for structural learning in Section 3.1. We survey methods of structural learning in ANNs already existing in literature, providing broad statistical overviews in Section 3.2 as well as detailed tables of features over our selected corpus at the end of this chapter, then diving deeper into each operator of our framework across implementations in Sections 3.3 and 3.4. We finally give suggestions on how to progress forward on current challenges in Section 3.5.

The aim of this work is to shed light on a set of similar research directions which are already well established in disjoint fields but not yet often linked. Using a common language for structural learning may help connect the many subcommunities of ANN research that are already researching similar approaches to structural learning with different implementations, abstractions, and goals.

This work is heavily based on Maile et al. [16]. Recent related works within the scope of this survey may not be included after its original publication.

## 3.1 Framework Definitions

In order to understand structural learning, we propose a definition which, in this work, will be used to define the scope of study. We consider structural learning to be a change to a neural network during its life-cycle through any of the following four atomic operators, referred to subsequently as the **neural operators**:

* **Neuron creation:** the addition of a unit to an ANN,

* **Neuron removal:** the removal of a unit from an ANN,

* **Synaptogenesis:** the creation of a non-zero weight between two units,

* **Synaptic pruning:** the removal, or change to zero, of a non-zero weight between two units.

We purposefully open the definition of neuron creation and removal to different unit types: in many structural learning algorithms, entire groups or “units” of parameters are added and removed together. Units may be as simple as a neuron in an ANN, or be another higher level structure such as a convolutional filter, channel, layer, or module of layers. A unit contains a portion of the parameters within the network and provides basic organization to make the construction of a network more tractable. While
the basic computational units in both the brain and in ANNs are conventionally called neurons, we use unit as a more generic term that can include any structure within a network being used as nodes in a graphical representation of the structure.

The terminology used in this framework is not yet adopted in structural learning literature. For example, what we define as neuron creation is often referred to as neurogenesis, although the latter process is much more involved in the brain than in the artificial case as discussed in Section 2.2.2. The differences between existing and often inconsistently used terms and our own will be discussed subsequently.

The dominant form of learning in ANNs is weight training through error backpropagation, which optimizes synaptic weights and neuron biases in an objective function landscape using gradient descent while keeping the other characteristics of each neuron constant: the layer type, activation function, and pattern of non-zero connections. In this chapter, we aim to characterize the impact of these neural operators on learning, which permit changes to connectivity and are often coupled with weight training in a bilevel optimization process.

To understand the role of these neural operators in the context between biological and artificial neural networks, we next review relevant implementations of them in existing ANN literature.

## 3.2 Characterizing Structural Learning in ANNs

Using our neural operators of neural creation, synaptogenesis, neural pruning, and synaptic pruning allows for characterization of these algorithms from a structural learning perspective, drawing links between different implementations of the neural operators. Some instances of these neural operators are brain-inspired by design, while others are incidentally correlated to biological operations.

In general, ANNs are constructed as a composition of individual components, layers, in a directed and usually acyclic graph. We refer readers to Goodfellow et al. [23] for a full review of the different basic components of modern ANNs, such as convolutional, pooling, and normalization layers.

### 3.2.1 Scope of Study

Our 59 collected papers from ANN literature each demonstrate at least one of the four neural operators, neural creation, synaptogenesis, neural pruning, and synaptic pruning, over the course of training an individual ANN via an optimization method. We focus on standard ANNs with parameterized connections, generally with feed-forward, convolutional, or recurrent layers.

Our focus on standard ANNs excludes time-dependent networks, like spiking neural networks. While more biologically inspired, these have not been as powerful nor as common as time-independent ANNs so far on standard computational hardware.

ANNs simplify the biological complexity of connection strength between two neurons into a single continuous value. We consider all non-binary changes to this weight, which could represent biological structural change, to be weight training, not structural learning. This includes soft structural methods that do not make any discrete changes to the architecture and thus no changes to the effective dimensionality of the objective function. Only hard structural methods are evaluated, including those which involve a discretization step after soft structural learning, such as Liu et al. [19].

While synaptogenesis necessarily follows neurogenesis in the brain through different processes, we consider the non-learned creation of connections of a new unit to be encapsulated within neuron creation in ANNs, such as adding a neuron with default connections to all neurons in the previous and subsequent layers. Thus, in our characterization, algorithms are only labeled with both neuron creation and synaptogenesis if there is a connection learning component for existing units in addition to a process for creating new units with a default set of connections.
<figure><img src="image_4.png" /><figcaption>**Figure 2:** Levels of graph abstractions found in structural learning algorithms. From the level of (a) feed-forward neurons, there are the two levels of convolutional paradigms: (b) on the same level as FFNNs with weight-sharing and systematically pruned connections, and (c) on a higher level with a parameter matrix on each synapse and an activation matrix within each neuron, compared to singular values respectively on the lower level. The shared weights in (b) are referred to as a kernel, which are the small matrices parameterizing each edge in the higher level paradigm in (c). On an even higher level is the NAS super-network paradigm (d), where each edge may contain parameters based on the operation type, like the parameter-less skip-connection and pooling and the parameterized convolution operations shown. Higher levels have higher dimensionality, structure, and complexity within each node and edge.</figcaption></figure>

Many structural search papers enumerate all possible options within their prescribed search space throughout the learning process in order to select the best combination of structures to build the final architecture. For example, some NAS algorithms initialize all possible layers with all possible connections and learn which ones are best. We consider this as entity pruning, rather than entity creation.

A biologically inspired family of algorithms for discovering ANN structures automatically are evolutionary algorithms. These algorithms model biological evolution by maintaining a population and recombining the genes, which may directly or indirectly encode architectures, of successful candidates to produce more candidates. Some such methods meet our criteria for structural learning by persisting network weights across individuals within the population [71, 72]. However, many evolutionary methods maintain a population of individuals with different structures and weights, such as NSGA-Net [73] and NEAT [74]. These methods include structural learning operators; in NEAT, for example, new neurons or connections can be added through mutation. However, to compare with structural learning in single networks, changes through mutation from one generation to another would have to be studied on specific individuals. We further discuss evolutionary methods in Section 3.5, but in the following sections, we only include search methods that which persist weights and study structural changes on individual networks. This limits the collected corpus to one-shot algorithms, which pursue a more continuous path in the changing objective function space during the co-optimization of architecture and parameters.

We exclude architectures with self-gating mechanisms, such as LSTMs [75] and transformers [76], from this framework and corpus. These neural networks change their own structure in a transient and input-dependent manner, occurring on a shorter timescale than structural learning. This is more akin to neural circuitry gating and modulation [77]. We discuss such dynamic architectures further in Section 3.5.2.

The abstraction from individual biological neurons to artificial feed-forward neurons, the most basic type, is clear. However, convolutional neural networks (CNNs) have two levels of abstraction, depicted in Figure 2. The first is by considering a CNN as a special case of a feed-forward neural network
<figure><img src="image_5.png" /><figcaption>(a) Occurrence of operators over time.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Co-occurrence of operators within the collected corpus.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(c) Occurrence of unit type over time.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(d) Co-occurrence of unit type within the collected corpus.</figcaption></figure>

<caption>**Figure 3:** Characterization of neural operators and unit type used in our selected corpus of papers.</caption>

(FFNN) with non-local weights zeroed out and local weights shared. The other paradigm moves the neuronal unit up to a higher level, considering a channel in each convolutional layer to be the neuron abstraction. Connections between channels represent synapses and are parameterized by a small matrix kernel rather than a single value, and the activation within the neuron is a larger matrix rather than a single value. While the lower level paradigm continues to multiply the singular activation coming as input to each synapse by the single shared synaptic weight value, the higher level paradigm performs a convolution of the kernel parameter matrix over the incoming activation matrix before summation across connections. Using the lower level paradigm of Figure 2b for structural learning enables partial-area convolution or changing kernel characteristics such as size and striding across neurons and is denoted as "Kernel shape" in Table 1. Using the higher level paradigm of Figure 2c becomes adding and removing channels as neurogenesis and neural pruning, respectively, denoted as "Filter/channel" in Table 1. An even higher level paradigm is super-networks in NAS, shown in Figure 2d, denoted as "Layer(s)" in Table 1. We consider works on structural learning which function on these different levels of abstraction.

### 3.2.2 Characterization of the Collected Corpus

The goal of this work is to characterize structural learning, rather than document all uses of any of the four neural operators in the vast ANN literature. Our selection of 59 seminal works is therefore not comprehensive and is rather intended to be a qualitatively representative sample, as there are
orders of magnitude more papers in the subfields of NAS and pruning than in the developmental structural learning space. Completing a more rigorous collection process for complete quantitative characterization would be prohibitively nuanced, for reasons including inconsistent vocabularies surrounding structural learning used across subfields and subtle differences between novel structural algorithms versus applications of existing techniques to new architecture types and task domains, and thus is beyond our scope. The 59 works in the collected corpus provide a diverse and informed view of the structural learning literature.

We refer the reader to other reviews for a more complete view of specific domains. An overview of early works in structural learning and their comparison to biological learning is presented in Quinlan [70]. More recently, Deng et al. [78] covers pruning and many other methods for model compression. Hoefler et al. [79] discusses sparsity in neural networks, which is usually achieved via pruning. Elsken et al. [80] describes the state of NAS as a burgeoning sub-field, while Xie et al. [81] is more recent and focuses on weight-sharing NAS. He et al. [31] covers AutoML, which encompasses methods that contribute to automating the machine learning pipeline beyond parameter tuning and thus includes structural learning as well as data preparation, hyperparameter optimization, and other types of architecture search. Stanley et al. [82] gives an overview of the use of evolution for neural network optimization, including methods which change neural structure over generations. Evolutionary developmental methods which evolve rules for structural learning are reviewed in Miller [83]. Parisi et al. [84] discusses methods for continual learning in ANNs, many of which use forms of structural learning to increase the information capacity of networks performing multiple tasks. Han et al. [85] covers dynamic ANNs, considering adaptability of the network beyond just connectivity during training as in our framework. Our goal in this chapter is to offer a coherent synthesis of works from these different domains under the framework of structural learning.

The general statistics for neural operators and units across the collected corpus are shown in Figure 3. These plots show trends and frequencies of characteristics across the collected corpus over time, although not necessarily representative of structural learning without the aforementioned biases in our sample collection. Neural creation and synaptic pruning appeared earlier, while synaptogenesis and neural pruning have become more popular in recent years, shown in Figure 3a. Most papers perform a single operator, but a significant portion perform a combination, shown in Figure 3b. Operating on units of standard neurons has been consistently used over time, while the arrival of deep learning architectures is evident in Figure 3c with structural learning over layers and convolutional units of kernel shape and channels becoming more popular after 2010. The impact of deep learning is also evident in Figure 3, with a significant increase in pruning methods while the automatic creation of new units or connections did not undergo a similar expansion of interest. We expect this is partially due to the relative complexity of the creation decision, which we explore in Section 3.4. Finally, as with neural operators, operating on a single unit type is common, but more algorithms are permitting structural learning across different units and layer types, shown in Figure 3d.

Further trends over the qualities of each algorithm in the collected corpus sorted by publication year are detailed in Tables 1-2 at the end of the chapter. The earliest algorithms were designed before convolutions and recurrent layers, but now most algorithms are designed for and demonstrated on ANNs with convolutional layers, with some additionally applied to dense layers or recurrent layers. Regarding tasks, nearly all algorithms are demonstrated with supervised learning tasks, particularly image classification in more recent years. Some specific techniques are applied to multi-task, continual learning, more specialized computer vision, natural language processing, sequential data, and reinforcement learning tasks: all of these pose additional challenges above the relatively simple image classification or tabular dataset classification baselines. We discuss existing structural learning methods in the context of each operator in Sections 3.3-3.4 and relevant implementation trends in Section 3.5.1.

Each of the four neural operators have different considerations and effects within structural learning. Performing synaptic pruning or neuron removal on an ANN is more tractable than their additive
<figure><img src="image_9.png" /><figcaption>**Figure 4:** Neural pruning and synaptic pruning in a fully connected ANN [86]. When a neuron is pruned, as in structured pruning, all incoming and outgoing synapses are also removed. Synaptic pruning additionally removes synapses between unpruned neurons and is also referred to as unstructured pruning.</figcaption></figure>

counterparts, because their structural learning search space is naturally defined by the existing network structure rather than being open-ended. As the two pruning operators have also been more commonly used in literature, we will consider them first in Section 3.3 before discussing the two creation operators in Section 3.4.

## 3.3 Synaptic Pruning and Neural Pruning

Our main questions to characterize pruning in our selected papers are:

* **Goals of Pruning:** Why remove connections or units from an ANN?

* **Biological Inspiration:** How can biological synaptic pruning and programmed neural cell death inspire artificial methods?

* **Architecture Specificity:** How has pruning been adapted to different ANN architectures, such as dense layers and convolutional layers?

* **Measuring Entities:** How is the search performed? How is importance of a connection or unit measured in order to make the pruning selection?

* **Scheduling Pruning:** When and how often does pruning occur with respect to optimization of the unpruned parameters?

* **Handling Pruned Entities:** How should pruned connections and units be handled?

* **Impact on Weight Optimization:** What is the impact of pruning on optimization of parameter weights?

The following subsections consider the implementations of synaptic pruning and neural pruning found within our corpus. We will first discuss a subset of these questions for pruning in general in Section 3.3.1 and then dive further into specifics of all questions for synaptic pruning in Section 3.3.2 then neural pruning in Section 3.3.3. Finally, we will draw conclusions and connect our discussion back to biology in Section 3.3.4.
### 3.3.1 Pruning Generalities

Pruning aims to remove parameters in an optimal manner, but their effect is not independent, and evaluating the effect of all possible combinations of parameters to prune on the network performance is not computationally feasible. Not only do the combinations grow exponentially with the size of the network, but the cost of completely evaluating each one by locating the new optimum through optimization to convergence is prohibitively expensive. Thus, algorithms generally make an approximation by estimating the effect of removing each parameter independently from the local landscape of the objective function on training data [79]. This naturally leads to greedy pruning decisions from the static set of weights, so parameters that currently seem beneficial to remove are selected at the time of pruning. Algorithms can only assume that this process approximates the globally optimal solution for what to remove from a network. Many Neural Architecture Search (NAS) papers formalize the difficulty of optimizing architectures and network weights simultaneously as a bilevel optimization problem, to be discussed further in Section 3.3.3.

**Goals of Pruning:** Pruning an ANN, or the removal or zeroing of portions of the network, to make it more efficient in space or time while maintaining performance is one of the many orthogonal ways to perform network compression. Using smaller networks with the same performance is particularly desirable for real-time applications and resource-limited devices such as mobile. Pruning large, over-parametrized networks to find their best-performing sub-networks is also a method for finding a high-performing architecture for a given network size. These naturally lead to applications of pruning in multi-task learning, where portions of the network are shared between tasks and others are task-specific, and NAS, where the architecture is optimized on a higher level.

Beyond efficiency, pruning can also yield generalization benefits [79, 87]. Due to over-parameterization of the base model, pruning can remove learned noise. This is why many pruning schemes show slightly increased test performance at low to moderate pruning levels. However, the benefit only applies within the initial distribution: pruning may come with a cost of reduced robustness to distribution shifts, but this may be ameliorated by explicit regularization [88].

**Biological Inspiration:** Activity-based synaptic pruning is a fundamental component of structuring the information circuitry of the biological brain, integral in varied processes such as memory formation and motor skill learning [51, 89]. Similar to ANN pruning methods [38, 90, 91], synaptic pruning in the adult brain uses measures of information passage in synapses to remove redundant connections and form sparse functional patterns. Less common is neuron removal, which is most prevalent during early development when neuron removal is the default for new neurons and only those with useful activation patterns survive to maturity [42]. In both cases, the brain relies on the creation of an excess of neural circuitry which is refined into the pertinent functional circuitry through pruning and removal.

**Measuring Entities:** Each entity must be evaluated in order to make an informed pruning selection. The final effect of changing an entity at a given training step cannot be predicted. Entities to prune may be selected by a salience metric computed per-entity designed to estimate the effect of removing the entity on the performance. Masks can also be used, introducing a new parameter for each entity that controls whether it is active or not. This is particularly useful for ephemeral pruning, where pruned entities may be reactivated later in the course of learning. The mask may be discretely controlled by a salience metric or relaxed to continuous values that can be trained via optimization. Finally, regularization can be used to encourage sparsity of network or mask parameters during gradient descent steps by adding a magnitude-penalizing term to the objective function.

Once the selection metric or mask is evaluated, the algorithm may then use either ranking or thresholding, at either a local or global scale, to discretize the pruning decision for each parameter. Ranking allows a defined proportion to be pruned, which may be particularly desirable for hard time or space constraints of the final ANN, while thresholding allows for only parameters meeting a score criteria to be pruned, which can be more consistent over iterative pruning phases. Local pruning makes the ranking or thresholding decision within a structure such as a layer, which makes comparisons more
homogeneous, while global pruning makes these comparisons over the entire architecture, which
allows the algorithm to determine the pruning level for each structure automatically.

**Scheduling Pruning:** The schedule of pruning determines how it is interwoven with weight updates, including when and how often. Iterative pruning, rather than all-at-once, has generally been more effective [90]. It allows the network to evaluate entities in a more intermediate state of pruning, lessening the effects of the assumptions of pruning independence between parameters, high order terms ignored in most metrics, and local objective function evaluations. On the other hand, fully pre-training a network before pruning may be expensive but can be completed independently of the pruning, such as using an off-the-shelf pre-trained model. Most pruning algorithms begin with a large, dense network, while some perform dynamic sparse training, where the initial network is sparsely initialized [90, 92–94].

**Handling Pruned Entities:** After pruning, the pruned entities may be either permanently disregarded or allowed to be revived, which we name ephemeral pruning. We distinguish ephemeral pruning from entity creation if the revival is done with the same method as the pruning and thus is bounded to the same search space as pruning.

**Impact on Weight Optimization:** Because the objective of pruning is often to at least maintain performance, this is synonymous to reducing the dimensions of the objective function space as much as possible while maintaining or decreasing the cost of the found minimum upon convergence of the objective function. Convergence to a globally optimal point is not guaranteed for standard gradient descent techniques in structurally static networks, but the local minimum found usually has a low enough cost in practice [23]. Because pruning potentially changes the cost and location of each optimum, this is necessarily an even more difficult optimization than training a structurally static network. Techniques mentioned so far such as iterative pruning and regularization intuitively help ameliorate negative effects of the discrete changes in the shape of the objective function while performing gradient descent in parallel to pruning, thus aiding in the search [79].

Pruning may be categorized by the level of structure used in the pruning process, as shown in Figure 4. Unstructured pruning, which we discuss further in Section 3.3.2, is the zeroing of any parameters within an ANN and thus is generally considered synaptic pruning for our neural operator definitions. Structured pruning, which we discuss further in Section 3.3.3, is the removal of entire units and their associated parameters within a network, such as channels, neurons, or layers, so it is synonymous with neural pruning.

### 3.3.2 Pruning Connections

**Goals of Pruning:** Synaptic pruning is the lowest level and simplest form of pruning; it intends to remove parameters, which each represent a connection, without any pattern or structure. The earliest pruning papers began with unstructured pruning [38, 95]. Pruning synapses in a network is accomplished by zeroing connections and may allow a lower space complexity of storage by storing the remaining parameters in sparse matrices. However, it does not yield significant improvements in time complexity of training or inference without specialized software for accelerating sparse matrix multiplications [90]. ANNs after unstructured pruning can outperform dense models of the same memory footprint and architectural skeleton[96], showing that the pruned models can be more specialized and effective. The main applications of synaptic pruning are for network compression [38, 86, 90, 95, 97, 98] and for counterbalancing synaptogenesis [68, 93, 94, 99–101]. It also benefits multitask learning such as in Peng et al. [91], where only important connections are saved for the current task and the pruned connections are reused for future tasks.

**Biological Inspiration:** Synapses are continuously being generated and pruned in the adult brain as a part of learning [89]. Synaptic pruning in the biological brain can result in weight change, removal of synapses in otherwise connected neurons, or wiring changes, complete disconnection of two neurons
<figure><img src="image_10.png" /><figcaption>(a) Single parameter stochastic gradient descent, where continuing descent from $s_1$ results in a local minimum.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(b) Double parameter stochastic gradient descent, where continuing descent from either $s_1$ and $s_2$ results in descending to the global optimum.</figcaption></figure>

**Figure 5:** Comparison of stochastic gradient descent in a toy loss landscape without and with an extra dimension (figure reproduced with permission from Hoefler et al. [79]). Transitioning from (a) to (b) represents creation, while transitioning from (b) to (a) represents pruning. $s_1$ and $s_2$ are potential parameter value(s) at the time of structural change.

[102]. The latter is the equivalent of ANN pruning, where continuous weight values are removed or set to zero. In the brain, this decision is based on activity correlation [57], where neurons with higher absolute correlation in their activity stay connected while other connections are pruned. This can be modeled as a locally-available measure of importance based on Fisher information [103], similar to unstructured pruning methods in ANNs. However, other decision factors in biological synaptic pruning could be further explored in ANNs. Glial cells have been shown to tag potential synapses for removal using chemical signals over regions of the brain, using spatially and temporally dependent tags [56, 60, 61]. This allows for coordinated pruning of multiple synapses on different neurons which share characteristics such as a response to certain activity patterns.

**Architecture Specificity:** Unstructured pruning is simple to apply to different neuron and layer types, since it can be agnostic to the layer type and just removes single parameters at a time. For convolutional layers, synaptic pruning can mean removing single values within a 2-D kernel [68, 86, 93, 97, 98, 100, 101] or the entire kernel itself [94]. Although each kernel contains more than a single parameter and is thus a higher structure, it can abstractly represent a connection between channels in convolutional layers, similar to a connection between neurons in dense layers [94], and cannot be explicitly removed from the 4-D convolutional filter matrix, only zeroed, so it is more akin to unstructured pruning.

**Measuring Entities:** Salience metrics try to estimate the importance of each connection in order to prune unimportant connections. They can be as simple as the absolute parameter value itself such that weights near zero are pruned [68, 86, 100], which is intuitive as they would have a lower impact on the ANN than weights with stronger magnitudes. More complex calculations include gradients, which measure sensitivity of the network outputs with respect to each parameter, so low values signify unimportance. Any metrics requiring a derivative of the objective function must be computed with respect to training data. For most modern datasets, computing them for the entire dataset is not feasible, so mini-batches are often used. A local estimate of the effect of removing a single parameter on the objective function may be derived using a Taylor expansion [104]. The earliest pruning methods often used up to the second order term, including a full Hessian matrix, within their saliency calculations [38, 95], while current methods usually use cheaper but less accurate approximations with at most first-order derivatives over a single mini-batch [91, 98, 101, 105]. For example, the diagonal Fisher information matrix is a first-order metric, scaling linearly with dimensions in computational complexity, that is often used as an approximation of the Hessian matrix, a second-order metric [91]. The choice of salience metric is usually a trade-off between efficiency and effectiveness, but exactly how useful the precision of saliency is may be confounded with the greedy, iterative pruning process
often used: such metrics can only measure local information without interdependencies, so using
more expensive methods may be futile.

Masks are also utilized in synaptic pruning. The mask may either be relaxed to be continuous so it
can be trained along with the other network weights through optimization [93, 94, 97, 106] or remain
discretely binary and be controlled with metrics [68, 100].

**Scheduling Pruning:** Regarding the training portion of the schedule, pre-training the large unpruned network is a common first step to a pruning, but some synaptic pruning algorithms avoid this for training time or space efficiency. Lee et al. [98] forgoes the pre-training step and instead prunes a newly-initialized model using a first-order derivative metric of connection sensitivity. Hassibi et al. [95] determines the optimal weight update for the remaining parameters after each pruning, but the inverse Hessian required is too expensive to compute exactly for modern ANNs, although estimations can be used [79]. Most neuron-level synaptic pruning algorithms use fine-tuning optimization steps after pruning to arrive at the final architecture with optimized weights.

**Handling Pruned Entities:** Most metric-based synaptic pruning papers do not allow discrete synapse revival, as computing metrics for inactive parameters often is not possible. However, masked selection techniques are more amenable to ephemeral pruning of synapses, which may help ameliorate the negative effects of greediness. Dynamic connection search allows previously pruned connections to be revived if they show a resurgence in parameter importance later in training [93, 94, 97, 100].

**Impact on Weight Optimization:** Synaptic pruning may decrease the dimensionality of the objective function search as finely as a single dimension at a time. The proceeding effect conjectured by Hoefler et al. [79] is demonstrated in Figure 5. This artificial example supports performing more gradient descent in the higher dimensional space and pruning later in the course of training, at least for traditional pruning occurring intermittently during training. If the network in Figure 5b is initialized to $s_1$, pruning $x_2$ before the value of $x_1$ surpasses the coordinate of the local maximum will result in convergence to a local minimum, but pruning after should allow convergence to the global minimum. Pruning non-zero parameters results in a displacement in the search space based on how large in magnitude the pruned parameters were. Magnitude-based pruning has a minimal but not negligible effect on the displacement of the location in the search space, and may also be more greedy. The performance change due to this displacement does not have clear consequences: the final performance cannot be tractably predicted for any given structural operation. Bartoldson et al. [87] found that the immediate performance drop after an iterative pruning step was positively correlated with improved generalization and performance of the final network, while Laurent et al. [104] had conflicting findings, showing loosely inverse correlation. While the objective function is an intuitive tool for optimizing the pruning selection, the interacting effects of the many local assumptions and approximations as well as hyperparameters such as algorithmic details confound its use.

### 3.3.3 Pruning Units

**Goals of Pruning:** Neural pruning is the removal of units within an ANN, such as neurons in fully-connected layers, filters or channels in convolutional layers, or entire layers themselves. Thus, neural pruning has different selection and scheduling considerations compared to synaptic pruning. Structured pruning easily yields both time and space efficiency benefits: entire portions of the network can be removed, thus skipping their computational steps and storage requirements.

Beyond network compression, structured pruning is also a very common abstraction for performing Neural Architecture Search (NAS), an emerging sub-field pursuing the automation of architecture engineering at the layer level. ANN architectures are selected from a predefined search space of possible architectures and evaluated according to a performance estimation strategy. This performance estimation then informs the search, which can be based on evolutionary algorithms, reinforcement learning, gradient descent, or other search methods. Structural learning can be used for NAS by using any of the four neural operators as search operators on persistent networks. Some existing NAS methods
<figure><img src="image_12.png" /><figcaption>**Figure 6:** Structured pruning in convolutional ANNs (figure reproduced with permission from Li et al. [107]). Each stack represents the channels of the activation state, while the matrices represent the kernels, which themselves are nxn matrices of parameters, for each convolutional layer, with each row being the kernels for each channel in the previous activation and each column being the kernels for each channel in the next activation. Decreasing the incoming channel count in the first convolution is represented in orange. The blue pruning decreases the outgoing channel count of the first convolution and the incoming channel count of the second convolution. This is synonymous with pruning both the incoming and outgoing connections of a neuron in a dense layer. The projection shortcut performs a 1x1 convolution and is pruned accordingly as necessary for dimensionality agreement.</figcaption></figure>

function as structural learning, integrating weight tuning with architectural change to arrive at a final static network in terms of both parameters and architecture. Layer-searching NAS algorithms that use structural learning generally employ a hand-designed super-network that contains all possible layer types over all possible interconnections, then perform the operator of neural pruning after optimizing the super-network to derive the final architecture.

**Biological inspiration:** In the adult brain, programmed neural cell death is most common following neurogenesis. During early development, neurons must develop sufficient active connections to avoid the default fate of removal [65]. This is similar to one-shot NAS methods such as DARTS [19] where large super-networks are created at initialization with only a minority of neurons remaining in the final architecture; however, biological cell removal is progressive and occurs at different rates in different regions of the brain, more similar to Maile et al. [18]. Neural cell death also follows neurogenesis in the adult brain; in the olfactory bulb, only 50% of newly generated neurons survive to be integrated into existing circuitry, with neural activity being a critical factor in determining cell survival [55]. This is most similar to neuron-level grow-and-prune methods such as NeST [68] and activity-based neural pruning [69, 105, 108], but further research of structural learning methods inspired by adult neurogenesis and subsequent neuron removal is needed.

**Architecture Specificity:** The structural types present within a network require some consideration for structured pruning, depending on the desired level of structure to prune. Many convolutional structural learning algorithms use the higher level paradigm, shown in Figure 2(c), for pruning [101, 107, 109, 110], as depicted in Figure 6. This allows an algorithm to be applied to convolutional and dense layers, even within the same architecture. Some sub-unit pruning in convolutional layers can still be effective for computational efficiency in contrast to unstructured pruning, such as pruning via striding within kernels [111] and limiting the area of convolution [68]. Beyond each layer's type, the surrounding structure and functionality may also be utilized in more specialized pruning: Kang and Han [69] prunes channels that tend to have low signal after the standard subsequent batch normalization and ReLU operations, while Gordon et al. [112] regularizes and prunes channels via the batch normalization scaling parameters. Exploiting such known structures in the limited architecture space of those including these structures has a trade-off between search space flexibility and effectiveness within those search spaces. When performing structural learning on units, the algorithm must ensure that all intermediate activations maintain compatibility with all incoming and outgoing layers: for example, neurons tied by skip-connections can be grouped during structural operations [112].
**Measuring Entities:** Structured pruning with heuristic search requires metrics or selection methods that can be measured or enforced at the level of desired unit of pruning. Similarly to unstructured pruning, early methods used perturbation-based selection [99, 113], where the algorithm measures the performance change from removing each neuron individually and retraining the rest of the network, but this does not scale well with the size of the network. One recent exception used a greedy forward search, where all neurons in a layer are pruned and then useful units are individually added back in [114]. For metric-based pruning of units, a norm, across parameters within a unit, of any metrics discussed previously for the unstructured case in Section 3.3.2 can be used [101, 107, 110] or group-sparse regularizers can tie parameters within a unit and encourage sparsity at the unit level [112, 115]. Selection methods that are only possible on the unit-level include activation-based pruning, where units with low or infrequent activations are removed [69, 105], and relative metrics of redundancy, where units with redundant functionality compared to others are marked for removal [109]. One issue with the latter approach is time complexity, since it requires a pairwise comparison of all units within each layer. Unit-wide masks, where the mask controls whether each unit is active or not, are also common, such as in Chen et al. [116], Wan et al. [117], Guo et al. [118] at the convolutional neuron level and in most NAS works at the layer level.

The two main mask-based developmental NAS approaches at the layer level are continuous NAS and path-sampling NAS, as shown in Figure 7. These are used in order to both train the network parameters and evaluate different architecture possibilities, which may be represented as architecture parameters. In continuous NAS, a continuously structured super-network is trained and then discretized into the desired architecture, as shown in Figure 7a [19, 106, 119–125]. The structural learning, namely neural pruning of the units of layers, occurs at discretization, which may happen once at the end of the search phase or progressively throughout. Before the super-network is fully discretized, it evaluates mixtures of potential layers over potential connections as weighted sums at each activation state, and uses back-propagation of the loss error to strengthen or weaken the architectural weight of each layer. Selecting the architecture from the super-network by the end of the search using learned architecture weights often looks very similar to magnitude-based pruning, selecting the strongest options such that a valid architecture is formed.

For path-sampling NAS, potential paths are discretely sampled from the super-network for training the network's parameters within each path, as shown in Figure 7b [72, 118, 126–129]. During the search process, a discrete path is selected for the training step of each mini-batch, and only the network weights and architecture parameters along that path are updated. Thus, each path is equivalent to performing ephemeral neural pruning for a single forward and backward pass, while persisting and updating weights of the super-network. Path selection may use the architectural parameters as sampling or state-transition probabilities within the super-network [118, 126–128] or use a uniform distribution [72]. At the end of the search process, the highest probability path is selected as the final architecture. While path-sampling NAS allows training passes to occur in a discretized network more structurally similar to the final desired architecture than the continuous mixtures in a super-network, only the parameters on the sampled path, rather than all parameters, have gradient information and thus can be updated per mini-batch.

**Scheduling Pruning:** As with synaptic pruning, neural pruning has been trending towards iterative methods, even as often as every iteration [130]. In NAS, the earliest continuous NAS methods had a single discretization where the pruning neural operator occurred, after searching the continuous super-network and before evaluating the discretized architecture [19]. However, this can lead to a discretization gap, where the shallower continuous super-network and deeper discretized architecture are too structurally different and have uncorrelated performance, due to the continuous parameters not being close enough to the discretized parameters [81]. More NAS algorithms are incorporating progressive discretization [122, 124], among other techniques [81], to avoid this gap.

As for the training portion of scheduling, NAS methods also tend to do a full reinitialization and retraining of weights after discretization, whereas most other structural learning methods only fine-
<figure><img src="image_13.png" /><figcaption>**Figure 7:** Example of a repeated cell of a super-network in NAS (figure reproduced with permission from Yao et al. [129]). Within each cell, the blue nodes represent intermediate activations between layers. Each colored edge represents a different layer type, such as a skip-connection or convolutional layer. This cell, $c_k$, receives the output of the two previous cells, $c_{k-2}$ and $c_{k-1}$. During network training, in continuous NAS, all layer options are used at all connections, and a weighted sum is computed at each node using the architecture parameters. In path-sampling NAS, a path representing a valid architecture is selected within the super-network, which is a form of ephemeral discretization. A common architecture constraint for cell-based NAS is at most one layer option per connection and exactly two inputs to each node. For both types of NAS, the final goal is to optimally prune the network into a discrete architecture within this constraint like (b).</figcaption></figure>

tune the network after the search and pruning process, as noted in Table 2. As for network compression, Liu et al. [131] finds that complete reinitialization and training is generally superior while Ye et al. [114] finds fine-tuning inherited weights is better, provably for the case of shallow networks and empirically for deeper networks.

Avoiding the pretraining step is more difficult in structured pruning than unstructured, since structural pruning selection techniques often assume all units are trained enough in order to differentiate their function from each other and fairly compare their utility. Pretraining is particularly important for comparing parameterized units with unparameterized units in NAS, like comparing a convolutional layer with a skip-connection at the same location within an architecture. Some relatively zero-cost NAS metrics that measure entire architectures without training, such as measuring saliency or sensitivity to perturbation [132] or nonlinearity alignment at initialization [133], have been shown to be effective in identifying good candidate architectures, which is particularly useful to avoid the expensive search training process when the final network is often retrained from scratch.

**Handling Pruned Entities:** At the extreme end of ephemeral pruning on the unit level in frequency is Dropout [134], which is a technique of randomly and independently selecting units to momentarily prune for each batch during training, preventing co-adaptation and overfitting. Structural pruning algorithms use ephemeral pruning in a more principled and informed manner, as in path-sampling NAS. This allows the modules of the super-network to learn more independently than in the continuous case. This idea is also used by Yuan et al. [130] at the neuron level, but most other neuron pruning works choose to instead reinitialize new neurons as the creation operation, which will be discussed in Section 3.4.

**Impact on Weight Optimization:** Structured pruning is less intuitive to consider in the objective function search space, since dimensions are tied through the units containing their corresponding parameters. Thus, multiple dimensions are removed at once for each unit removed. Pruning metrics often gain information through continued training, but this must be balanced with the cost of training units that will be pruned and thus not used in the final network or in further pruning decisions.

Many NAS works pose the architecture search as a bi-level optimization problem, where the archi-
tecture parameters $\alpha$ are optimized given that the weight parameters $w$ are optimal for any given architecture and constrained to certain sparsity rules $S$ that only permit valid architectures [18]:

$$ \min_{\alpha} L_{val}(w^*(\alpha), \alpha) \qquad (1a) $$

$$ \textbf{s.t.} \quad w^*(\alpha) = \arg\min_{w} L_{train}(w, \alpha), \quad \alpha \in S. \qquad (1b) $$

This problem is intractable to solve directly, so the process is usually approximated by continuous relaxation of $\alpha$, persisting weights across the architecture optimization search, and alternating weight updates with architecture updates. The gradient updates to the architectures variables may only be locally approximated: using a second-order estimate of the gradient of $\alpha$ that incorporates an inner update of $w^*$ only gives minor improvements in performance for a higher computational cost compared to first order estimates that approximates $w^*$ as the current $w$ [19]. See Liu et al. [135] for a further discussion on bi-level optimization in ANNs.

### 3.3.4 Pruning Conclusions

In ANNs and in the brain, more information is available for existing neurons and synapses to be pruned versus creating new units. This trend is noted in artificial algorithms, both in the popularity of pruning versus creation as well as more specifically using masking to propagate gradient information to inactive units, thus turning a structural learning problem into a pruning problem.

Pruning in both the brain and ANNs can improve performance relative to cost: the main costs in both medias are time and space, but the realization of these costs change how efficiency is achieved. For example, an important difference between brains and ANNs is the locality of physical biological signals versus the globality of ANN addressing in memory. Biological neurons are limited by physically transported molecular signals, but do not have a scaling time cost for enumerating the local options. ANNs algorithms have no distance-based cost and thus may make decisions globally, but do have a linearly scaling time cost for each entity that is evaluate due to the hardware's bound on floating point operations per second.

Pruning, however, does come with costs, notably the wasted training time of the associated parameters and potential disruption to the learning process. Pruning algorithms must thus strike a balance of information that could be useful for pruning and later utility in the network if the unit is not pruned versus the cost of obtaining this information.

Pruning has been used throughout the history of deep ANNs to partially automate the structural design. However, it relies upon a predefined and instantiated search space. Currently, most methods explicitly hand-design this search space as the initial architecture. Others, as well as the brain, use creation to dynamically create the search space. These creation methods and their synergies with pruning will be discussed in the following section.

## 3.4 Neuron creation and synaptogenesis

Adding to a search space is naturally more difficult than removing, because an indefinite number of ways to add new elements exists, but removal is limited to only existing elements. For the unit operators of structural learning, this means that neural creation is a much different problem from neural pruning. For the connection operators, because synaptogenesis occurs between existing structures, it is of the same search complexity as synaptic pruning, although metrics for where to add connections are not as straight-forward as where to remove them. Thus, synaptic pruning and synaptogenesis often look very similar, especially for ephemeral structural changes during training.

Nearly parallel to our pruning questions, our main questions for characterizing creation are:

* **Goals of Creation:** Why add connections or units to an ANN?
* **Biological Inspiration:** How can biological neurogenesis and synaptogenesis inspire artificial methods?

* **Architecture Specificity:** How has creation been adapted to different ANN architectures, such as dense layers and convolutional layers?

* **Measuring Entities:** How are the new entities to be added selected over other options?

* **Scheduling Creation:** When and how often does creation occur with respect to optimization of the existing parameters?

* **Operator Interactions:** What is the impact of using pruning along with creation, versus just one such modality?

* **Impact on Weight Optimization:** What is the impact of creation on optimization of parameter weights?

In discussing these questions, we focus on the growth aspects of the papers in our corpus: neural creation and synaptogenesis. We discuss the few topics that span both modalities of creation in Section 3.4.1, then cover each question specifically for synaptogenesis in Section 3.4.2 and for neural creation in Section 3.4.3 before concluding in Section 3.4.4.

### 3.4.1 Creation Generalities

Not as many shared characteristics exist between the modes of creation compared to those between the modes of pruning. We detail these few general characteristics below.

**Goals of Creation:** Growing in an ANN allows the architecture to be even more automatically customized: pruning-only algorithms require all entities in the search space to already exist in the initial network. Using the growing neural operators allow a much larger architectural search space to be explored over the course of structural learning, since this search space does not have to be explicitly predefined or instantiated. It is rather implicit from the algorithm's creation operations.

**Biological Inspiration:** Synaptogenesis is a frequent event in the biological brain which defines the possible neural circuitry using information from genetic cues and neural activity [51]. Neurogenesis, on the other hand, mostly occurs during early development, creating the critical structures of the brain largely through genetic influence [42, 44, 46]. Neural activity regulates both processes but is especially important for synaptogenesis [7, 136]; physical proximity and chemical neuromodulatory signals also influence these processes. As in ANNs, creation is used to expand the possible neural circuitry with relatively uniformed propositions; in both cases, creation is a more difficult problem than removal with less information available.

**Impact on Weight Optimization:** Creating entities in an ANN expands the dimensionality of the parameter search space. Network morphisms are methods of adding entities to an ANN with a specific weight initialization that preserves the functionality of the network [137]. After using a network morphism to grow units or connections, the expanded ANN may then begin gradient descent at an equivalent location of the previous objective function space but now with expanded dimensions. This guarantees that the new optimal point will have at most the same cost if not lower, since the previous optimum will necessarily be included in the new search space within the subspace where all new parameters are nullified. However, adding too many dimensions increases the time and space costs for the ANN, including in search time if the search space becomes unnecessarily complex. Similarly as for pruning as discussed in Section 3.3.1, creation is generally composed of greedy search decisions in a local search, which can only find local optima.

In the following section on synaptogenesis, we focus on algorithms that have a specific method for growing connections, rather than just reviving previously pruned connections, previously discussed as ephemeral pruning in Section 3.3.2, or creating units with their associated connections, to be discussed in Section 3.4.3.
### 3.4.2 Growing Connections

**Goals of Creation:** Synaptogenesis can be used in an ANN to create new connections between existing units. It has been used with pruning for discovering more effective flexible wirings between channels or neurons within ANNs, particularly without an initial overparameterized dense architecture [68, 92–94, 99, 100]. In other applications, Kim et al. [138] augments an ANN’s performance after training by adding neuron-level connections directly from hidden neurons to the output, while Elsken et al. [71] searches for layer-level skip connections, among other network morphisms, concurrent with continual training during evolutionary architecture search.

**Biological Inspiration:** Biological synaptogenesis is an integral part of learning which has been demonstrated in memory and control tasks [89, 139]. While a considerable part of biological synaptogenesis only reinforces the existing connection between neurons through adding new synapses, previously unconnected neurons can also be linked through synaptogenesis. The information used for this process is based on activity and chemical signatures [136]; nearby neurons will exchange signaling molecules to make them more receptive to synaptogenesis [52]. Physical proximity is an important factor in biological synaptogenesis; for example, in a cortical column, two neurons can connect through a short extension, where connection to a neuron to a more distant region requires investing space and energy in growing axon branches with a more daunting problem of connection choice [102]. There are zones of higher connection density, such as cortical columns which have 10⁹ synapses for 10⁵ neurons, as opposed to the overall higher sparsity of the brain, 10¹⁵ synapses for 10¹¹ neurons. This sparsity means that, through rewiring, a neuron can drastically change its role in information passage [102], and synaptogenesis is the main catalyst of rewiring change. Further work similar to Dai et al. [68], Mocanu et al. [92], Wortsman et al. [94], which connect previously unconnected neurons during learning, warrants exploration, as the brain demonstrates the importance of forming new connections during learning. Neural activity could form the basis for possible connection, as in the brain [140], by detecting neurons or layers in ANNs which have correlated activity and could be directly linked.

**Architecture Specificity:** Neuron-level synaptogenesis algorithms can generally be applied in both convolutional and standard layers. For convolutional layers, synaptogenesis is generally done on the kernel level [68, 94, 100]. Only Bellec et al. [93] performs synaptogenesis on individual parameters within each kernel, but does so randomly on masked connections to balance informed synaptic pruning. Wortsman et al. [94] introduces a rewiring algorithm that is agnostic to whether connections are convolutional or dense and removes the constraint of layer organization that engineered deep architectures follow.

**Measuring Entities:** Metrics for synaptogenesis are less straightforward than for synaptic pruning, since the potential connections do not have active information flow. Thus, a common approach is to use a mask that inactivates connections during the forward pass but allows propagation of the gradient to them during the backward pass, providing many of the same metrics as are normally available [68, 94, 100]. To shrink the quadratic search space of all possible connections, Dai et al. [68, 100] only allow connections between neurons in neighboring layers, while Wortsman et al. [94] allows connections across multiple layers but still limited within modules of similar depth. Bellec et al. [93] also uses a mask, but rather performs random synaptogenesis of a currently dormant connection each time another connection is pruned in order to maintain the sparsity level of the network. While masks bound the search space, that is not necessarily limiting to synaptogenesis since it already can only occur between existing units, whether or not a mask is used.

Beyond mask-based approaches, algorithms often use improved performance as the final metric for adding connections. Puma-Villanueva et al. [99] ranks all possible connections with a mutual information heuristic amenable to synaptic creation and adds those that improve the performance. Kim et al. [138] evaluates the effects on performance exhaustively for a smaller search space, only allowing hidden neurons to be connected directly to output neurons. Using improved performance as the selection criteria is intuitive, but is a greedy selection towards the goal of optimizing performance
and is expensive to complete exhaustively without heuristics, limited search spaces, or randomized
evaluations.

**Scheduling Creation:** Synaptogenesis is usually performed iteratively and greedily. New connections need to be optimized after a random initialization for optimal performance, so synaptogenesis phases are generally followed by weight optimization phases of either only the new parameters or the entire network.

**Operator Interactions:** Most synaptogenesis algorithms also implement synaptic pruning [68, 93, 94, 99, 100]. Implementing synaptogenesis with synaptic pruning can create dynamically balanced connectivity, which may counteract any negative effects of earlier greedy synaptic additions to escape local optima. The remaining synaptogenesis algorithms only use synaptogenesis out of our two connection operators [71, 138]. This simplifies the search process, but may approach local optima due to greediness.

**Impact on Weight Optimization:** Synaptogenesis adds dimensions to the search space. Because any added synaptic parameters can be set to zero to nullify their function and thus preserve the objective function of the ANN, synaptogenesis can guarantee the new global optimum has at most the same cost. However, depending on the initialization and training algorithms used, the synaptogenesis may displace the current location in the objective function space far enough to descend towards a different, possibly worse or better local optimum. Additionally, adding too many dimensions increases the time and space costs for the ANN, including in search time if the search space becomes unnecessarily complex.

### 3.4.3 Growing Units

**Goals of Creation:** Neuron creation, or adding units to an ANN, is the most open-ended neural operator. The general aims of neuron creation are to improve performance and to create an architecture that is specifically effective for the desired task without manual design. The resulting networks should have either reduced space and time complexity for deployment at a desired performance level or a better performance potential. Beyond algorithms with these general aims, some NAS approaches incorporate neural creation in the form of changing the architectural search space over the course of training [72, 119, 125], which provides a larger search space than traditional pruning-only approaches. Both the general and NAS-specific applications allow for a more customized architecture with less hand-design.

**Biological Inspiration:** Neurogenesis in the biological brain is more constrained than in ANNs; neural progenitor cells must be properly located to create new neurons, there must be enough space, and the associated energy cost is non-negligible. The most common period of neurogenesis, early development, is largely guided by genetic cues [42], with only highly irregular neural activity patterns disrupting neurogenesis [7]. In humans, adult neurogenesis is constrained to a few regions, notably the hippocampus [47, 48], and it is common in other organisms [141]. Adult neurogenesis could serve as further inspiration for structural learning in ANNs as there are fewer constraints. For example, new neurons in the mature brain first develop input synapses which observe incoming connections before firing action potentials [55]; in other words, new neurons are well-initialized based on surrounding activity, similar to recent studies in ANNs [20, 142]. The conditions leading to adult neurogenesis are the subject of current research in neuroscience [143] and could lead to further inspiration in ANNs.

**Architecture Specificity:** Neuron-level creation usually adds neurons of a predetermined type to existing or new layers. However, since selection techniques are often agnostic to layer type, many neural creation algorithms can be applied to architectures with layers of various types like convolutional or dense [20, 101, 110, 142, 144]. Similarly to pruning, growing new filters in a convolutional layer also involves a mapping in the subsequent layer [101], as shown in Figure 6. Layer-level creation often extends layers with the same type as previous layers [145, 146]. Roberts et al. [125] avoids
pre-engineering the layer type selection by parameterizing the layer function in a search space that includes most commonly used layer types in NAS among many others.

**Measuring Entities:** Selecting where and how to add new units to a network is not straight-forward. In order to avoid explicitly bounding the search space, a mask cannot be used as it requires all elements to be instantiated to be associated to an element of the mask. Most approaches limit the instantaneous search space of possibilities, such as iteratively adding a single neuron to an existing layer [68, 110, 147-152], adding a single new layer beyond the current architecture [110, 146, 148, 151, 153], or splitting existing neurons [101, 110, 144]. This results in a simpler problem at each step, but still gives the algorithm the power of an unbounded search space over the course of training.

The simplest approaches, often combined with some form of pruning, use a generic creation scheme that adds some default number of new units [112, 113]. Slightly more informed approaches use the network's objective function as a heuristical measure of success, adding the units that reduce the error the most out of generated options to the network [68, 153]. Evci et al. [142] adds neurons that maximize the immediate improvement in performance, while Maile et al. [20] dynamically adds neurons with unique activations or weights to avoid redundancy. Liang et al. [154] adds special exponentially activated neurons that provably improve the loss landscape.

**Scheduling Creation:** The objective function is used not only for creation selection but also for iterative scheduling and termination. In early construction algorithms, often a single neuron was added at a time upon convergence of the current network until the desired performance is reached [99, 147-152]. This approach of a single neuron added between training to convergence does not scale very well with the size of the network or complexity of the task. Many of the recent methods still use performance stagnation during training as a trigger for a neural creation phase, but for adding either many new neurons or an entire layer at a time [71, 144, 153]. Otherwise, a manual schedule for iterative phases may be implemented [101, 113, 118]. Most of these schedules introduce hyperparameters, such as thresholds or durations per each phase of the schedule. These require hand-tuning or optimization for each specific dataset, task, and initial architecture. This is not desirable for generality and automation of algorithms across applications, but selecting hyperparameters is a much smaller search space for manual exploration than that of the architecture being automatically designed. Maile et al. [20] uses a dynamic schedule that adds neurons while novel directions to explore in either the activations or weights in that layer still exist.

**Operator Interactions:** Most of the recent neural creation algorithms also use neural pruning, while the older ones did not. This dynamic approach not only can ameliorate effects of greedy structural changes, but also leaves the size of the final ANN as either an objective to optimize in tandem to performance or as an open-ended result. For example, Guo et al. [118] can adapt initial seed networks to any computational budget, whether smaller, approximately the same, or larger than the original network. Connection operators may also be used in addition to unit operators to make finer adjustments to the network [68, 99, 101].

**Impact on Weight Optimization:** Neural creation expands the dimensionality of the search space of gradient descent. To determine the starting point in the newly expanded space, the weights of new units need to be initialized before they are trained. Unit-splitting algorithms [101, 110, 144, 145, 155] copy the weights and may add a small perturbation to ensure the units are not redundant if split in parallel. Network morphisms, named by [137] and used in many other works [20, 71, 119, 142, 146, 152], are structural operations that initialize new structures to be null, which generally means initializing new weights to be zero. Most other neural creation algorithms use a similar initialization scheme to the original network, such as random weight initialization. Both unit-splitting without random noise and network morphisms perturb the location of the network in the new space less than methods that use random noise in their initialization. Both cases can be beneficial: staying in a similar location with newly added dimensions allows the path of gradient descent to continue with less disturbance or risk of regressing from the current level of performance, while a small perturbation may help escape local
optima or saddle points [110, 144], as demonstrated in Figure 5.

Liang et al. [154] theoretically shows that the addition of special exponentially-activated neurons to a
basic ANN can make all minima globally optimal. However, no empirical work has shown whether
these simple additions affect other aspects of training dynamics and performance. For example,
additional saddle points may confound the benefits of no bad local minima [156].

3.4.4 Creation Conclusions

Creation has been less well-studied than pruning in ANNs, due to the additional complexity imposed by
the open-ended search space of creation, which is less cooperative with back-propagation than static
architectures and even pruning. Hand-designing neural architectures has also made it less practical
and imperative, but the desire for it is continuing to increase alongside computational capabilities.

The brain has rather distinct phases of early development and adult life. These correspond roughly to
defining the neural architecture search space, or hand-design of an ANN architecture for the case of
conventional non-structural learning, and searching within this space over the process of learning.
Most structural learning research focuses on automating and improving the latter process, leaving
further optimization of search space design as an open problem. However, the search space design is
also synonymously influenced by the genetic priors used in the brain that have been optimized over
the course of human evolution, making search space design a much broader problem with a larger
scope than one-shot learning. The creation operations allow for a more dynamic approach to search
space design.

The brain uses Hebbian learning, or reinforcing effective connection strengths in response to corre-
lations in activity between interacting neurons, as one of many mechanisms for dynamic structural
learning [57]. Synaptic scaling, or the post-synaptic homeostatic plasticity [157], and non-synaptic
plasticity, such as membrane conductance changes [50], counterbalance this strengthening and are
analogous to weight decay and batch normalization, respectively, in ANNs. The explicit use of Hebbian
learning in ANNs, which could be implemented as increasing weight magnitude between artificial
neurons with correlated activations, is thus far not as popular as back-propagation but has been done
[158]. Standard ANNs are not time-dependent, which changes the nature of correlations in activity
compared to that of the brain, and ANNs have global information access, permitting back-propagation
of errors more easily than in the brain, although there are theories of supervised learning via target
activity patterns [159]. Dai et al. [68] applies this technique to the creation operations and shows that
growing connections via larger values of the gradient of the network loss with respect to dormant
masked weights is a Hebbian-like rule. This suggests further exploration into appropriate application
for artificial media of Hebbian and other plasticity theories such as behavioral timescale synaptic
plasticity [160] and .

As previously discussed in Section 3.3.4, the physical brain excels at local communication without
indexing cost while ANNs can operate globally but with a cost per index. This has implications for
effective creation in ANNs: to maintain search efficiency, non-random creation in ANNs, where each
potential entity must be indexed and evaluated, should be strategically bounded at a given structural
step without overly restricting the size of the overall search space. Creation is not as explicitly bounded
in brains: unit creation, including migration, is guided by genetic and developmental factors while
synaptogenesis is guided locally by chemical signals, yet there are no restrictions such as having
predefined patterns of connections for each neuron. For synapse measurement, the brain does not
use simulated heuristics like ANNs often do, but instead selects connections by dynamic processes of
repeated synaptogenesis and synaptic removal as directed by various types of plasticity [50, 57, 157].
Similarly for neurons, the heightened neurogenesis of early development is balanced by the automatic
removal of new neurons unless they prove their utility. This suggests an approach for ANNs to make
sub-optimal but less expensive additions balanced with more careful pruning to dynamically and
efficiently construct an informed architecture.
## 3.5 Perspectives

With our deepened understanding of how the four neural operators work in ANNs, we present our perspectives on the state of structural learning. We first note implementational trends that shape current works in Section 3.5.1. We then discuss current challenges with future directions in Section 3.5.3.

### 3.5.1 Implementations and Trends

In order to successfully implement structural learning on modern machines rather than biological media, we note several approaches used commonly across algorithms. The ongoing development of computer hardware and software, which provide the media on which ANNs are implemented, has potentiated each of the innovations in ANNs throughout history. Present computational technology is particularly adept at handling array-based calculations. For example, GPUs drastically speed up ANN training wallclock time particularly with stochastic gradient descent (SGD) and tensor-based architecture parameterizations and data structures. However, they also impose a limit in random-access memory (RAM): ANN training is much more efficient and easier to implement if the entire memory-intensive backward pass of SGD can fit on the GPU's RAM at once. The inference computational cost of an ANN is often measured in the number of Multiply-Adds, or equivalently floating point operations (FLOPs), required per inference of a single input. This value can be used as part of a multi-objective approach, maximizing performance while minimizing the inference cost. The training cost is usually measured in GPU-days, the number of days to train on a single GPU using current hardware and software. Low training costs are a key benefit of structural learning over techniques that instantiate and train a statically structured model for many individual architectures. The following implementation trends are noted across our structural learning corpus, specified in Table 2.

**Modularity:** Dividing a design problem into multiple hierarchical levels and reusing modules learned at lower levels makes it more tractable. This is seen biologically, where symmetry and modularity can be seen from the genetic level, reusing sections of DNA for many proteins that can each have many functions, through the nervous system level, showing bilateral and other symmetries, and even further, as in convergent evolution where unrelated organisms evolve similar structures or functionalities. For ANN architectures, this entails searching for smaller modular structures that can be used as building blocks to build larger architectures. This technique has already been used in the hand-design of architectures, like ResNet [29]. It is also used in structural learning, notably unit creation and pruning where groups of parameters are added or removed as a unit. Modularity has especially been used in NAS in order to reduce the complexity of the search space. Most NAS algorithms that allow flexible interconnectivity use repeated cells of the same architecture during the search process. Some recent NAS approaches incorporate structural learning on multiple levels, such as searching for both layer-level and neuron-level structures [106]. NAS algorithms incorporating creation take this even further by also optimizing the layer options used [72, 119, 125], thus expanding the search space. Modularity is particularly capitalized in algorithms that either duplicate or split units [101, 110, 144, 145, 155]. These algorithms reuse neurons, layers, or branches of the network that have already been trained on the present dataset and task as initialization points.

**Masking and Super-networks:** A common approach to allow structural learning is to use masking within the ANN implementation. This mask is an encoding of the structure, enumerating all possible entities and controlling which ones are actively used in a given structure. By relaxing this mask to be continuous, the mask can permit continuous learning where the mask itself is treated like parameters of the network that can be optimized and discretized, such as in pruning or continuous neural architecture search.

When implemented on the layer level for NAS, masking also naturally leads to weight-sharing, where many different architectures are represented with shared parameter data structures but the mask imposes different functionality by specifying the architectural path and thus the parameters used within the super-network. The architectures parameters used for this specification in continuous
NAS are usually activated by softmax, gumbel-softmax, or another balancing function in order to
enforce scaling across parallel options. Gumbel-softmax and annealing are more recent innovations
in activation of NAS parameters to smooth the transition to the final discretization [69, 117, 123, 128].
Other NAS papers have introduced non-competitive activation functions [161]. Once activated, the
architecture parameters may then be used for weighted sums of the activations from parallel layer
options in continuous architecture search. For path-sampling architecture search, the activated
parameters are often used as a sampling probability.

When implemented on the neuron level, masking for synaptic pruning, such as in unstructured
pruning algorithms, allows for simpler computations by element-wise multiplication, but does not
lead to any computational speed-up without specialized software. Because masks for neural pruning
control multiple synaptic parameters per unit with a single masking parameter, they may require
regularization to enforce group-wise operations on the parameters during the algorithm, or else the
mask may be implemented across dimensions of the parameter matrices if that is compatible with
the parameter data structure representation, such as batch normalization scaling factors [112]. Like
NAS, some neuron-level masking implementations also use temperature in their implementation to
smooth the transition from the continuous relaxation to discrete masking [93].

Masking naturally turns architectural selection problems into a pruning problem with a bounded
search space. This often creates a limitation in neural architecture search and pruning methods where
these methods are used: the maximum size network over the course of search should be able to fit
its forward and backward passes on a GPU at the same time, even if the desired final network will be
much smaller. For example, super-networks in NAS are often limited by the GPU size: for continuous
NAS methods, all options are at least partially activated, resulting in a very large network to evaluate
and optimize. In order for the final network discretized from the super-network to also approach the
GPU size, many continuous NAS works search in a shallower architecture with fewer channels and
then implement the discretized structure in a deeper architecture by repeating cells, which is another
benefit of using modular cells. However, this restrains the architecture search to repeatable cells
with channels, which is not always applicable for tasks outside of image classification. It also further
deepens the discretization gap by contributing to the structural differences between the super-network
and the discretized network, as discussed in Section 3.3.3.

Dynamically sparse training methods avoid the large initial model by sparsely initializing the network
and dynamically adjusting which entities are active, allowing the network to maintain a maximum
density level [92–94]. This results in a predefined structural search space without limitations on the
size of the full search space, but requires structural learning processes that are not memory-bound to
fully reap the benefits over standard pruning techniques.

**Interconnectivity:** The brain shows connectivity patterns that are much more flexible and complex than modern standard feed-forward networks, where neurons are organized in layers and each layer is only connected to its immediate neighbors. Neural architectures tended towards this organized structure in the early decades of backpropagation, particularly for the multi-layer perceptron. Recently, the skip-connection broke this simple patterning and improved the state of the art in computer vision and many other tasks [162]. It allows for much deeper networks to be trained without vanishing gradients and for higher and lower-level features to be used together. Many structural learning algorithms like NAS allow for interconnectivity flexibility on the layer level, while Bellec et al. [93], Wortsman et al. [94] allow even finer interconnectivity outside of layer organization on the neuron level. Including the potential for such flexible interconnectivity in structural learning, rather than using a rigid connectivity backbone, greatly expands the search space and thus adds complexity to the search, but also yields the potential for more powerful networks.

**Weight Reinitialization:** Most neural-level structural learning algorithms are end-to-end, where weights learned in tandem to the architecture are used in the final ANN either without any further training or with only fine-tuning. In contrast, most layer-level continuous NAS approaches require full weight reinitialization and training after architecture discretization. While this does allow the
super-network and discretized architecture to have different shapes and training hyperparameters, it is generally more expensive and has a risk of uncorrelated performance between the two stages. This trend is also seen in Frankle and Carbin [12], although here the pruned network returns to the initial initialization for the remaining weights.

### 3.5.2 Adjacent Domains

In addition to the external review papers cited in Section 3.2.2, we build upon our analysis to mention closely related domains and techniques that are not quite covered by our structural learning framework. They generally cover neural timescales or non-structural operations outside of the scope of our neural operators, but could be used in tandem with or inspire other forms of structural learning.

**Attention and LSTMs:** In contrast to the methods we have considered that specifically use dynamic connectivity and structures during the training process, networks incorporating self-gating mechanisms, such as recurrence and attention, exhibit ephemerality at inference time [75, 76]. This is most analogous to computation on the neuromodulatory timescale. Attention operates similarly to regulatory firing, instantaneously adjusting computation for a single input without longer term effects using feedback connections [163]. Simpler recurrence structures can also operate on this very short timescale, while LSTMs are more akin to chemical neuromodulation [164].

**Biologically-inspired Networks:** There are a number of artificial neural model and network types which are further inspired by biology. Spiking neural networks (SNNs) use the transmission of discrete spikes, sometimes along multiple synapses between a given pair of neurons, to transmit information [165, 166], thus allowing for spike-timing dependent plasticity and stochasticity [167]. Liquid state machines, a recurrent version of SNNs, have been studied using similar neural operators as the ones studied in this work [168]. As SNNs imitate the spiking behavior of biological cells, they can be used to model biological networks, including network structure [169]. Simple organisms such as *C. Elegans* with mapped neural circuits can be modelled with high levels of detail [170]; neural circuit policy models, inspired by the brain structure of *C. Elegans*, use cellular dynamics that model biological neurons, including time dependence and sparsity [171]. Modelling biological neural structure can help understand how such structures form and their role in learning, and biologically-inspired networks are well-positioned for this.

**Evolutionary Algorithms:** Evolutionary algorithms are a logical choice for NAS due to their flexible problem encoding and high parallelization; in addition, searching for the evolutionary priors for effective learning is a clear inspiration from biology. There is a long history of using evolution to find ANN structure: Miller et al. [172] was among the first to describe architecture search using a Genetic Algorithm; in Richards et al. [173], ANNs were evolved to play Go; and NEAT [74] has been applied in domains from image generation [174] to Atari game playing [175]. Contemporary methods combine evolutionary architectural search with gradient descent on weights [72, 73]. In these methods, structural changes usually happen at the generational level, for example as mutations of existing architectures; while this makes them difficult to study in the context of individual neural operators, they can provide insight and comparison for structural learning methods, as highlighted in Stanley et al. [82].

A different evolutionary domain which takes direct inspiration from the biological evolution of nervous systems is the evolution of structural learning rules [83, 176, 177]. In these works, structural learning decisions such as performing neural operators are taken by evolved rule sets, in the form of L-systems [178], grammars [176], or functional graphs [177]. In these works, individuals in a population representing different structural learning rules are used to develop ANNs; structural characteristics of the resulting ANNs or the performance of the ANN on tasks are used to inform the fitness and optimization of the various rules. Miller [83] presents a comprehensive overview of this domain. As demonstrated in this chapter, structural learning decisions can be difficult to design, with aspects like timing, relation to learning, and information from other parts of the network contributing to the
decision. Evolution can aid in this design, and we aim in this chapter to provide information about structural learning which may help inform these evolutionary approaches.

**Meta-learning:** The fields of AutoML and more specifically meta-learning go beyond optimizing a single model and rather tackle optimization of how a model learns [31, 179]. Many meta-learning works aim for fast learning, using only a few samples to learn a new task. Humans and other animals developed fast learning, such as easily learning how to toss a new object after learning how to toss other objects, over the course of evolution; we cannot nor need to conscientiously change brain structure to learn a new skill, while genes and structural operations in the brain are separated by many processes that prevent any strong direct effects. Thus, while fast learning itself happens on a timescale between neuromodulation and structural learning, the technique of fast learning was discovered over the course of the much longer timescale of evolution.

Similar hierarchies of automation occur in meta-learning and large language models, even going towards networks building networks [180, 181]. In these approaches and evolutionary search, there must be hand-design at some level, but engineering at a higher level allows for higher complexity to occur throughout the subsequent lower levels. In the biological case, operations within neural networks are abstracted away from the two controllable variables of consciousness and genes. This suggests continuing setting up artificial systems from progressively higher levels, but also comes with the disadvantage of turning the abstraction between levels into a black box. Similarly, meta-learning works often by training ANNs in the pursuit of learning how to improve learning [181, 182], although the recent trend of massive models with attention mechanisms allowing sparse module activations trained on massive amounts of data and tasks exhibit fast learning as well [183].

With the current state of structural learning and adjacent domains in mind, we next present challenges and future directions within structural learning.

### 3.5.3 Current Challenges and Future Directions for Development

As computational speed and power, as well as datasets, continue to increase, the desire to automate the engineering of ANN architectures is increasing as well. Human engineering is currently a rate-limiting step of ANN architecture innovation. Among discussions already presented throughout this work on the past and current state of structural learning, we will end our discussion with present challenges and future directions for development.

Many of the benchmark applications that structural learning algorithms are implemented and demonstrated for are well-studied supervised tasks over datasets with static and representative distributions across the training and test partitions. While these provide a consistent benchmark against hand-designed networks and are useful for algorithm development, their utilization is almost paradoxical for one of the powerful potentials of structural learning: automating structural design even for tasks that are not well-studied. Structural learning may allow for more power in less studied and engineered applications, bypassing the need for specific architectural innovations to be designed by hand for every such task. This phenomenon is particularly noted in NAS; automating the engineering of architectures could potentiate the application of ANNs to less-studied tasks, but the majority of NAS papers only report results on the same few image classification datasets for which architectures have already been thoroughly researched, each trial trained independently from scratch, with image-specific search spaces like convolutional layers in repeatable cells. Learning environments that go beyond the basics include multi-task learning [91, 100, 128, 155], continual learning [91, 100, 110, 120, 184], and reinforcement learning [185–187]. These may also approach the dynamic learning environment of the human brain more closely, where representations, labels, and rewards must be internally implemented [188, 189]. Few structural learning papers have implemented their algorithms in these environments thus far, but they represent a deeper level of artificial intelligence.

Our definition of structural learning implies dimensionality changes to the objective function space of ANNs. Structural optimization alongside weight optimization thus prompts a multi-level optimization,
with the structure of the network being the outer-most level that defines the space of parameters being optimized on a lower level [81]. This is explicitly considered in many continuous super-network NAS papers as a bilevel problem [19], but the optimization is often simplified to alternating gradient descent steps without theoretical guarantees but with reasonable performance relative to time and computation cost in practice. Multi-level optimization methods are continuously being developed [190] and may be applied to structural learning, but the trade-off of performance with complexity cost must be considered for effective use.

The level of abstraction, demonstrated in Figure 2, has resulting trade-offs for the structural learning algorithm employing it. Lower level abstractions, such as the direct paradigm of feed-forward neurons, allow for fine-grained tuning of the architecture, but may be too minute for architectures with multiple higher orders of structure and billions of parameters. Higher level paradigms, towards NAS super-networks, impose difficulty by requiring additional structural design in the search space, such as selecting or even structurally optimizing the layer options present in each connection as modules. Combining multiple levels within a single algorithm has thus far been rare [106], but presents a powerful direction for structural learning, requiring coordination of each level that have mostly been studied in isolation thus far.

Neural creation is the most difficult form of structural learning because it is open-ended and slow with current methods. However, such a large search space gives it the potential to be the most fruitful in the pursuit of automating architectural design of ANNs. It is represented not only in selecting where and how to add new units, but also in widening the search spaces of potential layer types and connectivity patterns, which are currently rather limited to discrete, hand-engineered choices. In order to surpass this limited space, Bronstein et al. [191] presents a geometric framework to unite diverse ANN architectures, layer types, and data structure types. Applying these ideas to ANN structural learning could expand the search space by utilizing symmetry-inducing invariances with less restrictions on the input data structure, thus supporting applications even to less naturally tensor-like data structures.

In biology, the genetic code potentiates each neural operator within biological media through transcription then translation to proteins underlying the cellular processes that govern brain structure dynamics. In ANNs, the components of the learning algorithm can be considered like the genetic sequence which determines how ANN learning, both basic weight optimization and the more difficult structural learning, can operate within the computational media of hardware and software given an individual in a learning scenario with training data. The developmental search space is encoded in the algorithm. Most such genetic sequences are hand-designed, but some efforts have been made towards recipe search, where the algorithm itself is automatically designed and optimized. Evolution was the crucial process for discovering the current foundation for biological structural learning, such as discovering useful modular structures and defining the phases of high neurogenesis in early development to dynamic connectivity later in life. Some efforts towards replicating this process are in evolutionary recipe search, where an evolutionary algorithm tries to optimize the entire procedure of structuring and training a neural network. This represents learning on multiple timescales, beyond just the lifetime of a single ANN.

Beyond providing a biologically-inspired framework for structural learning in ANNs, we also intend for this work to strengthen connections between the various sub-communities performing structural learning. We noted a lack of cross-citations both across time and across sub-communities. Many pruning algorithms in the last five years implement the same main abstract methodology as those from three decades ago, but now show drastic improvements in performance [192]. While pruning and NAS works are somewhat well-connected within their respective corpora, developmental works using the creation operations are not, due to both a lack of common vocabulary and a relatively low frequency of new methods. Further, algorithms for more dynamically structured ANNs often compare against their more static counterparts, but not vice versa, and a similar pattern occurs for some of the adjacent domains mentioned in Section 3.5.2 versus conventional networks. Our provided framework can help
bridge the abstract similarities between algorithms to promote collaboration, not only between the neuroscience and machine learning communities, but also between the sub-communities of the latter.

In conclusion, structural learning in ANNs is a dynamic and diverse field with a vast potential. Our biologically-inspired framework of neural operators, consisting of neural creation, synaptogenesis, neural pruning, and synaptic pruning, provides a means for synergy not only between neuroscience and machine learning, but also between subcommunities within structural machine learning such as pruning, AutoML, neural architecture search, and developmental neural networks. Gaps thus far identified in the current foundational understanding of structural learning for ANNs are tackled in the following chapters.
<table><caption>**Table 1:** Operator type, unit type, layer type, and task qualities across our corpus.</caption><thead><tr><th rowspan="2">Reference</th><th colspan="3">Operator type</th><th colspan="3">Unit type</th><th colspan="3">Layer type</th><th colspan="5">Task</th></tr><tr><th>Neural creation</th><th>Synaptogenesis</th><th>Neural pruning</th><th>Synaptic pruning</th><th>Neuron</th><th>Kernel shape</th><th>Filter/channel</th><th>Layer(s)</th><th>Dense</th><th>Conv/pooling</th><th>Recurrent</th><th>Multi-task</th><th>Continual</th><th>Image classification</th><th>Other computer vision</th><th>NLP/sequence</th><th>Supervised learning</th></tr></thead><tbody><tr><td>Ash [147]</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Fahlman and Lebiere [148]</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Frean [149]</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>LeCun et al. [38]</td><td></td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Hassibi et al. [95]</td><td></td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Lehtokangas [150]</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Ma and Khorasani [151]</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Narasimha et al. [113]</td><td>+</td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Islam et al. [152]</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Puma-Villanueva et al. [99]</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Han et al. [86]</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Guo et al. [97]</td><td></td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Pan et al. [115]</td><td></td><td>+</td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Siegel et al. [109]</td><td></td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Anwar et al. [111]</td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Cortes et al. [153]</td><td>+</td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>+</td></tr><tr><td>Elsken et al. [71]</td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Gordon et al. [112]</td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Li et al. [107]</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Lu et al. [155]</td><td>+</td><td></td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Bellec et al. [93]</td><td></td><td>+</td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td></tr><tr><td>Cai et al. [126]</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Chen et al. [116]</td><td></td><td></td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Liu et al. [19]</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td></tr><tr><td>Mocanu et al. [92]</td><td></td><td>+</td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Veniat and Denoyer [127]</td><td></td><td></td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td></tr><tr><td>Bian et al. [193]</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Dai et al. [100]</td><td></td><td>+</td><td>+</td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td>+</td><td></td><td>+</td><td>+</td><td>+</td><td></td><td>+</td></tr><tr><td>Dai et al. [68]</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Du et al. [101]</td><td>+</td><td>+</td><td>+</td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Laube and Zell [119]</td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td>+</td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Lee et al. [98]</td><td></td><td></td><td>+</td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td></tr><tr><td>Li et al. [120]</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td></tr><tr><td>Liu et al. [144]</td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td></tr><tr><td>Mei et al. [121]</td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td>+</td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Wortsman et al. [94]</td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Yan et al. [106]</td><td></td><td>+</td><td>+</td><td></td><td>+</td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Bi et al. [122]</td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Ci et al. [72]</td><td></td><td>+</td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Dong et al. [145]</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Guo et al. [128]</td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Kang and Han [69]</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Noy et al. [123]</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Wan et al. [117]</td><td></td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Wen et al. [146]</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Wu et al. [110]</td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td>+</td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Yao et al. [129]</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td>+</td></tr><tr><td>Ye et al. [114]</td><td></td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Guo et al. [118]</td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Kim et al. [138]</td><td></td><td>+</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Peng et al. [91]</td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td></tr><tr><td>Roberts et al. [125]</td><td></td><td>+</td><td>+</td><td></td><td></td><td>+</td><td>+</td><td>+</td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Sinha and Chen [194]</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Wang et al. [124]</td><td></td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td>+</td><td></td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td></tr><tr><td>Yuan et al. [130]</td><td></td><td>+</td><td></td><td></td><td></td><td>+</td><td></td><td></td><td>+</td><td>+</td><td></td><td></td><td></td><td>+</td><td>+</td><td></td><td>+</td></tr></tbody></table>
<table><caption>**Table 2: Method and implementation qualities across our corpus.**</caption><thead><tr><th rowspan="2">Reference</th><th colspan="10">Method</th><th colspan="4">Implementation</th></tr><tr><th>Mask descent</th><th>RI generator</th><th>Evolutionary</th><th>Magnitude-based</th><th>1st order metric</th><th>2nd order metric</th><th>Other metric-based</th><th>Improvement-based</th><th>Regularization</th><th>Splitting/joining</th><th>Modularity</th><th>Weight-share/Mask</th><th>Interconnectivity</th><th>Final reinitialization</th></tr></thead><tbody><tr><td>Ash [147]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Fahlman and Lebiere [148]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Frean [149]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>LeCun et al. [38]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Hassibi et al. [95]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Lehtokangas [150]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Ma and Khorasani [151]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Narasimha et al. [113]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Islam et al. [152]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Puma-Villanueva et al. [99]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Han et al. [86]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Guo et al. [97]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Pan et al. [115]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Siegel et al. [109]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Anwar et al. [111]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Cortes et al. [153]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Elsken et al. [71]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Gordon et al. [112]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Li et al. [107]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Lu et al. [155]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Bellec et al. [93]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Cai et al. [126]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Chen et al. [116]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Liu et al. [19]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Mocanu et al. [92]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Veniat and Denoyer [127]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Bian et al. [193]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Dai et al. [100]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Dai et al. [68]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Du et al. [101]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Laube and Zell [119]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Lee et al. [98]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Li et al. [120]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Liu et al. [144]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Mei et al. [121]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Wortsman et al. [94]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Yan et al. [106]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Bi et al. [122]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Ci et al. [72]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Dong et al. [145]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Guo et al. [128]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Kang and Han [69]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Noy et al. [123]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Wan et al. [117]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Wen et al. [146]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Wu et al. [110]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Yao et al. [129]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Ye et al. [114]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Guo et al. [118]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Kim et al. [138]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Peng et al. [91]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Roberts et al. [125]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Sinha and Chen [194]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Wang et al. [124]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Yuan et al. [130]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>
# 4 Equivariance-aware Architectural Optimization

Constraining neural networks to be equivariant to symmetry groups present in the data can improve their task performance, efficiency, and generalization capabilities [191], as shown by translation-equivariant convolutional neural networks [8, 195] for image-based tasks [9]. Seminal works have developed general theories and architectures for equivariance in neural networks, providing a blueprint for equivariant operations on complex structured data [196–199]. However, these works design model constraints based on an explicit equivariance property. Furthermore, their architectural assumption of full equivariance in every layer may be overly constraining; e.g., in handwritten digit recognition, full equivariance to 180° rotation may lead to misclassifying samples of “6” and “9”. Weiler and Cesa [200] found that local equivariance from a final subgroup convolutional layer improves performance over full equivariance. If appropriate equivariance constraints are instead learned, the benefits of equivariance could extend to applications where the data may have unknown or imperfect symmetries.

Learning approximate equivariance has been recently approached through novel layer operations [201–205]. Separately, the field of neural architecture search (NAS) aims to optimize full neural network architectures [19, 71, 180, 206, 207]. Existing NAS methods have not yet been developed for explicitly optimizing equivariance, although partial or soft equivariant approaches like those proposed by Romero and Lohit [208] and van der Ouderaa et al. [209] do allow for custom equivariant architectures. An important aspect of NAS is network morphisms: function-preserving architectural changes [137] which can be used during training to change the loss landscape and gradient descent trajectory while immediately maintaining the current functionality and loss value [16]. Developing tools for searching over a space of architectural representations of equivariance would allow for existing NAS algorithms to be applied towards architectural optimization of equivariance.

**Contributions** First, we present two mechanisms towards equivariance-aware architectural optimization. The *equivariance relaxation morphism* for group convolutional layers partially expands the representation and parameters of the layer to enable less constrained learning with a prior on symmetry. The [*G*-mixed *equivariant layer*] parameterizes a layer as a weighted sum of layers equivariant to different groups, permitting the learning of architectural weighting parameters.

Second, we implement these concepts within two algorithms for architectural optimization of partially-equivariant networks. Evolutionary Equivariance-Aware NAS (EquiNAS<sub>E</sub>) utilizes the equivariance relaxation morphism in a greedy evolutionary algorithm, dynamically relaxing constraints throughout the training process. Differentiable Equivariance-Aware NAS (EquiNAS<sub>D</sub>) implements [*G*-mixed] equivariant layers throughout a network to learn the appropriate approximate equivariance of each layer, in addition to their optimized weights, during training.

Finally, we analyze the proposed mechanisms via their respective NAS approaches in multiple image classification tasks, investigating how the dynamically learned approximate equivariance affects training and performance over baseline models and other approaches.

## 4.1 Related Works

**Approximate equivariance** Although no other works on approximate equivariance explicitly study architectural optimization, some approaches are architectural in nature. We compare our contributions with the most conceptually similar works to our knowledge.

The main contributions of Basu et al. [205] and Agrawal and Ostrowski [210] are similar to our proposed equivariant relaxation morphism. Basu et al. [205] also utilizes subgroup decomposition but instead
algorithmically builds up equivariances from smaller groups, while our work focuses on relaxing
existing constraints. Agrawal and Ostrowski [210] presents theoretical contributions towards network
morphisms for group-invariant shallow neural networks: in comparison, our work focuses on deep
group convolutional architectures and implements the morphism in a NAS algorithm.

The main contributions of Wang et al. [201] and Finzi et al. [202] are similar to our proposed [G]-mixed equivariant layer. Wang et al. [201] also uses a weighted sum of kernels, but uses the same group for each kernel and defines the weights over the domain of group elements. Finzi et al. [202] uses an equivariant layer in parallel to a linear layer with weighted regularization, thus only using two layers in parallel and weighting them through regularization rather than parameterization.

In more diverse approaches, Zhou et al. [203] and Yeh et al. [204] represent symmetry-inducing weight sharing through learnable matrices. Romero and Lohit [208] and van der Ouderaa et al. [209] learn partial or soft equivariances for each layer.

**Neural architecture search** Neural architecture search (NAS) aims to optimize both the architecture and its parameters for a given task. Liu et al. [19] approaches this difficult bi-level optimization by creating a large super-network containing all possible elements and continuously relaxing the discrete architectural parameters to enable search by gradient descent. Other NAS approaches include evolutionary algorithms [71, 206, 207] and reinforcement learning [180], which search over discretely represented architectures.

## 4.2 Background

### 4.2.1 Symmetries and Group Theory

A symmetry of an object is a mapping of the object onto itself such that structure is preserved. A *symmetry group* $G$ is a set of such mappings along with a binary operation $\cdot: G \times G \to G$, known as the *group product*, that satisfies axioms for closure, associativity, the identity, and the inverse [211]. A group $G$ acts on a set $\mathcal{X}$ via the *group action* $\cdot: G \times \mathcal{X} \to \mathcal{X}$, $(g, x) \mapsto g \cdot x$ that satisfies axioms for identity and compatibility: $\mathcal{X}$ is called a *G-space*.

Equivariance is the property of a mapping such that transformation of the input results in equivalent transformation of the output. Formally, a mapping $h: \mathcal{X} \to \mathcal{Z}$ between two $G$-spaces is $G$-equivariant if for all $g \in G$ and $x \in \mathcal{X}$ we have: $h(g \cdot x) = g \cdot h(x)$. For example, an image segmentation neural network should be $T(2)$-equivariant: shifting the input should result in the same shift in the output.

Invariance is a special case of equivariance, where the output of the function is completely independent of transformation of the input. Formally, a mapping $h: \mathcal{X} \to \mathcal{Z}$ is $G$-invariant if for all $g \in G$ and $x \in \mathcal{X}$ we have: $h(g \cdot x) = h(x)$. For example, an image classification network should be $T(2)$-invariant: shifting the input should not change the output. Symmetries leave objects invariant.

For two groups $G$ and $H$ with group products $\cdot_G$ and $\cdot_H$ respectively where $H$ acts on $G$ with group action $\cdot$, the (outer) semi-direct product $G \times H$ of $H$ acting on $G$ is a group composed of the set of elements $G \times H$ with group product $(g, h) \cdot (g', h') = (g \cdot_G (h \cdot g'), h \cdot_H h')$ and inverse $(g, h)^{-1} = (h^{-1} \cdot G \cdot h^{-1}, h^{-1})$.

A *subgroup* $H$ of $G$ is a nonempty subset with the same group product that also fulfills the group axioms. Then, $gH = \{g \cdot h | h \in H\}$ and $Hg = \{h \cdot g | h \in H\}$ denote the *left coset* and *right coset*, respectively, of $H$ with representative $g$.
<figure><img src="image_14.png" /><figcaption>**Figure 8:** Visualizations of (A) the equivariance relaxation morphism and (B) the [G]-mixed equivariant layer, using the C₄ group. In (A), the learnable parameters of a C₄-equivariant convolutional layer are expanded using the group element actions, such that the expanded filter can be used in a standard convolutional layer. Applying the equivariance relaxation morphism reparameterizes the layer to only be architecturally constrained to C₂ equivariance, initialized to be functionally C₄ equivariant. In (B), convolutional operations equivariant to subgroups of C₄ are summed with learnable architectural weighting parameters.</figcaption></figure>

### 4.2.2 Group Convolutional Neural Networks

Let G be a discrete group. The lth G-equivariant group convolutional layer [196] of a group convolutional neural network (G-CNN) convolves the feature map $f: G \to \mathbb{R}^{C_{l-1}}$ output from the previous layer with a filter with kernel size k represented as learnable parameters $\psi: G \to \mathbb{R}^{C_l \times C_{l-1}}$. For each output channel $d \in [C_l]$, where $[C] := \{1, ..., C\}$, and group element $g \in G$, the layer's output is defined via the convolution operator<fnref n="1" />:

$$[f \star_G \psi]_d(g) = \sum_{h \in G} \sum_{c=1}^{C_{l-1}} f_c(h) \psi_{d,c}(g^{-1}h). \qquad (2)$$

The first layer is a special case: the input to the network needs to be lifted via this operation such that the output feature map of this layer has a domain of G. In the case of image data, an image x with C channels may be interpreted as a function $x: \mathbb{Z}^2 \to \mathbb{R}^C$ mapping each pixel in coordinate space to a real number for each channel, where the cth channel of x is referred to as $x_c$. The input is $x: \mathbb{Z}^2 \to \mathbb{R}^{C_0}$, so the layer is instead a lifting convolution:

$$[x \star_G \psi]_d(g) = \sum_{y \in \mathbb{Z}^2} \sum_{c=1}^{C_0} x_c(y) \psi_{d,c}(g^{-1}y). \qquad (3)$$

We present our contributions in the group convolutional layer case, although similar claims apply for the lifting convolutional layer case.

## 4.3 Towards Architectural Optimization over Subgroups

We propose two mechanisms to enable search over subgroups: the equivariance relaxation morphism and a [G]-mixed equivariant layer, shown in Figure 8. The proposed morphism, described in Section 4.3.1, changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The [G]-mixed equivariant layer, presented in Section 4.3.2, allows for a single layer to represent equivariance to multiple subgroups through a weighted sum.

<footnote n="1">¹We identify the correlation and convolution operators as they only differ where the inverse group element is placed and refer to both as "convolution" throughout this work.</footnote>
### 4.3.1 Equivariance Relaxation Morphism

The equivariance relaxation morphism reparameterizes a G-equivariant group (or lifting) convolutional layer to operate over a subgroup of G, partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.

Let $G' \leq G$ be a subgroup of G. Let R be a system of representatives of the left quotient (including the neutral element), so that $G' \setminus G = \{G'r | r \in R\}$, where $G'r := \{g'r | g' \in G'\}$. Given a G-equivariant group convolutional layer with feature map $f$ and kernel $\psi$, we define the relaxed feature map $\tilde{f}: G' \to \mathbb{R}^{(C_{l-1} \times |R|) \times (C_{l-1} \times |R|)}$ and relaxed kernel $\tilde{\psi}: G' \to \mathbb{R}^{(C_{l-1} \times |R|) \times (C_{l-1} \times |R|)}$ as follows. For $c \in [C_{l-1}]$, $s, t \in R$, $d \in [C_l]$:

$$ \tilde{f}_{(c,s)}(g') := f_c(g's), \quad (4) $$

$$ \tilde{\psi}_{(d,t),(c,s)}(g') := \psi_{d,c}(t^{-1} g's). \quad (5) $$

We define the equivariance relaxation morphism from G to G' as the reparameterization of $\psi$ as $\tilde{\psi}$ (Eq. 5) and reshaping of $f$ as $\tilde{f}$ (Eq. 4). We will show that the new output layer, $[\tilde{f} \star_{G'} \tilde{\psi}]_{(d,t)}(g')$, is equivalent to $[f \star_G \psi]_d(g't)$ down to reshaping. Since the mapping $G' \times R \to G$, $(g', t) \mapsto g't$, is bijective, every $g$ can uniquely be written as $g = g't$ with $g' \in G'$ and $t \in R$. For $g \in G$, $G'g \in G' \setminus G$ has a unique representative $t \in R$ with $G'g = G't$, and $g' := gt^{-1} \in G'$. By the same argument, $h \in G$ may be written as $h = h's$ with unique $h \in G'$ and $s \in R$. With these preliminaries, we get:

$$ [f \star_G \psi]_d(g't) = [f \star_G \psi]_d(g) \quad (6) $$

$$ = \sum_{h \in G} \sum_{c=1}^{C_{l-1}} f_c(h) \psi_{d,c}(g^{-1}h), \quad (7) $$

$$ = \sum_{h' \in G'} \sum_{s \in R} \sum_{c=1}^{C_{l-1}} f_c(h's) \psi_{d,c}(t^{-1} g'^{-1} h's), \quad (8) $$

$$ = \sum_{h' \in G'} \sum_{c=1}^{C_{l-1}} \sum_{s \in R} \tilde{f}_{(c,s)}(h') \tilde{\psi}_{(d,t),(c,s)}(g'^{-1} h'), \quad (9) $$

$$ = [\tilde{f} \star_{G'} \tilde{\psi}]_{(d,t)}(g'), \quad (10) $$

which shows the claim. Thus, the convolution of $\tilde{f}$ with $\tilde{\psi}$ is equivariant to G but parametrized as a G'-equivariant group convolutional layer, where the representatives are expanded into independent channels. This morphism can be viewed as initializing a G'-equivariant layer with a pre-trained prior of equivariance to G, maintaining any previous training.

Standard convolutional layers are a special case of group-equivariant layers, where the group is translational symmetry over pixel space. Regular group convolutions are often implemented by relaxation to the translational symmetry group by expanding the kernel via the appropriate group actions, allowing a standard convolution implementation from a deep learning library to be used. The equivariance relaxation morphism generalizes this concept to any subgroup. With the given preliminaries and the case of $G' = T(2)$, $\tilde{f}$ and $\tilde{\psi}$ are computed such that $\tilde{f}_{(c,s)}(g') := f_c(g's)$ and $\tilde{\psi}_{(d,t),(c,s)}(g') := \psi_{d,c}(t^{-1}g's)$ for each $g' \in T(2)$, $c \in [C_{l-1}]$, $s, t \in R$, and $d \in [C_l]$.

Let $S_G := |R|$. The learnable parameters of the $G_l$-equivariant lth layer with $C_l$ output channels, corresponding to $\psi$, are stored as a tensor of size $C_l \times C_{l-1} \times S_{G_l} \times K_l \times K_l$. The kernel transformation expands this kernel tensor by performing the action of each $r \in R$ on another copy of the tensor to expand its shape along a new dimension, resulting in a tensor of size $C_l \times S_{G_l} \times C_{l-1} \times S_{G_l} \times K_l \times K_l$, which is reshaped to $C_l S_{G_l} \times C_{l-1} S_{G_l} \times K_l \times K_l$. The input tensor to the lth layer, corresponding to $f$, is in the shape of $B \times C_{l-1} \times S_{G_l} \times H_{l-1} \times W_{l-1}$, which is reshaped to $B \times C_{l-1} S_{G_l} \times H_{l-1} \times W_{l-1}$ and convolved with the expanded kernel. The output of shape $B \times C_l S_{G_l} \times H_l \times W_l$ is reshaped to $B \times C_l S_{G_l} \times H_l \times W_l$.

To implement the equivariance relaxation morphism, the new kernel tensor is initialized by applying Equation 5 such that result of applying the preceding kernel transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements:
**Algorithm 1 Evolutionary equivariance-aware neural architecture search.**

**procedure** EQUINAS<sub>E</sub>(Initial symmetry group G)

Initialize population with a G-equivariant group convolutional network.

**for** each generation **do**

**for** each network in population **do**

Add children of network with relaxed equivariance constraints into population.

**for** each network in population **do**

Partially train network on dataset.

Select Pareto-efficient and high accuracy networks as new population.

**return** population

to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion. The new kernel tensor has a shape of $C_l|R| \times C_{l-1}|R| \times S_{G_l}/|R| \times K_l \times K_l$.

### 4.3.2 [G]-Mixed Equivariant Layer

Towards learning equivariance, we additionally propose partial equivariance through a mixture of layers, each constrained to equivariance to different groups and applied in parallel to the same input then all combined via a weighted sum. The equivariance relaxation morphism provides a mapping of group elements between pairs of groups where one is a subgroup of the other. For a set of groups [G] where each group is a subgroup or supergroup of all other groups within the set, we define a [G]-mixed equivariant layer as:

$$[f \star_{[G]} [\psi]]_{(d,t)}(g) = \sum_{G \in [G]} z_G [f \star_{G'} \tilde{\psi}^G]_{(d,t)}(g) \quad (11)$$

$$= \left[ f \star_{G'} \sum_{G \in [G]} z_G \tilde{\psi}^G \right]_{(d,t)}(g), \quad (12)$$

where each element $z_G$ of $[z] := \{z_G | G \in [G]\}$ is an architectural weighting parameter such that $\sum_{G \in [G]} z_G = 1$, $G'$ is a subgroup of all groups in [G], each element $\tilde{\psi}^G$ of $[\psi]$ is a kernel with a domain of G, and $\tilde{\psi}^G$ is the transformation of $\tilde{\psi}^G$ from a domain of G to $G'$ as defined in Equation 5. Thus, the layer is parametrized by $[\psi]$ and $[z]$, computing a weighted sum of operations that are equivariant to different groups of [G]. The layer may be equivalently computed by convolution of the input with the weighted sum of transformed kernels, shown in Equation 12. The implementation for the [G]-mixed equivariant layer can be built on top of that of the equivariance relaxation morphism, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.

## 4.4 Equivariance-Aware Neural Architecture Algorithms

We present two neural architecture search methods that utilize the presented mechanisms for discovering appropriate equivariance during neural network training: Evolutionary Equivariance-Aware NAS (EquiNAS<sub>E</sub>) and Differentiable Equivariance-Aware NAS (EquiNAS<sub>D</sub>). Both methods optimize an architecture while learning network weights, returning a final trained network adapted to the equivariances present in the training data. However, they differ in NAS paradigm and approximate equivariance representation: EquiNAS<sub>E</sub>, described in Section 4.4.1, searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNAS<sub>D</sub>, described in Section 4.4.2, searches for smooth mixtures of equivariant layers.